Scraping Sysinternals

download-thumbRecently I was conversing with someone about my PowerShell code that downloads tools from the live Sysinternals site. If you search the Internet, you’ll find plenty of ways to achieve the same goal. But we were running into a problem where PowerShell was failing to get information from the site. From my testing and research I’m guessing there was a timing issue when the site is too busy. So I started playing around with some alternatives.

I knew that I could easily get the html content from http://live.sysinternals.com through a few different commands. I chose to use Invoke-RestMethod for the sake of simplicity.

Because $html is one long string with predictable patterns, I realized I could use my script to convert text to objects using named regular expression patterns. In my test script, I dot source this script.

Now for the fun part. I had to build a regular expression pattern. Eventually I arrived at this:

If you have used the Sysinternals site, you’ll know there is also a Tools “subfolder” that appears to be essentially the same as the top level site. My pattern is for the top level site. Armed with this pattern, it wasn’t difficult to create an array of objects for each tool.

sysinternals

Next, I check my local directory and get the most recent file.

Then I can test for files that don’t exist locally or are newer on the site.

If there are needed files, then I create a System.Net.WebClient object and download the files.

The end result is that I can update my local Sysinternals folder very quickly and not worry about timing problems using the Webclient service.

Test Subnet with PowerShell

keyboardanalyzeA few years ago I published a PowerShell function to test IP addresses on a given subnet. I had an email the other day about it and I decided to refresh it. My new version adds a few bells and whistles that I think you might like. For example, you can now run it from a remote computer. In other words, you can ping IP addresses from another computer which might be helpful.

I’ve also added an option to resolve the IP address to a hostname using DNS. Because I don’t want to require anything on a remote computer, I am using the .NET DNS class to resolve the name. In addition, I have included a fallback resolution using NETBIOS. If you ask for resolution, and DNS doesn’t return a name, you can opt to use the NBTSTAT command to resolve the hostname. I use a regular expression pattern to pull the computername.

Here’s version 2.0 of Test-Subnet.

And to be clear, “subnet” may be a bit of a misnomer as I’m not calculating any addresses with a subnet mask. Instead you enter the base IPAddress, like 10.10.1.0 and then a range of host numbers between 1 and 254, which is the default by the way. The command will then test 10.10.1.1 through 10.10.1.254, or whatever you entered.

My script employs some other techniques you might find help such as splatting, parameter sets, Write-Progress and parameter validation. Here’s a screenshot of results that I’ve sent to Out-Gridview and then customized.

test-subnet

But since the command writes objects to the pipeline you could do whatever you wanted. I trust you’ll let me know what you think. Enjoy!!

Find and Replace Text with PowerShell

magnifying-glass I’ve just finished up a series of tweets with a follower who had a question about finding and replacing a bit of data in a text file. In his case it was a web.config file but it really could be any text file that you can view in PowerShell. In PowerShell 3.0 and later this isn’t too difficult. Although I haven’t tested with really large files to determine if there are any limitations. Before you try anything I’m going to show you, please have a backup copy of of the original file.

Let’s say I have a file like this:

I need to change the IP source address in the connection string. As you probably guessed Get-Content is going to come into play. Normally when you use Get-Content you get an array of strings which adds a little complexity. But starting in v3 Get-Content has a -Raw parameter which writes the entire file as a single string. If we have a single string then we can easily use -Replace.

Your replacement pattern could even be a regular expression pattern. This could even be done as a one-liner:

For many of you, especially with unstructured text files, this technique should get the job done. But in this case, because the text file is also an XML document we could tackle the problem a different way. This is especially useful if you have several changes you want to make to the file.

First, create an XML document from the file.

We can navigate the document until we find the entry we need to change.

xmlconfig

We can assign a new value with revised connection string:

All that remains is to save the file back to disk.

Or, if you didn’t necessarily know the structure of your document, you could use an XPath query with Select-XML to find the node and then change the value.

The node properties will vary depending on your XML file so you will need to look at the object.
xmlnode

The XML approach is a bit more involved but is predictable and you don’t have to save the file until you have the XML just the way you want it.

One thing I will say about both techniques is that the more you understand how to use regular expressions in PowerShell the more you can accomplish. If you are feeling a little wobbly on the topic, there is an entire chapter on regular expressions in PowerShell in Depth.

Friday Fun: The Measure of a Folder

rulerLast week, I demonstrated how to measure a file with PowerShell. This week let’s go a step further and measure a folder. I’m going to continue to use Measure-Object although this time I will need to use it to measure numeric property values.

Here’s the complete function after which I’ll point out a few key points.

The command will use the current path or you can pipe in a directory name, or the output of a Get-ChildItem expression as I show in the help examples. One thing I added is that I test the path to make sure it is a file system path because anything else wouldn’t really work.

I resolve the path so that I can get the actual name of the current location (.) and then test the provider. The other interesting feature of this function is that I format the results on the fly.

The function has a Unit parameter which has a default value of bytes. But you can also specify one of the PowerShell numeric shortcuts like KB or GB. In the function I use a Switch construct to create a custom property on the fly.

Because there can only be a single value for $Unit, I’m including the Break directive so that PowerShell won’t try to process any other potential matches in the Switch construct. Realistically, there’s a negligible performance gain in this situation but I wanted you to see how this works.

measure-folder

I’ve taken some basic commands you could use interactively such as Get-ChildItem and Measure-Object and built a tool around them using a hashtable of properties to create a custom object. I hope you pick up a tip or two from this. If you have any questions about what I’m doing or why, please don’t hesitate to ask because someone else might have the same question.

Enjoy your weekend!

Convert Text to Object with PowerShell and Regular Expressions

squarepatternA few weeks ago I was getting more familiar with named captures in regular expressions. With a named capture, you can give your matches meaningful names which makes it easier to access specific captures. The capture is done by prefixing your regular expression pattern with a name.

When you know the name, you can get the value from $matches.

This also works, and even a bit better, using a REGEX object.

With the REGEX object you can get the names.

Because the names include index numbers, I usually filter them out. Once I know the names, I can use them to extract the relevant matches.

Then I realized it wouldn’t take much to take this to the next step in PowerShell. I have a name and a value, why not create an object? It isn’t too difficult to create a hashtable on the fly and use that to create a custom object. Eventually I came up with ConvertFrom-Text.

The function requires a regular expression pattern that uses named captures. With the pattern you can either specify the path to a log file, or you can pipe structured text to the function. By “structured text” I mean something like a log file with a predictable pattern. Or even output from a command line tool that has a consistent layout. The important part is that you can come up with a regular expression pattern to analyze the data. I also wanted to be able to pipe in text in the event I only wanted to process part of a large log file.

Here’s an example using the ARP command.

In this particular example, I’m trimming the ARP output to remove any leading or trailing spaces from each line and then converting each line to an object, using the regular expression pattern.

convertfrom-text

If you haven’t jumped to why command is useful, is that once I have objects I can easily filter, sort, group, export, or just about anything else. By converting a log file into a collection of objects I can do tasks like this:

convertfrom-text-2

I hope some of you will try this out and let me know what you think. What works? What is missing? What problem did this solve? Inquiring minds, well at least mine, want to know. Enjoy.