Skip to content
Archive of entries posted on September 2009

Flexible Filtering

When writing a “get-” function in PowerShell, you often run into the issue of filtering your data.  Do you want to include any filtering parameters?  Do you want to allow lists of values?  Do you want to provide “include” or “exclude” parameters?  What about wildcards?  I got tired of writing the same kind of code over, so I wrote a fairly general-purpose filter-list function.

Here it is (example usages follow):

function filter-list($list, $filterString,$propertyName,[switch]$help ){
 if ($help)
 {
 $msg = @"
Filter a list using a filterstring and a property name. Wildcards are allowed, as well as
a prefix of Not:, which means to include values that don't match the pattern.
If filterString is an array, it lists property values that will remain in the list.

Usage: filter-list list filterString propertyname [-help]
"@
 Write-Host $msg
 return
 }
 if (!$list) {return}
 if ($filterString -is [Array]){
 $list = @($list | ?{ $filterString -icontains $_.$propertyName})
 } else
 {
 if (($filterString) -and ($filterString -ne '')){
 if ($filterString -ilike 'Not:*'){
 $list = @($list | ? {$_.$propertyName -inotlike $filterString.Replace('Not:','')})
 } else {
 $list = @($list | ? {$_.$propertyName -ilike $filterString})
 }
 }
 }
 return $list
}

Here’s a simple function (not really useful) that uses this:

function new-dir{
 Param($path, $extension)
 $list=dir $path
 $list=filter-list $list $extension Extension

 return $list
}

new-dir -path c:\temp -ext '.LOG','.TXT'
new-dir -path c:\temp -ext 'Not:.LOG'
new-dir -path c:\temp -ext 'Not:.c*'

Now, without really writing any new code, we have a parameter that can take simple values, lists of values, wildcards, and include/exclude logic.

Let me know what you think of this.

Mike

  • Digg
  • Slashdot
  • Reddit
  • Tumblr
  • Delicious
  • Twitter
  • Google Bookmarks
  • StumbleUpon
  • Technorati Favorites
  • Google Reader
  • Share/Bookmark

A Handy Trick I’ve Started to Use a Lot

If you’re like me, you hate to do the same thing over and over.  That’s what programming is for, right?  To handle automating tedious procedures?  Unfortunately, it’s not at all appropriate to run off and build an app every time you need to do the same thing 3 times.  If you try that, you’ll have a lot of chances to write apps,  but probably will be looking for a new job because it takes you way too long to get anything accomplished.

Scripting is the short answer to the dilemma above.  PowerShell is one of the latest entries into the scripting world, and to my tastes, one of the best.

Here’s something I used several times in the last few days.  I can’t remember quite where I saw it first, but it was in a PowerShell blog about looping (I think).

Anyway, the problem is that I needed to edit config files on the servers in a farm.  Fortunately, the servers were numbered sequentially.  So, what I wrote was (suitably sanitized for public consumption):

1..9 | % { notepad "\\server$_\c$\path_to_config_file\config.file" }

That popped the first 9 files up in notepad, ready to be edited.  The trick is to use the range syntax to create a list of numbers, and use % to loop through them.

If you need a longer range (with leading zeroes, of course), it’s not too hard.

1..20 | % {notepad ("\\server{0:D2}\c$\path_to_config_file\config.file" -f $_)}

Here, we use the format operator with a D2 format spceifier (2 digits, leading zeros).  See here for more examples of format operators in PowerShell.

When you’re dealing with dozens of servers, tricks like this can save you a lot of time.

Let me know what you think.  What “idioms” in PowerShell do you find yourself using a lot?

Mike

  • Digg
  • Slashdot
  • Reddit
  • Tumblr
  • Delicious
  • Twitter
  • Google Bookmarks
  • StumbleUpon
  • Technorati Favorites
  • Google Reader
  • Share/Bookmark

Is it just me? (Or does PowerShell remind you of SQL?)

When preparing a PowerShell training class for a group of DBAs, I realized that there were some parallels between basic SQL and basic PowerShell commands.

A (very) basic SQL statement has the form:

SELECT <COLUMNS>
FROM <TABLE>
WHERE <CONDITION>
ORDER BY <EXPRESSION>

I noticed that a very common idiom for PowerShell pipelines* was:

<data source cmdlet> | select-object <properties> | where-object <CONDITION> |  sort-object <EXPRESSION>
 

By “<data source cmdlet>”, I mean some cmdlet that puts a bunch of objects in the pipeline, like get-childitem, get-process, get-task, etc.

Part of the power of SQL is that it doesn’t matter what kind of data is in the tables, the same form of SQL statement works the same way (predictability).  This is one of the things I love about PowerShell.  It doesn’t matter what kinds of data is returned by a cmdlet.  The same form of PowerShell pipeline* will perform the same kind of predictable operations on it.  I know that this is often mentioned in tutorials and videos about PowerShell, but this was when it really struck me. 

A few other SQL/PowerShell comparisons might be:

SQL

PowerShell

GROUP BY group-object
SUM(), AVG(), etc. measure-object
Cursors foreach-object loops
SELECT DISTINCT select-object –unique
SELECT TOP n select-object –first n

 

Obviously, this comparison breaks down pretty quickly.  There isn’t really a parallel that I can find to JOIN statements, which make SQL so powerful, and clearly there’s a lot of powershell scripts that don’t fit the pattern I’m describing.  I think, though, that it’s a useful comparison and can help get people “over the hump” in their quest to master PowerShell.

Let me know what you think.

Mike

* A Pipeline in PowerShell is a sequence of cmdlets where each takes the output of the previous cmdlet as its input.

  • Digg
  • Slashdot
  • Reddit
  • Tumblr
  • Delicious
  • Twitter
  • Google Bookmarks
  • StumbleUpon
  • Technorati Favorites
  • Google Reader
  • Share/Bookmark

Executing SQL the Right Way in PowerShell

We all know that using string concatenation (or substitution) in SQL is a “bad thing”.  If you google “SQL Injection”, you’ll find hundreds of pages that tell you not to do things this way and that you need to use parameterized queries.  However, I still see a lot of code in PowerShell that does this:

$cmd.ExecuteNonQuery("delete from Table1 where Column1='$value'")

Since code like this is obviously prone to SQL injection attacks, it must be that doing it the right way is difficult, right?  Actually, no.  Here’s a simple function that allows you to run parameterized queries easily using dictionaries.

function exec-query( $sql,$parameters=@{},$conn,$timeout=30,[switch]$help){
 if ($help){
 $msg = @"
Execute a sql statement.  Parameters are allowed.
Input parameters should be a dictionary of parameter names and values.
Return value will usually be a list of datarows.
"@
 Write-Host $msg
 return
 }
 $cmd=new-object system.Data.SqlClient.SqlCommand($sql,$conn)
 $cmd.CommandTimeout=$timeout
 foreach($p in $parameters.Keys){
 [Void] $cmd.Parameters.AddWithValue("@$p",$parameters[$p])
 }
 $ds=New-Object system.Data.DataSet
 $da=New-Object system.Data.SqlClient.SqlDataAdapter($cmd)
 $da.fill($ds) | Out-Null

 return $ds
}

Example usage:

$conn=new-object data.sqlclient.sqlconnection "Server=servername;Integrated Security=True"
$conn.open()
exec-query 'select * from sys.databases' -conn $conn
exec-query 'select FirstName,LastName from AdventureWorks2008.Person.Person where FirstName=@fname' -parameter @{fname='Mike'} -conn $conn

As you can see from the second example, using a parameterized function wasn’t much harder than it would have been to do string concatenation or substitution.  And you don’t have to worry about little Bobby Tables.

The code above still needs some work to take care of output parameters, but I’ll save that for another post.

Let me know if you have any questions, comments, or complaints.

Mike

  • Digg
  • Slashdot
  • Reddit
  • Tumblr
  • Delicious
  • Twitter
  • Google Bookmarks
  • StumbleUpon
  • Technorati Favorites
  • Google Reader
  • Share/Bookmark

PowerShell ETS (Extended Type System)

In a recent post , I showed how to get a list of Scheduled Tasks as objects using PowerShell and Import-CSV.   In that, I included the following line of code:

$task.PSObject.TypeNames.Insert(0,"DBA_ScheduledTask")

In this post, I’ll try to explain why I did that.

There are several approaches to making objects in PowerShell.

  • Use new-object to create an object from an existing class (.NET, COM, ADSI, WMI)
  • Use add-member to add properties and methods to an existing object
  • Use ETS (the Extended Type System) to add properties and methods to an existing class

Obviously, the first approach (new-object)  makes sense if there is an existing class that you know fits the purposes you have in mind.  It helps if there’s a constructor that takes arguments that you have lying around, too.  For example:

 $conn=new-object data.sqlclient.sqlconnection "Server=laptop\sqlexpress;Integrated Security=True"

But often, you’re working with a domain-specific object that doesn’t exist in the .NET framework or anything else.

The second approach (using add-member) is useful, for example,  if you need to create a record from a collection of properties.  You can also add methods using add-member.   One thing that you’ll run into with using add-member is that you have to remember to add the same members (the same set of properties and methods) to each similar object that you’re dealing with.  This is a lot different than when you create a class definition and instantiate objects from the class.

The third approach is similar to the second, in that you specify properties and methods to add, but in this case, you specify them in a configuration file ( *.ps1xml) that is added with the update-typedata cmdlet.   In this file, you name the class that you want to extend, and list the property and method definitions that you want to add.  After the update-typedata cmdlet is issued, all objects of the type listed will have the new properties and methods.

One especially nice feature of the third approach is that the name of the class you want to extend does not have to be the actual name of the type of the objects.  What it really does is match the values in the object’s PSObject.TypeNames property.  For most objects, this property is a list of the inheritance chain for the object, going back to System.Object.  However, as in the above example, you are free to add (or remove) items from this list.  Above, I added “DBA_ScheduledTask” to the list.  That means that if I have a .ps1xml configuration file that has ETS info in it for the type “DBA_ScheduledTask”, that set of properties and methods would be available from the objects I create.

I often use functions that return data out of a database.  PowerShell is nice enough that it acts like DataRows are objects, using column names for properties.  This makes it very easy to use the data in PowerShell without doing much to it.  By adding a specific “type name” to the TypeNames collection, I can instantly make the DataRow records into full-fledged objects (by including an appropriate update-typdata command in my profile).  In the post about Scheduled Tasks, it would make sense to use ETS to specify the Run and Delete methods.  Here’s what that would look like:

<?xml version="1.0" encoding="utf-8" ?>
<Types>
 <Type>
 <Name>DBA_ScheduledTask</Name>
 <Members>
 <ScriptMethod>
 <Name>Run</Name>
 <Script>
 schtasks.exe /RUN /TN $this.TaskName /S $this.HostName
 </Script>
 </ScriptMethod>
 <ScriptMethod>
 <Name>Delete</Name>
 <Script>
 schtasks.exe /DELETE /TN $this.TaskName /S $this.HostName
 </Script>
 </ScriptMethod>
 </Members>
 </Type>
</Types>

With this in a file (for instance c:\types\scheduledTasks_type.ps1xml), you would issue the command “update-typedata c:\types\scheduledTasks_type.ps1xml”, and the Run and Delete methods would be added to items of that type.  Of course, you’d want to modify the code in the previous post to not add those methods with add-member.

I tend to use a mix of add-member and ETS to create the objects I want.  You have to be careful when using ETS to remember that if you want the extra properties, you have to have loaded the ps1xml file in the session you’re running.  This often isn’t the case if you’re remoted into a server troubleshooting something.  For that reason, it’s usually preferrable (in my opinion) to use add-member most of the time.  I use a custom PowerShell host that allows me to use ETS data to specify context menus for different types of objects and customize them through the ps1xml files.  Since I don’t expect to find those context menus when I’m in the text console on a server, it doesn’t get me into trouble.

Let me know if you have any questions, comments, or complaints.

Mike

  • Digg
  • Slashdot
  • Reddit
  • Tumblr
  • Delicious
  • Twitter
  • Google Bookmarks
  • StumbleUpon
  • Technorati Favorites
  • Google Reader
  • Share/Bookmark

PowerShell Tools and Books That I Use

Tools I Use (note…these are all free!):

  • PowerGUI Script Editor (I haven’t ever gotten the hang of PowerGUI itself)
  • Powershell Community Extensions 1.2  (PSCX)
  • PowerTab
  • SQL PowerShell Extensions 1.61 (SQLPSX)
  • PrimalForms Community Edition

Books:

  • PowerShell In Action by Bruce Payette
  • Professional Windows PowerShell Programming: Snapins, Cmdlets, Hosts and Providers by  Arul Kumaravel et. al.
  • Mastering PowerShell by Dr. Tobias Weltner

I’ve tried a lot of other tools (several IDE’s, for example), but this is the list I keep returning to.

  • Digg
  • Slashdot
  • Reddit
  • Tumblr
  • Delicious
  • Twitter
  • Google Bookmarks
  • StumbleUpon
  • Technorati Favorites
  • Google Reader
  • Share/Bookmark

Getting Scheduled Tasks in PowerShell

There are a few approaches to manipulating scheduled tasks in PowerShell.

  • WMI – Useful if you are only going to manipulate them via script.  The tasks will not be visible in the control panel applet.
  • SCHTASKS.EXE – Works ok, but has a somewhat arcane syntax, and is a text-only tool.
  • Task Scheduler API -Best of both worlds, but only on Vista (not XP).

My current environment is an XP laptop, and hundreds of W2k3 and W2k8 servers.  I really need to be able to hit the tasks from the control panel (don’t have PowerShell installed everywhere…but that’s in process).  Since I have to use SCHTASKS.EXE , I figured I should write a PowerShell interface that makes things a bit easier.

The main thing I wanted was to get objects back.  Once you’ve started using PowerShell, you realize that “everything returns objects” makes for a very powerful scripting experience.  So, when you have to fall back to an external command like SCHTASKS.EXE, you really feel the pain.  The approach I took was to ask SCHTASKS for a verbose output (no reason not to get all of the properties), and also to write the output as CSV.  I then send the output into a file.  A problem that arises when trying to get PowerShell to import the CSV is that the column headers are not very friendly (e.g. “Repeat: Until: Time”).  So, before importing we need to “massage” a little bit.  I simply removed spaces (leaving Pascal-cased words) and replaced colons with underscores.  Another issue was that different versions of SCHTASKS.EXE would sometimes include an extra blank line in the file.  Finally, I decided that I should add some methods using Add-Member to allow me to run or delete the task without having to remember the syntax.

I hope you find this script useful.  If you have suggestions for improvements, or questions about how part of it works, feel free to leave a comment.

function get-tasks($server="", $taskname="",[switch]$help){
  if ($help)
    {
        $msg = @"
Get scheduled tasks from a remote server as objects.  Optionally you may supply a substring to be found in the task names.

Usage: get-tasks [Server] [substring] [-help]
ex:  get-tasks MACHINE1 LOGS    #to get all scheduled tasks from MACHINE1 containing LOGS
"@
        Write-Host $msg
        return
    }
    $filename = [System.IO.Path]::GetTempFileName()
 if ($server){
     schtasks /query /fo csv /s $server /v &gt; $filename
 } else {
    schtasks /query /fo csv /v &gt; $filename
 }
    $lines=Get-Content $filename
 if ($lines -is [string]){
    return $null
 } else {
        if ($lines[0] -ne ''){
   Set-Content -path $filename -Value ([string]$lines[0]).Replace(" ","").Replace(":","_")
   $start=1

  } else {
   Set-Content -path $filename -Value ([string]$lines[1]).Replace(" ","").Replace(":","_")
   $start=2
  }
  if ($lines.Count -ge $start){
   Add-content  -Path $filename -Value $lines[$start..(($lines.count)-1)]
  }
  $tasks=Import-Csv $filename
  Remove-Item $filename
  $retval=@()
  foreach ($task in $tasks){
   if (($taskname -eq '') -or $task.TaskName.contains($taskname)){
    $task.PSObject.TypeNames.Insert(0,"DBA_ScheduledTask")
    Add-Member -InputObject $task -membertype scriptmethod -Name Run -Value { schtasks.exe /RUN /TN $this.TaskName /S $this.HostName}
    Add-Member -InputObject $task -membertype scriptmethod -Name Delete -Value { schtasks.exe /DELETE /TN $this.TaskName /S $this.HostName}
    $retval += $task
   }
  }
  return $retval
 }
}

Links:

  • Digg
  • Slashdot
  • Reddit
  • Tumblr
  • Delicious
  • Twitter
  • Google Bookmarks
  • StumbleUpon
  • Technorati Favorites
  • Google Reader
  • Share/Bookmark

Why PowerShell?

Why am I writing a blog about PowerShell?  The answer is simple. I haven’t been as excited about a technology since I first learned SQL.  PowerShell allows me to do my job in a much more consistent, flexible, and scalable way.

What is it about PowerShell that has me so fascinated?

  • Ability to work with multiple technologies in a seamless fashion (.NET, WMI, AD, COM)
  • Dynamic code for quick scripting, strongly-typed code for production code (what Bruce Payette calls “type-promiscuous”)
  • High-level language constructs (functions, objects)
  • Consistent syntax
  • Interactive environment (REPL loop)
  • Discoverable properties/functions/etc. Example
  • Great variety of delivered cmdlets, even greater variety of community cmdlets and scripts
  • On a similar note, a fantastic community that shares results and research
  • Extensible type system MSDN
  • Everything is an object
  • Powerful (free) tools like PowerGUI, PSCX, PowerShell WMI Explorer, PowerTab, PrimalForms Community Edition, and many, many more.
  • Easy embedding in .NET apps including custom hosts.
  • The most stable, well-thought out version 1.0 product I’ve ever seen MicroSoft produce.
  • An extremely involved, encouraging community..

I have to admit that it wasn’t until the third time I looked at PowerShell that I finally “got it”.  I guess I didn’t have a problem in mind when I looked at it and was a bit overwhelmed by the syntax.  I think that trying to explain PowerShell as a “replacement for cmd.exe” really made it hard to see how all the new concepts (noun/verb combinations, for example) were going to pay off.  After all,cmd.exe is pretty painless to work with.  You can’t do much, so it’s not so hard.  With PowerShell, on the other hand, the sky’s the limit.

My first extensive use of PowerShell was to pull lists of shares from all of the servers that our group manages (about 450) along with Share and NTFS security.  Even to someone who hadn’t really ever used WMI, it was not too bad.

Once I had finished that, I started writing administration scripts for a purchased software package that used several different technologies and again spanned several servers.  We have about 60 different installations of this package on about 200 servers.  Managing them all through remote desktop and scheduled tasks was working, but was rapidly  becoming a maintenance nightmare.  PowerShell scripting has made the automation of most of our administrative tasks fairly painless.

I realize that this post was pretty low on technical details, but I hope to be following with several posts that include actual code. I’ve written several thousand lines of script and a custom PowerShell host, so I hope that something I write about is interesting and informative to you.

Feel free to comment if you have suggestions for topics you’d like to see me discuss.

Mike

  • Digg
  • Slashdot
  • Reddit
  • Tumblr
  • Delicious
  • Twitter
  • Google Bookmarks
  • StumbleUpon
  • Technorati Favorites
  • Google Reader
  • Share/Bookmark