If I were writing a script that needed to get a “bunch of things” from somewhere (perhaps several different sources) and return all of them, I might be tempted to do something like this. Please forgive my PowerShell pseudocode:

function get-stuff{
param($parm1 )
    $results=@()
    foreach ($source in $sources){
        $results += ($source | where { $_ -and "Some condition exists"  })
    }
    foreach ($source in $someothersources){
        $results += ($source | where { $_ -and "Some condition exists"  })
    }
 return $results
}

I’ve used several permutations of that kind of code using arrays of some sort to collect the results as I go along and eventually return the collection from the function. I’m not sure that there’s anything wrong with doing it this way. That is, I’m not sure that you’re likely to have issues with doing it this way.

On the other hand, it’s more idiomatic (i.e more in the style of the PowerShell language) to do something like this (again, pardon the pseudocode):

function get-stuff{
param($parm1 )
    foreach ($source in $sources){
        $source | where { $_ -and "Some condition exists"  }
    }
    foreach ($source in $someothersources){
        $source | where { $_ -and "Some condition exists"  }
    }

}

All I’m doing here is sending the output of the inner statements (which are pipelines) to the output stream of the function. Note that there isn’t any need for anything to accumulate the results into.  Using the output stream makes this function work more like the built-in cmdlets in PowerShell as it won’t be blocking the pipeline.

The only thing that I have against this code is that it goes against rule #2 that I wrote last time about writing values to the output stream. I said there that if you were going to write to the output stream, you should explicitly use write-output. We could modify the code above to use write-output, but that would involve using parentheses (around the pipelines), messing up the flow of the code, and even blocking the pipeline while the expressions in the parentheses were collected (as an argument to write-output).

That brings me to what I was saying about “another pointless function”. About a year ago I wrote a post about the identity function, which doesn’t really do anything except return the input. It is a really useful function for creating lists and such, allowing you to skip on providing a bunch of punctuation. It’s not a pointless function, but it’s not one that is getting much press, either. I was thinking about how to make the “pipeline” version of the code work nicely and not make it ugly and thought of the following function.

function out-output{
    process{ $_ }
}

Like the identity function (or ql, as I’ve seen it referred to), out-output doesn’t do anything but emit values that are provided. Out-output, however, gets its values from the pipeline rather than the argument list. This function allows us to be explicit about our intent to use the output stream.

function get-stuff{
param($parm1 )
    foreach ($source in $sources){
        $source | where { $_ -and "Some condition exists"  } | out-output
    }
    foreach ($source in $someothersources){
        $source | where { $_ -and "Some condition exists"  } | out-output
    }
}

I’m not sure if this is a good idea, and I know that it’s just adding a tiny bit of processing to the script. My thought is that making the operation of the script explicit is worth it in the long run.

What do you think?

-mike

In most languages, if you see “return 1″ as the only return in a function, you can know that the function is going to the value 1 to the caller. In fact, I’m not sure I’ve ever seen a language that didn’t work that way. That is, until I found PowerShell. Generally speaking, the return statement works just as expected. In the absence of any statements writing to the output stream (with write-output) or “dropping” their values, “return 1″ will cause the caller to receive the value “1″. Using write-output is pretty obvious, and I’d recommend using it explicitly if you intend to add objects to the output stream (thereby including them in the eventual function value). Expressions that don’t capture their return values, however, are not quite so easy to spot.

For example, examine this code to add a parameter to an ADO.NET command object looks fine:

$cmd.Parameters.AddWithValue('@demographics','$demoXML)

This is a straightforward translation of one of the lines of code in the example code here.  The problem with the code is that AddWithValue not only adds a parameter, it also returns the parameter.  Since we didn’t assign it to a variable, cast it to [void] or pipe it to out-null, the output of this function (AddWithValue) gets added to the output of the function it’s in.

Several “add” functions in the .net framework follow this pattern, either returning the object that was added or the index of that object in the collection.  The DBConnection.Open method (inherited by SQLConnection, among others) returns the opened connection.  I’m sure that with time I could find more examples than I’d feel like sharing.

Another way that I’ve seen the output stream getting messed up is when a function uses strings to output information without using write-host.  For example, this function outputs “progress” information as it goes:

function get-filelength{
param($filename)
	"reading file"
	$len=0
	$file=get-content $filename
	foreach($line in $file){
	   "adding a line"
	   $len+=$line.length
	}
	return $len
}

I realize that this is not the best way to find the length of a file, but it works for the purpose of illustration. If you simply call the function (without assigning it to a variable), you’ll see a bunch of text, followed by the “value” of the function. The “returned value” of the function is clearly intended to be $len, but instead all of the text is included. The type of the function output is Object[], rather than a number.

There are some good reasons to use the output stream in a function (the next post will examine one such use), but it’s very easy to accidentally put something in the stream without intending to. When this happens, it often leads to unexpected results and long debugging sessions.

Here are my recommendations for avoiding this kind of error:

1. Use one of the “write” functions to provide text messages to the user (write-host, write-debug, write-verbose)
2. Use write-output if your intent is to add something to the output stream
3. Check msdn to see if the methods you use have output. If you’re looking at the C# version, methods that return void do not need to be “handled”. In the VB.NET version, they’ll be declared as Sub, rather than Function.
4. Be consistent with how you “handle” methods or functions whose values you want to discard. Options are to cast the result as [void] or pipe it to out-null
5. If you’re using write-output to put objects in the output stream, use return with no value (i.e., don’t return $val). Seeing a value returned explicitly implies that that is the only value returned.
6. If all else fails, use write-host to output the type of variables when debugging (or look at their values in the watch window if you’re using an IDE with a debugger).

For those of you that are with me in the PowerShell camp, I have something that I’d like to discuss.  You probably have hundreds (dozens?) of scripts scheduled on multiple servers, possibly in multiple domains or geographical locations to perform things like these:

• Gather information about servers
• Generate reports about application usage
• Copy information from one place to another
• Validate security setup
• Start and Stop processes
• Scan log files for error conditions
• Lots of other things (you get the point)

How do you know that the scripts that you have written carefully and scheduled are actually running successfully?  At first, this seems like a silly question.  When you deployed the script, surely you ran it once to make sure it worked.  What could have gone wrong?

Here are some examples that come to mind:

• A policy was pushed which set the execution policy to Restricted
• The credentials you scheduled the script with have been revoked
• A file share that the scripts depend on is unavailable
• Firewall rules change and now WMI queries aren’t working
• The Task Scheduler service is stopped

You can probably think of a lot more examples of things that would keep scripts from working, but you get the idea.  I’ve given some thought to how to do this, but haven’t come to any real conclusions.  Obviously, having your scripts log results is helpful, but only if you monitor the logs for success/failure.  Also, if you have a script which is supposed to run every 10 minutes, it doesn’t help if you don’t get alerted when it only runs once in a day, even if  it runs successfully.  Also, if there is more than one person writing scripts, how do you make sure that everyone is using the same techniques to log progress?

Here are some of my thoughts:

• Use a “launcher” to run scripts (see below)
• Keep a database of processes with an expected # of runs per day
• Monitor matching start/end of scripts
• Log all output streams (example)

The first item in the list (the launcher) has been something I’ve been considering because it’s not trivial to run a PowerShell script in a scheduled task.  Even with the -file parameter which was added in PowerShell 2.0, it can involve a fairly long command-line.  With the added difficulty of trying to capture output streams (most of which are not exposed to the command shell) it becomes a process that is almost hard to get right every time.  Some features I’m planning for the launcher are:

• Load appropriate profiles
• Log all output streams (with timestamps) to file or database
• Log application start/end times
• Email owner of script if there are unhandled exceptions

I know this topic is not specific to PowerShell, but as Windows administrators get more used to scripting solutions to their automation problems with PowerShell (which I am confident that they are doing), it’s something that every organization will need to consider.  I’ll try to follow up with some posts that have some actual code to address some of these points.

Mike

P.S. I’m specifically not discussing “enterprise job scheduling” solutions like JAMS because of the high cost involved. I’d like to see the community come up with something a little more budget-friendly.

I selected MongoDB from the list because it looked more like something that would be used on smaller-scale projects.

I then googled “MongoDB PowerShell” and found this link from Doug Finke about using MongoDB with PowerShell (and F#, which is another “cool thing” I haven’t managed to find a need for).  Doug links to another article which explains setting up MongoDB and an open-source .Net driver for MongoDB called mongo-csharp.  He then follows up with a straight-forward script showing simple usage of MongoDB.  It looks like an almost literal translation of the C# code from the article he references.  With those in hand, I thought it was going to be a slam dunk.

It was, but I had a few hurdles to get over before I could get it working.  There weren’t any problems with the code;  it was written about a year ago, so it was using PowerShell 1.0 and an older version of mongo-csharp.  I had to update the script in a couple of places to make it work.  I probably wouldn’t even write it up, given how minor the changes are, but I was somewhat disappointed with the number of hits I got for “MongoDB PowerShell”.

Here’s the updated script:

add-type -Path .MongoDB.Driver.dll

$mongo=new-object mongodb.driver.mongo
$mongo.connect()
$db=$mongo.GetDataBase("movieReviews")
$movies=$db.GetCollection('Movies')

$movie=new-object Mongodb.driver.document
$movie['title']='Star Wars'
$movie['releaseDate']=get-date
$movies.Insert($movie)

$spec=new-object Mongodb.driver.document
$spec['title']='Star Wars'
$movies.FindOne($spec)

The two changes were

1. added the parameter name (-path) to add-type, since the default param is looking for an assembly name, not a path to a DLL
2. changed $mongo.GetDB to $mongo.GetDatabase to reflect a change in the driver

Worked fine. Not sure how I’ll use this, but if I need to, I know I can from PowerShell.

-Mike

I usually avoid nullable columns, but sometimes date fields make sense to be null (rather than use sentinel values like 1/1/1900).  In this case, I had a nullable date column and I needed to check in PowerShell whether the field was in fact null or not.  In SQL, I would have just used an IS NULL, or used the IsNull() function to replace the null value with something a little easier to deal with.  My first (feeble) attempt was to do this:

if (!$_.completedDate){
# it’s null
}

Unfortunately for me, that doesn’t work.  Next, I used this (which worked, but wasn’t very satisfactory either):

if ($_.completedDate.ToString() -eq ''){
# it’s null
}

Realizing that I was being stupid, I googled “PowerShell SQL NULL and after looking at several pages which didn’t really address the issue, I found this. A little work to change it into a function, and voilà.

function is-null($value){
  return  [System.DBNull]::Value.Equals($value)
}

A few quick tests and this is what I wanted. Now, my code looks like this:

if (is-null $_.completedDate){
# it’s null
}

I find it hard to believe I haven’t written this function before (or seen it).

By the way…be watching the SQL PowerShell Extensions project.  Chad released version 2.1, which includes SQL mode for the ISE (really nice).  I know he and several others are collaborating on an update which should be out sometime soon.

-Mike

What I posted yesterday as a bug in PowerShell (having to do with recursive functions, dot-sourcing, and parameters) seemed during my debugging session to clearly be a bug. After all, I watched the parameter value change from b to a, didn’t I? Sure did. And in almost every language I’ve ever used, that would be a bug. On the other hand, PowerShell is the only language that I know of that has dot-sourcing. Here’s a much simpler code example which shows my faulty thinking:

function f($x){
   if ($x -eq 1){
      write-host $x
      . f ($x+1)
      write-host $x
   }
}

f 1

Here, we have a simple “recursive function” which uses dot-sourcing to call itself. In my mind, how this would have worked is as follows:

• We call the function, passing 1 for $x
• The if condition is true, so it prints 1 and calls the function, passing 2 for $x
• In the inner call, the if condition is false, so nothing happens
• We pop back to the calling frame, where $x is 1 and print it

If it weren’t for that pesky dot operator, that would have been accurate.

The problem is, the dot operator changes the scoping of the inner call.  Here’s what the about_operators help topic, has to say about the dot sourcing operator:

        Description: Runs a script so that the items in the script are part of the calling scope.

Which is not a surprise…really.  The reason I was using the dot operator in my package management code was to make sure that functions defined in the scripts it was calling would be included in the existing scope, rather than their script scope.  The problem was one of nearsightedness.  I was so focused on the fact that the dot sourcing was making the functions part of the caller’s scope that I didn’t consider that variable declarations (including parameters) would also be in the caller’s scope.

So, the correct interpretation of the above script is:

• We call the function, passing 1 for $x
• The if condition is true, so it prints 1 and calls the function, passing 2 for $x
• The parameter is named $x, so $x in is set to 2 (overwriting the $x that was set to 1)
• In the inner call, the if condition is false, so nothing happens
• We pop back to the calling frame, where $x is 2 and print 2.

The trick here is that the function f dot-sourced something that set $x to 2.  The fact that it was f is incidental.  It didn’t have to be.

Maybe this example will make it more clear:

function f($x){
    write-host $x
    . g
    write-host $x
 }

function g{
   $x = "Hello, World!"
}
f 1

If we were doing this without dot-sourcing, we would expect to see the number 1 printed out twice. However, since we dot-sourced g, the assignment in the function body of g happens in the scope of f. In other words, it’s as if the $x=”Hello, World!” were executed inside f. Thus, the output of this script is 1, followed by “Hello, World!”.

So, it wasn’t a bug, it was just me not being thorough in applying my understanding of dot-sourcing.

Now, on with Package Management.
First, to fix the problem caused by the parameter being overwritten (which it is, it’s just that it’s expected to be). I hadn’t worked out a way to fix the problem before I went to bed last night, but as I was rolling this stuff around in my head (which is when I figured out that it wasn’t really a bug), I thought of a simple solution. Since we can expect that sometimes the $filename parameter in the require (and reload) function will be overwritten by the a value in the dot-sourced script, we just need to make sure we’re done using it at that point. So, I simply made the assignment to the dictionary before dot-sourcing.  Here’s the updated code:

$global:loaded_scripts=@{pkg_utils='INITIAL'}

function require($filename){
	if (!$global:loaded_scripts[$filename]){
	   $global:loaded_scripts[$filename]=get-date
	   . scripts:$filename.ps1
	}
}
function reload($filename){
	$global:loaded_scripts[$filename]=get-date
	. scripts:$filename.ps1
}

To add modules, we need to do a few extra things:

• We need to detect if we’re running in 2.0 or not
• We need to see if there is a module with the given name
• We need to see if the module is already loaded or not (in the case of require…it won’t matter for reload

Fortunately, none of those are very difficult.  Here’s the updated code (including modules). I even added some comments to make the flow more clear:

$global:loaded_scripts=@{pkg_utils='INITIAL'}

function require($filename){
    if ($global:loaded_scripts[$filename]){
          # this function has already loaded this (script or module)
          return
    }
    if ($psversiontable){
       # we're in 2.0
       if (get-module $filename -listavailable){
               #the module exists in the module path
         	   $global:loaded_scripts[$filename]=get-date
               import-module $filename
               return
       }
    }
    #it wasn't a module...so dot-source the script
    $global:loaded_scripts[$filename]=get-date
    . scripts:$filename.ps1

}
function reload($filename){
    if ($psversiontable){
        # we're in 2.0
        if (get-module $filename -listavailable){
           #the module exists in the module path
           $global:loaded_scripts[$filename]=get-date
           import-module $filename
           return
        }
    }
    # it wasn't a module...so dot-source the script.
  	$global:loaded_scripts[$filename]=get-date
	. scripts:$filename.ps1
}

That’s it for today. Let me know what you think.

-Mike

For a long time now, I’ve been dissatisfied with what I call “package management” in PowerShell.  Those of you who know me will be shocked that anything in PowerShell is less than perfect in my eyes, but this is one place that I feel let down.  Modules in 2.0 remedy the situation somewhat, but it still isn’t quite what I want or am used to in other languages.

Let me give an example.  In VB.NET, if you need to use the functions in an assembly, you put “Imports AssemblyName” at the top of your script.  In C#, you would have “Using AssemblyName”.  In Python, there would be “Import Something”.

In PowerShell 1.0, you had nothing.  In 2.0, you could create a module manifest which would specify either RequiredModules or ScriptsToProcess (or several other things to do upon loading the module).  The problems I see  with using the module manifest are:

• What if I’m not writing a module?  There’s no such thing as a “script manifest”
• What if the script or module that is required performs some initialization that should only be done once per session?
• What if the script or module that is required performs expensive initialization?

Because of these reasons (and because I only started using 2.0 when it went RTM) I wrote a couple of quick functions to do what I thought made sense.

$global:loaded_scripts=@{pkg_utils='INITIAL'}

function require($filename){
	if (!$global:loaded_scripts[$filename]){
	   . scripts:$filename.ps1
	   $global:loaded_scripts[$filename]=get-date
	}
}
function reload($filename){
	. scripts:$filename.ps1
	$global:loaded_scripts[$filename]=get-date
}

To use these you need to create a psdrive called scripts: with code like this (probably in your profile):

New-PSdrive -name scripts -PSprovider filesystem -root PathToYourLibraries | Out-Null

Then, also in your profile, you’ll want to dot-source the file you put these functions in (for example, package_tools.ps1):

. scripts:package_tools.ps1

Once you have those set up, you can dot-source the require function to make sure that a script has been loaded as such:

. require somelibrary

I have the functions I use divided by “subject” into several library scripts, and make sure that at the top of each script, I use “. require” to ensure that any prerequisites are already loaded.

Now for the PowerShell bug (which took me a long time to track down).
Create 2 files, a.ps1 and b.ps1 in your scripts: directory.

# a.ps1
write-host "this is script a"

#b.ps1
write-host "this is script b"
write-host "this script loads a"
. require a

After dot-sourcing package_tools, run the following commands:

. require b

You should get output that looks something like this:

this is script b
this script loads a
this is script a

Everything looks good until you inspect the $global:loaded_scripts variable:

ps> $loaded_scripts

Name                           Value
----                           -----
a                              1/19/2010 11:23:09 PM
package_tools                  INITIAL

Although b.ps1 was indeed dot-sourced (you can see the output), and the only code-path through the require function that would dot-source it would also add an entry to $loaded_scripts, there is no such entry. The problem is that when b.ps1 called the require function (to load a.ps1), the $filename variable in the calling context (where it should have been “b”) was overwritten by the call with “a” as a parameter. Walking through the code in a debugger confirms the problem.

Have you ever seen problems with recursion and dot-sourcing in PowerShell? Can you see any way around the problem I’ve described? For instance, saving the $filename in a variable and restoring it after the dot-source call (line 5 above) doesn’t help, because the same code-path is followed in the recursive call, and that variable is overwritten as well.

Even with this bug, I find the require function (and reload, which I didn’t discuss, but always loads the script in question) to be very helpful. I also have extended them to include importing modules, if they exist. I’ll discuss them in my next post, coming soon.

-Mike

P.S. Here‘s a question I posted to StackOverflow.com about these functions back in November of 2008.

Chad Miller, the driving force behind SQLPSX, has put a lot of effort into this release, and you’ll find really good examples of advanced functions (with comment-based help, even).

If you deal with SQL Server in any way, you’ll almost certainly be able to use this set of modules to streamline your scripting experience (and probably learn something about SMO in the process).

You can find the release here.

The event message we were interested in looked something like “Login succeeeded for uesr ‘UserName’ ….”.  The code we were trying to use was:

get-eventlog -computername $servername -logname Application -message "Login succeeded for user*" -after ((get-date).AddDays(-1))

I expected that, given a date parameter and a leading string to match wouldn’t be too bad, but this ended up taking several minutes per server.  As there are over a hundred servers to scan, that didn’t work well for us.

We ended up falling back to get-wmiobject.

$BeginDate=[System.Management.ManagementDateTimeConverter]::ToDMTFDateTime((get-date).AddDays(-1))
get-wmiobject -class win32_ntlogevent -computerName $servername -filter "(EventCode=18453)  and (LogFile='Application') and (TimeGenerated >'$BeginDate')"

Cons:

• We have to encode the date parameter (instead of using a nice datetime parameter like get-eventlog has)
• We have to write a WQL where-clause to match the parameters
  

Pros:

• We get to use the event code (rather than a string match)
• The code is orders of magnitude faster (39 servers in 13 minutes as a test case)
  

I think that you might have a positive experience using get-eventlog if you need to scan a range of time (for instance if you’re reporting on what happened on the server), but if you need to look for a specific event (or set of events) you’re probably going to want to use get-wmiobject.

-Mike

I had never noticed that parameter and was also annoyed by writing this all the time:

 get-something | foreach {$_.SomeProperty}

It was an idiom that I was using a lot that felt like it didn’t fit.</p> <p>As he points out, this can be replaced with the non-looping:

get-something | select-object -expandProperty SomeProperty

It’s longer if you don’t use aliases (and prefix-shortened parameternames), but I think it reads a lot better.

Let me know what you think. Was this a surprise to you, or have you used the -expandProperty parameter before?

Mike

P.S. You should definitely follow the PowerShell Team Blog…it is always worthwhile.