For those of you that are with me in the PowerShell camp, I have something that I’d like to discuss.  You probably have hundreds (dozens?) of scripts scheduled on multiple servers, possibly in multiple domains or geographical locations to perform things like these:

• Gather information about servers
• Generate reports about application usage
• Copy information from one place to another
• Validate security setup
• Start and Stop processes
• Scan log files for error conditions
• Lots of other things (you get the point)

How do you know that the scripts that you have written carefully and scheduled are actually running successfully?  At first, this seems like a silly question.  When you deployed the script, surely you ran it once to make sure it worked.  What could have gone wrong?

Here are some examples that come to mind:

• A policy was pushed which set the execution policy to Restricted
• The credentials you scheduled the script with have been revoked
• A file share that the scripts depend on is unavailable
• Firewall rules change and now WMI queries aren’t working
• The Task Scheduler service is stopped

You can probably think of a lot more examples of things that would keep scripts from working, but you get the idea.  I’ve given some thought to how to do this, but haven’t come to any real conclusions.  Obviously, having your scripts log results is helpful, but only if you monitor the logs for success/failure.  Also, if you have a script which is supposed to run every 10 minutes, it doesn’t help if you don’t get alerted when it only runs once in a day, even if  it runs successfully.  Also, if there is more than one person writing scripts, how do you make sure that everyone is using the same techniques to log progress?

Here are some of my thoughts:

• Use a “launcher” to run scripts (see below)
• Keep a database of processes with an expected # of runs per day
• Monitor matching start/end of scripts
• Log all output streams (example)

The first item in the list (the launcher) has been something I’ve been considering because it’s not trivial to run a PowerShell script in a scheduled task.  Even with the -file parameter which was added in PowerShell 2.0, it can involve a fairly long command-line.  With the added difficulty of trying to capture output streams (most of which are not exposed to the command shell) it becomes a process that is almost hard to get right every time.  Some features I’m planning for the launcher are:

• Load appropriate profiles
• Log all output streams (with timestamps) to file or database
• Log application start/end times
• Email owner of script if there are unhandled exceptions

I know this topic is not specific to PowerShell, but as Windows administrators get more used to scripting solutions to their automation problems with PowerShell (which I am confident that they are doing), it’s something that every organization will need to consider.  I’ll try to follow up with some posts that have some actual code to address some of these points.

Mike

P.S. I’m specifically not discussing “enterprise job scheduling” solutions like JAMS because of the high cost involved. I’d like to see the community come up with something a little more budget-friendly.

I selected MongoDB from the list because it looked more like something that would be used on smaller-scale projects.

I then googled “MongoDB PowerShell” and found this link from Doug Finke about using MongoDB with PowerShell (and F#, which is another “cool thing” I haven’t managed to find a need for).  Doug links to another article which explains setting up MongoDB and an open-source .Net driver for MongoDB called mongo-csharp.  He then follows up with a straight-forward script showing simple usage of MongoDB.  It looks like an almost literal translation of the C# code from the article he references.  With those in hand, I thought it was going to be a slam dunk.

It was, but I had a few hurdles to get over before I could get it working.  There weren’t any problems with the code;  it was written about a year ago, so it was using PowerShell 1.0 and an older version of mongo-csharp.  I had to update the script in a couple of places to make it work.  I probably wouldn’t even write it up, given how minor the changes are, but I was somewhat disappointed with the number of hits I got for “MongoDB PowerShell”.

Here’s the updated script:

add-type -Path .MongoDB.Driver.dll

$mongo=new-object mongodb.driver.mongo
$mongo.connect()
$db=$mongo.GetDataBase("movieReviews")
$movies=$db.GetCollection('Movies')

$movie=new-object Mongodb.driver.document
$movie['title']='Star Wars'
$movie['releaseDate']=get-date
$movies.Insert($movie)

$spec=new-object Mongodb.driver.document
$spec['title']='Star Wars'
$movies.FindOne($spec)

The two changes were

1. added the parameter name (-path) to add-type, since the default param is looking for an assembly name, not a path to a DLL
2. changed $mongo.GetDB to $mongo.GetDatabase to reflect a change in the driver

Worked fine. Not sure how I’ll use this, but if I need to, I know I can from PowerShell.

-Mike

What I posted yesterday as a bug in PowerShell (having to do with recursive functions, dot-sourcing, and parameters) seemed during my debugging session to clearly be a bug. After all, I watched the parameter value change from b to a, didn’t I? Sure did. And in almost every language I’ve ever used, that would be a bug. On the other hand, PowerShell is the only language that I know of that has dot-sourcing. Here’s a much simpler code example which shows my faulty thinking:

function f($x){
   if ($x -eq 1){
      write-host $x
      . f ($x+1)
      write-host $x
   }
}

f 1

Here, we have a simple “recursive function” which uses dot-sourcing to call itself. In my mind, how this would have worked is as follows:

• We call the function, passing 1 for $x
• The if condition is true, so it prints 1 and calls the function, passing 2 for $x
• In the inner call, the if condition is false, so nothing happens
• We pop back to the calling frame, where $x is 1 and print it

If it weren’t for that pesky dot operator, that would have been accurate.

The problem is, the dot operator changes the scoping of the inner call.  Here’s what the about_operators help topic, has to say about the dot sourcing operator:

        Description: Runs a script so that the items in the script are part of the calling scope.

Which is not a surprise…really.  The reason I was using the dot operator in my package management code was to make sure that functions defined in the scripts it was calling would be included in the existing scope, rather than their script scope.  The problem was one of nearsightedness.  I was so focused on the fact that the dot sourcing was making the functions part of the caller’s scope that I didn’t consider that variable declarations (including parameters) would also be in the caller’s scope.

So, the correct interpretation of the above script is:

• We call the function, passing 1 for $x
• The if condition is true, so it prints 1 and calls the function, passing 2 for $x
• The parameter is named $x, so $x in is set to 2 (overwriting the $x that was set to 1)
• In the inner call, the if condition is false, so nothing happens
• We pop back to the calling frame, where $x is 2 and print 2.

The trick here is that the function f dot-sourced something that set $x to 2.  The fact that it was f is incidental.  It didn’t have to be.

Maybe this example will make it more clear:

function f($x){
    write-host $x
    . g
    write-host $x
 }

function g{
   $x = "Hello, World!"
}
f 1

If we were doing this without dot-sourcing, we would expect to see the number 1 printed out twice. However, since we dot-sourced g, the assignment in the function body of g happens in the scope of f. In other words, it’s as if the $x=”Hello, World!” were executed inside f. Thus, the output of this script is 1, followed by “Hello, World!”.

So, it wasn’t a bug, it was just me not being thorough in applying my understanding of dot-sourcing.

Now, on with Package Management.
First, to fix the problem caused by the parameter being overwritten (which it is, it’s just that it’s expected to be). I hadn’t worked out a way to fix the problem before I went to bed last night, but as I was rolling this stuff around in my head (which is when I figured out that it wasn’t really a bug), I thought of a simple solution. Since we can expect that sometimes the $filename parameter in the require (and reload) function will be overwritten by the a value in the dot-sourced script, we just need to make sure we’re done using it at that point. So, I simply made the assignment to the dictionary before dot-sourcing.  Here’s the updated code:

$global:loaded_scripts=@{pkg_utils='INITIAL'}

function require($filename){
	if (!$global:loaded_scripts[$filename]){
	   $global:loaded_scripts[$filename]=get-date
	   . scripts:$filename.ps1
	}
}
function reload($filename){
	$global:loaded_scripts[$filename]=get-date
	. scripts:$filename.ps1
}

To add modules, we need to do a few extra things:

• We need to detect if we’re running in 2.0 or not
• We need to see if there is a module with the given name
• We need to see if the module is already loaded or not (in the case of require…it won’t matter for reload

Fortunately, none of those are very difficult.  Here’s the updated code (including modules). I even added some comments to make the flow more clear:

$global:loaded_scripts=@{pkg_utils='INITIAL'}

function require($filename){
    if ($global:loaded_scripts[$filename]){
          # this function has already loaded this (script or module)
          return
    }
    if ($psversiontable){
       # we're in 2.0
       if (get-module $filename -listavailable){
               #the module exists in the module path
         	   $global:loaded_scripts[$filename]=get-date
               import-module $filename
               return
       }
    }
    #it wasn't a module...so dot-source the script
    $global:loaded_scripts[$filename]=get-date
    . scripts:$filename.ps1

}
function reload($filename){
    if ($psversiontable){
        # we're in 2.0
        if (get-module $filename -listavailable){
           #the module exists in the module path
           $global:loaded_scripts[$filename]=get-date
           import-module $filename
           return
        }
    }
    # it wasn't a module...so dot-source the script.
  	$global:loaded_scripts[$filename]=get-date
	. scripts:$filename.ps1
}

That’s it for today. Let me know what you think.

-Mike

For a long time now, I’ve been dissatisfied with what I call “package management” in PowerShell.  Those of you who know me will be shocked that anything in PowerShell is less than perfect in my eyes, but this is one place that I feel let down.  Modules in 2.0 remedy the situation somewhat, but it still isn’t quite what I want or am used to in other languages.

Let me give an example.  In VB.NET, if you need to use the functions in an assembly, you put “Imports AssemblyName” at the top of your script.  In C#, you would have “Using AssemblyName”.  In Python, there would be “Import Something”.

In PowerShell 1.0, you had nothing.  In 2.0, you could create a module manifest which would specify either RequiredModules or ScriptsToProcess (or several other things to do upon loading the module).  The problems I see  with using the module manifest are:

• What if I’m not writing a module?  There’s no such thing as a “script manifest”
• What if the script or module that is required performs some initialization that should only be done once per session?
• What if the script or module that is required performs expensive initialization?

Because of these reasons (and because I only started using 2.0 when it went RTM) I wrote a couple of quick functions to do what I thought made sense.

$global:loaded_scripts=@{pkg_utils='INITIAL'}

function require($filename){
	if (!$global:loaded_scripts[$filename]){
	   . scripts:$filename.ps1
	   $global:loaded_scripts[$filename]=get-date
	}
}
function reload($filename){
	. scripts:$filename.ps1
	$global:loaded_scripts[$filename]=get-date
}

To use these you need to create a psdrive called scripts: with code like this (probably in your profile):

New-PSdrive -name scripts -PSprovider filesystem -root PathToYourLibraries | Out-Null

Then, also in your profile, you’ll want to dot-source the file you put these functions in (for example, package_tools.ps1):

. scripts:package_tools.ps1

Once you have those set up, you can dot-source the require function to make sure that a script has been loaded as such:

. require somelibrary

I have the functions I use divided by “subject” into several library scripts, and make sure that at the top of each script, I use “. require” to ensure that any prerequisites are already loaded.

Now for the PowerShell bug (which took me a long time to track down).
Create 2 files, a.ps1 and b.ps1 in your scripts: directory.

# a.ps1
write-host "this is script a"

#b.ps1
write-host "this is script b"
write-host "this script loads a"
. require a

After dot-sourcing package_tools, run the following commands:

. require b

You should get output that looks something like this:

this is script b
this script loads a
this is script a

Everything looks good until you inspect the $global:loaded_scripts variable:

ps> $loaded_scripts

Name                           Value
----                           -----
a                              1/19/2010 11:23:09 PM
package_tools                  INITIAL

Although b.ps1 was indeed dot-sourced (you can see the output), and the only code-path through the require function that would dot-source it would also add an entry to $loaded_scripts, there is no such entry. The problem is that when b.ps1 called the require function (to load a.ps1), the $filename variable in the calling context (where it should have been “b”) was overwritten by the call with “a” as a parameter. Walking through the code in a debugger confirms the problem.

Have you ever seen problems with recursion and dot-sourcing in PowerShell? Can you see any way around the problem I’ve described? For instance, saving the $filename in a variable and restoring it after the dot-source call (line 5 above) doesn’t help, because the same code-path is followed in the recursive call, and that variable is overwritten as well.

Even with this bug, I find the require function (and reload, which I didn’t discuss, but always loads the script in question) to be very helpful. I also have extended them to include importing modules, if they exist. I’ll discuss them in my next post, coming soon.

-Mike

P.S. Here‘s a question I posted to StackOverflow.com about these functions back in November of 2008.

Chad Miller, the driving force behind SQLPSX, has put a lot of effort into this release, and you’ll find really good examples of advanced functions (with comment-based help, even).

If you deal with SQL Server in any way, you’ll almost certainly be able to use this set of modules to streamline your scripting experience (and probably learn something about SMO in the process).

You can find the release here.

• PowerGUI Script Editor (I haven’t ever gotten the hang of PowerGUI itself)
• Powershell Community Extensions 1.2  (PSCX)
• PowerTab
• SQL PowerShell Extensions 1.61 (SQLPSX)
• PrimalForms Community Edition

Books:

• PowerShell In Action by Bruce Payette
• Professional Windows PowerShell Programming: Snapins, Cmdlets, Hosts and Providers by  Arul Kumaravel et. al.
• Mastering PowerShell by Dr. Tobias Weltner

I’ve tried a lot of other tools (several IDE’s, for example), but this is the list I keep returning to.