If I were writing a script that needed to get a “bunch of things” from somewhere (perhaps several different sources) and return all of them, I might be tempted to do something like this. Please forgive my PowerShell pseudocode:

function get-stuff{
param($parm1 )
    $results=@()
    foreach ($source in $sources){
        $results += ($source | where { $_ -and "Some condition exists"  })
    }
    foreach ($source in $someothersources){
        $results += ($source | where { $_ -and "Some condition exists"  })
    }
 return $results
}

I’ve used several permutations of that kind of code using arrays of some sort to collect the results as I go along and eventually return the collection from the function. I’m not sure that there’s anything wrong with doing it this way. That is, I’m not sure that you’re likely to have issues with doing it this way.

On the other hand, it’s more idiomatic (i.e more in the style of the PowerShell language) to do something like this (again, pardon the pseudocode):

function get-stuff{
param($parm1 )
    foreach ($source in $sources){
        $source | where { $_ -and "Some condition exists"  }
    }
    foreach ($source in $someothersources){
        $source | where { $_ -and "Some condition exists"  }
    }

}

All I’m doing here is sending the output of the inner statements (which are pipelines) to the output stream of the function. Note that there isn’t any need for anything to accumulate the results into.  Using the output stream makes this function work more like the built-in cmdlets in PowerShell as it won’t be blocking the pipeline.

The only thing that I have against this code is that it goes against rule #2 that I wrote last time about writing values to the output stream. I said there that if you were going to write to the output stream, you should explicitly use write-output. We could modify the code above to use write-output, but that would involve using parentheses (around the pipelines), messing up the flow of the code, and even blocking the pipeline while the expressions in the parentheses were collected (as an argument to write-output).

That brings me to what I was saying about “another pointless function”. About a year ago I wrote a post about the identity function, which doesn’t really do anything except return the input. It is a really useful function for creating lists and such, allowing you to skip on providing a bunch of punctuation. It’s not a pointless function, but it’s not one that is getting much press, either. I was thinking about how to make the “pipeline” version of the code work nicely and not make it ugly and thought of the following function.

function out-output{
    process{ $_ }
}

Like the identity function (or ql, as I’ve seen it referred to), out-output doesn’t do anything but emit values that are provided. Out-output, however, gets its values from the pipeline rather than the argument list. This function allows us to be explicit about our intent to use the output stream.

function get-stuff{
param($parm1 )
    foreach ($source in $sources){
        $source | where { $_ -and "Some condition exists"  } | out-output
    }
    foreach ($source in $someothersources){
        $source | where { $_ -and "Some condition exists"  } | out-output
    }
}

I’m not sure if this is a good idea, and I know that it’s just adding a tiny bit of processing to the script. My thought is that making the operation of the script explicit is worth it in the long run.

What do you think?

-mike

In most languages, if you see “return 1″ as the only return in a function, you can know that the function is going to the value 1 to the caller. In fact, I’m not sure I’ve ever seen a language that didn’t work that way. That is, until I found PowerShell. Generally speaking, the return statement works just as expected. In the absence of any statements writing to the output stream (with write-output) or “dropping” their values, “return 1″ will cause the caller to receive the value “1″. Using write-output is pretty obvious, and I’d recommend using it explicitly if you intend to add objects to the output stream (thereby including them in the eventual function value). Expressions that don’t capture their return values, however, are not quite so easy to spot.

For example, examine this code to add a parameter to an ADO.NET command object looks fine:

$cmd.Parameters.AddWithValue('@demographics','$demoXML)

This is a straightforward translation of one of the lines of code in the example code here.  The problem with the code is that AddWithValue not only adds a parameter, it also returns the parameter.  Since we didn’t assign it to a variable, cast it to [void] or pipe it to out-null, the output of this function (AddWithValue) gets added to the output of the function it’s in.

Several “add” functions in the .net framework follow this pattern, either returning the object that was added or the index of that object in the collection.  The DBConnection.Open method (inherited by SQLConnection, among others) returns the opened connection.  I’m sure that with time I could find more examples than I’d feel like sharing.

Another way that I’ve seen the output stream getting messed up is when a function uses strings to output information without using write-host.  For example, this function outputs “progress” information as it goes:

function get-filelength{
param($filename)
	"reading file"
	$len=0
	$file=get-content $filename
	foreach($line in $file){
	   "adding a line"
	   $len+=$line.length
	}
	return $len
}

I realize that this is not the best way to find the length of a file, but it works for the purpose of illustration. If you simply call the function (without assigning it to a variable), you’ll see a bunch of text, followed by the “value” of the function. The “returned value” of the function is clearly intended to be $len, but instead all of the text is included. The type of the function output is Object[], rather than a number.

There are some good reasons to use the output stream in a function (the next post will examine one such use), but it’s very easy to accidentally put something in the stream without intending to. When this happens, it often leads to unexpected results and long debugging sessions.

Here are my recommendations for avoiding this kind of error:

1. Use one of the “write” functions to provide text messages to the user (write-host, write-debug, write-verbose)
2. Use write-output if your intent is to add something to the output stream
3. Check msdn to see if the methods you use have output. If you’re looking at the C# version, methods that return void do not need to be “handled”. In the VB.NET version, they’ll be declared as Sub, rather than Function.
4. Be consistent with how you “handle” methods or functions whose values you want to discard. Options are to cast the result as [void] or pipe it to out-null
5. If you’re using write-output to put objects in the output stream, use return with no value (i.e., don’t return $val). Seeing a value returned explicitly implies that that is the only value returned.
6. If all else fails, use write-host to output the type of variables when debugging (or look at their values in the watch window if you’re using an IDE with a debugger).

For those of you that are with me in the PowerShell camp, I have something that I’d like to discuss.  You probably have hundreds (dozens?) of scripts scheduled on multiple servers, possibly in multiple domains or geographical locations to perform things like these:

• Gather information about servers
• Generate reports about application usage
• Copy information from one place to another
• Validate security setup
• Start and Stop processes
• Scan log files for error conditions
• Lots of other things (you get the point)

How do you know that the scripts that you have written carefully and scheduled are actually running successfully?  At first, this seems like a silly question.  When you deployed the script, surely you ran it once to make sure it worked.  What could have gone wrong?

Here are some examples that come to mind:

• A policy was pushed which set the execution policy to Restricted
• The credentials you scheduled the script with have been revoked
• A file share that the scripts depend on is unavailable
• Firewall rules change and now WMI queries aren’t working
• The Task Scheduler service is stopped

You can probably think of a lot more examples of things that would keep scripts from working, but you get the idea.  I’ve given some thought to how to do this, but haven’t come to any real conclusions.  Obviously, having your scripts log results is helpful, but only if you monitor the logs for success/failure.  Also, if you have a script which is supposed to run every 10 minutes, it doesn’t help if you don’t get alerted when it only runs once in a day, even if  it runs successfully.  Also, if there is more than one person writing scripts, how do you make sure that everyone is using the same techniques to log progress?

Here are some of my thoughts:

• Use a “launcher” to run scripts (see below)
• Keep a database of processes with an expected # of runs per day
• Monitor matching start/end of scripts
• Log all output streams (example)

The first item in the list (the launcher) has been something I’ve been considering because it’s not trivial to run a PowerShell script in a scheduled task.  Even with the -file parameter which was added in PowerShell 2.0, it can involve a fairly long command-line.  With the added difficulty of trying to capture output streams (most of which are not exposed to the command shell) it becomes a process that is almost hard to get right every time.  Some features I’m planning for the launcher are:

• Load appropriate profiles
• Log all output streams (with timestamps) to file or database
• Log application start/end times
• Email owner of script if there are unhandled exceptions

I know this topic is not specific to PowerShell, but as Windows administrators get more used to scripting solutions to their automation problems with PowerShell (which I am confident that they are doing), it’s something that every organization will need to consider.  I’ll try to follow up with some posts that have some actual code to address some of these points.

Mike

P.S. I’m specifically not discussing “enterprise job scheduling” solutions like JAMS because of the high cost involved. I’d like to see the community come up with something a little more budget-friendly.

• Write a post or 2 about remoting
• Kill off the powershellworkbench project (I can’t stand writing WPF)
• Start a new host project using Windows Forms and the Puzzle.SyntaxBox edit control (probably using C#)
• Update the Books/Tools/Sites pages (and keep them more or less up to date)

I was really hoping that there would be some interest in the powershellworkbench project, but I’ve not heard anything (except for a comment that there were some missing icons).  Since I really didn’t enjoy writing WPF code (I might like it if I did a lot of it, but this is a hobby project), I don’t see much of a future for that project.  I am comfortable with Windows Forms, though, so I’ll be rebooting that project with a new name.  I’m also going to switch to the Puzzle.SyntaxBox control, because I know it will do the things I need to and not get in the way.  I haven’t found a similar component that wasn’t extremely complicated.  My goal with the project is to make something that an administrator with limited time could customize for their environment.  I’m afraid that the complexity of WPF and the Avalon controls would make that improbable at best.

I was also hoping that there would be more feedback.  I’m sure that this is a common frustration among bloggers (especially bloggers in a niche like this), so I’m not going to worry about that.  I am, however, going to make a concerted effort to comment on blogs that I read.  If you have any comments, they are always welcome.  If you have suggestions on topics, I’m open to that, as well.

One final thing.  I’ve been thinking about moving my blog to “No Support Linux Hosting”.  I’m not a Linux expert, but they have a WordPress setup tutorial, and you can’t beat $1 per month.  If I do change, I hope I can do it without any disruption, but who knows?

-Mike

What I posted yesterday as a bug in PowerShell (having to do with recursive functions, dot-sourcing, and parameters) seemed during my debugging session to clearly be a bug. After all, I watched the parameter value change from b to a, didn’t I? Sure did. And in almost every language I’ve ever used, that would be a bug. On the other hand, PowerShell is the only language that I know of that has dot-sourcing. Here’s a much simpler code example which shows my faulty thinking:

function f($x){
   if ($x -eq 1){
      write-host $x
      . f ($x+1)
      write-host $x
   }
}

f 1

Here, we have a simple “recursive function” which uses dot-sourcing to call itself. In my mind, how this would have worked is as follows:

• We call the function, passing 1 for $x
• The if condition is true, so it prints 1 and calls the function, passing 2 for $x
• In the inner call, the if condition is false, so nothing happens
• We pop back to the calling frame, where $x is 1 and print it

If it weren’t for that pesky dot operator, that would have been accurate.

The problem is, the dot operator changes the scoping of the inner call.  Here’s what the about_operators help topic, has to say about the dot sourcing operator:

        Description: Runs a script so that the items in the script are part of the calling scope.

Which is not a surprise…really.  The reason I was using the dot operator in my package management code was to make sure that functions defined in the scripts it was calling would be included in the existing scope, rather than their script scope.  The problem was one of nearsightedness.  I was so focused on the fact that the dot sourcing was making the functions part of the caller’s scope that I didn’t consider that variable declarations (including parameters) would also be in the caller’s scope.

So, the correct interpretation of the above script is:

• We call the function, passing 1 for $x
• The if condition is true, so it prints 1 and calls the function, passing 2 for $x
• The parameter is named $x, so $x in is set to 2 (overwriting the $x that was set to 1)
• In the inner call, the if condition is false, so nothing happens
• We pop back to the calling frame, where $x is 2 and print 2.

The trick here is that the function f dot-sourced something that set $x to 2.  The fact that it was f is incidental.  It didn’t have to be.

Maybe this example will make it more clear:

function f($x){
    write-host $x
    . g
    write-host $x
 }

function g{
   $x = "Hello, World!"
}
f 1

If we were doing this without dot-sourcing, we would expect to see the number 1 printed out twice. However, since we dot-sourced g, the assignment in the function body of g happens in the scope of f. In other words, it’s as if the $x=”Hello, World!” were executed inside f. Thus, the output of this script is 1, followed by “Hello, World!”.

So, it wasn’t a bug, it was just me not being thorough in applying my understanding of dot-sourcing.

Now, on with Package Management.
First, to fix the problem caused by the parameter being overwritten (which it is, it’s just that it’s expected to be). I hadn’t worked out a way to fix the problem before I went to bed last night, but as I was rolling this stuff around in my head (which is when I figured out that it wasn’t really a bug), I thought of a simple solution. Since we can expect that sometimes the $filename parameter in the require (and reload) function will be overwritten by the a value in the dot-sourced script, we just need to make sure we’re done using it at that point. So, I simply made the assignment to the dictionary before dot-sourcing.  Here’s the updated code:

$global:loaded_scripts=@{pkg_utils='INITIAL'}

function require($filename){
	if (!$global:loaded_scripts[$filename]){
	   $global:loaded_scripts[$filename]=get-date
	   . scripts:$filename.ps1
	}
}
function reload($filename){
	$global:loaded_scripts[$filename]=get-date
	. scripts:$filename.ps1
}

To add modules, we need to do a few extra things:

• We need to detect if we’re running in 2.0 or not
• We need to see if there is a module with the given name
• We need to see if the module is already loaded or not (in the case of require…it won’t matter for reload

Fortunately, none of those are very difficult.  Here’s the updated code (including modules). I even added some comments to make the flow more clear:

$global:loaded_scripts=@{pkg_utils='INITIAL'}

function require($filename){
    if ($global:loaded_scripts[$filename]){
          # this function has already loaded this (script or module)
          return
    }
    if ($psversiontable){
       # we're in 2.0
       if (get-module $filename -listavailable){
               #the module exists in the module path
         	   $global:loaded_scripts[$filename]=get-date
               import-module $filename
               return
       }
    }
    #it wasn't a module...so dot-source the script
    $global:loaded_scripts[$filename]=get-date
    . scripts:$filename.ps1

}
function reload($filename){
    if ($psversiontable){
        # we're in 2.0
        if (get-module $filename -listavailable){
           #the module exists in the module path
           $global:loaded_scripts[$filename]=get-date
           import-module $filename
           return
        }
    }
    # it wasn't a module...so dot-source the script.
  	$global:loaded_scripts[$filename]=get-date
	. scripts:$filename.ps1
}

That’s it for today. Let me know what you think.

-Mike

For a long time now, I’ve been dissatisfied with what I call “package management” in PowerShell.  Those of you who know me will be shocked that anything in PowerShell is less than perfect in my eyes, but this is one place that I feel let down.  Modules in 2.0 remedy the situation somewhat, but it still isn’t quite what I want or am used to in other languages.

Let me give an example.  In VB.NET, if you need to use the functions in an assembly, you put “Imports AssemblyName” at the top of your script.  In C#, you would have “Using AssemblyName”.  In Python, there would be “Import Something”.

In PowerShell 1.0, you had nothing.  In 2.0, you could create a module manifest which would specify either RequiredModules or ScriptsToProcess (or several other things to do upon loading the module).  The problems I see  with using the module manifest are:

• What if I’m not writing a module?  There’s no such thing as a “script manifest”
• What if the script or module that is required performs some initialization that should only be done once per session?
• What if the script or module that is required performs expensive initialization?

Because of these reasons (and because I only started using 2.0 when it went RTM) I wrote a couple of quick functions to do what I thought made sense.

$global:loaded_scripts=@{pkg_utils='INITIAL'}

function require($filename){
	if (!$global:loaded_scripts[$filename]){
	   . scripts:$filename.ps1
	   $global:loaded_scripts[$filename]=get-date
	}
}
function reload($filename){
	. scripts:$filename.ps1
	$global:loaded_scripts[$filename]=get-date
}

To use these you need to create a psdrive called scripts: with code like this (probably in your profile):

New-PSdrive -name scripts -PSprovider filesystem -root PathToYourLibraries | Out-Null

Then, also in your profile, you’ll want to dot-source the file you put these functions in (for example, package_tools.ps1):

. scripts:package_tools.ps1

Once you have those set up, you can dot-source the require function to make sure that a script has been loaded as such:

. require somelibrary

I have the functions I use divided by “subject” into several library scripts, and make sure that at the top of each script, I use “. require” to ensure that any prerequisites are already loaded.

Now for the PowerShell bug (which took me a long time to track down).
Create 2 files, a.ps1 and b.ps1 in your scripts: directory.

# a.ps1
write-host "this is script a"

#b.ps1
write-host "this is script b"
write-host "this script loads a"
. require a

After dot-sourcing package_tools, run the following commands:

. require b

You should get output that looks something like this:

this is script b
this script loads a
this is script a

Everything looks good until you inspect the $global:loaded_scripts variable:

ps> $loaded_scripts

Name                           Value
----                           -----
a                              1/19/2010 11:23:09 PM
package_tools                  INITIAL

Although b.ps1 was indeed dot-sourced (you can see the output), and the only code-path through the require function that would dot-source it would also add an entry to $loaded_scripts, there is no such entry. The problem is that when b.ps1 called the require function (to load a.ps1), the $filename variable in the calling context (where it should have been “b”) was overwritten by the call with “a” as a parameter. Walking through the code in a debugger confirms the problem.

Have you ever seen problems with recursion and dot-sourcing in PowerShell? Can you see any way around the problem I’ve described? For instance, saving the $filename in a variable and restoring it after the dot-source call (line 5 above) doesn’t help, because the same code-path is followed in the recursive call, and that variable is overwritten as well.

Even with this bug, I find the require function (and reload, which I didn’t discuss, but always loads the script in question) to be very helpful. I also have extended them to include importing modules, if they exist. I’ll discuss them in my next post, coming soon.

-Mike

P.S. Here‘s a question I posted to StackOverflow.com about these functions back in November of 2008.

I had never noticed that parameter and was also annoyed by writing this all the time:

 get-something | foreach {$_.SomeProperty}

It was an idiom that I was using a lot that felt like it didn’t fit.</p> <p>As he points out, this can be replaced with the non-looping:

get-something | select-object -expandProperty SomeProperty

It’s longer if you don’t use aliases (and prefix-shortened parameternames), but I think it reads a lot better.

Let me know what you think. Was this a surprise to you, or have you used the -expandProperty parameter before?

Mike

P.S. You should definitely follow the PowerShell Team Blog…it is always worthwhile.

Before actually writing anything, it’s good to ask yourself…why in the world would I write a host when there are so many out there already (ISE and PowerGUI are notable free examples)?  This is a really important question and one that will stop most projects in their tracks.  Most people can get what they need using an existing host.  Here are some of the reasons I chose to write  a host:

• I wanted complete control over the environment, as I knew (hoped) that I would be spending a lot of time using it
• I wanted to be able to interact with the environment in ways that the existing tools didn’t allow
• I was constrained to use PowerShell 1.0 (which eliminates the ISE)

But probably the most pressing reason in reality was:

• I had a book (link) that explained the technology and I wanted to play 

Unlike most (some?) administrators, I have a development background and even have Visual Studio installed on my machine, so testing the waters of writing a host wasn’t a big investment of time, and the pleasure of seeing something like this come together was well worth it.

Here are the posts that got my mind going again:

Create your own IDE in 10 minutes

How to Host PowerShell in a WPF Application

In the next post, I’ll start the project and give you something to look at.

Let me know if there’s anything specific you’d like to see (or have experience implementing).

Mike

Scripting is the short answer to the dilemma above.  PowerShell is one of the latest entries into the scripting world, and to my tastes, one of the best.

Here’s something I used several times in the last few days.  I can’t remember quite where I saw it first, but it was in a PowerShell blog about looping (I think).

Anyway, the problem is that I needed to edit config files on the servers in a farm.  Fortunately, the servers were numbered sequentially.  So, what I wrote was (suitably sanitized for public consumption):

1..9 | % { notepad "server$_c$path_to_config_fileconfig.file" }

That popped the first 9 files up in notepad, ready to be edited.  The trick is to use the range syntax to create a list of numbers, and use % to loop through them.

If you need a longer range (with leading zeroes, of course), it’s not too hard.

1..20 | % {notepad ("server{0:D2}c$path_to_config_fileconfig.file" -f $_)}

Here, we use the format operator with a D2 format spceifier (2 digits, leading zeros).  See here for more examples of format operators in PowerShell.

When you’re dealing with dozens of servers, tricks like this can save you a lot of time.

Let me know what you think.  What “idioms” in PowerShell do you find yourself using a lot?

Mike

A (very) basic SQL statement has the form:

SELECT <COLUMNS>
FROM <TABLE>
WHERE <CONDITION>
ORDER BY <EXPRESSION>

I noticed that a very common idiom for PowerShell pipelines* was:

<data source cmdlet> | select-object <properties> | where-object <CONDITION> |  sort-object <EXPRESSION>
 

By “<data source cmdlet>”, I mean some cmdlet that puts a bunch of objects in the pipeline, like get-childitem, get-process, get-task, etc.

Part of the power of SQL is that it doesn’t matter what kind of data is in the tables, the same form of SQL statement works the same way (predictability).  This is one of the things I love about PowerShell.  It doesn’t matter what kinds of data is returned by a cmdlet.  The same form of PowerShell pipeline* will perform the same kind of predictable operations on it.  I know that this is often mentioned in tutorials and videos about PowerShell, but this was when it really struck me. 

A few other SQL/PowerShell comparisons might be:

Obviously, this comparison breaks down pretty quickly.  There isn’t really a parallel that I can find to JOIN statements, which make SQL so powerful, and clearly there’s a lot of powershell scripts that don’t fit the pattern I’m describing.  I think, though, that it’s a useful comparison and can help get people “over the hump” in their quest to master PowerShell.

Let me know what you think.

Mike

* A Pipeline in PowerShell is a sequence of cmdlets where each takes the output of the previous cmdlet as its input.