PowerShell Station issues

Posted by on

I’m officially an idiot.  I am in good company, though.  The funny (not really funny) thing, is that I read Jeff’s post 2 years ago and did absolutely nothing about it.  So when my hosting provider had some issues and rebuilt the server this site is on, I was completely unprepared (i.e. I had no recent backups).

Let me say that I’m very pleased with nosupportlinuxhosting and especially how they handled this outage.  I was informed promptly about the issue, the solution they were pursuing, and was notified when they server was back online.  For a “no support” company, I can definitely say that I’ve had better responses from them than the previous company that was handling the site.

I hope to have (most of) the posts back online in the next day or 2.  Fortunately I did have a backup from january (ouch), but that was just posts, pages, and comments.  If you receive some posts in your feed reader over the next few days, that’s probably me adding posts from the last 9 months or so.  I doubt I’ll be able to reconstruct the comments, so I apologize for that lack.

The easy “take away” from this is:  Make sure you have offsite backups for all of your systems.

The not so easy “take away” from this is: No, really, make sure you have offsite backups for all of your systems.

Maybe I’ll learn my lesson this time.

Mike

PowerShell’s Problem with Return

Posted by on

I think that PowerShell is a fairly readable language, once you’re used to its unique features. Naming functions (cmdlets) with an embedded hyphen, using -eq instead of the equals sign (and similarly for other operators) and not using commas to delimit parameters in a function call (but using them in a method call) are all things that you get used to pretty quickly. There is one feature of PowerShell, however, that I don’t think will ever come naturally to me, and that’s how it handles return values from functions.

In most languages, if you see “return 1″ as the only return in a function, you can know that the function is going to the value 1 to the caller. In fact, I’m not sure I’ve ever seen a language that didn’t work that way. That is, until I found PowerShell. Generally speaking, the return statement works just as expected. In the absence of any statements writing to the output stream (with write-output) or “dropping” their values, “return 1″ will cause the caller to receive the value “1″. Using write-output is pretty obvious, and I’d recommend using it explicitly if you intend to add objects to the output stream (thereby including them in the eventual function value). Expressions that don’t capture their return values, however, are not quite so easy to spot.

For example, examine this code to add a parameter to an ADO.NET command object looks fine:

$cmd.Parameters.AddWithValue('@demographics','$demoXML)

This is a straightforward translation of one of the lines of code in the example code here.  The problem with the code is that AddWithValue not only adds a parameter, it also returns the parameter.  Since we didn’t assign it to a variable, cast it to [void] or pipe it to out-null, the output of this function (AddWithValue) gets added to the output of the function it’s in.

Several “add” functions in the .net framework follow this pattern, either returning the object that was added or the index of that object in the collection.  The DBConnection.Open method (inherited by SQLConnection, among others) returns the opened connection.  I’m sure that with time I could find more examples than I’d feel like sharing.

Another way that I’ve seen the output stream getting messed up is when a function uses strings to output information without using write-host.  For example, this function outputs “progress” information as it goes:

function get-filelength{
param($filename)
	"reading file"
	$len=0
	$file=get-content $filename
	foreach($line in $file){
	   "adding a line"
	   $len+=$line.length
	}
	return $len
}

I realize that this is not the best way to find the length of a file, but it works for the purpose of illustration. If you simply call the function (without assigning it to a variable), you’ll see a bunch of text, followed by the “value” of the function. The “returned value” of the function is clearly intended to be $len, but instead all of the text is included. The type of the function output is Object[], rather than a number.

There are some good reasons to use the output stream in a function (the next post will examine one such use), but it’s very easy to accidentally put something in the stream without intending to. When this happens, it often leads to unexpected results and long debugging sessions.

Here are my recommendations for avoiding this kind of error:

  1. Use one of the “write” functions to provide text messages to the user (write-host, write-debug, write-verbose)
  2. Use write-output if your intent is to add something to the output stream
  3. Check msdn to see if the methods you use have output. If you’re looking at the C# version, methods that return void do not need to be “handled”. In the VB.NET version, they’ll be declared as Sub, rather than Function.
  4. Be consistent with how you “handle” methods or functions whose values you want to discard. Options are to cast the result as [void] or pipe it to out-null
  5. If you’re using write-output to put objects in the output stream, use return with no value (i.e., don’t return $val). Seeing a value returned explicitly implies that that is the only value returned.
  6. If all else fails, use write-host to output the type of variables when debugging (or look at their values in the watch window if you’re using an IDE with a debugger).

PowerShell Tools and Books That I Use (Revisited)

Posted by on

A friend of mine found this blog yesterday and commented on “PowerShellToolsandBooksThatIUse“. I thought that it would be good to update the list since it’s almost 2 years old.  The changes are mostly incremental: version changes, a few new items, a few I don’t use much anymore.

Without further ado, here we go (now with linky-goodness!):

Tools I Use (note…these are all free!):

Books:

In addition to these, there are any number of blogs (I really need to update mylist) that I subscribe to in google reader. Also, there’s a very active community on StackOverflow that will usually be able to provide direction if you’re not sure what’s going on.

As I said the last time, there are lots of other tools and books, but these are the ones I keep coming back to.

Let me know if you think I missed something super-important.

Mike

A friend of mine found this blog yesterday and commented on “PowerShellToolsandBooksThatIUse“. I thought that it would be good to update the list since it’s almost 2 years old.  The changes are mostly incremental: version changes, a few new items, a few I don’t use much anymore.

Without further ado, here we go (now with linky-goodness!):

Tools I Use (note…these are all free!):

●      PowerShell ISE (I know it’s part of the install, but I use it a lot!)

●      PowerGUIScriptEditor (I still haven’t gotten the hang of PowerGUI itself)

●      PowershellCommunityExtensions 2.0 (PSCX)

●      SQLPowerShellExtensions 2.3 (SQLPSX)

●      PrimalFormsCommunityEdition (free registration required) Download under “Free Tools/32-bit”

Books:

●      PowerShellInAction,2ndEdition by Bruce Payette (just released yesterday, 5/18!)

●      ProfessionalWindowsPowerShellProgramming: Snapins, Cmdlets, HostsandProviders by Arul Kumaravel et. al. (PowerShell 1.0, but still has good info about building hosts)

●      MasteringPowerShell by Dr. Tobias Weltner

●      PowerShellinPractice, by Richard Siddaway

In addition to these, there are any number of blogs (I really need to update mylist) that I subscribe to in google reader. Also, there’s a very active community on StackOverflow that will usually be able to provide direction if you’re not sure what’s going on.

As I said the last time, there are lots of other tools and books, but these are the ones I keep coming back to.

Let me know if you think I missed something super-important.

Mike

A few functions for working with SQL Agent Job data

Posted by on

If you’ve ever looked at sysjobhistory, you know that the way SQL Server stores datetimes and durations in this table are not very friendly. Well, that’s not quite true. They are human-readable if they are small, but if you’ve got a job that runs for hours (days?), it becomes more and more difficult.
For example, a job which started at 9:05AM today (5/9/2011) would have a run_time of 905 and a run_date of 20110905. Both of those are recognizable, but since they are integers, it’s complicated to get them into a datetime variable in PowerShell in order to manipulate them. I have seen several posts on how to create a UDF (user-defined function) to convert these values to a datetime, and that’s usually a good solution. Unfortunately, it’s not always possible (politically) to create a UDF on a SQL instance. For example, if you are simply monitoring a box to make sure that backups are run, and don’t have much control over the box besides that function, you might not be allowed to create the necessary UDFs. Similarly, a SQL solution involving a select statement that kneads the data into the value you want is not a great solution because it’s not reusable, except in the cut-paste sense.
For that reason, I have written 4 different functions to assist with managing job data.

function convertfrom-SqlAgentDuration{
param($sql_duration)
    return New-TimeSpan -seconds ($sql_duration % 100) -minutes (($sql_duration / 100) % 100) -hours ($sql_duration /10000)
}

function convertfrom-SQLAgentDateTime{
param($sql_startdate,$sql_starttime)
    return (get-date -year ($sql_startdate/10000) -month ($sql_startdate/100 % 100) -day ($sql_startdate %100) -hour ($sql_starttime/10000 % 100) -minute ($sql_starttime/100 % 100) -sec ($sql_starttime % 100))
}

function convertto-SQLAgentDuration{
param([System.TimeSpan]$duration)
    return $duration.seconds+100*$duration.minutes+10000*($duration.hours+24*$duration.days)
}

function convertto-SQLAgentDateTime{
param([datetime]$date)
    return (new-object PSObject -property @{date=[int](get-date $date -uformat "%Y%m%d");time=[int](get-date $date -uformat "%H%M%S")})
}

Obviously, these are not high-performance functions, but given the volume of data you would be getting out of the SQL Agent, that shouldn’t pose any kind of problem. Here are some example usages:

PS>get-date

Monday, May 09, 2011 9:01:58 PM
_________________________________________________________________________________
PS>convertto-sqlagentdatetime (get-date) | ft -auto

    date   time
    ----   ----
20110509 210648
_________________________________________________________________________________
PS>convertfrom-sqlagentdatetime 20110509 210648

Monday, May 09, 2011 9:06:48 PM
_________________________________________________________________________________
PS>convertfrom-SqlAgentDuration 12345

Days              : 0
Hours             : 1
Minutes           : 23
Seconds           : 45
Milliseconds      : 0
Ticks             : 50250000000
TotalDays         : 0.0581597222222222
TotalHours        : 1.39583333333333
TotalMinutes      : 83.75
TotalSeconds      : 5025
TotalMilliseconds : 5025000

I hope these functions are useful to you. Let me know what you think.

Powershellstation.com has moved

Posted by on

I mentioned a while back that I was thinking about moving my site to NoSupportLinuxHosting.com.  They offer $1 per month basic shared hosting with the caveat that they don’t provide tech support.  Since I’m not doing anything fancy on this site (i.e. hosting wordpress only), and I’m not to the point where I need to be concerned about storage or bandwidth, it seemed like a good fit.  I’ve not had any real issues with my previous host (godaddy.com…yes, I know), but figured I could be saving a bit of money.

If you’re reading this, the transformation went well.  It only took 30-ish minutes to accomplish.

Let me know if you find something missing or not working.

Mike

Verifying Automation

Posted by on

If you’re anything like me, you’ve been bitten by the PowerShell bug and are using it among other automation sources to make you life in IT much more enjoyable. If this is not the case…you need to get started!  There’s no time like the present, and a PowerShell New Year’s resolution should be something to consider.

For those of you that are with me in the PowerShell camp, I have something that I’d like to discuss.  You probably have hundreds (dozens?) of scripts scheduled on multiple servers, possibly in multiple domains or geographical locations to perform things like these:

  • Gather information about servers
  • Generate reports about application usage
  • Copy information from one place to another
  • Validate security setup
  • Start and Stop processes
  • Scan log files for error conditions
  • Lots of other things (you get the point)

How do you know that the scripts that you have written carefully and scheduled are actually running successfully?  At first, this seems like a silly question.  When you deployed the script, surely you ran it once to make sure it worked.  What could have gone wrong?

Here are some examples that come to mind:

  • A policy was pushed which set the execution policy to Restricted
  • The credentials you scheduled the script with have been revoked
  • A file share that the scripts depend on is unavailable
  • Firewall rules change and now WMI queries aren’t working
  • The Task Scheduler service is stopped

You can probably think of a lot more examples of things that would keep scripts from working, but you get the idea.  I’ve given some thought to how to do this, but haven’t come to any real conclusions.  Obviously, having your scripts log results is helpful, but only if you monitor the logs for success/failure.  Also, if you have a script which is supposed to run every 10 minutes, it doesn’t help if you don’t get alerted when it only runs once in a day, even if  it runs successfully.  Also, if there is more than one person writing scripts, how do you make sure that everyone is using the same techniques to log progress?

Here are some of my thoughts:

  • Use a “launcher” to run scripts (see below)
  • Keep a database of processes with an expected # of runs per day
  • Monitor matching start/end of scripts
  • Log all output streams (example)

The first item in the list (the launcher) has been something I’ve been considering because it’s not trivial to run a PowerShell script in a scheduled task.  Even with the -file parameter which was added in PowerShell 2.0, it can involve a fairly long command-line.  With the added difficulty of trying to capture output streams (most of which are not exposed to the command shell) it becomes a process that is almost hard to get right every time.  Some features I’m planning for the launcher are:

  • Load appropriate profiles
  • Log all output streams (with timestamps) to file or database
  • Log application start/end times
  • Email owner of script if there are unhandled exceptions

I know this topic is not specific to PowerShell, but as Windows administrators get more used to scripting solutions to their automation problems with PowerShell (which I am confident that they are doing), it’s something that every organization will need to consider.  I’ll try to follow up with some posts that have some actual code to address some of these points.

Mike

P.S. I’m specifically not discussing “enterprise job scheduling” solutions like JAMS because of the high cost involved. I’d like to see the community come up with something a little more budget-friendly.

Happy (Belated) Anniversary!

Posted by on

I didn’t manage to post anything on the 1 year anniversary of powershellstation.com, but I did remember.  I was hoping to have posted more, but all in all, it wasn’t a bad year.  I have some plans for this year:

  • Write a post or 2 about remoting
  • Kill off the powershellworkbench project (I can’t stand writing WPF)
  • Start a new host project using Windows Forms and the Puzzle.SyntaxBox edit control (probably using C#)
  • Update the Books/Tools/Sites pages (and keep them more or less up to date)

I was really hoping that there would be some interest in the powershellworkbench project, but I’ve not heard anything (except for a comment that there were some missing icons).  Since I really didn’t enjoy writing WPF code (I might like it if I did a lot of it, but this is a hobby project), I don’t see much of a future for that project.  I am comfortable with Windows Forms, though, so I’ll be rebooting that project with a new name.  I’m also going to switch to the Puzzle.SyntaxBox control, because I know it will do the things I need to and not get in the way.  I haven’t found a similar component that wasn’t extremely complicated.  My goal with the project is to make something that an administrator with limited time could customize for their environment.  I’m afraid that the complexity of WPF and the Avalon controls would make that improbable at best.

I was also hoping that there would be more feedback.  I’m sure that this is a common frustration among bloggers (especially bloggers in a niche like this), so I’m not going to worry about that.  I am, however, going to make a concerted effort to comment on blogs that I read.  If you have any comments, they are always welcome.  If you have suggestions on topics, I’m open to that, as well.

One final thing.  I’ve been thinking about moving my blog to “No Support Linux Hosting”.  I’m not a Linux expert, but they have a WordPress setup tutorial, and you can’t beat $1 per month.  If I do change, I hope I can do it without any disruption, but who knows?

-Mike

My new favorite cmdlet: set-strictmode

Posted by on

If you’ve ever written Visual Basic or VB.Net code, you’re aware that it’s highly recommended that you use “Option Strict” in all of your code.  Similarly, Perl scripters have a “use strict” that comes highly suggested.

The idea of these options is that there’s quite a bit of flexibility built into these languages, and sometimes that flexibility backfires on you.  Using these options actually limits the flexibility of the languages in question in ways that help coders to keep from making certain types of mistakes in their code.  Writing code professionally in VB.Net (you may scoff, but it happens quite a lot) or Perl (it’s not my language of choice, but again, there’s a lot out there) without using these options is not a good idea at all.

I’ve been writing code in PowerShell for about 2 years, and have probably written about 10,000 lines, mostly code that’s used on a daily basis.  I’ve read several PowerShell books, online tutorials, and watched several webcasts.  I may have not been paying attention, but I’ve completely missed any emphasis on the analogous option in PowerShell, “set-strictmode”.  One of my co-workers asked if there was something like this in PowerShell, and I found it almost instantly.  Google, and the PowerShell help file both explain how the cmdlet works.  He and I started using it, and to our dismay, found dozens of errors in code that we had been trusting, in some case, for over a year.

First of all, let me explain that I’m not complaining that there’s not any information about how set-strictmode works in the “ecology” of the PowerShell community.  I’m confident that each book and tutorial explains how this cmdlet works.  What I’m concerned about is how I could have read as much as I have from as many people are talking about PowerShell without hearing anyone (or everyone) shouting at the top of their collective voices, “Add this to your profile…it will save you countless hours and tears in the long run”.  Hopefully, I’ve just missed it, and everyone has been saying this all along.  In case this has not been the case, let me say:

Add set-strictmode to your profile.

It will save you countless hours and tears in the long run.

So….how do I use set-strictmode, and what does it help with?  First, to turn on “strict mode” you need to decide which version of strictmode you want.  The options are:

Version Effect
1.0 References to uninitialized variables(except in a string) are errors
2.0
  • References to uninitialized variables(including in a string) are errors
  • References to undefined properties are errors
  • Calling a function using method-call syntax is an error
  • ${} (an un-named variable) is a syntax error
Latest Uses the most strict mode available for this version of PowerShell.  (Currently the same as 2.0)

I would advise that you use “set-strictmode –version Latest” in your profile.

Let’s look at the different restrictions.  First, if you’ve written any PowerShell scripts, you’re aware that you don’t have to declare your variables.  That’s a “good thing”, and strictmode 1.0 doesn’t change that in any way.  What it does do is make sure that you’re not retrieving values from variables you haven’t assigned anything to.  Here’s some sample code that strictmode will choke on.

$servername=read-host -prompt "Enter name of server"
$wmi=get-wmiobject -class Win32_Service -computername $server_name

This is a pretty simple example, but it’s almost certain that sooner or later you’ll typo a variable name in a script.  Strictmode will help keep you from getting in this kind of trouble.  With strictmode Version 1.0, you won’t be warned about using uninitialized variables in double-quoted string, e.g. “Server=$server_name”…(they’ll be replaced with $null).  Version 2.0 will cause references to uninitialized variablesinside strings to be an error as well.

Here’s an example of code that illustrates property restrictions in StrictMode 2.0.

$servername="Server123"
$length=$servername.lenght

I would write up an example of using the method-call syntax for a function, but Thomas Lee just blogged about this in his blog Under the Stairs.

The final “restriction” in strict mode 2.0 says that you can’t have a variable with no name.  The syntax ${varname} is a way of expressing $varname, when varname contains characters that are not generally allowed in variable names.  I haven’t ever had a really convincing use for this syntax (I’ve thought about using it to associate variables with filenames using the full path as the variable name, but it seems like a dictionary would be cleaner).

I should note that Lee Holmes has done a great thing in his recently published PowerShell Cookbook V2.  Each script in the cookbook starts with “set-strictmode –version Latest”.   In my opinion, this is a best practice.

If you’re just now hearing about set-strictmode for the first time, and already have a body of powershell scripts in your toolbox (I’m in this category), I’d recommend the following steps:

  1. Add set-strictmode –version Latest to the profile on the machine you do your script development on
  2. Add set-strictmode –version Latest  to any scripts you create from this time forward
  3. Add set-strictmode –version Latest to scripts that you have tested to make sure they comply with strictmode.

I’m going to avoid turning on strictmode on servers that I’m running PowerShell on for now, because I’m sure stuff will break.  I’m slowly refurbishing scripts as I run them on my development box and see the strictmode errors pop up, and since set-strictmode is a scoped change (if you set it in a script or function, it reverts to the old setting when you exit the script or function), I can take advantage of it in a script-by-script basis.

If you are just starting to build a PowerShell script library, I would recommend turning strictmode on in the profile on every machine that you run PowerShell scripts.  You’ll be glad you did.

Hopefully this was not too much of a rant.  Please let me know what experiences you’ve had with with set-strictmode.

Mike

PowerShell and MongoDB

Posted by on

I recently saw this link on using NoSQL with Windows.  Now, I’m a SQL Server DBA, so I haven’t really had any reason to use NoSQL.  I was curious about how easy it was going to be to set up and if I could get it working with PowerShell.

I selected MongoDB from the list because it looked more like something that would be used on smaller-scale projects.

I then googled “MongoDB PowerShell” and found this link from Doug Finke about using MongoDB with PowerShell (and F#, which is another “cool thing” I haven’t managed to find a need for).  Doug links to another article which explains setting up MongoDB and an open-source .Net driver for MongoDB called mongo-csharp.  He then follows up with a straight-forward script showing simple usage of MongoDB.  It looks like an almost literal translation of the C# code from the article he references.  With those in hand, I thought it was going to be a slam dunk.

It was, but I had a few hurdles to get over before I could get it working.  There weren’t any problems with the code;  it was written about a year ago, so it was using PowerShell 1.0 and an older version of mongo-csharp.  I had to update the script in a couple of places to make it work.  I probably wouldn’t even write it up, given how minor the changes are, but I was somewhat disappointed with the number of hits I got for “MongoDB PowerShell”.

Here’s the updated script:

add-type -Path .MongoDB.Driver.dll

$mongo=new-object mongodb.driver.mongo
$mongo.connect()
$db=$mongo.GetDataBase("movieReviews")
$movies=$db.GetCollection('Movies')

$movie=new-object Mongodb.driver.document
$movie['title']='Star Wars'
$movie['releaseDate']=get-date
$movies.Insert($movie)

$spec=new-object Mongodb.driver.document
$spec['title']='Star Wars'
$movies.FindOne($spec)

The two changes were

  1. added the parameter name (-path) to add-type, since the default param is looking for an assembly name, not a path to a DLL
  2. changed $mongo.GetDB to $mongo.GetDatabase to reflect a change in the driver

Worked fine. Not sure how I’ll use this, but if I need to, I know I can from PowerShell.

-Mike

The Identity Function

Posted by on

In mathematics, an identity function is a function that returns the arguments that are passed to it unchanged.  While the concept of an identity function is quite often useful in formulating proofs, it is not something that I ever expected to use in a programming environment.   Here’s the identity function written in PowerShell:

function identity{
    return $args
}

The surprising thing about this function is that it’s actually pretty useful. PowerShell’s array literal syntax is considerably more involved than it’s argument syntax. Using this function lets us do this:

identity arg1 arg2 arg3

instead of this:

@('arg1','arg2','arg3')