PowerShell Test Levels

The motivation for building an xUnit-style library such as PSExpect (http://www.codeplex.com/psexpect) for PowerShell is based on my belief and hope in its utility for automating tests.  Admittedly I don’t have the Admin Development Model in mind – my perspective leans towards systems program, but with a Tester’s twist.  Thus I use the term dynamic language to describe PowerShell and not scripting language

Unit Testing – If you’re writing a .NET application, then I’m assuming that you will use the unit test functionality built in Visual Studio, or NUnit.  You don’t need PowerShell (or PSExpect) for this.  If you’re writing a PowerShell script then you might want to use PSExpect to test your script, if you have time and if you’re doing systems programming. I actually like the idea of Admins using something like PSExpect to confirm so that any script they write verifies the intended action. But I understand the comments about the lack of time mentioned in the blog entry I’ve linked to above.  If you’re writing cmdlets, then use NUnit for unit testing and PSExpect for black-box functional testing of your cmdlet.  I’ll post on unit testing in a subsequent blog entry once I’ve built more examples.

Functional Testing – As mentioned above, PSExpect is useful for black-box functional testing of cmdlets.  The idea behind this is that you should be testing the new cmdlet in the environment that it will be used (or as close to it as possible).  In other words, you should write cmdlet test scripts in PowerShell.  PSExpect fits in then as the means of automating the pass/fail decision (what testers refer to as the verdict).  This isn’t always going to be easy since automating the verdict is one of the traditional challenges of test automation, however, the flexibility of PowerShell makes me hopeful.  Never before has it been so easy to confirm, in a script, that a specified service was running, that a WMI instance had expected properties, or that a data source contained a certain expected value or set of values.

If you’re writing a .NET application, then PowerShell and PSExpect might also have a role to play.  If you’ve structured the application with a layered architecture, then possibly, maybe, you might be inclined to jump out of your comfort zone in NUnit and use PowerShell/PSExpect to test the lower (non-GUI) layers.  It might actually give you better tests because of that very thing – jumping out of the ‘culture’ of the compiled languages and into the dynamic language culture.  The automated test itself would also be quicker to develop – dynamic languages such as PowerShell enjoy a fast turn-around time.

Deployment Testing – A ‘smoke test’ or ‘health check’ is a common testing requirement, especially when the team expects to do more than one release, or when they expect to do on-going configuration changes.  It is virtually a MUST HAVE requirement when the development team is using an agile process, since releasing early and often is one of the principles behind that development style.  Using PowerShell is going to be useful for this since as I mentioned earlier, you can quickly check the structure of the deployed application (services, process, folders, files, file versions, …) and through ADO.NET, the contents and structure of a database.  You can even check the contents of configuration files since PowerShell does a nice job of navigating XML and plain text files.  The strength is the flexibility of PowerShell – I’m betting more checks can be done in fewer lines of code as compared to a VBScript deployment test.

The above test levels are the initial focus of the PSExpect library (http://www.codeplex.come/psexpect) but as mentioned in the project introduction on CodePlex, there are other test levels that PowerShell would be useful for as well.  I’ll reserve working on those test levels for now so that I can focus on building examples of the three mentioned here first.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s