Cmdlet Functional Testing

One of the benefits of creating a language like PowerShell is to enable knowledge sharing (see http://blogs.msdn.com/powershell/archive/2007/01/01/the-admin-development-model-and-send-snippet.aspx) and to get better, together, faster.  From an administrator’s perspective, knowledge sharing means replacing a description of a point-and-click exercise with an script – concise, expressive, and executable.

We should have a similar line of thinking when we’re extending PowerShell with custom cmdlets.  We can provide a description of what the new cmdlet does and we can provide the help documentation that is accessing using get-help.  But we can provide concise, expressive, and executable details about the cmdlet if we provide functional tests written in PowerShell.  I’ll explain what I mean.

As a first step, let’s take a look at a cmdlet that is easy to test. I’ll choose join-string from the PowerShell Community Extensions (http://www.codeplex.com/PowerShellCX).  Join-string accepts an array of strings as input and joins them to create a single string.  It’s easy to test since it manipulates strings and returns a string.  A test oracle (the criterion for making  a pass/fail decision) that depends on the contents of a string is generally easy to automate.  So we have inputs that are easy to generate (strings) and an output that is easy to inspect.

First, create a test script for the cmdlet.  In my case, I created test-joinstring.ps1 and gave it the following structure.  The script relies on the PSExpect (http://www.codeplex.com/psexpect) library (the xUnit functions for PowerShell).

set-psdebug -strict -trace 0 # Tests the Join-String cmdlet function Test-Command { } # run the function library that contains the PowerShell Testing Functions # the functions are defined as global so you don't need to use dot sourcing if (!(Test-Path variable:_TESTLIB)) { ..srcTestLib.ps1 } # run the function defined above that demonstrates the PowerShell Testing Functions Test-Command

The script is the basic skeleton for running tests that use the PSExpect functions.  The first test case added to the script can be the simplest success case, when the bare minimum number of inputs is provided.

# no separator $contents = "Prefix","Suffix" $expected = "PrefixSuffix" $joined = join-string -Strings $contents AssertEqual -Expected $expected -Actual $joined -Label "TC-JS-1"
-Intent $Intention.ShouldPass RaiseAssertions

This has the standard elements of any test case – set up the inputs, exercise the target, and compare the actual results to the expected results.  In this case, the ‘AssertEqual’ function does the comparison.  The only required parameters to the call to AssertEqual are -Expected and -Actual.  The -Label parameter is for tracing back to test cases that fail (when the script is run with many test cases in it and logged).  The last parameter is a bit strange, something that I added to support test-driven development.  You use the -Intent parameter to indicate whether or not you expect the test to pass or fail.  In this case, since we’re writing a script that is a functional test of a released cmdlet, we expect it to pass.  In other situations, we might be writing a test that we expect to fail since we haven’t made the configuration change or developed the functionality yet.  In that case, we would use $Intention.ShouldFail.  The difference is in the output – when $Intention.ShouldPass fails, the output is RED.  When $Intention.ShouldFail fails, the output is YELLOW.  Using the traffic light metaphor, it is a warning but not a problem.

In this particular case, we get GREEN output since the test case passes and the log entry on the console shows the following:

1/7/2007 9:39:28 AM,SHOULDPASS,PASSED,TC-JS-1

In the next test cases, we can explore the success scenarios when using the -Separator and -NewLine parameters to join-string.  ‘Separator’ allows you to place a custom delimiter between the strings, while ‘NewLine’ causes a line break between them.  These test cases follow the same pattern as the previous test case:

# custom separator $separator = "|" $joined = join-string -Strings $contents -Separator $separator $expected = "Prefix|Suffix" AssertEqual -Expected $expected -Actual $joined -Label "TC-JS-2"
-Intent $Intention.ShouldPass # newline separator $joined = join-string -Strings $contents -NewLine $expected = "Prefix`nSuffix" AssertEqual -Expected $expected -Actual $joined -Label "TC-JS-3"
-Intent $Intention.ShouldPass

 

 

But the output isn’t as we might expect.  The ‘NewLine’ test case fails, even though the output looks to be identical.

1/7/2007 9:44:10 AM,SHOULDPASS,PASSED,TC-JS-1
1/7/2007 9:44:10 AM,SHOULDPASS,PASSED,TC-JS-2
1/7/2007 9:44:11 AM,SHOULDPASS,FAILED,TC-JS-3,Prefix
Suffix expected but was Prefix
Suffix

The failing test case demonstrates what I mean by providing additional documentation about the cmdlet that may not be obvious.  In generating script output, we would use the `n escape sequence if we needed a line break, but that’s not what the cmdlet does. It uses (quite appropriately) the [System.Environment]::NewLine static property instead of a simple `n.  Maybe another way of saying this is that `n is not equivalent to [System.Envrionment]::NewLine.  So my initial guess as to how join-string worked was incorrect, and replace the `n with the appropriate separator resolved the test failure.

1/7/2007 9:52:53 AM,SHOULDPASS,PASSED,TC-JS-3

The three test cases now collectively summarize the functionality of the join-string cmdlet.  In a concise, expressive, and executable manner, that is, in PowerShell script itself.  In my opinion, sharing these test cases is even better than sharing non-test examples if the test suite is really the test suite used to test the cmdlet.  You’re telling the consumers of the cmdlet exactly how you planned the cmdlet to work and you’re inviting them to extend the test cases for behaviours that you didn’t think of.  They might even find a bug and that is always a good thing.

If we continue the investigation of the join-string cmdlet, we know that it should throw exceptions given certain input.  For example, providing both the -Separator and -NewLine parameters causes a conflict of parameter sets:

# custom and newline separators on the same command - check the exception $separator = "|" $blk = {$joined = join-string -Strings $contents -Separator $separator -NewLine} $blk | AssertThrows
-ExceptionExpected "System.Management.Automation.ParameterBindingException" -MessageExpectedRegExpr "Parameter set cannot be resolved" -Label "TC-JS-5"

In this test case, the ‘AssertThrows’ function verifies both the type of Exception thrown and the message within it.  Providing null input throws a different exception so a test case can also be devised for that behaviour.  The complete test script – at least as far as I took it – is shown below.

set-psdebug -strict -trace 0 # Tests the Join-String cmdlet function Test-Command { # no separator $contents = "Prefix","Suffix" $expected = "PrefixSuffix" $joined = join-string -Strings $contents AssertEqual -Expected $expected -Actual $joined -Label "TC-JS-1"
-Intent $Intention.ShouldPass # custom separator $separator = "|" $joined = join-string -Strings $contents -Separator $separator $expected = "Prefix|Suffix" AssertEqual -Expected $expected -Actual $joined -Label "TC-JS-2"
-Intent $Intention.ShouldPass # newline separator $joined = join-string -Strings $contents -NewLine $expected = "Prefix" + [System.Environment]::NewLine + "Suffix" AssertEqual -Expected $expected -Actual $joined -Label "TC-JS-3"
-Intent $Intention.ShouldPass # custom and newline separators on the same command - check the result $separator = "|" $joined = join-string -Strings $contents -Separator $separator -NewLine $expected = "Prefix" + [System.Environment]::NewLine + "Suffix" AssertEqual -Expected $expected -Actual $joined -Label "TC-JS-4"
-Intent $Intention.ShouldPass # custom and newline separators on the same command - check the exception $separator = "|" $blk = {$joined = join-string -Strings $contents -Separator $separator -NewLine} $blk | AssertThrows
-ExceptionExpected "System.Management.Automation.ParameterBindingException" -MessageExpectedRegExpr "Parameter set cannot be resolved" -Label "TC-JS-5" # null input $contents = $null $blk = {$joined = join-string -Strings $contents} $blk | AssertThrows
-ExceptionExpected "System.Management.Automation.ValidationMetadataException" -MessageExpectedRegExpr "Cannot validate argument because it is null." -Label "TC-JS-6" # elements of zero length $contents = "","" $expected = "" $joined = join-string -Strings $contents AssertEqual -Expected $expected -Actual $joined -Label "TC-JS-7"
-Intent $Intention.ShouldPass # empty array $contents = [Array]::CreateInstance([string],0) $expected = "" $joined = join-string -Strings $contents AssertEqual -Expected $expected -Actual $joined -Label "TC-JS-8"
-Intent $Intention.ShouldPass # larger array $contents = [string[]](@("a") * 10000) $expected = "" for ($i=1;$i -le 10000;$i++) { $expected += "a" } $joined = join-string -Strings $contents AssertEqual -Expected $expected -Actual $joined -Label "TC-JS-9"
-Intent $Intention.ShouldPass $expected = $null $joined = $null $contents = $null RaiseAssertions } # run the function library that contains the PSExpect Testing Functions # the functions are defined as global so you don't need to use dot sourcing if (!(Test-Path variable:_TESTLIB)) { ..srcTestLib.ps1 } // run the function defined above that demonstrates the PowerShell Testing Functions Test-Command

The output is all GREEN indicating that all the test cases were passed:

1/7/2007 9:52:53 AM,SHOULDPASS,PASSED,TC-JS-1
1/7/2007 9:52:53 AM,SHOULDPASS,PASSED,TC-JS-2
1/7/2007 9:52:53 AM,SHOULDPASS,PASSED,TC-JS-3
1/7/2007 9:52:53 AM,SHOULDPASS,PASSED,TC-JS-4
1/7/2007 9:52:53 AM,SHOULDPASS,PASSED,TC-JS-5-ExceptionType
1/7/2007 9:52:53 AM,SHOULDPASS,PASSED,TC-JS-5-ExceptionMessage
1/7/2007 9:52:53 AM,SHOULDPASS,PASSED,TC-JS-6-ExceptionType
1/7/2007 9:52:53 AM,SHOULDPASS,PASSED,TC-JS-6-ExceptionMessage
1/7/2007 9:52:53 AM,SHOULDPASS,PASSED,TC-JS-7
1/7/2007 9:52:53 AM,SHOULDPASS,PASSED,TC-JS-8
1/7/2007 9:52:54 AM,SHOULDPASS,PASSED,TC-JS-9

Now we have documentation of the intended functionality in a language that we understand (PowerShell) and in a format that we can learn from.  In the spirit of knowledge sharing, we have also built a test suite for the cmdlet in the environment that it is intended to run in.  I say that’s a win for both the producers and consumers of custom cmdlets.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s