Web Service Functional Testing

* Updated – the Jan 21/2007 source code version of the PSExpect library now works with cultures other than ‘en-US’.

In a previous blog entry, I demonstrated how to obtain the test data from Excel using PowerShell.  I’m using the January 21/2007 version of the PSExpect library.  This version adds functions for working with Excel and should work with culture settings other than ‘en-US’.

In this blog entry, I will demonstrate testing a web service using PowerShell.  The test data will be in Excel so that the tests are data-driven instead of run directly from script (at least as many as I can).  The web service that will be the target of the test was the subject of a blog entry by Jeffrey Hicks and highlighted by Jeffrey Snover in the PowerShell Team Blog.

Start by first designing the data-driven part.  We should design a table that contains the required inputs to the we service and our expected output.  The following table lists some of the test cases for the web service based on its required inputs and its output. 

TestCase Remark Zip Units ExpectedTitle
TC-1 Success in Celsius 35801 C Conditions for Huntsville, AL
TC-2 Success in Fahrenheit 35801 F Conditions for Huntsville, AL
TC-3 Success – invalid units 35801 Z Conditions for Huntsville, AL
TC-4 Success – default units 35801 $null Conditions for Huntsville, AL
TC-5 Failure – no Zip $null F City not found
TC-6 Failure – invalid Zip 2 F City not found


The key concept here is that you are designing this table. You choose the inputs based on the specifications of the web service, and you won’t have much control over that, true enough.  The rest, you choose.  In other words, it’s a design task, and the output is the design of the test fixture that you will code up in PowerShell and call from the test script.

In testing terminology, you are specifying the test oracle, that is, the way that you choose to automate the pass/fail decision.  In the case of the above table, the test oracle is simple and consists of only the Expected Title in the results of the call to the web service.  We will test the web service in the live production environment, so we can’t cook up the web-service to return a predictable temperature or wind speed.  So we can’t use that data as part of the test oracle.  As an aside, if we were really doing this in a test environment, then you might in fact consider working with a known data set so that you could use more fields than just the title as the test oracle.

In test automation terminology, by creating this table, you have designed the text fixture for the web service.  The next step is to code this test fixture in PowerShell.  The test fixture needs to accept the inputs required to call the web service and return a single output value, the ExpectedTitle.  The ‘TestCase’ and ‘Remarks’ fields are used for test documentation only and won’t show up as inputs – they will show up in the log of the tests as they run.

# Exercises the target of the test - the get-process cmdlet # and responds with a pre-defined set of responses that were # on the worksheet as the expected results function TestGetWeatherFixture([string] $zip, [string] $units) { write-host $zip $units if ($units -eq "$null") { $units = $null } if ($zip -eq "$null") { $zip = $null } [string]$urlbase="http://xml.weather.yahoo.com/forecastrss" [string]$url=$urlbase + "?p="+$zip+"&u="+$units write-host Connecting to $url # create .NET Webclient object and call the web service $webclient=New-Object "System.Net.WebClient" [xml]$weather=$webclient.DownloadString($url) # translate the results to match the expected output return $weather.rss.channel.item.Title.ToString() }

The function first translates the units since it only receives text input values – a blank cell would also have worked but isn’t as easy to work with.  I like testers to be deliberate, so I don’t mind forcing them to enter ‘$null’ instead of leaving the cells blank.  Then the function builds the URL required to call the web service, and calls it.  The result is cast as an XML document for each reference, that is, you can use the dot notation to access the various items within the document that is returned from the web service.  Accessing XML content this way was one of the nicest surprises I noticed in PowerShell.  That will become extremely useful to me in many projects, I know it.

The next step is to create the test script that first retrieves the contents of the worksheet from Excel and then loops through each row, calling the test fixture each time.


function TestGetWeather { # specify the workbook full path $theFolder = get-location $WorkbookFullPath = $theFolder.ToString() + "TestGetWeather.xls" # identify the worksheet that the test data is on $WorksheetName = "Sheet1" # identify a range - it's a good idea to use Excel's named ranges # so that you don't have to change your script if you add test cases. # You just have to re-define the named range. $RangeName = "TestCases" # specify the names of each column - doesn't have to agree with what is # on the spreadsheet $FieldNames = "TestCase","Remark","Zip","Units","Title" # retrieve a list of the contents of the range $fileExists = test-path $WorkbookFullPath if (!$fileExists) { return $null } # start Excel $excel = New-Object -comobject Excel.Application # get the data $workbook = open-workbook $excel $WorkbookFullPath $worksheet = get-worksheet $workbook $WorksheetName $range = get-range $worksheet $RangeName $rangeAsList = collect-range $range $FieldNames # make sure it's not null AssertNotNull $rangeAsList "GWXL-1"
# continue
if ($rangeAsList -ne $null) { write-host "There are " $rangeAsList.Count " test cases." # for each row in the test cases range, foreach ($item in $rangeAsList) { # exercise the target of the test $proc = TestGetWeatherFixture $item.Zip $item.Units # check the actual results against the expected results on the worksheet # and use the label from the worksheet's first column $actual = $proc.PadRight(30," ").Substring(0,29).Trim() $expected = $item.Title AssertEqual $expected $actual $item.TestCase } RaiseAssertions } else { write-host "Couldn't find the workbook." } # this is how you ensure that no references are left on your COM object so # that it does actually quit $excel.Quit() $a = release-range $range
$a = [System.Runtime.InteropServices.Marshal]::ReleaseComObject($worksheet) $a = [System.Runtime.InteropServices.Marshal]::ReleaseComObject($workbook) $a = [System.Runtime.InteropServices.Marshal]::ReleaseComObject($excel) }

This function uses the DataLib.ps1 functions from PSExpect in order to retrieve the data from Excel.  Another difference from FIT to highlight is the use of the $FieldNames array to specify how each of the fields will be referred to later in the test script.  I did think of retreiving this automatically from the Excel worksheet, but on thinking about it and trying this out for a bit, I realized that it was easier specifying the field names in the test script since that is where they will be used.  So it’s a matter of scrolling up to see the field names instead of ALT-TAB to Excel to see them.  As long as the order is the same, the field names in the test script do not have to match what is in Excel.  Notice in the script the single-valued test oracle – the line that begins with ‘AssertEqual …’.

Here’s the output you get when you run this script:


There are 6 test cases.
35801 C
Connecting to http://xml.weather.yahoo.com/forecastrss?p=35801&u=C
35801 F
Connecting to http://xml.weather.yahoo.com/forecastrss?p=35801&u=F
35801 Z
Connecting to http://xml.weather.yahoo.com/forecastrss?p=35801&u=Z
35801 $null
Connecting to http://xml.weather.yahoo.com/forecastrss?p=35801&u=$null
$null F
Connecting to http://xml.weather.yahoo.com/forecastrss?p=$null&u=F
2 F
Connecting to http://xml.weather.yahoo.com/forecastrss?p=2&u=F
1/12/2007 3:47:51 PM,SHOULDPASS,PASSED,GWXL-1
1/12/2007 3:47:54 PM,SHOULDPASS,PASSED,TC-1
1/12/2007 3:47:54 PM,SHOULDPASS,PASSED,TC-2
1/12/2007 3:47:55 PM,SHOULDPASS,PASSED,TC-3
1/12/2007 3:47:56 PM,SHOULDPASS,PASSED,TC-4
1/12/2007 3:47:57 PM,SHOULDPASS,PASSED,TC-5
1/12/2007 3:48:02 PM,SHOULDPASS,PASSED,TC-6


As an aside, implementing these tests before the web service exists would lead to a data-driven, test-first development style for the web service.  You would run the tests and watch them all fail (you might have to code the test fixture to make it handle the non-existent web service better) and then use the failing tests as a "to do" list.  Code only enough to get the first test case to pass.  Then proceed to the next one, refactoring (and unit testing) the code as you go.  This is called story-driven or acceptance-test development since the tests specified here mirror the requirements, not necessarily the structure of the code that implements the web service.  In the above example, the web service is trivial enough that the functional tests we’ve provided look a lot like unit tests – but that isn’t always the case.  Nor are these functional tests covering the full range of the requirements of this particular web service.  But if you did create a script and Excel worksheet that did cover all the requirements – that would be excellent documentation for the web service!

In summary, functional testing of a web service using PowerShell isn’t much different from functional testing of a cmdlet.  Your first step is to design the text fixture by creating a table with the necessary inputs and the outputs that will help you to make the pass/fail decision.  The next step is to build the test fixture in PowerShell, and then build a test script that retrieves the data from Excel and runs it through the test fixture.  Using PSExpect, the functions to retrieve the Excel data and to automate the pass/fail decision are provided – you have to supply the test fixture and the test script that loops through the data.  Future versions of DataLib component within PSExpect may reduce the amount of test code you have to write (making it more like FIT) but in the meantime, it does offer a way to get data-driven testing happening sooner.

Happy testing!


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s