Lately testing people have been talking about testing heuristics – tips, techniques, tactics, strategies for testing something. I have one that I call MTT.
M – Mission
T – Targets
T – Tests
But I remember it as "mean time to testing". Exploratory testing excels in part because the mean time to testing is small compared to (caution: wiki-speak ahead), say, BigUpFrontTestManagement. As an example of minimizing the mean time to testing, the example below uses PowerShell to explore the configuration and functionality of SQL Server Reporting Services 2005.
Mission
Encourage the team to build consensus on why testing will be conducted by paying attention to the project context – who tests and what their background is, what is the system under test, who supports the test environment, what are the communication patterns between testers and developers, … post the essence of this as a ‘test mission’ in the team’s work area and let them mark it up as the project evolves. The more people that are involved in the test effort, the more useful a ‘testing working agreement’ becomes. Testing faster requires that all the testers understand why they are testing so that they get to the point.
Targets
If the mission indicates why we are testing, then the targets drill deeper and identify what will be tested. I see a lot of testing without clearly-defined test targets. Some heuristics are test targets, and that helps a lot. Exploratory testing, even when automated, fits really well with test targets because listing them gives you a testing backlog, making citing progress and indicating expected completion dates simpler using something like testing burndown charts.
Consider automated exploratory testing with PowerShell – let’s use the Microsoft Reporting Services installation test as an example. To use MTT heuristic and the mission stated above, the second step would be to create a testing backlog of the targets of the tests – in other words, what exactly would you want to test? As with all things agile, this list doesn’t have to be complete because you have every right to revise this backlog and prioritize it as necessary, within the context of the test mission. It is important to avoid listing how these targets might be tested – that’s left to exploration.
Microsoft Reporting Services Installation Test – Targets (Testing Backlog)
ReportServer – Service Control
ReportServer – Service Properties
Database – Reporting Services Db – Topology
Database – Reporting Services Db – Physical Configuration
Database – Reporting Services Db – Authentication
Database – Reporting Services Db – Authorization
Operations – Performance Monitors – Availability
Operations – Microsoft Operations Manager – Access
Operations – Microsoft Operations Manager – Alerts
Web Service – Availability
Web Service – Authentication
Web Service – Authorization
Web Service – Functional – Run Report
Report Manager – Availability
Report Manager – Authentication
Report Manager – Authorization
Report Manager – Functional – Reports
Report Manager – Functional – Data Sources
Report Manager – Capacity – Concurrency
Report Manager – Capacity – Reponse Time
Given this list of test targets, an exploratory test effort would start by estimating each one of the targets to create a testing burndown chart, and then diving in according to the priorities assigned in the backlog. PowerShell is incredibly useful for this given the ‘get-member’ cmdlet.
Tests
Continuing the Reporting Services example, the first thing that a tiny bit of investigating reveals is that SQL Server Reporting Services has a WMI class that exposes the configuration. Let’s get that class and see what we can do with it.
$targetcomputer = "localhost"
$rs = get-wmiobject -computer $targetcomputer -namespace `
"rootMicrosoftSqlServerReportServerv9Admin" MSReportServer_ConfigurationSetting
$rs | get-member
| Name | MemberType | |||
| —- | ———- | |||
| BackupEncryptionKey | Method | |||
| ConfigureSharePointExclusion | Method | |||
| CreateApplicationPool | Method | |||
| CreateVirtualDirectory | Method | |||
| DeleteEncryptedInformation | Method | |||
| DeleteEncryptionKey | Method | |||
| GenerateDatabaseCreationScript | Method | |||
| GenerateDatabaseRightsScript | Method | |||
| GenerateDatabaseUpgradeScript | Method | |||
| InitializeReportServer | Method | |||
| ListReportServersInDatabase | Method | |||
| ReencryptSecureInformation | Method | |||
| RemoveUnattendedExecutionAccount | Method | |||
| ResetVirtualDirectoryMappings | Method | |||
| RestoreEncryptionKey | Method | |||
| SetDatabaseConnection | Method | |||
| SetDatabaseLogonTimeout | Method | |||
| SetDatabaseQueryTimeout | Method | |||
| SetEmailConfiguration | Method | |||
| SetPortNumber | Method | |||
| SetSecureConnectionLevel | Method | |||
| SetServiceState | Method | |||
| SetUnattendedExecutionAccount | Method | |||
| SetWebServiceIdentity | Method | |||
| SetWindowsServiceIdentity | Method | |||
| ApplicationPoolActual | Property | |||
| ApplicationPoolConfigured | Property | |||
| ConnectionPoolSize | Property | |||
| DatabaseLogonAccount | Property | |||
| DatabaseLogonTimeout | Property | |||
| DatabaseLogonType | Property | |||
| DatabaseName | Property | |||
| DatabaseQueryTimeout | Property | |||
| DatabaseServerName | Property | |||
| InstallationID | Property | |||
| InstanceName | Property | |||
| IsInitialized | Property | |||
| IsSharePointExclusionConfigured | Property | |||
| IsSharePointInstalled | Property | |||
| IsWebServiceEnabled | Property | |||
| IsWindowsServiceEnabled | Property | |||
| PathName | Property | |||
| SecureConnectionLevel | Property | |||
| SenderEmailAddress | Property | |||
| SendUsingSMTPServer | Property | |||
| ServerPort | Property | |||
| ServiceName | Property | |||
| SMTPServer | Property | |||
| UnattendedExecutionAccount | Property | |||
| VirtualDirectory | Property | |||
| VirtualDirectoryHasSSLCertificate | Property | |||
| WebServiceIdentityActual | Property | |||
| WebServiceIdentityConfigured | Property | |||
| WebSite | Property | |||
| WindowsServiceIdentityActual | Property | |||
| WindowsServiceIdentityConfigured | Property | |||
| __CLASS | Property | |||
| __DERIVATION | Property | |||
| __DYNASTY | Property | |||
| __GENUS | Property | |||
| __NAMESPACE | Property | |||
| __PATH | Property | |||
| __PROPERTY_COUNT | Property | |||
| __RELPATH | Property | |||
| __SERVER | Property | |||
| __SUPERCLASS | Property | |||
| ConvertFromDateTime | ScriptMethod | |||
| ConvertToDateTime | ScriptMethod | |||
| Delete | ScriptMethod | |||
| GetType | ScriptMethod | |||
| Put | ScriptMethod | |||
From an exploratory tester’s perspective, this list is super useful. Now you can explore the configuration settings interactively using the $rs variable (they are all listed as Property in the above listing), or create a test script and use data-driven testing to check all the Reporting Services configuration parameters against expected values. The ‘get-history’ cmdlet is also useful to retrieve and save the commands that you have experimented with in a format that you can use later for building a script.
To do the data-driven testing in Excel, create a table with three columns and all the properties that you want to test for. Name the range TestRsConfig and save the worksheet.
| TestCase | Property | Value |
| Security – Service Identity – As Configured | WindowsServiceIdentityConfigured | NT AuthorityNetworkService |
| Security – Service Identity – Actual | WindowsServiceIdentityActual | NT AuthorityNetworkService |
| Service – Instance | InstanceName | SQLEXPRESS |
| Service – ServiceName | ServiceName | ReportServer$SQLEXPRESS |
| ServicesDependedOn – Database – Server | DatabaseServerName | (LOCAL)SQLEXPRESS |
| IIS – WebSite | WebSite | 1 |
| Security – IIS – WebServiceIdentity – Configured | WebServiceIdentityConfigured | ALAPTOPASPNET |
Next create a script that includes a function called TestRsConfig (the same name as the named range above) as follows (this script uses the PSExpect testing library for PowerShell):
set-psdebug -strict -trace 0 $targetcomputer = "localhost" $rs = get-wmiobject -computer $targetcomputer ` -namespace "rootMicrosoftSqlServerReportServerv9Admin" ` MSReportServer_ConfigurationSetting # Exercises the target of the test - the # and responds with a pre-defined set of responses that were # on the worksheet as the expected results function TestRsConfig() { param([string]$TestCase, [string]$RsConfigProperty, [string]$ExpectedValue) $ActualValue = $rs.PSBase.GetPropertyValue($RsConfigProperty).ToString() if ([string]::IsNullOrEmpty($ActualValue)) {$ActualValue = $null} if ($ExpectedValue.Trim() -eq "N/C") { AssertNull $ActualValue -Label $TestCase -Intention $Intention.ShouldPass } else { AssertEqual $ExpectedValue $ActualValue -Label $TestCase -Intention $Intention.ShouldPass } RaiseAssertions } # run the function library that contains the PowerShell Testing Functions # the functions are defined as global so you don't need to use dot sourcing if (!(Test-Path variable:_XLLIB)) { ..srcDataLib.ps1 } if (!(Test-Path variable:_TESTLIB)) { ..srcTestLib.ps1 } # Configuration and installation testing ... # Confirm the Reporting Services configuration matches expectations documented in # the Excel workbook 'TestRsConfig.xslx' write-host "Configuration testing ..." $FieldNames = ("TestCase", "RsConfigProperty", "ExpectedValue") start-test ((get-location).ToString() + "TestRsConfig.xlsx") "Sheet1" ` "TestRsConfig" $FieldNames
The line of note contains ’$rs.PSBase.GetPropertyValue’ – a PowerShell base method that is proving to be really useful in writing test scripts without hard-coding the property names. In my script, I’ve used ‘N/C’ to identify Reporting Services properties that are not configured at all on my laptop. If I had left them out and therefore avoided the if-then-else code section, the script would only have been two lines. (The start-test function from PSExpect runs the test and matches up the named range with the test function). Not bad!
But that’s only part of the picture – the configuration of Reporting Services. The other part of the picture is basic functional testing. Reporting Services exposes its functionality as the Report Manager, a web application but also as Report Server, a web service. So – what if we used PowerShell to explore and test the Reporting Services web service? Using the same technique as was used with the Reporting Services WMI class, we can use get-member to help with the exploration. This doesn’t mean that we don’t have to test the Report Manager, but it does mean that we can get a quick check of the functionality without a manual test.
This requires a small bit of .NET working knowledge regarding web services. The trick is to use the .NET SDK ‘wsdl.exe’ utility to generate a proxy class for the web service, and then wrap that proxy class into an assembly that PowerShell can access:
wsdl http://localhost/ReportServer/ReportService2005.asmx?wsdl csc /target:library ReportService2005.cs
Now load this library into PowerShell and use get-member again to investigate the methods that are available:
[void][Reflection.Assembly]::LoadFile("C:...ReportingService2005.dll") # Instantiate the proxy object for the web service $repsvc = new-object ReportingService2005 $repsvc.Credentials = [System.Net.CredentialCache]::DefaultCredentials $rs | gm
(I won’t list all the methods here because there are many).
... CancelAsync CancelBatch CancelBatchAsync CancelJob CancelJobAsync CreateBatch CreateBatchAsync CreateDataDrivenSubscription CreateDataDrivenSubscriptionAsync CreateDataSource CreateDataSourceAsync CreateFolder CreateFolderAsync CreateLinkedReport CreateLinkedReportAsync CreateModel CreateModelAsync CreateObjRef CreateReport CreateReportAsync CreateReportHistorySnapshot CreateReportHistorySnapshotAsync CreateResource CreateResourceAsync CreateRole CreateRoleAsync CreateSchedule ...
The (almost complete) example included in the PSExpect download uses the ListChildren, CreateDataSource, and CreateReport methods of the web service to check the functionality of Reporting Services. The test script uses functions that wrap the calls to the web service in the name of making the test easier to read for testers not familiar with the Reporting Services web service interface. This is not the complete test – ideally there would also be a ‘render-report’ and ‘test-report’ set of functions provided for the testers so that actually generating a report is also part of the test (work in progress).
write-host "Functional testing ..." $DsBefore = get-datasourcecount new-datasource -Name "FromPsScript" test-datasourcecount -Expected ($DsBefore + 1) -Label "Datasources.Count.AfterNew" remove-datasource -Name "FromPsScript" test-datasourcecount -Expected ($DsBefore) -Label "Datasources.Count.AfterRemove" # Verify the report create/delete functionality $RsBefore = get-reportcount new-report -Name "ReportFromPsScript" test-reportcount -Expected ($RsBefore + 1) -Label "Reports.Count.AfterNew" remove-report -Name "ReportFromPsScript" test-reportcount -Expected $RsBefore -Label "Reports.Count.AfterRemove"
The example is available for download in the latest source code version of PSExpect. All together, this wasn’t very much code. The ‘mean-time-to-testing’ is low considering the automation – and I have an artifact that other people can use for running the tests at any time.
RsLib.ps1 – Contains the wrapper functions new-datasource, test-datasourcecount, remove-datasource, etc.
Test-RsConfig.ps1 – the test script for both configuration and functional tests described above
Test-RsConfig.xslx – Excel spreadsheet containing the named range for properties and expected values used in configuration testing.
Summary
The ‘get-member’ cmdlet is extremely useful for exploratory testing, especially when combined with the easy access to WMI classes and to web services in PowewrShell. In the example, I’ve used the SQL Server Reporting Services WMI class to check the configuration and installation against expected values, and then used a proxy class around the Reporting Services web service to check basic functionality. What ‘get-member’ and PowerShell do for us in these cases is give us the tools and the means for exploring the targets of our tests. Oh yeah – everything is in a script that we can share and evolve as our understanding of the test target increases.
