Mission-Targets-Tests

Lately testing people have been talking about testing heuristics – tips, techniques, tactics, strategies for testing something.  I have one that I call MTT.

M – Mission
T – Targets
T – Tests

But I remember it as "mean time to testing".  Exploratory testing excels in part because the mean time to testing is small compared to (caution: wiki-speak ahead), say, BigUpFrontTestManagement.  As an example of minimizing the mean time to testing, the example below uses PowerShell to explore the configuration and functionality of SQL Server Reporting Services 2005.

Mission

Encourage the team to build consensus on why testing will be conducted by paying attention to the project context – who tests and what their background is, what is the system under test, who supports the test environment, what are the communication patterns between testers and developers, … post the essence of this as a ‘test mission’ in the team’s work area and let them mark it up as the project evolves.  The more people that are involved in the test effort, the more useful a ‘testing working agreement’ becomes.  Testing faster requires that all the testers understand why they are testing so that they get to the point.

Targets

If the mission indicates why we are testing, then the targets drill deeper and identify what will be tested. I see a lot of testing without clearly-defined test targets.  Some heuristics are test targets, and that helps a lot.  Exploratory testing, even when automated, fits really well with test targets because listing them gives you a testing backlog, making citing progress and indicating expected completion dates simpler using something like testing burndown charts.

Consider automated exploratory testing with PowerShell – let’s use the Microsoft Reporting Services installation test as an example.  To use MTT heuristic and the mission stated above, the second step would be to create a testing backlog of the targets of the tests – in other words, what exactly would you want to test?  As with all things agile, this list doesn’t have to be complete because you have every right to revise this backlog and prioritize it as necessary, within the context of the test mission.  It is important to avoid listing how these targets might be tested – that’s left to exploration.

Microsoft Reporting Services Installation Test – Targets (Testing Backlog)

ReportServer – Service Control
ReportServer – Service Properties
Database – Reporting Services Db – Topology
Database – Reporting Services Db – Physical Configuration
Database – Reporting Services Db – Authentication
Database – Reporting Services Db – Authorization
Operations – Performance Monitors – Availability
Operations – Microsoft Operations Manager – Access
Operations – Microsoft Operations Manager – Alerts
Web Service – Availability
Web Service – Authentication
Web Service – Authorization
Web Service – Functional – Run Report
Report Manager – Availability
Report Manager – Authentication
Report Manager – Authorization
Report Manager – Functional – Reports
Report Manager – Functional – Data Sources
Report Manager – Capacity – Concurrency
Report Manager – Capacity – Reponse Time

Given this list of test targets, an exploratory test effort would start by estimating each one of the targets to create a testing burndown chart, and then diving in according to the priorities assigned in the backlog.  PowerShell is incredibly useful for this given the ‘get-member’ cmdlet.

Tests

Continuing the Reporting Services example, the first thing that a tiny bit of investigating reveals is that SQL Server Reporting Services has a WMI class that exposes the configuration.  Let’s get that class and see what we can do with it.

$targetcomputer = "localhost"
$rs = get-wmiobject -computer $targetcomputer -namespace `
    "rootMicrosoftSqlServerReportServerv9Admin" MSReportServer_ConfigurationSetting
$rs | get-member

Name

MemberType

—-

———-

BackupEncryptionKey

Method

ConfigureSharePointExclusion

Method

CreateApplicationPool

Method

CreateVirtualDirectory

Method

DeleteEncryptedInformation

Method

DeleteEncryptionKey

Method

GenerateDatabaseCreationScript

Method

GenerateDatabaseRightsScript

Method

GenerateDatabaseUpgradeScript

Method

InitializeReportServer

Method

ListReportServersInDatabase

Method

ReencryptSecureInformation

Method

RemoveUnattendedExecutionAccount

Method

ResetVirtualDirectoryMappings

Method

RestoreEncryptionKey

Method

SetDatabaseConnection

Method

SetDatabaseLogonTimeout

Method

SetDatabaseQueryTimeout

Method

SetEmailConfiguration

Method

SetPortNumber

Method

SetSecureConnectionLevel

Method

SetServiceState

Method

SetUnattendedExecutionAccount

Method

SetWebServiceIdentity

Method

SetWindowsServiceIdentity

Method

ApplicationPoolActual

Property

ApplicationPoolConfigured

Property

ConnectionPoolSize

Property

DatabaseLogonAccount

Property

DatabaseLogonTimeout

Property

DatabaseLogonType

Property

DatabaseName

Property

DatabaseQueryTimeout

Property

DatabaseServerName

Property

InstallationID

Property

InstanceName

Property

IsInitialized

Property

IsSharePointExclusionConfigured

Property

IsSharePointInstalled

Property

IsWebServiceEnabled

Property

IsWindowsServiceEnabled

Property

PathName

Property

SecureConnectionLevel

Property

SenderEmailAddress

Property

SendUsingSMTPServer

Property

ServerPort

Property

ServiceName

Property

SMTPServer

Property

UnattendedExecutionAccount

Property

VirtualDirectory

Property

VirtualDirectoryHasSSLCertificate

Property

WebServiceIdentityActual

Property

WebServiceIdentityConfigured

Property

WebSite

Property

WindowsServiceIdentityActual

Property

WindowsServiceIdentityConfigured

Property

__CLASS

Property

__DERIVATION

Property

__DYNASTY

Property

__GENUS

Property

__NAMESPACE

Property

__PATH

Property

__PROPERTY_COUNT

Property

__RELPATH

Property

__SERVER

Property

__SUPERCLASS

Property

ConvertFromDateTime

ScriptMethod

ConvertToDateTime

ScriptMethod

Delete

ScriptMethod

GetType

ScriptMethod

Put

ScriptMethod

From an exploratory tester’s perspective, this list is super useful.  Now you can explore the configuration settings interactively using the $rs variable (they are all listed as Property in the above listing), or create a test script and use data-driven testing to check all the Reporting Services configuration parameters against expected values.  The ‘get-history’ cmdlet is also useful to retrieve and save the commands that you have experimented with in a format that you can use later for building a script.

To do the data-driven testing in Excel, create a table with three columns and all the properties that you want to test for.  Name the range TestRsConfig and save the worksheet.

TestCase

Property

Value

Security – Service Identity – As Configured

WindowsServiceIdentityConfigured

NT AuthorityNetworkService

Security – Service Identity – Actual

WindowsServiceIdentityActual

NT AuthorityNetworkService

Service – Instance

InstanceName

SQLEXPRESS

Service – ServiceName

ServiceName

ReportServer$SQLEXPRESS

ServicesDependedOn – Database – Server

DatabaseServerName

(LOCAL)SQLEXPRESS

IIS – WebSite

WebSite

1

Security – IIS – WebServiceIdentity – Configured

WebServiceIdentityConfigured

ALAPTOPASPNET

Next create a script that includes a function called TestRsConfig (the same name as the named range above) as follows (this script uses the PSExpect testing library for PowerShell):

set-psdebug -strict -trace 0 $targetcomputer = "localhost" $rs = get-wmiobject -computer $targetcomputer ` -namespace "rootMicrosoftSqlServerReportServerv9Admin" ` MSReportServer_ConfigurationSetting # Exercises the target of the test - the # and responds with a pre-defined set of responses that were # on the worksheet as the expected results function TestRsConfig() { param([string]$TestCase, [string]$RsConfigProperty, [string]$ExpectedValue) $ActualValue = $rs.PSBase.GetPropertyValue($RsConfigProperty).ToString() if ([string]::IsNullOrEmpty($ActualValue)) {$ActualValue = $null} if ($ExpectedValue.Trim() -eq "N/C") { AssertNull $ActualValue -Label $TestCase -Intention $Intention.ShouldPass } else { AssertEqual $ExpectedValue $ActualValue -Label $TestCase -Intention $Intention.ShouldPass } RaiseAssertions } # run the function library that contains the PowerShell Testing Functions # the functions are defined as global so you don't need to use dot sourcing if (!(Test-Path variable:_XLLIB)) { ..srcDataLib.ps1 } if (!(Test-Path variable:_TESTLIB)) { ..srcTestLib.ps1 } # Configuration and installation testing ... # Confirm the Reporting Services configuration matches expectations documented in # the Excel workbook 'TestRsConfig.xslx' write-host "Configuration testing ..." $FieldNames = ("TestCase", "RsConfigProperty", "ExpectedValue") start-test ((get-location).ToString() + "TestRsConfig.xlsx") "Sheet1" ` "TestRsConfig" $FieldNames

The line of note contains ‘$rs.PSBase.GetPropertyValue’ – a PowerShell base method that is proving to be really useful in writing test scripts without hard-coding the property names. In my script, I’ve used ‘N/C’ to identify Reporting Services properties that are not configured at all on my laptop.  If I had left them out and therefore avoided the if-then-else code section, the script would only have been two lines.  (The start-test function from PSExpect runs the test and matches up the named range with the test function). Not bad!

But that’s only part of the picture – the configuration of Reporting Services.  The other part of the picture is basic functional testing.  Reporting Services exposes its functionality as the Report Manager, a web application but also as Report Server, a web service.  So – what if we used PowerShell to explore and test the Reporting Services web service?  Using the same technique as was used with the Reporting Services WMI class, we can use get-member to help with the exploration.  This doesn’t mean that we don’t have to test the Report Manager, but it does mean that we can get a quick check of the functionality without a manual test.

This requires a small bit of .NET working knowledge regarding web services.  The trick is to use the .NET SDK ‘wsdl.exe’ utility to generate a proxy class for the web service, and then wrap that proxy class into an assembly that PowerShell can access:

wsdl http://localhost/ReportServer/ReportService2005.asmx?wsdl csc /target:library ReportService2005.cs

Now load this library into PowerShell and use get-member again to investigate the methods that are available:

[void][Reflection.Assembly]::LoadFile("C:...ReportingService2005.dll") # Instantiate the proxy object for the web service $repsvc = new-object ReportingService2005 $repsvc.Credentials = [System.Net.CredentialCache]::DefaultCredentials $rs | gm

(I won’t list all the methods here because there are many). 

... CancelAsync CancelBatch CancelBatchAsync CancelJob CancelJobAsync CreateBatch CreateBatchAsync CreateDataDrivenSubscription CreateDataDrivenSubscriptionAsync CreateDataSource CreateDataSourceAsync CreateFolder CreateFolderAsync CreateLinkedReport CreateLinkedReportAsync CreateModel CreateModelAsync CreateObjRef CreateReport CreateReportAsync CreateReportHistorySnapshot CreateReportHistorySnapshotAsync CreateResource CreateResourceAsync CreateRole CreateRoleAsync CreateSchedule ...

The (almost complete) example included in the PSExpect download uses the ListChildren, CreateDataSource, and CreateReport methods of the web service to check the functionality of Reporting Services.  The test script uses functions that wrap the calls to the web service in the name of making the test easier to read for testers not familiar with the Reporting Services web service interface.  This is not the complete test – ideally there would also be a ‘render-report’ and ‘test-report’ set of functions provided for the testers so that actually generating a report is also part of the test (work in progress).

write-host "Functional testing ..." $DsBefore = get-datasourcecount new-datasource -Name "FromPsScript" test-datasourcecount -Expected ($DsBefore + 1) -Label "Datasources.Count.AfterNew" remove-datasource -Name "FromPsScript" test-datasourcecount -Expected ($DsBefore) -Label "Datasources.Count.AfterRemove" # Verify the report create/delete functionality $RsBefore = get-reportcount new-report -Name "ReportFromPsScript" test-reportcount -Expected ($RsBefore + 1) -Label "Reports.Count.AfterNew" remove-report -Name "ReportFromPsScript" test-reportcount -Expected $RsBefore -Label "Reports.Count.AfterRemove"

The example is available for download in the latest source code version of PSExpect.  All together, this wasn’t very much code.  The ‘mean-time-to-testing’ is low considering the automation – and I have an artifact that other people can use for running the tests at any time.

RsLib.ps1 – Contains the wrapper functions new-datasource, test-datasourcecount, remove-datasource, etc.
Test-RsConfig.ps1 – the test script for both configuration and functional tests described above
Test-RsConfig.xslx – Excel spreadsheet containing the named range for properties and expected values used in configuration testing.

Summary

The ‘get-member’ cmdlet is extremely useful for exploratory testing, especially when combined with the easy access to WMI classes and to web services in PowewrShell.  In the example, I’ve used the SQL Server Reporting Services WMI class to check the configuration and installation against expected values, and then used a proxy class around the Reporting Services web service to check basic functionality.  What ‘get-member’ and PowerShell do for us in these cases is give us the tools and the means for exploring the targets of our tests.  Oh yeah – everything is in a script that we can share and evolve as our understanding of the test target increases.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s