A Heuristic Approach to Test Charters

One of the elements of Session-Based Test Management (SBTM) that I have found testers new to the approach struggle with the most is the creation of effective test charters. I believe this is also a skill that is key to SBTM, as crafting an achievable charter that is sufficiently directive but not overly restrictive is essential to productive testing sessions.

In teaching an Essential Testing class in Christchurch recently, I was prompted by the attendees' struggles with this technique to reflect on the skills and approaches I had unconsciously developed in writing charters, and this led me to identify a number of heuristics that may help testers to craft effective charters for their test sessions.

Probably my main resource for advising testers new to SBTM on writing effective charters is an excellent EuroSTAR webinar from Michael Kelly that is available on both YouTube and SlideShare. Amongst a good deal of other useful advice around chartering, Michael offers the following as a basic template for writing effective test charters:

My mission is to test <insert risk here> for <insert coverage here>.

Using this template, the "risk" is the type of problem you're testing for, whereas the "coverage" is the area or function that you're testing, for example: "My mission is to test customers with multiple policies for assessment calculations".

Of course, there may be occasions when these three central elements are not useful in constructing a useful charter. At the beginning of a testing project, you may conduct a series of reconnaissance style sessions, and so you won't necessarily know the specific risks or coverage that are important to drive your activity.

As such, this template is a nice simple heuristic for writing a charter, in that it will often help you to identify a simple driver for an effective testing session, but sometimes it won't be applicable.

Personally though, I predominantly work from a functional model of an application - not always, and I'll use heuristics to challenge and broaden my model and approach, but this is my instinctive tendency - so I often opt to start with the coverage, then specify the risk/s. To do this, I ask myself two basic questions:

• What am I testing?
• What could go wrong?

Each time I answer the first question, I can usually answer the second question several times over - and I have usually done this in the process of building a functional model of the application. This means that for each coverage area, I have identified several risks. This is typically displayed visually by my model, but could be shown as follows:

What am I testing?	What could go wrong?
Assessment Calculations	Customers with multiple policies
	Customers with outstanding premiums
	Customers with grandfathered benefits
	Customers with no claims bonuses

This gives me two of the three raw ingredients for a number of potential charters - coverage (what am I testing?) and risk (what could go wrong?). The third pillar of effective charters, as described by Michael Kelly, is time - what can you achieve during the time allocated for your test session?

In this example, I might end up with three charters:

My mission is to test customers with multiple policies for assessment calculations

My mission is to test customers with outstanding premiums for assessment calculations

My mission is to test customers with outstanding premiums and customers with grandfathered benefits for assessment calculations

You'll note that the third charter here combines two risk areas. This is because I deem that I could effectively test both of those risks during one session. Perhaps this is because they're relatively simple functions, or are deemed low priority, or are unlikely to have a negative impact, or simply that they're very closely related processes or functions.

In some cases, it might pay to go into more detail in your charter surrounding the coverage and risk areas, so that you've got a more clearly defined mission. For instance, consider the following two charters:

My mission is to test customers with multiple policies correctly receive their multi-policy discount on their premiums for assessment calculations processed automatically by our web-based quote calculator.

My mission is to test customers with multiple policies aren't sent multiple quote documents for assessment calculations processed automatically by our web-based quote calculator.

Both charters address the basic risk and coverage areas identified above, but become quite different once the detail of the risk is fleshed out a little further.

Sometimes this level of detail is not required, or may not actually be valuable - perhaps this becomes too specific a charter in some contexts, prescribing exactly the testing that will be performed. In other situations though, there may be several different and complex methods that could conceivably see the stated risks realised, and so spending a session exploring these is of value.

What these two charters also highlight is a variance in the tester's likely approach to investigating the product - the style with which they approach the performance of testing.

In the first, the charter adopts a positive, confirmatory tone - the main drive of the session is to test whether the product does what some set of expectations suggest it should do, given a particular set of circumstances. The second is more negative, seeking instead to discover whether or not a suspected or potential problem will manifest itself in a certain situation.

This highlights a potential fourth element of an effective charter that might be worth considering: the style of testing to be performed. The way that we approach testing can have a significant effect on what we observe and thus the type of information that we harvest, and so consciously controlling our style might be a useful tool in some situations.

The impact of a style of testing is particularly prevalent in factory-style script-driven testing - where we are prone to inattentional blindness and other limiting psychological flaws. However, the exploratory tester is not immune, with many an explorer suffering from confirmation bias - continually validating their theories without ever really challenging them.

As such, writing charters that dictate a style of testing may be useful. In the above examples, the two charters look at broadly similar risks and coverage areas, but they approach the testing with a very different perspective. One is focused on positive affirmation, the other is negatively hunting for failure. These approaches will likely yield different behaviours, different observations and thus, if both are conducted responsibly, potentially a broader and more useful picture of the coverage area to inform stakeholder decisions.

Indeed, it might sometimes be valuable to actually substitute style for risk in Kelly's charter template described earlier:

My mission is to test <insert style here> for <insert coverage here>.

This can help the tester to remove themselves from a risk-focused view of the product - which is probably the predominant way that testers approach a product. Instead, they may see value in touring a product or feature, or using a product or feature whilst adopting one or several specific personas. These are less directly focused on the risk (though an assumed potential for risk is inherent in all testing) and thus may help the tester again to observe the product in new and useful ways.

Of course, sometimes a session may be overly limited by the conscious application of a singular style. Part of the value in an exploratory approach to testing is to enhance the variability of the testing performed - it is a style of testing in and of itself that values the ability to observe and adapt methods and styles and approaches dynamically.

As such, consider the inclusion of the style of the testing to be performed in a charter a heuristic approach to effective chartering. Applied with skill and judgement in the right context, it can enhance your charters and in turn increase the value of your testing. But it is not a rule that can and should always be applied.

In summation, chartering is a definite skill and takes practice and patience to master. Much depends on the instinct and judgement of the tester to assess how much freedom they require to explore, or how much explicit direction is useful, in tandem with the ever present difficulty of estimating how long any activity is likely to take.

But for the new tester trying out charter writing for the first time, or even for the old hand looking for some tips to hone their instincts, here's a recap of some heuristics for effective test chartering:

• Remember the three central elements of a charter: risk, coverage and time
• Ask yourself "what am I testing?" and "what could go wrong?"
• Consider the depth of your charter: seek the balance of directed exploration
• Consciously consider the style of your testing, whether consistent or varied

My thanks go to Kathleen Cosgrove whose question prompted the introspection that shaped this post. In addition, Aaron Hodder as ever proved an inspiring colleague and triggered my consideration of the importance of style in test charters.

Further, sparked by the discussions that led to this post, Aaron and I have developed a simple tool for generating heuristic test charters as a thinking tool for testers. We call it the Chartersaurus and you can check it out here.

If you have thoughts on the content of this post, or have other useful heuristics for effective test chartering, please share them in the comments.