What Do People Think of Your Site Design? Here's How To Find Out.

This chapter is from the book 

Creating Test Plans

If you had to conduct a test tomorrow, there are three basic elements you'd need (beyond a group of users, a place to conduct the test, and—oh, yeah—a set of designs to test). The simplest test plan—described in the elements of layer 1—consists of a set of goals, a description of the logistics and method, and a set of questions to ask the participant. As stated, these are pretty broad categories, and therefore suitable for informal testing. The more formal the testing, the more details you'll need. Fleshing out the test script is described in layer 2. Further details around the logistics, described in layer 3, seem somewhat extraneous but your stakeholders may demand them.

Layer 1: When You Need to Test Tomorrow

The basic elements of a usability test answer three questions: What do we want out of this test, how will we conduct the test, and what will we ask users during the test? With those questions answered, you've got the barest information necessary to make a test happen.

Test Objectives

In planning a test it's easy to let things get out of hand. Without any direction, you'll find yourself asking users questions of increasing irrelevance. Explicitly stating a direction in the test plan, as in other documents, allows you to keep the design of the test on track. The script should be derived from the objectives and everything you ask the user should take you closer to meeting your objectives.

Unfortunately, people designing usability tests frequently gloss over this part because (1) it's hard, and (2) they want to dig into the script. Consider this: Your script is meaningless unless it is working toward a specific purpose.

Because the test objectives drive the discussion between the test facilitator and participant, you can make them pretty specific. Too often, the test objective looks something like this:

To determine whether our web site is easy to use.

You'll forgive me if I refer you to the Wikipedia entry on "duh." You can use a simple strategy to make your objectives more specific—think about why people are coming to your site. (If you created the personas described in the previous chapter, this should be easy.) By basing test objectives on user motivations, you get objectives that look more like these examples for a fictitious health encyclopedia:

To determine whether people who come to the site with a specific illness in mind are satisfied with how they located relevant information.

To determine whether people who are doing research on behalf of another person can easily refer someone else to the infor mation they find.

To understand how people who have just been diagnosed with an illness want to revisit their research.

These objectives may be used together—nothing says your test must have only one objective. Once you've written up a couple, you can do a quick litmus test. If you state the objective in the form of a question, is it an answerable question? To take the first (bad) example, the test designers could ask themselves "Is this site easy to use?" which would not be an easy question to answer. On the other hand, you could probably easily answer the question "Are people who have a specific illness in mind satisfied with how they found relevant information?"

Test Logistics

Your plan needs to describe the who-what-when-where-how of the test, if for no other reason than to make sure you've thought of everything you need before recruiting and scheduling participants. Even the most basic tests have lots of questions that need answering: Where will you be conducting the test? How many people do you need to recruit? When will the tests take place? Who from your team is involved? What roles will people play? The list goes on. The more you have planned out, the less you have to worry about when the test happens.

For stakeholders and team members who have never participated in a usability test (or at least one organized by you) the test plan gives a good picture of what's going to happen. The logistics section is also a good place to describe a high-level agenda for the test itself—sort of an outline of the test script. If each participant is scheduled for an hour, you can describe how you break up that hour.

Another topic you might cover in the logistics section is methodology. There are several different kinds of usability testing methods, and this section is your opportunity to define and defend the one you've chosen.

Finally, your stakeholders may want to know what happens after the test, in which case you can include a description of the type of analysis you will do. Another approach is to describe the outputs—the documents and artifacts that are by-products of the test. For an informal test your outputs may be the raw notes and a short report. For more formal tests you may have additional outputs, like audio and video recordings, or more detailed reports, like an observations report and a recommendations report.

Test Scenarios

Test scenarios run from the general to the specific. For example, in some cases your scenarios may be as straightforward as "present screen to user and ask for impressions." In other cases, you may want to ask the participant to imagine himself in a specific situation and use the web site to address the situation: "You have just come down with the flu and you want to see if any of your symptoms put you at high risk."

The list of these scenarios forms the test script. Layer 2 describes a more detailed test script, but at the basic level the purpose is to give an impression of what the facilitator will ask the user to do.

Layer 2: Further Background

If you've got a little more time to put into planning, there's plenty more detail you can add to provide a clearer picture of what the testing is for and what you'll do during the test.

User Profiles and Screener

Although the logistics element from layer 1 should reference the kinds of people you'd like to recruit, you can always provide more detail. The more complex the test, the more detail you'll want to give because you may be testing different user groups. Defining profiles for each group allows you to effectively design an appropriate test for each group. If you've done user personas (described in the previous chapter) you can draw straight from those. Here's a user profile for a parenting web site:

The Day Job Parent needs to work outside the home. He sees his kids in the morning, maybe drops them off at the bus stop, and then sees them again in the evening, when he comes home from work. The Day Job Parent may steal a glance at this web site during a lunch break, seeking advice on specific child-rearing problems. He may spend a bit more time on the site in the evening after the kids are in bed. In the Day Job Parent's house, both spouses work. They are both familiar with Internet technologies, and have shopped and researched health issues online.

High-level descriptions of users are one thing, but actually finding people who meet those criteria is another. If your test is formal—that is, you're recruiting a large number of people outside your immediate user group—you need to put together a screener. A screener is a set of questions to ask potential participants to see if they fit your needs for the usability test.

Some screener questions are pretty straightforward, asking about experience with using the Internet or shopping online. Some will be more specific to your particular application. For example, for the fictitious online health encyclopedia, your screener might seek out people who have recently been to the doctor, or who visited a health site in the last three months. Such questions will help narrow the pool of applicants to those who best match your target user group.

Usually, recruiting from the general public is done by a recruiting agency. They have a large list of people to draw from. If you're using an outside agency, you'll need to provide as much detail in the screener as possible. Indicate when a particular answer to a question should eliminate a potential participant. Indicate quotas for particular answers. You may want, for example, half your participants to have visited a health site in the last three months, and half to never have visited a health site. Recruiting methodologies aside, you need to communicate the makeup of your ideal group of participants to the recruiter. A screener is the best way to do this.

Pre-Test and Post-Test Questions

Usability testing best practices suggest that you ask users questions before and after the actual test. At the beginning of the test, before showing participants any screens or prototypes, you can establish their expectations with respect to the web site and their general level of experience. You can clarify their motivations for using your site and flesh out your profile of them.

The format for these questions will vary depending on your methodology. Some usability testers ask the questions of the user in person, recording their answers by hand. Others give participants a questionnaire that they can answer separately. (This approach allows the facilitator to spend less time with each participant.) If you decide to let participants answer the questions on their own, you may want to separate these questions into a different document because it will be something you hand to them. In that case, you may need to format it differently, and—if the participant is to answer the questions without intervention from the facilitator—you'll certainly want to be explicit about directions.

Pre-test questions can help you get a handle on the user's expectations and experience before seeing the site for the first time. Questions after the test can gauge overall impressions and allow general pain points and suggestions to surface. Formatting for pre-test and post-test questions can include open-ended questions and questions with predefined responses.

Examples of open-ended questions:

We're developing a new system to support your sales process. What are the most challenging tasks in the sales process for you?

The web site we're building is called FluffyPuppies.com. What do you think you'd find on this web site?

What system do you currently use to manage your family videos?

Examples of questions with predefined responses:

On a scale of 1 to 7, where 1 is strongly disagree and 7 is strongly agree, how would you rate this statement: I found the web site easy to use.

What level of Internet user are you? Beginner, Intermediate,

Advanced

How long have you been involved with the sales process?

Regardless of how you deliver the questions to the participants, you should write the questions exactly as you intend to ask them. This can helpg ensure that you ask the question the same way every time.

The Script

Layer 1 calls for defining the scenarios for the usability test, though it leaves the format pretty open. For the second layer, you can add more detail, depending on the complexity of the test. You can look at the script as a hierarchy of information. At the highest level are the scenarios, which re-create situations in which participants might actually use the product. Each scenario includes a series of tasks, and each task describes what will be read to the participant and the background material for the facilitator. The task or scenario may also include specific follow-up questions and notes to the people developing the prototype.

Layer 3: Adding Further Detail

You may have seen tests that go into even more detail than described in the first two layers. In some cases, this information is important because it reflects a larger-scale test. In other cases, the value of this information is measured only by your team's and client's need to have it spelled out.

Background

Your test plan can provide an overview of the project and the product being tested. The background describes the purpose of the product and offers a high-level business case for it. This section of the test plan can be used to rationalize the testing, or describe the design process to date. There are two main purposes to this content: First, it makes the plan internally comprehensive, a stand-alone document. Second, background information helps make stakeholders who care about this sort of thing more comfortable with usability testing. If you don't have a need for either of these, the background section may not be essential to conducting a good usability test.

Functional Details

If you are testing a prototype of the site you may need to provide instruction to the facilitator that describes how it should behave during the test. Prototypes are notorious for being hopelessly incomplete at the time of testing and the facilitator can easily misstep if not adequately prepared. In a test plan, the functional details should be embedded in the script adjacent to the relevant scenarios. Figure 3.4 shows how functional details might be embedded with the rest of the script.

For example, in running a test on an internal web application designed to support a specific business process, the facilitator will need some direction about how the application is supposed to behave. For a scenario where the user needs to check on incoming responses from users, the test plan might include directions like: "The user must click the checkboxes next to the incoming responses before clicking 'Export to Text File'. Clicking this button before selecting responses will lead to an error."

Expected Behaviors

In addition to functional details, the script may also include notes to the facilitator describing the expected response from the participant. This helps the facilitator gauge whether the participant is going in the right direction. Depending on your methodology, you may want the facilitator to steer participants right if they stray too far off course.

Of course, expected behaviors should appear adjacent to the relevant scenarios. At the same time, they should be formatted to distinguish them from the part that gets read aloud to the participant. (Giving them the answer during the course of the test just wouldn't be fair, would it?) Depending on the facilitator's comfort level with the prototype, you may need to be very explicit about the expected behaviors.

Table 3.1. Comparing Explicit and Non-Explicit Expected Behaviors

Explicit	Non-Explicit
User clicks checkbox next to responses	After users select responses, they should click Export to Text File and select the Attachments option. User clicks "Export to Text File" Export options dialog box appears User clicks checkbox for "Export with Attachments" User clicks button "Export" Browser window appears with text file

Preparing a Test Plan: The Basics

In simplest terms, the function of a test plan is to tell people what to do in anticipation of and during a usability test. It is a set of instructions, grounded in project-specific objectives. Like any document in this book, the test plan will be shaped by a situation analysis—an assessment of the purpose, timing, and audience for the document.

The Why and When of Usability Testing

For most other documents in this book, the situation analysis calls for assessing the purpose of the document itself. In this case, the question isn't "Why should I do a test plan?" as it might be with wireframes or personas. Instead, the more relevant question is "Why should I do usability testing at all?" because if you're doing a usability test, you need to do a test plan. Like any good business meeting, you need to go in prepared, and a test plan gives you that preparation.

Still, the answer to that question—"Why do usability testing?"—will define the content of the test plan. Usability testing is an essential component of design, but it can't be done for the sake of itself. To understand how the purpose of the test affects the content of the test plan, we need to look at the two main kinds of testing: formative and summative. Formative tests take place during the design process, allowing the design team to do research on early versions of a design as it's taking shape. Summative tests occur at the end of the design process, with a more or less final product.

Plans for formative tests: You'll find that most formative testing is to assess the usability of a specific part of a web site—the home page and high-level pages, the checkout process, or the navigation labeling. The test plan needs to recognize that participants will only experience part of the entire interface. It may be more rigidly scripted to focus on specific issues and avoid users' missteps into unfinished parts of the interface.

Plans for summative tests: When conducted with a final product, usability testing can be more open ended. Rather than identifying particular tasks, your script may focus more on the post-test questions to glean the participant's opinion of the product.

Adjusting the Test Plan for Your Audience

The two main audiences for usability test plans are the project stakeholders and the test facilitator—the person doing most of the orchestration during the test. There may be other players in the test—people taking notes, people managing the facility, people working the recording equipment—or there may be one person who does it all. The facilitator may be someone on your team (i.e., you) or it may be a contractor or freelancer you've hired for this purpose. In the latter case, the script needs to be as specific as possible.

Frankly, even if you're doing the test yourself, making the script specific can be very helpful. During the test you'll be thinking about so many things at the same time—what the participant is doing right then, whether the recording devices are working, what will happen when the user clicks a button or link—that you won't want to have to worry about what to do next.

A specific script is also useful for stakeholders. By being explicit about what the facilitator will say to the participants, the script can help set the stakeholders' expectations, especially if they've never been involved with usability testing.

If the facilitator is outside your project team—a contractor you brought in just for tests, for example—your test plan may need to be explicit in other areas as well, like the purpose of the test, the test procedure, and the project background. It's easy to take this information for granted when you're working with the same set of people throughout the process. But for a new or unfamiliar team, a fully fleshed-out test plan can help build a common understanding of the project, the methodology, and how they fit together.

Making Test Plans Unboring

The situation analysis—understanding the who, when, and why of usability testing—helps define the contents of the test, providing direction for what kinds of information you need to provide and how much detail to give. With these decisions made, you'll need to think about how to format the document itself. Although this isn't a document heavy on the visuals, there are still several considerations for formatting.

Prose or bullets: Items like test objectives, background, and method can be described in straight paragraphs or through bullet points. The decision is more than aesthetic. The primary consideration is your audience, since you can communicate the same message with either bullets or prose. Bullets will be more effective if you do not need to go into a lot of detail. The objectives for the imaginary test at the beginning of this chapter might be rendered in prose like this:

Users researching health information online are a committed and persistent audience. Despite their keen interest—or maybe because of it—they are especially picky about the web sites they'll use to find information. A web site should not just have the most up-to-date information. It needs to be easy to find, written well, and in a format that gives users lots of flexibility in how they use it. The purpose of this usability test is to see how well our site lives up to the expectations of our users, by putting potential users in mock scenarios and observing the effectiveness of the web site design.

Specifically, we want to learn whether people with a specific illness can quickly find useful information. They should be able to locate it quickly and it should be compelling enough to make them want to stick around. This test will also measure whether the site makes it easy to refer other people to information—our users typically are doing research on someone else's behalf. Finally, we recognize that our users don't come to the site once. They return again and again to learn more or to focus on a new aspect of their condition. This test will identify how the design can be improved to more effectively meet this need.

Distinguishing "say aloud" from instruction: There are two types of information in the script—things to expose to the participant and things that give the facilitator background on a particular task or scenario. In some cases, the facilitator needs to read things out loud to the participant; in others, the participant must read to him- or herself. Figure 3.4 illustrates all the different parts of a usability script.

Making Test Plans Airtight

Here are a couple tips for making sure you cross all the t's and dot all the i's in your test plan. The best strategy, however, is not in document preparation, but in the planning itself. If you've made your testing methodology airtight, your test plan will follow.

Save the Script for Last

Before digging into the script details, you should make sure the test's objectives and logistics are more or less nailed down. By saving detailed script development until after these other pieces are in place, you ensure that your work on the script is compatible with the test objectives and with the methodology. For example, if you're conducting a test on a semifunctional prototype, that may limit the number and type of questions you'll ask participants. If you do not have the time, budget, or personnel to develop the prototype further, those facts will establish boundaries for the tasks participants can perform.

Format the Document Appropriately

Consider how the facilitator will use the actual document during the test. Even if the sessions are being recorded, for example, the facilitator may want to take notes on the script itself. Different facilitators have different preferences, so ask yours what he or she wants. (If you are the facilitator, make sure no one catches you talking to yourself.) A document designed to be a note-taking tool will look different than one that's simply a series of prompts. If the test includes pre- and post-test questions for participants to answer on their own, perhaps these can appear on separate pages so they can be easily removed and distributed.

Risks in Creating Test Plans

When putting your test plan together, you may run into a couple different obstacles. Realizing these shortcomings during the test itself is way too late, so you'll want to diagnose and mitigate these risks as early as possible.

Incomplete Script

A script can be incomplete for any number of reasons: The read-aloud part does not adequately describe the scenario, the expected behaviors aren't explicit, there are crucial scenarios missing. The single best remedy for these problems is to conduct a dry run, which takes the usability test out of the abstract and into the real. Anything missing from the plan appears quite starkly when in the thick of a test. If you don't have time to test the test, as it were, you can also cross reference the script with the user profiles to identify potential holes in the scenarios. You can do a cheap version of the dry run by reading the scenarios aloud to colleagues and seeing if they understand them. A set of expected behaviors is simply a set of instructions for doing a particular activity on the web site. Hand someone the list of expected behaviors for a scenario and see if they can follow it to the goal. These are low-impact ways to diagnose potential holes in the script.

Document Unusable for Testing

It's one thing to review a test plan in a meeting and quite another to bring it into a testing situation. The content of the document may work, but the formatting may not be appropriate for use during an actual test. Again, the best way to mitigate this risk is to do a dry run of the test. By putting everyone (the facilitators and note-takers) and everything (the web site or prototype and the test plan) through its paces, you can establish whether the document format is suitable.