- Questioned: Simplicity
- Questioned: Usability Testing
- Questioned: Process
- Tipping Sacred Cows
- Fire Away
Like this article? We recommend
Like this article? We recommend
Like this article? We recommend
Questioned: Usability Testing
In the late 90s, Rolf Molich (co-inventor of the heuristic evaluation method with Jakob Nielsen) started a series of Comparative Usability Evaluations (CUE) to establish a set of best practices for usability tests. To do this, he hired multiple teams to evaluate a single design and report back their results. And what he found was also quite surprising.
It turns out that when different teams look at the same design, they see incredibly different things. Nine usability teams evaluated Hotmail.com and, of the several hundred problems they discovered, only a small percentage were noted by more than half of the teams. Worse, most of the things that were called “serious” or “critical” were only reported one time.
The same thing happened when he hired 17 teams to evaluate a hotel reservation system.
Now, usability testing is great for a lot of things. It’s great for validating new design ideas. It’s great for feeding your instincts. There is no better way to learn how people work than to watch them work. It’s great for helping people on this side of the screen understand the people on the other side.
What is it not great for? It’s not great for finding the problems in an existing design to determine what you should try to fix. It’s terrible for that. If you’re using it for that purpose, you can spend a lot less time and money by having a lone reviewer do it instead, because what a lone reviewer finds will be no better or worse than what you find in an expensive round of usability tests.
Use it to test new ideas, not old ones.
By questioning how usability testing is done, Rolf Molich has shown us how and when we can put this valuable tool to its best use.