Reverse Engineering of Content to Find Usability Problems: A Healthcare Case Study
Shadi Ghajar-Khosravi, Flora Wan, Samir Gupta, and Mark Chignell
Journal of Usability Studies, Volume 8, Issue 1, November 2012, pp. 16 - 28
Article Contents
Tips for Usability Practitioners
We were interested in finding out if reverse engineering could be a useful alternative to task-based usability testing in this and perhaps other contexts. For a practitioner, would usability testing with reverse engineering be as easy to set up, administer, and analyze as a traditional test, and would it uncover a representative set of usability problems? Why should practitioners use the reverse engineering method when there are a number of methods already available that have a much more extensive track record?
Based on our experience, the reverse engineering method is worth a serious look. We found that participants had a positive attitude towards the reverse engineering method. Although the participants had to figure out how to carry out the task, the definition of the task seemed more straightforward and less arbitrary because they were trying to duplicate a meaningful artifact. Thus, we believe that the value of testing software using the reverse engineering method seems more self-evident to test participants than methods where they are told to carry out a series of tasks that have been defined for them.
We found that the reverse engineering method required more attention to detail by the participants, because of the requirement to make a copy of the target artifact. In some cases, this attention to detail might be beneficial in finding more problems, but in other cases it may find problems that are too detailed to concern most participants. Thus, they might put too much time and pay too much attention to some of the details that would not necessarily be confronted when using a task-based testing method. This is where practitioners have to exercise judgment in deciding whether or not reverse engineering may work for a particular tool and associated set of users. While it is possible that instructions to focus on the right level of detail might be beneficial in this regard, it was not an issue that we examined in our case study.
Our bottom line message for practitioners based on our experience with the case study is that the reverse-engineering approach is useful if you want to
- Find problems with the formatting/layout aspects of a tool.
- Avoid bias from the language used in task scenarios, for example, if an instruction says to “create a report” and the UI has a Create menu with a Report command.
- Encourage the participants to work more slowly and explore the tool more thoroughly at first.
On the other hand, the reverse-engineering method is not recommended if
- You want participants to use their own data or in other circumstances where participant’s motivation (or lack thereof) can have a significant effect.
- The goal/outcome has ambiguity; it’s not clear to people when they are done.
- Session time must be kept short. It will take longer for participants to complete tasks with reverse engineering.
Note that in any given study, the preferred approach might be to use a combination of usability testing methods. Reverse engineering could be used with other methods to allow participants to more fully explore products. However, based on our experience with the case study, we suggest that when reverse engineering is used along with other methods, it should generally be carried out first.
While this case study assesses the benefits of using reverse engineering for usability testing in a specific healthcare context, we believe that the benefits of the method found in this case study may apply more broadly.