upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

Reverse Engineering of Content to Find Usability Problems: A Healthcare Case Study

Shadi Ghajar-Khosravi, Flora Wan, Samir Gupta, and Mark Chignell

Journal of Usability Studies, Volume 8, Issue 1, November 2012, pp. 16 - 28

Article Contents


Introduction

The purpose of this paper is to present a case study of the use of reverse engineering as a usability testing tool instead of the traditional task-based scenario method. We have heard anecdotally of usability specialists using reverse engineering in their practice, but there seems to be a lack of published discussions of the use of the technique. Thus, in this paper we present a case study in the use of reverse engineering for usability testing in the hope that it will promote increased use of the reverse engineering technique.

Healthcare is the area of application context for this case study, and more specifically the creation of asthma action plans. These plans are important artifacts that provide guidance on how to manage asthma as a chronic condition. We chose this context to explore the value of reverse engineering.

Background

Many methods for general purpose usability evaluation have been described (e.g., Sharp, Rogers, & Preece, 2011), both quantitative (e.g., measuring task completion time and number of errors) and qualitative (e.g., collecting feedback on user preferences through questionnaires and focus groups; Rubin, Chisnell, & Spool, 2008). However, one method that has received little attention is the case where a software tool produces a specific artifact, and participants are asked to recreate the artifact (using the tool) through a process of reverse engineering.

The term reverse engineering as it applies to software engineering is defined by Chikofsky and Cross (1990) as the following:

The process of analyzing a subject system to (i) identify the system’s components and their interrelationships and (ii) create representations of the system in another form or at a higher level of abstraction.

From a usability testing perspective, the four phases of a typical reverse engineering process(Storey, Sim, & Wong, 2002) can be adapted as the following sequential steps:

In the case study reported in this paper, we compared the reverse engineering approach to usability testing with a traditional task-based scenario technique. The research involved an evaluative case study using an online tool for creating asthma action plans. The tool was developed to let three different groups of stakeholders (respirologists, patients, and asthma educators/primary care physicians) create asthma action plans collaboratively. Further information on the development of the tool was provided by Wan (2009) and by Gupta, Wan, Newton, Chignell, and Straus (2011).

An asthma action plan consists of a set of written instructions on how to treat asthma when the condition is stable, as well as when the symptoms are active (Plottel & Feldman, 2008). Asthma action plans usually use the “zone” system, patterned after the red-yellow-green lights of a traffic signal, to describe the severity of asthma symptoms and to determine the appropriate response (Sloane, Slatt, Ebell, & Jacques, 2008). Studies have shown that the use of asthma action plans for self-management of asthma symptoms greatly improves the health of adult patients (Gibson et al., 2003), and in the case of asthmatic children, significantly reduces the number of acute care visits per child (Zemek, Bhogal, & Ducharme, 2008).

The case study reported here was part of a larger project named Wikibreathe (a.k.a. Online Collaboration Tool for Asthma Action Plan with Usability [OCTAPUS]). The primary goal was to both determine the preferences of different stakeholders for the design and content of an asthma action plan and to use an online tool to collaboratively author a consensus asthma action plan.

 

Previous | Next