Reverse Engineering of Content to Find Usability Problems: A Healthcare Case Study
Shadi Ghajar-Khosravi, Flora Wan, Samir Gupta, and Mark Chignell
Journal of Usability Studies, Volume 8, Issue 1, November 2012, pp. 16 - 28
Article Contents
Methods
A set of available asthma actions plans were surveyed to create an exhaustive list of features to be included in the Wikibreathe tool. Based on those features, an initial prototype was created. Then, a series of focus groups were conducted to gather initial feedback on the design of the Wikibreathe tool prototype. Based on the feedback collected, the revised prototype was developed and then evaluated using two usability testing methods: task-based and reverse engineering testing methods.
Prototyping
The following sections discuss the initial requirements for gathering data and the focus group that helped develop our prototype.
Initial requirements gathering
Before starting to design the Wikibreathe tool, available asthma action plans written in English were surveyed in order to create an exhaustive list of features that could potentially be included in the tool. Sixty-nine unique asthma action plans from around the world were analyzed for both content and format differences. Based on the requirements collected as a result of this survey, expert opinion, and best evidence, it was decided that the Wikibreathe tool should be able to generate an asthma action plan with the following sections:
- Header: This section contains the basic administrative information, such as a line to write the date and patient name, as well as other pertinent information such as the severity of a patient’s overall condition.
- Zone Description: This section contains the description of asthma symptoms divided into different sections based on the severity of each symptom. For example, a description of “breathing” in the Green zone might be “Breathing is easy” whereas in the Red zone the description would be “Very short of breath."
- Zone Instruction: This section describes the actions an asthma patient should take when their symptoms match one of the zone descriptions. An example of an instruction would be “Take reliever inhaler, ____ puffs every 4 to 6 hours.” .3
- Footer: This section is at the bottom of the plan and may include a space for patient or physician signatures or important reminder statements about asthma.
Focus group
An initial prototype was developed based on the format and content features identified during the initial requirements gathering phase. The overall concept for the envisioned tool was that the user would choose from a list of options on the left side of the screen to build up the asthma action plan that would then be displayed on the right side of the screen. A series of moderated focus groups and interviews were then conducted to gather feedback on the initial design of the Wikibreathe tool. In order to clearly evaluate the preferences of the stakeholders and to avoid bias, each stakeholder group was evaluated separately.
The results of all three stakeholder groups showed that participants had a positive attitude towards the Wikibreathe tool. A major difficulty across all groups of participants was lack of proper organization in the navigation system. Based on the feedback collected during these focus groups, the initial prototype was modified, and a revised prototype was developed for testing using the two contrasting methods (i.e., the traditional task-based method and the novel reverse-engineering method). See Figure 1 for the revised prototype.
One point to keep in mind in reading the case study discussion is that while asthma action plans are used by patients, the Wikibreathe tool is designed for respirologists to make it easier for them to develop well-structured asthma action plans. Thus, the intended users of the Wikibreathe system were respirologists.
Figure 1. Revised prototype of the Wikibreathe tool
Usability Testing Methodology
This section discusses the process used to evaluate the revised version of the Wikibreathe tool using the two usability evaluation methods.
Participant selection
A convenience sample of 12 participants (4 Male, 8 Female; these participants were not the participants used in the focus group) between the ages of 20-40 were recruited from friends and family of the researchers. Their occupations ranged from a high school teacher and music therapists to a programmer and an engineer. They spent an average of 9 hours each week on the Internet at work and 10.4 hours each week on the Internet at home. All participants had a generally positive attitude towards computers. Thus, our participants tended to be well educated and computer savvy, as would be the respirologists who were the target users of the Wikibreathe system. If we were testing the effectiveness of the tool itself, a sample of respirologists would be needed. We were interested, however, in the testing method so we decided that for a case study, the sample we used would be appropriate.
User study process
Participants used the Wikibreathe tool with both a traditional task-based scenario approach and a reverse-engineering method. Participants were asked to use the tool to perform the same task of creating an asthma action plan twice (i.e., in a within-subjects design): once using the traditional task-based method and once using the reverse engineering technique.
During the user study sessions, participants were first introduced to the goal of the project and were told that the purpose of the study was to examine the usability of the Wikibreathe tool for designing asthma action plans. They were then given a brief introduction and training on the Wikibreathe tool. Each participant received the same amount and type of training. Lastly, they were shown three asthma action plan examples that we had created previously using the Wikibreathe tool.
Participants then used the Wikibreathe tool with one of the usability testing methods, then the other. The order of methods was counterbalanced such that six participants did the task-based method first and six did the reverse engineering method first. The following describes the methods used in this case study:
- Task-based testing method (TM): This method is based on the traditional task-based usability testing approach using invented scenarios. Participants were told to imagine that they were a respirologist who was designing an asthma action plan. They were given a set of tasks to perform using the Wikibreathe tool.
- Reverse engineering testing method (REM): In the reverse engineering method, participants were simply given a completed asthma action plan (that was previously created using the Wikibreathe tool which was already developed by our team as part of an earlier project) and asked to recreate it as closely as possible, using the tool.
The process of preparing for the reverse engineering portion of the case study consisted of creating an asthma action plan, using the Wikibreathe tool, that participants could then attempt to duplicate. Thus, once we created an asthma action plan instance to use as the target in the reverse engineering method, the only remaining step was preparing a few straightforward instructions on how to do the reverse engineering task using the Wikibreathe tool (see Appendix A). In contrast, a detailed set of instructions and a moderator script were required as preparation for the task-based test method. A more important challenge in task-based usability testing is that the test administrator not only has to select which tasks to include and in what order but also to carefully word them so that they are clear and unambiguous. Besides, with task scenarios, it is almost impossible to probe all aspects of a product. Therefore, test administrators have to choose what to probe and that may limit their ability to uncover all of the important problems. These challenges are sidestepped in the “make what you see” approach of the reverse engineering method (Appendix A).
Because participants were to use the same tool with each evaluation method, we decided to use two different asthma action plans so as to reduce the learning effect of having to create the same plan twice. We constructed the two plans in such a way that they were different overall, but had similar tasks, so that the results were comparable.
During each user study session, the task completion time and percentage of errors made during each of the testing methods were recorded. Examples of errors included deselecting an option while the option had to be selected or not recognizing the different page layouts (Table 1). Participants were encouraged to verbalize their thoughts while using each method.
After each method, a short questionnaire was administered to the participant. The questionnaire consisted of the System Usability Scale (SUS) questions and a number of open-ended questions inquiring about the participant’s attitude towards the tool (for example, if they felt confident or frustrated while using the tool). The SUS questionnaire used a 5-point Likert scale where 1 was Strongly Disagree and 5 was Strongly Agree (Brooke, 1996).
After both of the methods were completed by the participant, a semi-structured interview was conducted. The interview inquired about the issues raised during the participant’s interaction with the tool. Participants were also asked about their overall impression of the tool.