upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

RITE+Krug: A Combination of Usability Test Methods for Agile Design

Jennifer (Jen) McGinn and Ana Ramírez Chang

Journal of Usability Studies, Volume 8, Issue 3, May 2013, pp. 61 - 68

Article Contents


Our RITE+Krug Combination

Because the RITE method hadn’t been used before with this team, there were challenges and opportunities. The potential challenge would be to educate development and product management stakeholders about what the new method would entail and to get their buy-in for providing the necessary resources to make it work. We were fortunate that this team had no prior experience with the RITE method, so we could adapt it to include elements of the Krug method to better meet their needs and goals as an Agile team.

The next two sections discuss how we gained support for the RITE+Krug combination of methods. Then “The Method” section describes how we adapted, and currently use, the RITE and Krug methods for our usability testing.

Goals

We needed a method that was transparent and engendered trust with our stakeholders because that was a key issue with the first round of traditional usability testing. Other goals and constraints that this method needed to accommodate were to

Stakeholder Buy-In

The RITE method requires that stakeholders must observe the test sessions and participate in the discussions about what to fix and how to fix it (Medlock et al., 2005; Medlock et al., 2002). Krug suggests that stakeholders should observe at least one session to be allowed to attend the debrief meeting (Krug, 2010). We went a bit further and said that we would require attendance by key stakeholders at each session as well as at the debrief meeting. Because we were asking for management to commit the time of key people from product management, development, and design, our stakeholders had to approve and agree to the level of participation we were requiring.

We held a meeting with our stakeholders, including the VP of development, the design manager, and the product manager, to propose the new combination of methods. To be honest, we just called it RITE testing so we could focus on getting their buy-in and not get stuck debating the details of the method. Our goals were to get these stakeholders’ approval to move forward with the new “method” and to obtain a list of individuals who would be invited to each usability test session and debrief meeting. The conversation went surprisingly well. We described the method (see “The Method” section) and what we needed from the team. In the meeting, we proposed that we would give them as much or as little control as they wanted with task development, script review, recruitment requirements, and solution definition. In return, we needed key stakeholders to attend all sessions. If they couldn’t attend, they would need to send a proxy with decision-making authority. One way we got buy-in on the process was by noting that their presence in the sessions also gave them the ability to ask the participants questions. The VP of development approved the proposal, and the development and product management representatives were identified by the stakeholders.

The Method

Our RITE+Krug combination contains elements of both RITE and Krug methods, with a few adaptations of our own. For example, neither RITE nor Krug require testing in a lab environment. While our combination of RITE+Krug doesn’t either, we conduct our sessions in a traditional usability test lab, we record the sessions, and we use a think-aloud protocol as participants complete the tasks.

Like the Krug method, we began by testing three to four participants in a single day and held the debrief session an hour or two after the final participant. Both the RITE and Krug methods rely on research that small numbers of participants can uncover the most common problems for a given set of tasks, participants, and evaluators (Krug, 2006; Krug, 2010; Medlock et al., 2005; Medlock et al., 2002).

Like the RITE method, we made changes to the designs between participants if we found a usability problem. Changing the design after only one participant can be risky because the one participant could be an outlier; however, only design issues that the team thinks are not unique to the one participant are changed. Also like the RITE method, we require key stakeholders to attend each session (Medlock et al., 2005; Medlock et al., 2002). Any other observers are welcome, but we need at least one person from each department—development, product management, and design—to observe in real time.

Like the Krug method, rather than spending a week or two creating a long written report, the findings take more lightweight forms. As soon as possible after the final session (usually within 30 minutes), the user researcher emails a list of observations to the stakeholders. Everyone has the opportunity to ensure that any additional input or feedback of theirs is incorporated before the debrief meeting, where we go over the observations and determine what actions to take as a result (see the “Debrief Meeting and Results” section). A smaller set of people are invited to the debrief meeting than to the sessions, but we require that the attendees observe all of the test sessions.

Session Details

For each participant session, we set up a conference call and a web conference, and then send out a meeting invitation. Then after the meeting we post a link to a recording of each participant’s session. While not a requirement of our method, a huge side benefit of using tools to accommodate a team that is distributed around the globe is inclusion. We can recruit participants globally, not just from the geography where the researcher is located.

The use of low-fidelity wireframes in a slide deck and a distributed conference also allows for a wide audience of stakeholders and team members to observe the sessions. We only require three stakeholders to observe the session, but word has spread and we’ve recently had more than a dozen observers in each of the test sessions—something we’ve never achieved in local tests. We believe the large number of observers increases the chance that a recommended change will be agreed upon and implemented.

Participant Recruiting and Scheduling

While Krug (2006) promotes the notion of recruiting loosely and grading on a curve, we do not. We use the same rigor in recruiting participants for the RITE+Krug tests as we do for any other usability activity. The participants must meet the recruiting criteria for experience, job role, and product usage to participate in the tests. However, to increase recruiting efficiency, we do allow participants to test another part of the product in a subsequent RITE+Krug test if the new part of the system was not included in their first session and if that earlier interaction doesn’t create a bias for what we are testing in the subsequent session. As a result, we do not use team members to emulate users like some Agile teams (Sy, 2008).

Because our team is distributed across time zones, we schedule sessions to try to accommodate those differences. Inspired by Krug, we initially scheduled three 1-hour sessions with 30 minutes in between and the debrief meeting 1 hour after the third participant. That was convenient, but produced two problems. First, 30 minutes between sessions was often not enough time for the designer to create a solution for a problem and update the prototype. Second, if one participant had to reschedule, then we needed to run the session the following day and move the debrief meeting, which wreaked havoc with stakeholder schedules. As a result, we tend to recruit four participants (Krug, 2006; Krug, 2010; Nielsen, 2000). Three participants complete the test on day one, the fourth participant completes the test on day two, and then we hold the debrief meeting on day two with the stakeholders. This two-day format allows us to schedule at least an hour between participants to make design changes and to avoid rescheduling the debrief meeting if one participant can’t make the session. If we lose one participant, we still have our minimum of three or have time to reschedule a participant from the first day to the second.

Prototypes

We use both working code as well as low-fidelity drawings in our studies, depending on the state of the feature we want to study. The Agile methodology recommends short sprints where a feature is implemented over a series of sprints, depending on where it is added to the backlog (Schwaber & Beedle, 2002). If a feature has been implemented and refinement of the feature is coming up in the backlog, we test the current implementation of the feature to inform the refinement step.

When the team is working on a new feature, we test low-fidelity drawings in PowerPoint, essentially paper prototypes as the RITE method advocates. PowerPoint is great for testing low-fidelity or high-fidelity designs, it allows us to include navigational hot-spots, and it works far better than paper over a web conference. It’s also easy to change between test sessions and requires no development skills whatsoever, so no work is required by the development team to make changes to the designs.

Debrief Meeting and Results

The debrief meeting happens as soon as possible after the final session concludes—usually an hour later. At the meeting, we walk through the observations and discuss changes that need to be made. After the meeting, the user researcher updates the wiki page for the test with the list of observations and agreed-upon actions that will be taken, and then sends out a link to the rest of the extended team.

Because a RITE+Krug test takes less recruiting effort and only two days to conduct and report, we can run several RITE+Krug tests in the time it normally takes to run a single, traditional test. As a result, we can test more new features earlier in the design cycle and can make more changes faster, because the code is either scheduled to be changed or hasn’t been written yet. Then later, once the features have been fully developed, we still have time to run a full baseline test so we can validate fixes.

How This Works with Our Agile Development Process

Our development, design, and product management teams use a modified Agile development methodology (Schwaber & Beedle, 2002). We have 4-month iterations made up of six 3 week sprints. Four of the sprints are designated for feature implementation and two are designated for bug fixes and stability. Prior to using the RITE+Krug combination, the user research process and results had been divorced from the Agile processes, which resulted in the findings coming too late to be acted on. Because of this issue, we integrated the user research with the rest of the process, as illustrated in Table 1.

The team consists of developers, product managers, user experience designers, visual designers, quality assurance engineers, and a user researcher. The user experience designers and product managers work closely with the development team during the feature sprints—answering questions, giving feedback on progress, and fine tuning the feature as it is implemented. The bug fix sprints give the developers time to focus on product stability.

Meanwhile the product managers, user experience designers, visual designers, and the user researcher work on preparing the small set of features that will be implemented in the next iteration (see Table 1). This work includes feature selection, design, user testing, and redesigns. The whole team (including developers) gives feedback on the feature specification and design before it is ready to be implemented. Like others (McInerny, 2005; Nielsen, 2008; Sy, 2007), our design team stays an iteration ahead of the development team. Like Patton (2008, August) recommends, we iterate the UI before it ever reaches development, thereby turning what is traditionally a validation process into a design process (2008, May).

Table 1. Schedule of Design and Development Activities.

Table 1

 

Previous | Next