Resources: UPA 2004 Idea Markets
Remote Moderated Usability
Activator: Mark Safire
Director of User Experience Research, Sachs Insights, NY, NY
We discussed the pros of cons of conducting usability studies using webconferencing tools such as WebEx. These tools enable the moderator to observe respondents' computers from afar, typically while talking to them by phone.
Most of the participants in this discussion came from Fortune 1000 companies, had conducted some usability testing remotely before, thought the experience was a positive one, were interested in doing it again, and were interested in learning more about best practices for this technique.
The basic advantage of this approach was seen to be the savings in cost and time, due primarily to the need for less travel, but also to the possibility of conducting these sessions without a usability lab or research facility.
Other advantages included the ability to perform research that could not feasibility be conducted in-lab. Some studies require the use of participants from multiple, far-flung locations. In other cases, a company may simply have saturated the local population of customers and need to sample from other areas.
There were other advantages to remote moderating without a lab. In particular, a moderator could have observers in the same room without the respondent knowing. This enabled them to pass notes more easily. Also, observers could listen in at other locations more easily, since the webconferencing technology provided for it.
Nonetheless, there were still considered to be advantages of in-lab testing. Certain issues become more difficult when testing is done remotely. The most apparent one is that in a lab the respondent's face can be seen. The moderator can read "body language." The experience of seeing the respondent's face is more impactful for observers.
There are additional challenges to using remote respondents, which need to be accommodated by the moderator when working remotely:
- Labs are set up for video recording. A remote moderator may have to arrange for video recording in some other way. (Note however that a remote interview conducted from a lab facility can use the lab equipment, with some additional configuration.)
- Disclosing proprietary materials is more problematic, since the respondent can more easily make copies of the stimuli.
- Confirming a respondent's identity can be more difficult, since they are not on site with a photo ID.
- Distributing incentives can be challenging, since electronic coupons require more preparation than cash. For international populations this is even more of a challenge, since there is no one site that is popular in all countries.
- If respondents are in other time zones, the moderator may need to conduct interviews after typical working hours (though one moderator pointed out that it is possible to conduct an interview from home, if so desired).
- Signing a non-disclosure agreement requires an electronic signature of some sort, and should ideally be tailored to the laws and practices of the appropriate jurisdiction.
- Since the respondent's computer is used for the study, there is the possibility of harming their computer (or even the possibility that a respondent might make the claim that their computer has been harmed).
- Finally, the respondent needs to be able to install the webconferencing software easily, use it on a compatible software platform, and uninstall it afterwards.
Interviewing employees from ones own company can be done over an Intranet and can avoid some of the issues mentioned above, since the employee is a member of the same company.
Opinions differed about whether respondents were more or less at ease during a remote interview. Some felt that distance makes the experience more awkward; when respondents are in the lab, they may be more at ease because of the in-person contact with the moderator. Others felt that respondents could be more at ease being interviewed remotely because they were more relaxed in their home or office setting, and they were less threatened than they would be by a moderator interviewing them in person.
Approaches to putting a remote respondent at ease included taking longer during the introduction to explain the process and to establish rapport, and making more of an effort to be "chipper" and to avoid a monotonous speech pattern. For respondents who were speaking English as a second language, it was considered essential to speak slowly and clearly, though this would apply to in-lab interviews as well.
One discussion participant pointed out that the "Voice of God" approach may apply more readily to a remote interview, since less in-person dialog is involved.
Some intricacies of adapting the methodology were discussed. When asked about the trade-off between the quality of the screen image and the speed with which the interactivity is transmitted, the moderator pointed out that WebEx has a setting which allows the host to favor one or the other, based on which is more important for the study.
Another nuance involved deciding who should be sharing who's computer during the interview. It was agreed that it was better for the respondent to allow the moderator to see his computer. That way the respondent does not have to experience the loss in screen quality and the delay, which could impact the respondent's experience and bias the test results.
However, in the case of an elaborate or fragile prototype, it may be impossible to install on the respondent's machine. The moderator may need to host the stimuli and allow the respondent to view his machine.
The in-between situation of a "satellite lab" was discussed. This refers to one or more special locations that are set up remotely to facilitate interviews. Respondents can visit a local satellite lab at a facility nearby or in a special room at a company's remote office location. The room could be set up with videoconferencing equipment to allow for the moderator to view and record the respondent's face and screen. This situation shares some of the advantages and disadvantages of both the basic remote interview and the in-lab interview.
Some time was spent discussing webconferencing tools. While most reported using WebEx successfully, one person mentioned using Breeze, a Macromedia Flash-based product, which worked well with stimuli that could be put into Flash. Another person had used Microsoft LiveMeeting (formerly PlaceWare) and liked its ability to integrate recording through a phone bridge, its multiple channels of audio, scalability, and ease of scheduling.
One person had configured a prototype in Flash to log events so that a recording could be made by simply tracking these events and "playing them back" after the fact. This allowed for dramatically smaller recordings, but required a Flash-based prototype.
The free program Microsoft NetMeeting was thought to be a good solution for a local remote setup, from one room in an office building to another, for example, but not a realistic solution for a typical remote interview, since it does not go through routers.
Overall, it was generally agreed that the advantages of remote moderated usability outweighed the disadvantages, though it was not considered to be appropriate to conduct testing this way exclusively.