Home
[an error occurred while processing this directive]

Resources: UPA 2005 Idea Markets

Which methodologies are most effective when conducting research on handhelds and why?


Activator: Kim Oslob, Visto Corporation


The Activator's Initial Questions

There is little information available on the best methodologies to use when conducting research on handhelds. Traditional methodologies don’t always map to handheld testing needs. The goal of this discussion is to provide information on which methodologies work best, and how to tailor existing methodologies for handheld research.

The following starter questions were presented to attendees:

  • What methodologies have you used when conducting research on handhelds?
  • Which methodologies have you found the most effective and why?
  • What modifications have you made to existing methodologies to tailor them for handheld research?
  • What new handheld methodologies have you developed and how are they used?
  • What issues have you come across when conducting handheld field research and how have you handled them?

Attendees came from a variety of backgrounds to include the software industry, libraries, students, consulting firms, and mobile manufacturing companies. They provided the following information on the types of methodologies that have worked best when testing handhelds and which methodologies have been challenging.

Successful Methodologies

Field Research

  • Goal Based
    • Where do participants use their devices?
    • How do participants use their devices?
    • What types of tasks do participants use their devices for?
    • How do participants use your product on their device?
    • How do participants use their device?
  • In their natural environment
    • Elevators
    • Stairs
    • Walking down the street
    • In a café or coffee house
    • On the train
    • Airports
    • Shopping Malls

Lab Task Based Studies – When using the right equipment

  • Baseline Research
    • Discover what the average is for participants to complete a common task.
  • Qualitative Research
    • There is much to be gained by asking participants to think out loud when using handhelds. Unlike desktop software, participants use their mobile devices and software in many ways that are unique or outside of the box.

Focus Groups

  • Determine how users respond to a new product or design.

Journal Studies

  • Captures issues on a day to day basis for mobile users who typically use their device intermittently rather than for long stretches.
  • New Device and/or New Software Users
    • Determine how new users interact with a new device or new handheld software.
    • Discover what issues new users have when configuring a new device, learning a new device, and/or learning new device software.
    • Discover how new user use a device or device software that may deviate from how the development team expected them to use it.

Expert Reviews

  • Use good UI Principals and Standards for handhelds (specific to the OS and device) to determine issues with the device and/or device software.

Modifications

  • Field Studies/Work Observations are taken out of the office and home into the real world such as in coffee shops, parks, airports, shopping malls, etc.
  • When testing prototypes of software make them html based with interactive screens that can be downloaded to a device for a more realistic and interactive experience.

Methodology Issues

  • Prototype studies are difficult for handheld studies due to participants not being able to make the leap from a paper or html mockup to a device. You do not experience the interaction of both the actual device usage with software usage.
  • People use handhelds intermittently, not for long periods of time so traditional Field Studies/Work Observations do not work well in yielding true data.

In conclusion, a majority of the participants at this session have used a limited amount of methodologies and techniques when testing handhelds. There is very little information and resources on new or modified methodologies created for this space. Usability Practitioners in the mobile space are very limited in their resources on what works and what doesn’t. As this relatively new type of research becomes more common, we hope to see further information published and presented on researching handheld devices.

Usability Resources UPA Store UPA Chapters UPA Projects UPA Publications Conferences and Events Membership and Directories About UPA