Home

Resources: UPA 2006 Idea Markets

[Usability] Consumer Report: Can we make usability performance data more compelling and understandable?


Activator: Kevin Lee, GE Healthcare

Thought Starter Questions

  • How do you quantify usability attributes? What usability attributes are you able to quantify?
  • How do you report out the quantified data?
  • What are some ways in which you can consistently quantify usability attributes?\
  • What are your organization’s practices for reporting usability data (both qualitative and quantitative)?
  • What effort have you seen in your organization’s approach to reporting results?
  • How do you justify usability initiatives in your organization? (or do you?)
  • How do you track or quantify specific usability attributes (learnability, efficiency, low errors, memorability, satisfaction)?
  • What other usability or product attributes that should be tracked and measured?

Executive Summary

Justifying usability within organizations as well as in the marketplace has been challenging. One of reasons for such challenge was stemmed from the fact that quantifying usability attributes and presenting the data through use of metrics or ratings was considered novel but impractical idea. The intent of this discussion is to explore ways to make usability performance data more compelling and understandable for non-UCD business partners as well as consumers, and that a consumer-reports-style rating system may be a good solution.

 

The following are some of the key discussion summaries from the ideas developed during the session:

  • Data presentation format is critical and has to be carefully tailored to the audience.
  • Adopting a “consumer report” data format for presenting usability data can help make the data more compelling when presented both internally or externally (setting expectations). However, the establishment of baseline measurement criteria is still questionable in terms of consistency of data presentation
  • Efficiency and user satisfaction are easy to measure but need to standardize the measurement criteria to improve the repeatability of data
  • Simplify usability data presentation and voice it in terms of the “pain” users go through (i.e., severity rating based on the observation, likelihood of influencing product perception, safety & hazard relative to users physical and mental state, etc)
  • Need to establish appropriate metrics in advance across different competitors – requires establishment of usability body of knowledge group or product consortium to drive the consistency of usability performance data presentation including templates, methodologies or approaches to obtaining the data, minimum requirements for obtaining the valid performance data, etc
  • “Risk to business or stakeholder” data with respect to poor user performance data must be clearly presented in order to achieve the consistent expectation and understanding from non-UCD business partners as well as customers

Discussion Details

The comments of the participants collected during the Idea Market session are generally presented without editing. Some additional wording has been added to clarify some of these comments were captured for the publication purpose.

  • In general, there are three critical information required in the report of data:
    • High-level category or classification of what were measured (UI workflow or task flow in general, specific UI components, satisfaction, error rates, efficiency relative to task completion, number of keystrokes, time to complete tasks, etc)
    • Specific “target definition” within each high-level category which presents how the data was intended to be captured (lower target description such as 80% of all users complete [task A] with no assistance, etc)
    • Measurement unit description and relationship to overall usability goals
  • Data presentation is critical more than how usability testing is performed or executed
  • Try to simplify data presentation (charts and graphs) by calling out the most “risk to the business or the stakeholder” data
  • Learnability is important type of data but hard to measure
    • One successful way is to repeat the same task at the beginning and at the end of usability testing session
  • Learnability can be derived from conducting multiple trials over time starting with a novice user or someone who rarely use (infrequent users)
  • Performance data can be streamline by observing “lead users” or “early adopters” – extract consistent behaviors and feedback to establish the performance baseline
  • Learnability data reporting will need multiple reference points
  • Call center metrics can be used to describe learnability and other hidden usability related problems that are not necessarily observed during usability testing
  • Reports are usually based on study type
  • Only high confidence data and most critical to quality and risk should be reported
  • In your reporting use the terminology of users and consumers
    • Voice it in terms of the “pain” they are feeling
    • Keep sound bite size
    • Keep movie / photo of “pain” areas
  • Develop standard format (like consumer report) so people expect know what to expec
  • CIF (Common Industry Format) for usability test report is NOT conducive for “telling the story” of data. Successful storytelling technique using consistent and concise industry format is necessary
  • Ethical considerations will become important if usability data is developed and published “consumer reports” style – how to avoid appearance of impropriety as if endorsing a product
  • A big challenge is how to link product performance improvements to usability changes as a result of overall product integration and ROI
  • For ROI purposes – work with management and quality to agree on UCD metrics for a given project
  • Establish cross-functional core-team to establish consistent expectation and appreciation of usability performance metrics

Conclusions

Justifying usability within organizations as well as in the marketplace has been challenging. One of reasons for such challenge was stemmed from the fact that quantifying usability attributes and presenting the data through use of metrics or ratings was considered novel but impractical idea. And exploring ways to make usability performance data more consistent and repeatable by adopting and establishing a consumer-reports-style rating system has been accepted as a good solution at large.

However, there are several pre-requisite arrangements or considerations must be carefully established amongst practitioners as well as product manufacturers:

  • Simplify usability data presentation and voice it in terms of the “pain” users go through (i.e., severity rating based on the observation, likelihood of influencing product perception, safety & hazard relative to users physical and mental state, etc)
  • Need to establish appropriate metrics in advance across different competitors – requires establishment of usability body of knowledge group or product consortium to drive the consistency of usability performance data presentation including templates, methodologies or approaches to obtaining the data, minimum requirements for obtaining the valid performance data, etc
  • “Risk to business or stakeholder” data with respect to poor user performance data must be clearly presented in order to achieve the consistent expectation and understanding from non-UCD business partners as well as customers

Driving consistent usability data presentation across similar product categories (B2B websites, B2C websites, medical software / hardware UI, consumer office software UI, financial UI, etc) should be carefully but urgently considered by establishing a consortium such as product usability body of knowledge or product quality council, etc. This establishment will break the silo of each product manufacturers culture around conducting usability testing which in return, brings consistency in usability practice and standardize languages, methods, and process. At the end, it is necessary to help our end users to have more clear understanding and appreciation of what usability (a.k.a., ease-of-use) contributes to overall perception and user experience of products.

Usability Resources UPA Store UPA Chapters UPA Projects UPA Publications Conferences and Events Membership and Directories About UPA