The Effect of Experience on System Usability Scale Ratings
Sam McLellan, Andrew Muddimer, and S. Camille Peres
Journal of Usability Studies, Volume 7, Issue 2, February 2012, pp. 56 - 67
Abstract
Longitudinal studies have to do with testing over time and thus take into consideration previous user experience with a product or product versions. However, it is difficult to conduct these types of studies. Therefore the literature is sparse on examples of the explicit effect of user experience on user satisfaction metrics in industry-standard survey instruments. During a development experience in 2009, we used a cross-sectional method to look at the effects of user profiles on ratings for commercial products that use one such instrument, the System Usability Scale or SUS.
Recent research has reported finding that differences in user ratings could be based on the extent of a user’s prior experience with the computer system, a Web site being visited or a desktop application like Microsoft’s Office suite being used. Compared to off-the-shelf office products or personal Web applications, we were curious if we would find the same experience effect for domain specialists using geosciences products in the course of their daily professional job roles. In fact, from data collected with 262 end users across different geographic locations testing two related oilfield product releases, one Web-based and one desktop-based, we found results that were quite close to early assessment studies: Users having a more extensive experience with a product tended to provide higher, more favorable, SUS scores over users with either no or limited experience with a product—and by as much as 15-16%, regardless of the domain product type. This and other observations found during our product testing have led us to offer some practical how-to's to our internal product analysts responsible for managing product test cycles, administering instruments like the SUS to users, and reporting results to development teams.
Practitioner’s Take Away
Given a potential relationship between such factors as user experience and SUS ratings, we provided the following instructions to our internal company practitioners using SUS as a measure of usage satisfaction. We believe these same recommendations would be useful for anyone using the SUS with domain products.
- Ask users for their level of experience with the domain product being evaluated with the SUS.
- Regularly inspect the literature for other demographics (or survey changes) that might be useful to incorporate formally across their product centers using SUS as part of user test instruments.
- Report SUS results from the intended user community for a product.
- Be explicit to users about directions for correctly filling out the SUS.
- Be explicit to administrators about how to correctly administer the SUS to users.
- Be explicit to product teams about directions for appropriately reading the results of the SUS.
Article Contents
Download Article
The Effect of Experience on System Usability Scale Ratings