The U P A voice
April 2005 Contents

Featured UPA Links:

Job Bank

Member Benefits:
Discounts and Special Offers:

Event Discounts
Order your usability books now from the link from this site. Part of all sales help fund UPA.

Why Listening to Users Can Damage Your Website

by David Unsworth
Saga Services Ltd.
Hythe, Kent United Kingdom

The first time I noticed that people tend to say one thing and do another in a usability test was back in 2000. We had been building a new company website and testing it with real users brought us an unexpected problem. All the users liked the new design a lot more than the old one, but nobody could work out how to use it.

Over the course of the following year we repeatedly encountered the same type of problem. In one study where users performed a task repeatedly on 5 different websites, the majority reported they favoured one because they completed the task more quickly. Unfortunately, the timings we took indicated completion times for this site was the slowest of all five.

After looking at the usability literature, it became clear this was not an uncommon experience. In the evaluation of computer interfaces there is a history of mismatch between user’s success with an interface and their verbal assessment. In the field of commercial Web Site Usability, this mismatch is critical, as poor usability can alienate potential users and divert them to competitor sites.

Nielsen and Levy (1994), for example, admit that users may not always choose optimally, while Dillon and Morris (n.d.) quote a study by Bailey, claiming that “when choosing between design alternatives, users will rarely prefer the interface that bests supports their own performance”, an effect also found by Toub (1999), Walker et al. (1998) and Spool (1999), who found users preference ratings for web sites were almost in reverse order to their success. No wonder Nielsen in 2001 advised in his Alertbox "pay attention to what users do, not what they say. Self-reported claims are unreliable, as are user speculations about future behaviour."

Since Nisbett and Wilson’s influential paper on verbal reporting of cognitive processes in 1977, there has been serious doubt about whether a person can be relied upon to know why they took a particular action, what caused an error or confusion, or even whether a process was easy or hard for them. In their study, they sought to evaluate the view that “we may have no direct access to higher order mental processes such as those involved in evaluation, judgment, problem-solving and the initiation of behaviour”. To do so, they evaluated research from the fields of dissonance and attribution theory, and in a number of studies they found evidence to support the view that subjects were not only unaware of critical stimuli that altered their response, but also in some cases were unaware of any influence taking place.

In addition, they looked at work on learning without awareness, literature on subject abilities to accurately report judgments on weights, Maier’s work from 1931 on awareness of stimuli influencing problem solving, as well as conducting a number of experiments to ‘fill in’ some gaps in the experimental literature, exploring the ‘bystander effect’, ordering effects on purchase selection and others. Repeatedly, they found participants not only unable to correctly identify the critical stimuli influencing their responses, but also often either wrongly identifying other cues as critical influences, or denying any external influencing factor at all.

Nisbett and Wilson conclude that the “accuracy of subject reports about higher order mental processes may be very low”, proposing instead that, when asked for explanations of what stimuli caused this or that response, subjects employ a priori causal theories to evaluate the likely reason. This view is consistent with a large body of research that suggests a dominant role for unconscious processes in the initiation of behaviour, in the learning process, and in the processes of perception and cognition.

In fact it is this tendency to fabricate in the absence of knowledge that is at the heart of the mismatch problem, and one of critical significance to the evaluation of HCI, as there would seem to be a strong case against the use of verbal protocols either during or after a test for identifying what is happening or has happened cognitively during an interaction. As Dennett (1993) observes, “the cosy complicity of the resulting first-person plural perspective is a treacherous incubator of errors.”

The problems for usability testing are all too clear. Any approach is flawed if entrenched in the assumption that the user is consciously aware of their decision-making strategies and that we can discover the rationale either by questionnaire, interview or observation. All of which leaves us with the question of how to study usability without resorting to the hetero-phenomenology of the user's own perception, and, in pursuit of this, only approaches based on observation of actual user behaviour appear to offer hope.

  Usability Professionals' Association
140 N. Bloomingdale Road
Bloomingdale, IL 60108-1017
Tel: +1.630.980.4997
Fax: +1.630.351.8490
Contact the Voice