Resources: UPA 2004 Idea Markets
How many users with disabilities should you include when conducting a usability test for accessibility?
Activator: Mary Martinson Grossnickle
To answer the question, we first discussed whether you would use the four typical categories of disabilities (visual, auditory, physical, cognitive) to define participant characteristics when selecting users with disabilities for a usability test? The consensus of the group was that your first task in addressing this issue would be to determine the percentage of your target audience in each category. If your target audience includes a higher percentage of people with a certain disability, focus on the relevant characteristics.
For example, a usability test for a product that is marketed primarily to seniors should include seniors who have age-related disabilities in the usability test; and people with visual disabilities should be included in a test for a web site containing information on diabetes.
But how do you determine whether users with disabilities are party of your target audience? How would you know if a person with a physical disability, such as lack of fine motor control, would use your site? Although market research can give some sense of your audience, most people felt that there was no definitive answer to this question. The conclusion we reached was that you should include people with disabilities in user group profiles and personas to ensure that you incorporate accessibility issues, even if your market research doesn't show people with specific disabilities in your target audience.
A major concern is that there is vast variability within each category, both in type of disability and in severity. For example, a person who is blind and a person with low vision have very different needs. Some people have a lifelong disability and some people are newly disabled, or temporarily disabled. To add even more variability into the mix, some people have multiple disabilities across more than one category.
- The key considerations when determining how many users with disabilities to include in a
usability test are:
- Match user profiles
- Decide on the range of disabilities to include
- One option would be to include different types of disabilities in each usability test. As your product design goes through several iterations of usability testing, you can include participants with different characteristics in each round of testing. This approach is more manageable, in terms of time and resources than trying to fit participants representing all disability characteristics into one test.
- Recruiting and screening were major concerns. Our discussion included the following points:
- A usability professional from AARP noted that when she does usability testing with seniors, she doesn't need to include questions about disabilities in her recruiting screener. Her participants who are aging often have one or more age-related disabilities in each of the four categories, so she doesn't need to screen specifically for a disability.
- People with temporary disabilities, such as carpal tunnel surgery, can be recruited to do some testing for you.
- People with visual disabilities seem to be the largest population of users with disabilities. It is easiest to recruit people who are blind or have low vision. The danger is that we design strictly for this disability, forgetting about hearing, motor and cognitive impairments.
- One strategy to find usability test participants is to develop an ongoing relationship with local senior centers, disability groups, schools that have programs for students who are disabled, etc.
- Some people mentioned using agencies that specifically recruit people with disabilities.
- A Yahoo group called UVIP consists of people with visual disabilities who will do web site testing for developers.
- Screen reader technology has improved significantly, and that raises a concern that the client might say, "Oh, we don't need to worry about [whatever] because the screen reader can handle that for the user." However, a usability professional who has conducted a large study using participants with disabilities noted that her participants used the default settings in the screen reader. So even though functions are available with screen reader technology, users are not using those functions if it means they need to change the default settings. Depending on the assistive technology to give the user what he or she needs is not an option.
- We discussed whether visual disabilities are easiest to test for. Discussion included these points:
- Yes, because the issues are clear: the web site needs to work with a screen reader.
- No, because you need to purchase and learn how to use assistive technology, such as a screen reader and screen magnifier.
- Also with the wide range of visual disabilities, one solution does not fit all.
- Perhaps auditory easiest to test because you can provide a transcript, which is relatively easy to produce, at least for an informational site
- Most of the discussion participants voiced concerns about budget, and that the number of participants with disabilities would be limited by the extra amount of money it would take to expand the test to include them. A discussion participant from the BBC in London found that it didn't cost much more to include a person with a disability compared to the cost for a person without a disability. Another discussion participant said that they used employees with disabilities to test the design throughout the design process, and so did not incur any expenses for those participants.
- When we came back to the original question, how many people with disabilities should you include in a usability
test, the answers ranged from one to fifty.
- Only one would be needed if there's a major problem, because you'd find it with one person.
- Fifty in each category would be ideal so that we could generate some data about how many users we really need.
- A more realistic number would be 3-4 people with disabilities in each test, running more than one test, and including people with different types of disabilities in each test.
- Other points made in the discussion included:
- Being compliant with standards doesn't necessarily mean the product is usable.
- Usability and accessibility go hand-in-hand, but are not the same.
- Perhaps we should focus on auditory design; that is, we could approach design from an auditory standpoint rather than always thinking in visual terms.