Dicks discusses on the problems that he has noticed in testing practices and in teaching usability testing. At first, he reminds what characteristics are requited to call some testing a usability test. Dumas&Redish (1994; see references from the original paper) name the following requirements:
- the goal is to improve a product’s usability,
- the participants represent real users,
- the participants do real tasks,
- the testers or evaluators observe and record the participants, and then analyze the data and recommend changes to fix problems
Then he summarizes what most evaluators try to assess while conducting usability tests. A bit surprisingly, the factors are not from the ISO standards but from Gould&Lewis (1985):
“Most usability experts include four aspects of usability, taken from Gould and Lewis (1985), in their conception of the term: easy to learn, useful, easy to use and pleasant to use.”
Then re moves on to showing what sort of problems lie in the usability testing practices:
1. The evaluators do not know what usability means and do not know the difference between usability studies and empirical tests.
– Dicks is worried about the phenomena that people usability test whatever they like – even things that do not have attributes related to usability. As an example, he mentions usability tests on a college course – not on the course web-sites, but on the course itself. Can one really claim that some course is usable?
– asking someones’ opinions on some product is easily called usability testing, although this is just gathering user and usability data instead of “performing empirical, reproducible usability tests”
2. Statistics are not understood: “misusing statistical results, especially from tests performed without large enough user samples”
3. Verification tests are to easily called usability tests although no real users are used as participants
4. The limitations of usability testing are not understood (e.g.
Testing is always an artificial situation.2. Test results do not prove that a product works.3. Participants are rarely fully representative of the target population.4. Testing is not always the best technique to use.