Dicks, R.S. (2002) Mis-usability: on the uses and misuses of usability testing. In Proceedings of the 20th annual international conference on Computer documentation (SIGDOC ’02). pp. 26-30

DOI=10.1145/584955.584960 http://doi.acm.org/10.1145/584955.584960

Dicks discusses on the problems that he has noticed in testing practices and in teaching usability testing. At first, he reminds what characteristics are requited to call some testing a usability test. Dumas&Redish (1994; see references from the original paper) name the following requirements:

  1. the goal is to improve a product’s usability,
  2. the participants represent real users,
  3. the participants do real tasks,
  4. the testers or evaluators observe and record the participants, and then analyze the data and recommend changes to fix problems

Then he summarizes what most evaluators try to assess while conducting usability tests. A bit surprisingly, the factors are not from the ISO standards but from Gould&Lewis (1985):

“Most usability experts include four aspects of usability, taken from Gould and Lewis (1985), in their conception of the term: easy to learn, useful, easy to use and pleasant to use.”

Then re moves on to showing what sort of problems lie in the usability testing practices:

1. The evaluators do not know what usability means and do not know the difference between usability studies and empirical tests.

– Dicks is worried about the phenomena that people usability test whatever they like – even things that do not have attributes related to usability. As an example, he mentions usability tests on a college course – not on the course web-sites, but on the course itself. Can one really claim that some course is usable?

– asking someones’ opinions on some product is easily called usability testing, although this is just gathering user and usability data instead of  “performing empirical, reproducible usability tests”

2. Statistics are not understood: “misusing statistical results, especially from tests performed without large enough user samples”

3. Verification tests are to easily called usability tests although no real users are used as participants

4. The limitations of usability testing are not understood (e.g.

1. Testing is always an artificial situation.
2. Test results do not prove that a product works.
3. Participants are rarely fully representative of the target population.
4. Testing is not always the best technique to use.)
5. Usefulness is too easily forgotten in testing, and assumed to follow ease of use.
Dicks is quite understandably worried about the use of the term “usability testing” and the inflation it will have in the eyes of the ones doing business decisions if used with such varied meanings.

Testing is always an artificial situation.2. Test results do not prove that a product works.3. Participants are rarely fully representative of the target population.4. Testing is not always the best technique to use.