Good usability testing is similar to good empirical research: the use of improper procedures will result in invalid data, and thereby poor validity and reliability. Holleran categorises pitfalls in usability testing into three groups: sampling problems mainly in planning the tests, methodological problems in conducting the test sessions, and problems in interpreting the results. Sampling problems refer to the selection of subjects and test tasks, their number and representativeness, and also to the poor reporting of use characteristics as well as poor analyses of possible correlations between these characteristics and test results.
Methodological problems, on the other hand, cover issues related to having human test participants, such as motivation and moderator bias. He recommends to be cautious in analysing test users’ performance as test users may put more attempt to the tasks than normally. The moderators in the tests also easily affect the users’ performance, as well as the thinking aloud method if used in the test. All these factors affect to the validity of usability testing as it is not so clear what is actually measured and evaluated, and also to the reliability, as the results may be quite different although the same procedure was repeated.
The last pitfalls that Holleran mentions lie in the interpretation of the test results. He recommends to use quantitative measures, such as content analysis and observational coding procedures that allow also statistical analyses to avoid purely subjective interpretations of qualitative data.