Lindgaard, G. and Chattratichart, J. (2007) Usability testing: what have we overlooked?. In Proceedings of the SIGCHI conference on Human factors in computing systems (CHI ’07). ACM, New York, NY, USA, pp. 1415-1424.

DOI= 10.1145/1240624.1240839

The studies by Lindgaard and Chattratichart indicate a need to shift the focus from the number of test users to the number of test tasks in usability testing. Lindgaard and Chattratichart analysed the results of several usability teams that had conducted usability tests on the same web service with quite similar settings including same evaluation goals, using thinking-aloud method and reportng the findings in common problem format. The number of reported usability problems was restricted to maximum of 50, but the number of test users and test tasks was no way restricted. Therefore, Lindgaard and Chattratichart set the number of test users and number of test tasks as independent variables in their study, and analysed if these variables correlate with the proportion of problems found and proportion of new problems found. Their results showed no significant correlation between the number of users and proportion of problems nor with new problems found. Yet, the results showed significant correlations between the number of test tasks and proportion of both problems found and new problems found. Another important factor affecting to the quality of test results was the recruitment procedure of test participants: the web service was intended for a variety of users with varying skills and background knowledge, so it was important to recruit a heterogeneous sample of users to cover as many potential problems as possible.

Based on their results, Lindgaard and Chattratichart recommend to invest in wide task coverage and careful participant recruitment instead of large number of test users. The possible affect of limiting the number of reported problems into 50 was not discussed in the article, although the total number of problems from all the reported tests was as high as 176.

Posted by Sirpa

This entry was posted in Journal article and tagged , , . Bookmark the permalink.

Leave a Reply