Boren, T., Ramey, J. (2000) Thinking aloud: reconciling theory and practice. IEEE Transactions on Professional Communication. Vol. 43, No. 3, pp. 261-278.
Boren and Ramey have their backgrounds on technical communication and are actively involved in usability testing. As technical communicators they criticise the way thinking aloud is applied in usability testing. Ted Boren observed over 20 usability specialists in 2 software companies, and reported the findings in his doctoral thesis. The results include the following:
– The instructions to think aloud vary considerably. Most of the instructions did not include a description of the difference between thinking aloud and explanation, and some even requested for explanations, although Ericsson & Simon (1980) claim that explanations may alter the subjects’ performance.
– Usability practitioners do not prompt the subjects in the way Ericsson and Simon instruct as the subjects fall silent. E&S recommend just to say: “Keep talking”, whereas usability practitioners easily say something that redirects subject’s attention and interrupts the work flow.
– Usability practitioners often intervene the work flow in theoretically inconsistent way. As usability practitioners want to know more about some problems the subjects face in using the evaluated system, they ask more precise questions, although this is not allowed in E&S procedure. Specific questions may give valuable information to the development, but according to the underlying thinking aloud theory, it may also alter the way subjects continue with the current task and perform succeeding tasks.
– Usability practitioners do not use verbalizations as data or do systematic protocol analysis. Instead, they value the evaluative and explanatory comments that are usually parts of Level 3 thinking aloud, and thereby not very reliable information according to the E&S theoretical base.
Boren and Ramey point out several situations in which the usability practice violates the principles of E&S thinking aloud theory. They suggest some improvements on the basis of speech communication theories, but some problems are left for further research. For the ones that are planning to do usability tests, the list of problems and suggested improvements is well worth reading. In addition to giving food for thoughts in usability testing, the most important part for me in this article was summarized in the end: “We hope usability professionals will strive for a practice that is theoretically informed in deed, not merely in word“, and a bit earlier in the article: “If usability testing is to be considered a discipline, the methods employed by its practitioners must be theoreticaly motivated and systematically applied.” If we want usability engineering to be a convincing, reliable and valid discipline we need to be informed how the theories forming the basis for our methods support the ways we apply the methods.