Ericsson, K.A. & Simon, H.A. (1980) Verbal reports as data. Psychological Review, Vol. 87, no. 3, pp. 215-251.
Ericsson and Simon aimed to show in their article that verbal reports are data and can offer valuable information about the cognitive processes going on in people’s heads. They admit that the verbal information that people give on their actions is not always complete, but still adequate to give plausible guidelines on their cognitive processes.
For a researcher on usability and user testing, Ericsson&Simon’s article gives interesting thoughts on how to instruct test users to think-aloud, and what sort of information is to be expected and, furthermore, likely to be left out from the verbal reports.
Ericsson and Simon categorises verbalization into three levels:
1. When the information is told in the same way as it was processed in the short-term (or working) memory, it is called direct i.e. Level 1 verbalization.
2. In the Level 2 verbalization, the original information is not in verbal form, e.g. an image, and it has to be explained or translated into verbal form.
3. In Level 3 verbalization, the subject (or participant) is asked to do something more than just to tell one’s thoughts aloud. The subject may be asked to filter information from the contents according to given instructions (e.g. select most appropriate alternative), or to generate new principles or infer something with possibly incomplete information. If the subjects are asked to describe their motor activities that they would not otherwise pay attention to (e.g. routine actions), the verbalization falls into Level 3.
In addition to categorising the verbalization according to the form of stored information, Ericsson and Simon categorise it also on the basis of the form of probing:
1. The subjects are asked to think-aloud, i.e. to articulate their thoughts at the same time they are processing the information.
2. The subjects may perform their tasks silently, but the experimenter probes concurrently with the performance for some specific information. For this procedure, Ericsson and Simon advice to use general probes instead of specific ones if one is interested in information that is likely to still be in the subject’s short-term memory.
3. In the third class of verbalization procedures, the information is asked only after the subject has completed the task, i.e. retrospectively. If the subjects perform a number of tasks before the probing, Ericsson and Simon call it interpretive probing, and are skeptical on the quality and accuracy of the subjects’ memory of their cognitive processes during the tasks.
Ericsson and Simon address in their article some of the effects verbalization may have on the subjects’ performance and on the quality of results of verbalization studies. Here is a list of some of those effects (some of them relate to other studies; see references in the original article):
– Especially on the Level 3 kind of verbalization, the subjects may alter their normal behaviour to be more efficient in the following tasks, e.g. concentrate on some other information than usually.
– If the subject is asked to select an answer from ready-made alternatives, the alternatives may not include the one that came to the subject’s mind and therefore the subject has to choose the most appropriate and possibly change the actual thought process.
– Highly practiced and automated processes are not verbalized in thinking-aloud studies, since the actions do not require the use of the short-term memory.
– The Level 1 thinking aloud does not affect the course and structure of the cognitive processes.
– The level 2 thinking aloud (not explanation) does not change the way subjects perform the tasks, but may slow down the performance.
– If the subject is asked to explain the reasons for doing something, the effects on performance are predicted to be more substantial.
– Transfer: if the subjects are asked to verbalize the reasons for certain behaviour or solution, they are more efficient in solving similar tasks than those subjects that have acted silently.
– Subjects easily stop verbalizing as the cognitive load gets high.
– Information that leads directly to the solution without intermediate processing is not usually verbalized (Duncker 1945) and not even remembered when asked retrospectively (Maier 1931).
As a brief conclusion, Ericsson and Simon state: “In this article we have undertaken to show that verbal reports, elicited with care and interpreted with full understanding of the circumstances under which they were obtained, are a valuable and thoroughly reliable source of information about cognitive processes.”