Russo et al. listed several factors that could damage the validity of verbal protocols, and made experimental studies to show possible conflicts in Ericsson and Simon’s thinking-aloud model. They name two forms of protocol invalidity: reactivity and nonveridicality. Reactivity means that the protocol changes the the primary process, and nonveridicality means that the protocol does not reflect the underlying primary process accurately. If a protocol is reactive, veridicality is of little interest, so Russo et al. concentrate in reactivity in their studies.
Russo et al. made studies with 24 students that performed 4 different kinds of tasks using 4 different verbalization protocols. The protocols included a silent performance, concurrent thinking aloud (CTA), and a retrospective condition with a replay of eye fixations as an assistance (prompted). The fourth condition varied so that half of the subjects recalled their thoughts in retrospective condition on the basis of the given problem (response-cued) or on the basis of their problem solutions (stimulus-cued) right after each task.
The results of Russo et al. studies show significant reactivity in some tasks, since the accuracy of performance varied from better to worse performance depending on the task when it was compared to the accuracy in silent protocol. For example, the concurrent thinking aloud protocol significantly improved the accuracy in multiplying decimal numbers, and significantly decreased the accuracy of adding three three-digit numbers. Also the response time varied, but this was somewhat expected on the basis of the Ericsson and Simon model.
Russo et al. name some potential causes of reactivity: increased workload due to verbalizing, auditory feedback, enhanced learning due to repetition, and a motivational shift due to monitoring. Auditory feedback refers to the effect in which hearing an auditory stimulus facilitates recalling. Protocols and repetition give subjects an opportunity to reflect on their performance, and thereby enhance their learning. The motivational shift refers to the motivation that the subjects generate as they know their performance is observed and analysed. Therefore, they may want to minimise their errors and put more effort to their performance than usual. Even in retrospective protocols, the motivational shift is possible, as well as enhanced learning. Russo et al. also suggest that retrospective protocol may increase the memory load by pressuring the subjects to remember something worth explaining.