Kato, T. (1986) What “question-asking protocols” can say about the user interface. International Journal of Man-Machine Studies, Vol. 25, No. 6, pp. 659-673.

DOI= 10.1016/S0020-7373(86)80080-3

Takashi Kato presents in his article (1986) a question-asking protocol which he considers as a more natural way of behaviour than thinking aloud. In question-asking protocol, the test users are encouraged to ask questions from a tutor that sits near the test user in the test setting. In addition to the tutor, a test experimenter is present in the test situation and prompts both the test user and the tutor if needed.

The lists of asked questions give valuable data to the developers of the system on what parts should be clarified and what to include in the manual. Kato and his colleagues used the protocol in testing new systems with novice users who had to learn to use the system on their own without a manual. The test tasks were deliberately chosen to be so challenging that the users would most likely not be able to perform the tasks without the help of the tutor. As weaknesses of the protocol, Kato names inappropriateness in assessing manuals and in making quantitative analyses.

In my humble opinion, the method does not seem very fruitful in evaluating interactive systems with lots of visual data. The users might look for clues silently for a long time, and the tutor and the experimenter could only guess what they are looking for. If the users, on the other hand, asked the questions before even trying to look for an answer themselves, the intuitiveness of the user interface leaves without assessment.