Kjeldskov et al. decided to test if the analysis phase of usability testing could be cut down and thereby cut the costs of testing. They utilized the resources already used in testing, i.e. the moderator and a note taker, and reserved one more hour for them after the tests (all 4-6 tests in one day) to make a summary of their findings. A specific facilitator joined the meeting and made a summary of the discussion and most important findings after the meeting. All this took 3x1h (moderator, note taker and facilitator) plus 1h for the facilitator in wrapping up the findings, and validating the results with the moderator and the note taker. That is, 4 h in total.
The results were compared with a group doing traditional videotape analysis and spending 40 hours in the analysis. In total, these two analysis methods revealed 13 critical problems and 11 were found with the instant analysis and 12 with traditional videotape analysis. With serious problems, the division was surprising, because both methods revealed 15 problems, but there was less overlap, as the total sum was 22. The same phenomena applied to cosmetic problems (IA: 15, VA: 19, total: 27).
Closer investigation of the problems not found with instant data analysis revealed that most of the problems were confronted only by one user. In some studies, this kind of problems are ignored as “noise”, and thereby Kjeldskov et al. promote this possibility to minimize the amount of problems detected only with one user as a potential advantage of the analysis method.