Barkhuus, L. and Rode, J.A. 2007. From mice to men – 24 years of evaluation in CHI. In Extended Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Barkhuus and Rode analysed the type and scope of evaluations in 1983, 1988, 1994, 2000 and 2006 CHI conference papers. Their results showed that proportion of papers including evaluation has increased so that almost all the accepted papers include some type of evaluation. At the same time, the number of subjects in quantitative studies have decreased and the use of students as test subjects has become almost as a routine. As one reason for the increase of evaluation papers, Barkhuus and Rode suggest the CHI review criteria of validating the results of the paper to be accepted.
Barkhuus and Rode also studied the role of the evaluation in CHI papers. They divided the papers that include evaluation of a technology or a technological concept into analytical vs. empirical and qualitative vs. quantitative methods. Papers that concentrate on evaluation methods themselves were classified in their own category. These evaluation papers had diminished in 2000 to only two papers, and in 2006 there were none. The papers including an evaluation of some new technique, were mostly using empirical, quantitative methods, sometimes backed up with some qualitative studies, such as interviews or surveys.