Nielsen, J. and Levy, J. (1994) Measuring usability: Preference vs. performance. Communications of the ACM, Vol. 37, no. 4, pp. 66-75.

DOI= 10.1145/175276.175282

Nielsen and Levy (1994) made a meta-analysis to study the correlation between test users’ performance with the compared products and their preferences after the test. They found a strong positive correlation between performance and preferences, but several cases were left unexplained, why the users chose against their actual performance. The results also showed that the correlation between performance and preference was much weaker for the expert users than the novice users. The expert users used remarkably wider scale than the novice users in their evaluations, although their performance with the products did not differ much more than with the novice users.

Although this article is almost 20 years old, I added it to this blog, since it seems to have inspired several tracks of studies on the interaction between preferences or perceived usability and various factors, including aesthetics. The remark on the expert users’ tendency to use extreme evaluations is also something that seems to apply to my and my colleagues’ assessments on usability of various products. Still, I do not remember many papers addressing this issue so clearly.