Sengers and Gaver want to add new perspectives to both design and evaluation of new systems. In addition to, designing systems that “convey a single, specific, clear interpretation of what they are for and how they should be used“, one could intentionally design for multiple interpretations.
When evaluating these kind of systems, there is no point in checking whether the designer’s and users’ models are the same, but to explore the ways users start using it. Sengers and Gaver recommend ethnography, longitudinal studies and dynamic feedback (giving evaluation results back to the users for getting their reasoning for the results) as some examples of appropriate methods. The also emphasize the need for “variety of assessments from a diverse population of interpreters“.
The most valuable part for me in this paper was the division of interpretation into three levels. Although it was not the focus of this paper, it gave me a framework for categorizing users’ comments on a system under evaluation. The three levels of interpretation are briefly:
Operating the system: from recognizing a button, understanding its functionality, to figuring out how to do a certain task.
Utility of the system: from what is this system meant for, what is it appropriate for, to what role can it play in my life?
Value of the system: “What does it mean about me, my social group, my society, my culture?“