Benford, S., Greenhalgh, C., Giannachi, G., Walker, B., Marshall, J., Rodden, T. Uncomfortable User Experience. Communications of the ACM. Vol 56, No 9. 2013

CHI article on the cover of the CACM = must read.

Benford et al.’s article is about creating and utilizing uncomfortable user experiences in design. The authors derive examples and ideas from art, media and amusement parks. The main idea is that discomfort used in controlled ways can produce “entertaining, enlightening, and socially bonding cultural experiences”.

The idea of designing for discomfort is interesting. However, using discomfort outside art and amusement usage can be difficult. The authors identify four forms of discomfort and relating design tactics:

  • Visceral -> unpleasant wearables and tangibles, strenuous physicality, pain
  • Cultural -> challenging themes, culturally resonant devices
  • Control -> control to the machine, less or greater control than usual
  • Intimacy -> isolation, strangers, surveillance and voyerism

In addition to these design themes and tactics the article suggests utilizing structures from storytelling, theatre and movies to build the wanted experiences. Discomfort is not usually the experience that is the goal of the interaction/design. Instead discomfort is used as intermediate step to produce or communicate something else. As a result the discomforting part of the event or usage can and should not be too long. Discomfort is also an experience that emphasizes designers’ ethics. Important questions that the designer needs to answer to are is the discomfort justifiable in the case and how to ensure the rights of the users (i.e. right to withdraw, privacy, etc.). Authors leave the question whether uncomfortable user experiences are usable outside the domain of cultural applications open.

Posted by Petri

Journal article - Leave a comment

Siegel, D., Sorin, A., Thompson, M., Dray, S. Fine-Tuning User Research to Drive Innovation. Interactions. September-October 2013. pp. 42-49

The article focuses on interesting and difficult theme of innovating based on user-research. “User-centered innovation” is problematic area of design and engineering since traditional user research methods seem to produce basis for incremental improvements instead of new and novel ideas and out of the box concepts.

The article includes apt critique about different user research approaches as well as using affinity diagramming as user research data analysis method. However, the approach and methods selected by the authors for their design project do not seem to differ much from the criticized ones. All in all, the Interactions article is a good start but it would be very interesting to see more broader research results of the topic instead of experiences and examples from a single design project. For example Donald Norman has commented on the same topic on his Interactions column (volume 17, issue 2) so this article does not seem to contribute as much as it could.

Posted by Petri

Journal article - Leave a comment

Bodker, S., Klokmose, C.N. Dynamics in artifact ecologies. NordiCHI’12. pp. 448-457.

The paper aims to continue discussions of artifact ecologies, i.e. artifacts a person owns, has access to, and uses. The focus on the paper is especially on changes of one’s artifact ecology when new artifact is introduces. The authors have interviewed 12 iPhone users about their obtaining and usage of iPhones. Activity theory is used as theoretical background in the research.

The result of the study is that artifact ecologies seem to change between three states when new artifacts are obtained/introduced: unsatisfactory state, excited state, and stable state. This is somewhat similar to Carroll et al.’s [*] three levels of appropriation: first encounters, exploration, and long-term integration into everyday practices. However, dynamics of artifact ecologies are not an adoption process but a description of what happens in artifact ecology during adoption or appropriation of artifacts.

The article describes some interesting findings, but the results are just the beginning. It would be interesting to understand how the concept of artifact ecology and its dynamics could and should be taken into account during design. In addition a more detailed description of the states would be beneficial. Based on the article it is also unclear whether the list of artifact ecology states is comprehensive or if there is more states that relate to other usage processes such as abandoning an artifact.

Posted by Petri

Conference article - Leave a comment

Klein, G., Calderwood, R., Macgregor, D. Critical Decision Method for Eliciting Knowledge. IEEE Transaction on System, Man, and Cybernetics, vol 19, no 3, (1989)

Klein et al.’s paper describes an interview method that focuses on non-routine events. The method has been developed for studying decision making but it seems to be potentially applicable to UCD/HCI also.

The core idea of the method is to go through the same event multiple times with the interviewee. The paper suggests four rounds:

  1. unstructured account of the event
    The interviewee describes the event whit his/her own words
  2. timeline of the event
    Timeline of the event is constructed in collaboration between interviewee and interviewer. The timeline includes all the phases (sequence and duration) of the event.
  3. decision point identification
    Decision points, i.e. phases and places where decisions were made during the event are identified.
  4. deeper decision point analysis
    Understanding of the decisions and decision points are deepened by asking additional questions (article uses term: cognitive probes) such as “what did you see during…” or “what information did you use…” or “what where your specific goals in this time…”.

 

Posted by Petri

Journal article - Leave a comment

Isomursu, M., Kuutti, K. and Vainamo, S. (2004) Experience clip: Method for user participation and evaluation of mobile concepts. In Proceedings of the eighth conference on Participatory design: Artful integration: interweaving media, materials and practices (PDC 04), Vol. 1, pp. 83-92.

DOI=10.1145/1011870.1011881

In Experience Clip, a pair of users from the passers by is invited to participate in the evaluation of a mobile application in the use of which moving around is central. They gave the evaluated application to the other participant, and a mobile phone with video shooting capability to the other. The one with the mobile phone was instructed to take video clips as the other participant is using the application.

The video clips revealed typical usage patterns, but also expressions of emotions and usability issues. As the users seemed to enjoy observing each other, the usage situations in the video clips appeared natural even when they occassionally were performances created solely for the designers to make their point clear and recommending better solutions.

As requirements for the use of Experience Clip, the authors urge the participants to be well instructed on what they are expected to do and capture on the video clips, and also to have motivated and willing participants. The setting of having participants of equal status produces natural settings for acting and commenting, and for expressing emotions, expectations and even improvements for the applications.

Posted by Sirpa

Conference article - Leave a comment

Foelstad, A. and Hornbaek, K. (2010) Work-domain knowledge in usability evaluation: Experiences with Cooperative Usability Testing. The Journal of Systems and Software, Vol. 83, No. 11, pp. 2019-2030.

DOI= 10.1016/j.jss.2010.02.026

Foelstad and Hornbaek studied the use of Cooperative Usability Testing in the development of two work-domain specific systems. As modifications to the original method, they included an interpretation phase after each task, and used task-scenario walkthroughs instead of video recordings as the basis for discussions. The latter modification was made to bring more flexibility to the discussions and to enable comments also on parts of the system that were not used in performing the tasks. In the interpretation phase, the authors used a set of predefined questions focusing on potential problems, desired changes and well working solutions.

The results of the studies indicated that the interpretation sessions offered an opportunity to broaden the scope of usability issues to cover also user needs and requirements in addition to issues related to static design and interaction design brought out in the interaction sessions. Based on their experiences, the authors recommend to use the interpretation sessions to gain additional insight into observed problems without leading the user, and to look for new requirements and also new solutions.

Posted by Sirpa

Journal article - Leave a comment

Froekjaer, E. and Hornbaek, K. (2005) Cooperative usability testing: complementing usability tests with user-supported interpretation sessions. In CHI ’05 Extended Abstracts on Human Factors in Computing Systems (CHI EA ’05), pp. 1383-1386.

DOI=10.1145/1056808.1056922

Froekjaer and Hornbaek present a usability testing method called Cooperative Usability Testing. It consists of two parts: The first part is an interaction session in which user interacts with the system as in contextual inquiry or in thinking aloud test. The second part consists of an interpretation session in which the test user and the evaluators cooperatively discuss on the problems faced with the use of the system on the basis of a videotape recorded during the interaction session. The authors recommend to use two evaluators in the sessions, so that in the first part one takes notes as the other guides the user, and in the other part they switch roles and the note taker leads the conversation to problems and situations of interest based on the notes.

In their studies, the authors found out that the test users liked to reflect and comment their actions more than in traditional thinking aloud tests with even extensive debriefings. Also the evaluators valued the interpretation sessions, as they helped in clarifying and understanding the most important usability problems. Since the interpretation session is limited to max 45 minutes, the conversations are quite quick, and it is challenging for the former note taker to utilise the notes taken in the interaction sessions. The hastiness also brings out a risk of introducing new and potentially problematic interpretations without careful analysis.

Posted by Sirpa

Conference article - Leave a comment

Woolrych, A., Hornbaek, K., Froekjaer, E., and Cockton, G. (2011). Ingredients and meals rather than recipes: A proposal for research that does not treat usability evaluation methods as indivisible wholes. International Journal of Human-Computer Interaction, Vol. 27, No. 10, pp. 940-970.

DOI= 10.1080/10447318.2011.555314

Woolrych et al. nicely analyse the state of research and comparisons on usability evaluation methods. Too often, these methods are considered as precisely presented step-by-step procedures that almost automatically produce a list of usability problems regardless of the context of use, evaluators, test participants (if ones), design and test goals or many factors that affect to the evaluations in practical work, and to some extent, also to experimental settings.

Woolrych et al. make analogies between recipes and usability evaluation methods, ingredients and factors or resources affecting to the implementation of the method, and meals and usability work as a whole for example in organizational settings. Instead of keep on focusing on the methods, Woolrych et al. recommend to go deeper into the ingredients, and also broaden the scope to levels above the methods to the development and business context. The change in focus should, in their opinion, reflect both to the research and the way usability evaluation is taught to students.

Posted by Sirpa

Journal article - Leave a comment

Lindgaard, G. and Chattratichart, J. (2007) Usability testing: what have we overlooked?. In Proceedings of the SIGCHI conference on Human factors in computing systems (CHI ’07). ACM, New York, NY, USA, pp. 1415-1424.

DOI= 10.1145/1240624.1240839

The studies by Lindgaard and Chattratichart indicate a need to shift the focus from the number of test users to the number of test tasks in usability testing. Lindgaard and Chattratichart analysed the results of several usability teams that had conducted usability tests on the same web service with quite similar settings including same evaluation goals, using thinking-aloud method and reportng the findings in common problem format. The number of reported usability problems was restricted to maximum of 50, but the number of test users and test tasks was no way restricted. Therefore, Lindgaard and Chattratichart set the number of test users and number of test tasks as independent variables in their study, and analysed if these variables correlate with the proportion of problems found and proportion of new problems found. Their results showed no significant correlation between the number of users and proportion of problems nor with new problems found. Yet, the results showed significant correlations between the number of test tasks and proportion of both problems found and new problems found. Another important factor affecting to the quality of test results was the recruitment procedure of test participants: the web service was intended for a variety of users with varying skills and background knowledge, so it was important to recruit a heterogeneous sample of users to cover as many potential problems as possible.

Based on their results, Lindgaard and Chattratichart recommend to invest in wide task coverage and careful participant recruitment instead of large number of test users. The possible affect of limiting the number of reported problems into 50 was not discussed in the article, although the total number of problems from all the reported tests was as high as 176.

Posted by Sirpa

Journal article - Leave a comment

Holleran, P.A. (1991) A methodological note on pitfalls in usability testing. Behaviour & Information Technology, Vol. 10, no. 5, pp. 345-357.

DOI:10.1080/01449299108924295

Good usability testing is similar to good empirical research: the use of improper procedures will result in invalid data, and thereby poor validity and reliability. Holleran categorises pitfalls in usability testing into three groups: sampling problems mainly in planning the tests, methodological problems in conducting the test sessions, and problems in interpreting the results. Sampling problems refer to the selection of subjects and test tasks, their number and representativeness, and also to the poor reporting of use characteristics as well as poor analyses of possible correlations between these characteristics and test results.

Methodological problems, on the other hand, cover issues related to having human test participants, such as motivation and moderator bias. He recommends to be cautious in analysing test users’ performance as test users may put more attempt to the tasks than normally. The moderators in the tests also easily affect the users’ performance, as well as the thinking aloud method if used in the test. All these factors affect to the validity of usability testing as it is not so clear what is actually measured and evaluated, and also to the reliability, as the results may be quite different although the same procedure was repeated.

The last pitfalls that Holleran mentions lie in the interpretation of the test results. He recommends to use quantitative measures, such as content analysis and observational coding procedures that allow also statistical analyses to avoid purely subjective interpretations of qualitative data.

Posted by Sirpa

Journal article - Leave a comment

Orne, M.T. (1962) On the social psychology of the psychological experiment: With particular reference to demand characteristics and their implications. American Psychologist. Vol. 17, No. 11, pp. 776-783.

Accession Number= 00000487-196211000-00005

Finally I found an article presenting notes on social psychology studies regarding the behavior of people as test participants. Although usability tests are not generally treated as scientific or psychological experiments, many similar phenomenon as in Orne’s studies can be identified also in usability tests. For example, the test users are ready to “perform a very wide range of actions on request without inquiring as to their purpose, and frequently without inquiring as to their durations”, as Orne mentions. Furthermore, the tasks are carried out with care even for hours until the experimenter ends the session. It is unlikely that the test users would perform with similar care and persistence in contexts outside the tests.

In Orne’s studies, test participants have been shown to be eager to validate the experimental hypothesis that they infer from various cues ranging from the reputation of the research filed, rumors of the specific study and research group to the gestures and tones of the experimenter. Even if the tasks are clearly meaningless, the participants continue to do them and rely the test to have an important purpose. As the factors affecting to test participants’ expectations and behaviour cannot be eliminated, Orne recommends to identify them as systematically as possible to be able to assess their effect on the results and especially the extent to which the results can be generalized to contexts outside test situations.

Although the article is even older than me, people have not changed, so they behave much the same way in usability tests nowadays, so it is important to have at leas some idea of the numerous factors possibly influencing the test.

Posted by Sirpa

Journal article - Leave a comment

Kjeldskov, J., Skov, M.B. and Stage, J. (2004) Instant data analysis: conducting usability evaluations in a day. In Proceedings of the third Nordic conference on Human-computer interaction (NordiCHI ’04). ACM, New York, NY, USA, pp. 233-240.

DOI=10.1145/1028014.1028050

Kjeldskov et al. decided to test if the analysis phase of usability testing could be cut down and thereby cut the costs of testing. They utilized the resources already used in testing, i.e. the moderator and a note taker, and reserved one more hour for them after the tests (all 4-6 tests in one day) to make a summary of their findings. A specific facilitator joined the meeting and made a summary of the discussion and most important findings after the meeting. All this took 3x1h (moderator, note taker and facilitator) plus 1h for the facilitator in wrapping up the findings, and validating the results with the moderator and the note taker. That is, 4 h in total.

The results were compared with a group doing traditional videotape analysis and spending 40 hours in the analysis. In total, these two analysis methods revealed 13 critical problems and 11 were found with the instant analysis and 12 with traditional videotape analysis. With serious problems, the division was surprising, because both methods revealed 15 problems, but there was less overlap, as the total sum was 22. The same phenomena applied to cosmetic problems (IA: 15, VA: 19, total: 27).

Closer investigation of the problems not found with instant data analysis revealed that most of the problems were confronted only by one user. In some studies, this kind of problems are ignored as “noise”, and thereby Kjeldskov et al. promote this possibility to minimize the amount of problems detected only with one user as a potential advantage of the analysis method.

Posted by Sirpa

Conference article - Leave a comment

Sawyer, P., Flanders, A. & Wixon, D. (1996) Making a difference – The impact of inspections. In Proceedings of the ACM CHI’96 Conference on Human Factors in Computing Systems. pp. 376-382.

DOI=10.1145/238386.238579

Sawyer, Flanders and Wixon defined a metric called impact ratio to measure the effectiveness of usability evaluation methods. This numerical value presents the proportion of the problems that the development team commits to fix from all the problems found in the evaluations. They inspected ten software products that they had evaluated, and achieved an average impact ratio of 78%.

They found out that the following factors improved the impact ratio:
– Developers’ respect: The usability group conducting most of the evaluations was an internal group in the company, so it had a long-term relationship with the development groups and had earned the developers’ respect.
– Written reports: The usability group gave written reports of the findings to the development groups. The reports included descriptions of the problems and alternatives for fixing them.
– Written response: The group required a written response from the clients to report the commitment to fix the problems.
– Multiple methods: The evaluations involved the use of multiple methods for discovering usability problems.
– Easy response process: The group set up meetings where they went over the report with the clients.
– Specific recommendations: The group provided detailed and technically specific recommendations to fix specific problems.
– Severity level: The group rated each problem by its severity.
– Client participation: The group involved the development team in the evaluations by having a member of the team sit on an inspection or user test.
– Early involvement: The group tried to work with the development teams as early in the development cycle as possible.

To be able to give effective recommendations, Sawyer et al. recommend that the evaluators become familiar with the product and its design goals before the evaluations, so that the recommendations support the product goals. In addition, the evaluators should study the product within the context of its product family, so that the recommendations do not represent inconsistencies with other components in the product family.

Posted by Sirpa

Book chapter - Leave a comment

Hackman G.S. and Biers, D.W. (1992) Team Usability Testing: Are two Heads Better than One? Proceedings of the Human Factors and Ergonomics Society Annual Meeting, October 1992; vol. 36, 16: pp. 1205-1209.

DOI= 10.1177/154193129203601605

Hackman and Biers made studies to compare the performance of a single user alone, a single user with an observer and a pair of users all using the thinking aloud method. Their results showed that the presence of an observer did not have an effect on the users’ performance or quality of thinking aloud statements. If the number of statements of moderate to high value to designers were divided to the single users in a pair, there was no significant difference in the users’ performance or quality of statements between single or pair activity. Nevertheless, the same amount of valuable statements and parts of interface were detected in half the time in one paired-user test as in two single user tests.

Posted by Sirpa

Conference article - Leave a comment

Höysniemi, J., Hämäläinen, P. and Turkki, L. (2003) Using peer tutoring in evaluating the usability of a physically interactive computer game with children. Interacting with Computers, Vol. 15, No. 2, pp. 203-225.

DOI= 10.1016/S0953-5438(03)00008-0

This study used peer tutoring to evaluate an interactive computer game with children. They used either a pair of children or one child at a time to teach the use of the game to another child. This way, the interaction between an adult and a child was minimised, and the ease of learning and teaching to use the game could be assessed. The researchers noticed that especially the part where the children became tutors gave a lot of information about the usability of the game, whereas the role of the tutee kept the children pretty quiet.

Posted by Sirpa

Journal article - Leave a comment

Kennedy, S. (1989) Using video in the BNR usability lab. SIGCHI Bulletin. Vol. 21, No. 2, pp. 92-95.

DOI=10.1145/70609.70624

Co-discovery learning shares many principles with constructive interaction, but in addition, has a list of spesific tasks and includes a reflection on the task difficulty after each task. Sue Kennedy and her colleagues used this method in evaluating various communication systems by making two users cooperate and discovering how to use a new system by trial and error. The experimenter gives a list of tasks to the users and leaves them performing the tasks. After the users have completed a task, the experimenter immediately asks them to jointly assess the difficulty of the task. Both the discovery and the jointly made assessment give valuable information on the usability problems of the system.

Posted by Sirpa

Journal article - Leave a comment

O’Malley, C.E., Draper, S.W. & Riley, M.S. (1984) Constructive interaction: A method for studying human-computer-human interaction. In Shackel, B. (Ed.) Human-computer interaction – INTERACT’84. pp. 269-274.

Constructive interaction is a method involving two users at the same time in solving a problem. O’Malley et al. brought this method into the studies of human-computer interaction in the mid 1980’s. In this method, two subjects with comparable expertise explain their ideas and rationale behind their hypotheses to their partner. The subjects are encouraged to experiment with the studied system, and they are disturbed or interrupted only if the discussion ends. O’Malley et al. recommend this method to be used when emphasis is on understanding or developing system concepts instead of learning procedures.

Posted by Sirpa

Conference article - Leave a comment

Eppler, M.J., (2011) What is an Effective Knowledge Visualization? Insight from a Review of Seminal Concepts, 15th International Conference on Information Visualisation

Paper discusses different concepts related to knowledge visualization. My interest was on the concepts ‘conscription device’ and ‘boundary object’ (in particular, on conscription devices). Boundary objects have different meanings in different contexts, but their structure is common enough to more that one professional community to make them recognizable means of translation.

Paper lists three knowledge visualization principles that can be derived from conscription device concept and concepts close to it:

  • visual unfreezing – visualization must be able to be switched from a fixed mode to flexible, modifiable mode and back.
  • visual discovery – visualization must provide assistance for reasoning, reflection, and linking items in a new ways so as to facilitate new discoveries from the shared insights.
  • visul playfulness – the visual should provide playful mechanisms to reframe issues and cajole participants into a different mindset and thus generate new insights and intesify collaboration

Posted by Venlakaisa

Conference article - Leave a comment

Trudel, C-I. and Payne, S.J. (1995) Reflection and goal management in exploratory learning. International Journal of Human-Computer Studies. Vol. 42, No. 3, pp 307-339.

DOI= 10.1006/ijhc.1995.1015

In these experiments, Trudel and Payne studied the effect of constraining the number of keystrokes that subjects were allowed to make while they were learning to use a new interactive device. They also tried the effect of having a list of goals to manage their explorations with the device. In a later study, they added a condition in which they forced the subjects to study one part of the device at a time.

The results showed that
– both the keystroke limit and concentrating to one part at a time, yield to significant improvements in learning, whereas the goal list had only small effect.
– subjects with keystroke limit remembered significantly more about the modes and features of the device than subjects with the goal list or unstructured exploration
– subjects with keystroke limit also made significantly less excessive keystrokes than the other two groups
– the keystroke limit and forcing to study one mode at a time both had such a significant effect on learning that combining these conditions did not have significant additional effect, nor having goal list if keystroke limit was already in use. Only if the subjects had no keystroke or mode limits, the goal list improved learning to some extent.

Posted by Sirpa

Journal article - Leave a comment

Trudel, C-I. and Payne, S.J. (1996) Self-monitoring during exploration of an interactive device. International Journal of Human-Computer Studies, Vol. 45, No. 6, pp. 723-747.

DOI= 10.1006/ijhc.1996.0076

Trudel and Payne are interested in how people learn to use interactive devices and how this learning can be supported. Their studies relate to usability testing as they made experiments where “subject explored an unfamiliar interactive device without the benefit of assistance or instruction” as in usability test. They did not use concurrent thinking aloud, but asked the subjects to explain their doings or intentions with short intervals, as the comparison group worked silently without interruptions. In one experiment, they used different prompts for these explanations having some or no relevance to the tasks. In one experiment, they also tested, whether an external prompt was needed or if the subjects could independently interrupt their work to review their process.

The results showed that
– the subjects who were interrupted to verbalize what they had learned during past 2 minutes made significantly less errors than the group who were not interrupted
– also the subjects who were interrupted to verbalize their intentions performed significantly better than those without interruptions
– when compared, those reviewing their learning made significantly less errors than those stating their intentions, describing the current screen (DS) or answering unrelated questions (UQ). Still, the ones stating their intentions made significantly less errors than DS or UQ groups. Thereby, the content affects performance instead of mere verbalization and having breaks.
– if the subjects were asked to stop working whenever they felt that they had learned something and to state that aloud, they also made significantly fewer errors than the comparison group.

Posted by Sirpa

Journal article - Leave a comment