Commonalities of usability testing vs. field observations as a research method

I have used field observations with video recording and think aloud in my research on exploratory software testing (ET) work. To me the similarities to usability testing seem obvious, but I have not explicitly used usability testing as a basis for my research methods. However, it could be fruitful to have some discussion on the commonalities and distinctions of usability testing and more generic observation based research methods for studying people’s work.

Here is a brief initial comparison based on the definition of usability testing according to Dumas and Redish described in earlier post by Sirpa:

Usability testing
– the goal is to improve a product’s usability,
– the participants represent real users,
– the participants do real tasks,
– the testers observe and record the participants
– the testers analyze the data and recommend changes to fix problems

Field observations in the context of my ET research
– the goal is to understand how people perform ET (with computer)
– the participants are real testers
– the participants do their real tasks in the real environment
– the researchers observe and record the testing sessions
– the researchers analyze the data and report research findings (and possibly create new and document the observed practices and techniques)

In the methodology literature on empirical software engineering this type of video recorded filed observations are mostly ignored. Some examples can be found where this type of methodology has been applied. E.g. Salinger et al. (2008) studied pair programming using similar methodology as we used to study testing. Both for them and us the major challenge was the data analysis and especially coping with the large volume of very rich data when you have quite open research questions. What could we learn and generalize to a wider context of empirical research from the analysis methods for usability test data?

Salinger S, Plonka L, Prechelt L (2008) A Coding Scheme Development Methodology Using Grounded Theory for Qualitative Analysis of Pair Programming. Human Technology 4:9–25.

Itkonen, J. (2011) Empirical Studies on Exploratory Software Testing. Doctoral dissertation. (Article 5)