Research designs: Identifying the right methods for the study

In this, slightly longer text, I address a topic that I have wished to explain a long time: how good research plans can be created.

Research designs

Which methods should I use in my study? This is a question that one must answer in each and every study, from simple courseworks to complex PhD theses that may take several years to complete. The question about the right methods is often hard to solve. This article aims to give ideas on how to survive in this task.

The correct methods can be difficult to choose because everyone of us has their personal favourite methods that are more inspiring than others. The researcher may be good at some method in particular – be it interviewing, prototype-building, a quantitative survey, or something else – and they would wish to use it also in the upcoming study. However, the method may not fit with the goals of the study. In such a situation, and also universally, it is important to understand that a method has to be chosen based on the research question (RQ) that is asked. That is, the RQ has a higher importance than the method; methods are merely tools for finding answers. (In this text, I will solely talk about RQs, but the term always includes also design problems).

RQ–method combinations are called research designs. The term “research design” describes the meaning of the concept very well: it refers to the way in which one designs their research: how one goes about carrying out the study and achieving findings. Term is commonly used in quantitative studies (see e.g., Trochim, n.d.), but it is applicable to any other kind of research: qualitative, constructive, action research, even to theorising. 

Research designs can be excellent or terribly poor. There would be, for example, severe flaws in a study that would seek to prove that one user interface (UI #1) of a mobile navigation app would be better than another interface (UI #2) by presenting visual layouts to the two UIs to six participants, asking their opinions about them, and analysing which one gets more positive responses. Perhaps most obviously, such a research design would flawed because interviewing is not the right method for evaluating action-oriented UIs such as navigation apps. A better research design would involve navigation tasks that would be performed using the two UIs. Even after that correction, many smaller methodological decisions would need to be determined. A finished, well-planned research design takes some time to create.

Research designing as a cyclical process

Therefore, it is a good idea to consider research designing as a process that involves many attempts and restarts until a good design is found. The following diagram diagram in Figure 1 presents the main elements of a research design – contribution, RQs and the methods – that need to be fit to each other, as well as the main directions of effects that spark changes between these elements.

Figure 1. Research design planning cycle

Let’s examine the above-presented comparison study of two mobile navigation UIs using this diagram. The left side of the diagram presents the ambitions that the researcher has for the study. Starting from the top on that side, the study’s contribution relates to the superiority of one UI over another. Perhaps UI #1 is researcher’s own, new design, and they wish to prove that it si better than UI #2, which is typical (so called “baseline”) in the market at the moment. Moving to the middle, RQ of the study could be then be something like this: “Is UI #1 better than UI #2 when used for navigation?”. Finally, the method would be to show the two UIs as visual layouts to six people, listen to their opinions, and determine the winning UI. As was already pointed out above, navigation can be better studied by asking people to navigate in the physical world rather than by interviewing them about their opinions.

This is a point where ambition meets realism: It will not be possible to get a valid answer to the RQ using this method. The process now moves to the right side of the diagram where either a new method is chosen, or the RQs and contributions are adapted to match with the method. If the researcher wants to stick to interviewing as the research method, then the RQ has to be changed. It may be possible to ask something like this: “In what ways will experts’ opinions about two mobile navigation UIs differ from each other?”

Now the study asks about expert opinions and does not consider actual end users. This decreases the need for participants, since experts are probably able to provide valid analyses even by just looking at visual layouts – end users would not be able to do that. Experts are probably also more likeminded in their assessments. Therefore maybe six participants will be enough.

Also, the RQ is now framed as an inquiry about differences between expert opinions instead of straightforwardly about finding a winning UI. That suits better for interviews. What is however lost is the sharpness of the intended contribution: with the new RQ, the researcher will not be able to claim that one UI would be better for navigation than another. Such a claim can be validly studied only by observing navigation in action. Now the intended contribution will be somewhat softer: to show that a number of experts consider one UI better than another UI, at least in some respects.

As a result of these changes, we have a workable research design. We have reached it by softening our intended contribution. We have also decided that we need to recruit experts and not end-users as our participants. Finding such people will be harder, but will allow us to carry out fewer interviews, possibly only six. Figure 2 illustrates the transformations that have happened in the research design planning cycle.

Figure 2. Research design planning cycle applied to the example in the text.

What the analysis above tries to illustrate is that creation of a research design involves going back and forth between the contribution, RQs and methods. This is normal, and every experienced researcher is used to doing this kind of process before they start their actual research. The process may take a considerable amount of time, because in addition to just thinking, also reading may be needed, since literature may help in making the RQs sharper with the help of theoretical assumptions and existing empirical findings. The end of this article presents some common “tricks” by which dead ends in research designing can be avoided, and it is eventually possible to find a combination of contributions, RQs and methods that satisfies the necessary criteria of validity and the ambitions of the researcher.

How do I know what methods are required by the RQ?

The text above emphasised that it is the RQ that determines what the right methods are. But how does one know, given that one has a suggestion for an RQ, what are the possible methods to consider? It is not possible to give a universal rule for the question, but good examples can be found from prior studies that have been published about similar problems. It is usually a good idea not to start inventing one’s own methods but to use ones that have already been used successfully. That also gives an idea on the complexity and labouriousity of the study.

Most important, however, is to precisely to understand what one is asking in the RQ. The question should be understood word by word, and taken very literally. This helps figure out what kind of methods (for data gathering and its analysis) could possibly be considered proper what kind of a method could be considered as being proper. Here are some examples:

  • Is UI #1 better than UI #2 when used for navigation? This is a comparative question (because of the word “better” in it) and requires that two measurements, one from UI #1 and the other from UI #2, can be compared against each other. It is also asked matter-of-factly, in a universal and objectivistic manner: Is UI #1 better than #2, whereby it refers to all app users and assumes that the comparison is carried out from all possible viewpoints (navigation time, number of errors, stress level, etc.), and that it is independent of researchers’ or users’ subjective opinions. Also, the RQ talks about navigation, which points out that the study should consist of a navigation task and that not an interview, for example. Considering all of this together, a controlled experiment and quantitative methods would be considered the right way to answer to this RQ.
  • What are the strengths and weaknesses of UI #1 and UI #2? In this RQ, the comparison is removed, and replaced with an analysis of the two UIs independently from each other. That makes the RQ easier to study. Also qualitative methods, such as interviews and subjective opinions, are now applicable. What is sacrificed is the ability to say that one UI is better than another. Some level of qualitative comparison of the differences is however possible. For example, one can say that “it is possible that UI #1 would be less stressful than UI #2”.
  • How should a UI intended for stress-free navigation be designed? This question drops the comparison of two UIs altogether, and also focuses on the design process instead of the outcome of the design. It also narrows down the design focus on stress-freeness instead of focusing on UI’s design from every possible viewpoint. With this new orientation, now the focus must be on the ways in which stress-freeness should be taken into account in different design stages. One may think how stressful navigation situations could be studied in user research, how stress-freeness can be evaluated using prototypes, and so on. In the final design after the process, one can explain how different design choices are based on different attempts to stress-free navigation.

The RQ can be changed also in many other ways. These examples show that it is possible to formulate the RQ in different ways, seeking for a study that remains meaningful and interesting while making the methods realistic to carry out. This illustrates how to make use of Figure 1’s research design planning cycle: one starts from some contribution and an RQ that would suit it, then thinks about what they require from the methods. Then the RQ and the methods are reformulated until they match and the study is possible to carry out.

Triangulation: using several methods for one RQ

Most RQs can be answered using more than one method. This is good: If several methods are used to answer a question, their findings, when put together, are more reliable and valid than with any method individually.

In the best case, the methods that answer the same RQ triangulate each other. This means that their are different in such a way that they cancel each other’s weaknesses. The term’s meaning can be understood by considering the word’s original use in close-to-shore marine navigation. When the captain wants to know their ship’s position, it is best obtained by measuring the directions (“bearings”) to landmarks that are at a right angle (i.e., 90 degrees) to each other. In Figure 3, bearings to the Skyscraper and the Manufacturing plant offer such a situation: the errors of measuring the exactly correct directions to these landmarks cancel each other optimally. The second best solution is to use directions that are not at a right angle, such as bearings to the Skyscraper and the Peaky hill. The least favourable situation is to take the bearing to one landmark only and guess what the distance to it might be.

Figure 3. Triangulation in locating a ship’s exact position.

In research designs, triangulation is useful in an analogical manner. The answer to the RQ is matches with the question about the ship’s exact position. The methods that give answers to the RQ match with the bearings to the landmarks. Similarly as the bearings have their errors, also research methods have their individual errors and biases. Interviewing, which is so-called “self-report” method, for example, depends on participants’ truthfulness and verbalisation skills, and cannot be always trusted. The same applies to a diary-keeping and questionnaires – also they a
re self-report methods. Direct observation of people who carry out tasks without knowing that they are being observed, in contrast, does not have such problems on truthfulness or people’s verbalisation skills. On the other hand, it has its own problems as a method. Together, however, observation and interviewing triangulate each other excellently: if both methods offer the same findings, then our results are much more believable than with only either of the methods alone. Figure 4 illustrates the relationships between these methods using a similar diagram as Figure 3.

Figure 4: Triangulation in research design.

Several RQs and several methods

Finally, it is typical that a study has several, maybe 2 or 3, RQs that crystallize the main aspects of the intended contribution. One could think that these RQs triangulate, in their own way, the intended contribution. For each of these RQs, following the principle of triangulation described above, several methods may be considered. Although it may seem that a study with two RQs, each with two suitable methods that triangulate each other, would require 2 x 2 = 4 different methods, this does not need to be the case. Instead, often the same method can be used to answer two different questions. One can interview, for example, about matters related to RQ1 and then about RQ2. Thus, a research design of a full-fledged study about two mobile navigation apps may have the structure shown in Figure 5. In this example, only one method is used for answering the first RQ, which leads to somewhat weak findings. The second RQ, in comparison, is rather well addressed with three questions where two self-report methods and navigation tasks nicely triangulate each other.

Figure 5. Research design diagram for several RQs and methods.

This diagram is visual and easy to read. It is even possible to point out which methods triangulate each other and which ones involve redundancy (and are especially useful in increasing reliability in data collection) but are mostly susceptible for the same biases and weaknesses. Well-triangulating methods’ arrows can be placed in right angles with each other, while methods with redundancies have arrows that are almost parallel with each other.

In a research plan document, on the other hand, a more detailed format for the same diagram can be useful. Table 1 below shows one example for such a format. In the same way as in the diagrams above, it has the RQs in the left and methods in the right. One column has been added between the RQs and the methods: theoretical constructs and concepts. They are helpful contents from literature that are used to sharpen the focus of the study to most informative elements. In mobile navigation, such a concept can be, for example, multiple resources theory (Wickens, 2002) that presents a model for human attention and cognitive processing capacities in multi-tasking situations. It is informative since navigation using a mobile app involves always multitasking: division of attention between the surroundings and the app. Finally, Table 1 also divides methods into data gathering and analysis stages.

 

RQs Theoretical constructs and concepts Data/material that will be gathered to answer the RQ How this data/material will be analysed to obtain answer to the RQ
 …  …  …  …
Navigation test with end-user who use prototypes of UI #1 and #2 Multiple resources theory  Navigation task where every participant uses both UIs and where they have to continuously switch their attention between the surroundings and the app Statistical comparison of completion times using the two UIs
(same as above) Cognitive and physical load Participants rate their subjective task load using NASA-TLX questionnaire (Hart & Staveland 1988) Statistical comparison of between questionnaire answers
(same as above) Human orientation Interview on whether participants prefer maps that are oriented so that north is always in the top of the screen, or if the map should rotate based on the user’s facing direction Qualitative analysis of participants’ considerations
 …  …  …  …

Table 1. Research design matrix (incomplete).

When the research design has reached this stage, and if all the RQs have methods that have methods that involve triangulation, it is a good moment to start the practical preparations for the empirical parts of the study!

References

Trochim, W. M. K. (n.d.). Research design. In Research Methods Knowledge Base at https://conjointly.com/kb/ (accessed 17 April 2022).

Wickens, C. W. (2002). Multiple resources and performance prediction. Theoretical Issues in Ergonomics Science, 3(2), 159–177. https://doi.org/10.1080/14639220210123806

Hart, S. G. and Staveland, L. E. (1988). Development of NASA-TLX (task load index): Results of empirical and theoretical research. Human Mental Workload (1988).

How to include reflection to the text

Introduction

In ordinary English, “to reflect” and “reflection” refer to deeper-level thinking. Collins Cobuild English Dictionary describes that “when you reflect, you think deeply about something.” Apple’s dictionary, in turn, defines reflection as “serious thought or consideration: he doesn’t get much time for reflection” (emphasis is original).

In an academic research context, the meaning is quite similar. When a researcher reflects, they seek answers to questions on what research questions they should focus on, do they approach the research question with the right methods, what decisions they have to do during the research, what limitations they have to accept in their research, and what conclusions they can draw from their findings. Asking these questions and answering to them are examples of deep thinking.

These questions show that reflection is an important element in all stages of academic practice, such as

  • in the identification of a topic for research
  • in the selection of appropriate methods to study the topic
  • in the empirical part of the study where the collected data may deviate from what was expected, possibly directing attention to findings that may or may not be even more important than what was originally expected
  • in the analysis of the data and making sense of its findings
  • in the writing about the process, its findings, and what their significance is

Common to all of the examples in this list is the focus on “staying on track” of the research process: making sure all the time that there will be something meaningful – a research contribution – in the end of the process. Reflection is therefore an essential element of finding out how to navigate successfully past the challenges of the research process, and knowing what the findings imply.

In the spirit of this blog, which focuses on writing, this post will focus on how to write reflectively in a thesis or an academic article. This text works well with my earlier writing about an academic text’s storyline, where it is easy to notice, at least after reading this post, how reflection plays an important role in the narrative generation.

General characteristics of reflection

From the introduction above, I want to highlight the following characteristics:

  • Depth. Reflection explores matters beyond, above, or underlying the practical matters of research. By doing this, it supplements those parts of writing that focus on plain reporting (e.g., what has been written in literature, or how research methods were carried out, or what data was collected, or what stages a prototyping project included).
  • Explanatory style. Reflection gives an explanation for why the research was carried out in a certain way and not some other way. By explaining the decisions about methods, and the reasonings behind the conclusions, it makes the researcher’s process understandable for the reader.
  • Higher viewpoint. Reflection looks at matters from a higher vantage point, “from the outside”. This way it can explain the reasons for subjective choices in more objective terms, such as by linking it to other research writings, practical limitations that could not be overcome, or personal preferences that build on the authors’ prior experiences, for example.
  • Navigation in an unknown terrain. With all the characteristics above, the reflection helps in managing the unavoidable uncertainties of every research or design process. It prepares for expecting the challenges that lie ahead. During practical research work, such as when one gathers data or builds prototypes, it helps in noticing interesting features, unexpected outcomes, and needs for decision-making between alternative courses of action. In the writing up the findings, it helps in identifying the strengths and weaknesses of the work that was carried out, what the project’s findings are, and what implication and future considerations arise from them.

Reflection during a research or design process

From the last point, where I listed the benefits of reflection before, during and after the practical “execution” of research or design project, it is evident that researchers and designers should preferably reflect all the time. They should repeatedly ask questions such as “What should be the best course of actions now?”, “Are there alternative ways to do the things that I am now doing?”, “What matters can I leave out from my focus, and which ones should I focus particularly on?”, “Am I still going towards the goal that I had in the beginning, or have I shifted to a different goal? If I have, should I shift back, or change my goal?”

The questions above are examples of reflection-in-action: reflective questions that one can ask as a part of their practical work, and thereby make it part of their creative process. This is a viewpoint that design theorist Donald Schön (1983, 1992) has particularly endorsed: creative practice (applies both to research and design) can be described as “reflective conversation with the situation” where the designer/researcher proceeds by making conscious “framing experiments” by which they frame and re-frame the problem that lies in front of them, and learn by observing the results. Schön’s book The Reflective Practitioner (1983) contains a widely cited example of reflection-in-action from architectural practice that is worth reading.

Although Schön does not describe reflection-in-action using the terms “problem space” or “design space”, I find it very useful to think reflection-in-action as an ongoing exploration and re-definition of the problem space and the problem itself. The idea that a design process both explores and constructs the design space has been advocated especially in Kees Dorst and Nigel Cross’s writing (2001), who are other prominent design theorists. One way to conceptualize this reflective process is as follows, based on interaction designer Bill Buxton’s illustration (2007, p. 388) that I have modified in Figure 1 but whose caption I am copying in entirety from Buxton.

Figure 1. “Design is about exploring and comparing the relative merits of alternatives. There is not just one path, and at any given time and for any given reason, there may be numerous different alternatives being considered, only one of which will eventually find itself in a product.” (Buxton, 2007, p. 388).

The exclamation marks (!) in the diagram illustrate learning moments that are particularly important to note when describing the design or research process, such as a choice about a prototype’s shape, or a suitable methodology that should match a research question.

What is important here is that the entire design process can be seen as “branching exploration and comparison”. The same applies to research: also there the different stages of empirical work require their own critical analysis that ascertain that the process is progressing towards a solution. These stages are useful to be presented to the reader when the thesis or article is written from the process. They showcase the author’s skills in being able to work maturely with an open-ended problem and its challenges.

How are the alternatives and their solutions identified

Here are some strategies that help you reflect during the research or design process:

  • Analyse whether another researcher or designer could decide to proceed using different methods or approaches. Analyse both alternatives, find reasons why one would be better than another, and select the one that seems better.
  • Compare your project to similar ones in the literature. Decide whether you want to use the same approach, or choose a different path, and why.
  • Alternatively, use other external reference materials, such as benchmarking to other projects or products, and apply similar reflection as in the second bullet point.
  • Explicate your subjective preferences and identify the plausible alternatives (e.g., “I wanted to prioritise human values over utilitarian ones because…”)
  • Identify tensions, contradictions or tradeoffs in the design or research problem, and use them as decision-making junctions.

In order to avoid becoming paralysed by the number of possibilities that could be reflected, do not try to be exhaustive in everything that you do. The purpose, as explained by Schön, is to use reflection as your tool that generates meaningful possibilities, and helps you thereby advance further in the design process.

Reflection in a thesis or an article

Now, finally, we can discuss how reflection should be visible in the written text.

Obviously, all the elements, listed in various parts of the sections above, give already an idea what to include as reflections in a text. Thus, following the general characteristics given in this post’s Introduction, reflection should add depth, explanations, a higher viewpoint, and this way navigate the reader through an unknown terrain of your research/design process. It should also describe your answers to the questions that you have been asking during the process, thereby giving an idea for the reader as to what question or design problem you started with in the beginning, and what events possibly led you to change your focus, and what choices you had to do along the way, and how you made those choices.

These examples on what to reflect about in the text are very similar to many elements of my earlier blog post that presents storyline for an academic text. In a nutshell, here are examples from that blog post on reflection points that most academic texts are expected to contain:

Section of the text Topics to reflect on
Introduction Problematization: Critical analysis of the current state-of-affairs in the world that needs to be studied in more detail.

Motivation: Problematization’s constructive follow-up where an approach is suggested to the problematized topic.

Related research Identification of the bodies of knowledge: An analysis of the research question or design problem that focuses on identifying what knowledge is important to use from previous research.

Reviews of bodies of knowledge: A critical analysis of the knowledge in the end of each sub-section about a body of knowledge. This analysis shows the “research gap” that remains to be found out.

Summary of knowledge: A constructive wrap-up of all the knowledge, and an analysis on how realistic the research question / design problem is in light of all this knowledge, and how it needs to be adjusted.

Method Methodological reflection and choice of methods: A constructive analysis of possible methods that can be used to answer the question / problem. For example, may describe, referring back to related research, how other researchers/designers have solved similar problems before, and whether the same methods can be used also in the present study. After these reflections, a constructive conclusion can describe the approach and methods that you have chosen among the alternatives.
Results/Findings/Project Answers to the research question / design problem. This is the “execution” part of the text, and can be different depending on the nature of the study.

If the study is a design project, the description of the work done can contain reflections on what was learned during the project, what surprises emerged, what corrections to the approach were made, what insights were obtained and how, and how the framing of the design problem therefore changed. All these elements look at the design project from a higher viewpoint, compare it to its original goals, and help to navigate the project’s description towards its design solution or answer.

If the study is an open-ended qualitative study, the same element apply as above.

If the study is close-ended quantitative study, reflection may relate to outcomes that were unexpected and which therefore need further analysis.

Evaluation of the result. If the study contained a prototype that was tested, the test itself is a critical analysis of the design’s success, and needs to contain a reflection on whether the prototype successfully satisfied the original goals.

Additional analyses. If the findings contain surprises, the reasons for them can be discussed.

Discussion Expansion. After a nutshell summary of the study’s main findings, their wider discussion helps to contextualize them in relation to their environment. For example, if the finding presented a prototype, its suitability for a larger ecosystem of technologies, tools and human practices can be reflected.

Implications. Reflection on what further thoughts should be drawn from the findings.

Limitations. Reflection on the study’s shortcomings, to show that the author is able to understand the limits of their work, and point out what to focus on in the future to alleviate them.

If at least most, if not all, of these items are found in your  article or thesis, then it probably satisfies the needs of the readers who expect that the researcher is able to carry out reflection on their own research.

Concluding words for designers

The following words are written especially for designers, but they apply also to applied research fields in general, especially to those studies that have had exploratory, open-ended research questions (e.g., what is quite common in HCI).

Design is a discipline with more open-endedness than many other fields where academic texts are written. The design problems are more open-ended, for example, than in most scientific or engineering disciplines. This means that design processes can involve much more personal judgment and subjective decision-making in design than in these other fields. This is a reason why reflection is a particularly important part of writing about design: you need to be able to explain your design choices because there are no objective criteria on which other disciplines may be able to base their judgments.

One result from the requirement is that although design can be often visually represented (e.g., with pictures), it does not usually speak for itself: it is not guaranteed that the reader can understand what the designer has focused on in the design process, and how they have solved it. Therefore it rarely suffices to present a plain presentation of the starting point, a report about the stages of a project, and the outcome, such as a physical prototype. Instead, these stages require explanation that help the readers (or customers, in a company context), to appreciate the details that the design contains.

Reflection is a skill that can be practiced, and by remaining mindful about the principles and advices above, you will be better and better equipped with an ability to prove that what you have produced is based on solid, critical and productive analysis.

References

Buxton, B. (2007). Sketching User Experiences: Getting the Design Right and the Right Design. San Francisco, CA: Morgan Kaufmann.

Dorst, K. & Cross, N. (2001). Creativity in the design process: co-evolution of problem–solution. Design Studies, 22(5), 425–437.

Schön, D. A. (1983). The Reflective Practitioner: How Professionals Think in Action. New York, NY: Basic Books. 

Schön, D. A. (1992). Designing as reflective conversation with the materials of a design situation. Knowledge-Based Systems, 5(1), 3–14.

From table of contents to a finished text

Introduction

Writer’s block is a term for a feeling where a writer feels being stuck: unable to get anything written down or any progress made in the writing process. It is a frustrating feeling, and very hard to overcome. There are probably many reasons for the writer’s block. In this blog post I address one of them: the lack of process that would divide writing into meaningful, manageably sized steps.

I offer the following advice based on my personal experience. I have noticed that the steps described in this blog post work for me, and I am also able to explain why that is the case.

While some others may offer different guidance, since there can be many other ways to write and avoid writer’s block too, I am not alone with my ideas. Simon Peyton Jones from Microsoft Research Cambridge (https://www.microsoft.com/en-us/research/academic-program/write-great-research-paper/) illustrates the recommended process like this:

The diagram suggests that research should not be done before the paper has been written. But what if you already have the data and you need to write now? Is there anything useful in this blog post if the requirement is to write before doing the research (i.e., gathering data)? No worries: while Simon Peyton Jones seems to suggest that kind of a process, this blog post helps also in situations where the data already exists. You can find tips from Step 4 onwards below.

The research process (with emphasis on writing)

If I have to simplify, writing of an academic text (e.g., thesis, essay or a publication) is a process consisting of the following stages.

1. Exploration that finds a purpose for the study

This step can be a bit messy, but it has a clear goal: to identify possible contributions (i.e., findings) that my new study might make. This step can be very slow (even years) if there is no deadline, or it may be rushed, if my research has to be ready on a given date and I’m still even lacking all the data. In addition to thinking about possible contributions, this step also involves generation of possible research question(s) (RQs) and searching for background literature that can answer to them. It also includes the identification of research methods that would be suitable for that RQ. I need to adapt the contributions, RQs and methods until they are in alignment and seem to point towards a study that could produce some interesting as a result. This part’s outcomes often look like a collection of diagrams, lists, ideas, questions, etc., – that is, the outcomes do not look like a text at all yet.

I will not focus on Step 1 in this blog text. However, the linked posts above offer recommendations for dealing with this messy and open-ended early stage where the purpose gestates and gives the direction for the later steps.

2. Research plan

Now we get to the first step that resembles writing. In the previous step, I have managed to identify a promising contribution and RQs, as well as the methods that I can use to answer the RQs. I can therefore write a research plan for myself based on the outcomes from the Exploration step. This plan will be the outcome from this step. It has resemblance to the text that Simon Peyton Jones may mean in his “Write paper” step. This plan will have the sam contents as the beginnings of an IMRAD-style academic text has: an Introduction, a Related research section, and presentation of Methods.

In writing this text, I work in the same way as I describe below in Step 4 (Bullet point narrative).

At this point, the text may not be intended for anyone else’s eyes (unless I work as a part of a team). The plan can therefore quite sketchily written. The clarity of ideas is more important than how these ideas are expressed in words.

3. The empirical study

Now I gather the data that I have planned in the research plan. Especially in a qualitative study, this may overlap with the next steps in the process. I may adapt the focus of data collection if I notice needs for particular kinds of data.

4. A bullet point narrative

This step is the culmination point of the writing process. When I reach this point, I have at least some data that I have gathered, and it is time to analyse it and write about the findings.

If I have had a good plan and I have been able to follow it, I should now have data that I can use to answer the RQs, and reach the intended contribution. Usually the data does not have a perfect match with the intended outcomes, so I have to adapt my path, and find a slightly different contribution and RQs for which the data suits better. Maybe I have to change the type of contribution that my research is able to make, for example. Let’s assume that I manage to deal with this problem, and I identify the final, suitable contribution and the RQs.

A lot of uncertainty still exists even if I may have an idea about a finding that I could write about. For example, it is difficult to choose which order I should explain the things. I might wonder how much I should write about some background before I can tell a finding. Would that  sidetrack the text too much from the main plot that I want to tell? What if two things seem to require each other as their background? Which one should I write first?

Because of the puzzles like the ones above, it is important to minimize the time that I may  waste in writing long pieces of text that I may need to delete later. Therefore, the entire narrative of the text – from its very beginning to as far into to end – is better to write using bullet points.

Here is a mini-process of how this is done:

Preparations

If my research contribution has not significantly changed from the Research plan that I produced in Step 3, I can use that as my starting point for the paper. I can then extend that by adding the remaining section headings: Results/Findings and Discussion.

However, more commonly, the case is that my idea of the research contribution has changed because of the data that I collected and what my analysis found out. Then I will start from a new, empty document. I fill it with the main IMRAD section headings: Abstract, Introduction, Related research, Method, Results (or Findings) and Discussion.

Next, I fill the contents of each section with the names of the narrative elements (i.e., opening sentence, problematization, motivation, RQs, …) that most of the applied research follows. If I would not already remember these narrative elements, I could copy its contents from the table in my earlier blog post.

These preparations provide the main “skeleton” for the contents that I will need in different sections. This will help me find the appropriate place for different pieces of text that the text needs.

Writing using full-sentence bullets

After this preparation, I start the most significant part of the writing process. While until now I have been writing in first person, in this part I will switch to an imperative “you” style.

The idea is to work in a way that lets you make changes easily, and lets you focus on the content and logic. Therefore, the bullet points work excellently here. There are some rules that need to be followed, so that I don’t fool myself into thinking that I’m making progress while I’m not. Luckily the rules are quite simple:

Rule 1. Use only full-sentence bullets. These can be directives to yourself (“Explain here why interviewing would this time be the wrong method”) or claims (“Interaction design process involves a lot of trial and error.”). Aim for such a clarity that by reading the bullets, a knowledgeable outsider would be able to describe the logic of your paper.

This is why simple keywords (e.g., “participants”, “content analysis”, “limitations”) would not work: they would be only placeholders for content without the content itself. Writing sentences that express claims has the benefit that you are actually doing the mental effort of planning your logic, and working towards results. Keywords, instead, easily turn out to be to-do lists where actual thinking is postponed to some later time. Writing full sentences is harder, of course, but it is cost-efficient to do the thinking now than at a later time.

Rule 2. Do not leave logical gaps in the narrative. Try to find a solution to every open step: to the narrowing and scoping of your focus, to the selection of fields of literature that you are building on, to the selection of methods, and so on. These choices are things that your final text will need to explain, and now is the right moment to ensure that. Finding a way to justify all the decisions is easiest to do first using bullet points. In particular, pay attention to clear the openings and closings of each section and sub-section.

Rule 3. You can speculate about future results. It is clear that you don’t know everything about every part yet when you write these bullets. But that is not a problem: you can define goals for your analysis, for example, and writing down what you expect to find out or which you would like to write about. You can later see if your analysis lets you do that. You can also “foreshadow” these contents in earlier parts of the text, which makes the narrative stronger.

Therefore, write also speculative sentences. If you still lack information – for example, if you don’t have all the data for Results, or your Results are unfinished and you don’t know yet how to start your Discussion – sketch bullets nonetheless for these sections. These bullets are educated optimistic guesses on what you will tell about your research if things go well in your data collection and analysis. Here my work process and Simon Peyton Jones’s recommendations are again similar.

Rule 4. The narrative does not need to be perfect immediately. Don’t fall in despair! Writing bullets in this way is hard at times. This is because it is often hard to know how one idea should be followed with another one. You can soothe yourself, however, by thinking that these decisions are easiest to make using bullets. They are the least demanding way to find the best narrative for your text. It is easy to reorder them, delete them, and write new ones. The narrative does not need to be perfect – it is sufficient that you find one kind of narrative. Then you already have one plan for how to later write the full text. You can decide to improve the narrative later if you keep it in bullet point format.

When is this stage complete? The main goal in this stage is to generate a full overarching understanding about your entire written document. The resulting narrative should not have contradictions (e.g., starting with one research question but ending with a result that does not answer to it, but answers to something else).

Continue “massaging” your text’s bullet point outline until it meets the requirement above. Adapt the level of detail based on the clarity of your thinking: If you are confident that a certain sub-section will be easy to write, then bullets do not need to go into a detailed level. In contrast, if you are unsure about the narrative of a certain sub-section, use second-level and third-level indented bullets and clarify to yourself how that section can be written.

When this stage is finished, you have a narrative that only waits to be transformed into actual paragraphs. The most important work for the text has been completed!

Example

Below is an authentic example of bullet-style writing from a draft that we eventually developed into a paper that we presented in CHI 2017 conference. You are seeing our plan for a sub-section in the background section from an early stage of the writing process.

Tip

If you use Microsoft Word, make yourself familiar with Ctrl-Shift-arrows (Mac) or Alt-Shift-arrows (Windows) keyboard shortcuts. With up and down arrows, you can reorder entire paragraphs easily. With left and right arrows, in turn, you can quickly change the level of indentation in your bullet points.

5. From bullets to full text

After finishing the bullet point plan, the hard part of writing the paper is over. This is because the bullet point version already outlines for me how to tell the narrative, present the evidence and the findings, and other important matters in the text.

I can now take any bullet point and turn it into approximately one paragraph of final-looking text. I can work on any part I wish because I know what has been said before and what will be said later. Therefore, depending on the mood and inspiration, I can work on different parts of the text on different days, whichever seems best.

The first final-looking paragraphs of text do not need to be perfect. I usually do not mind about detailed terminology on this stage yet, for example. I focus on writing down the text so that it preserves the idea in the bullet point, even if the expressions are clumsy and have repetition.

6. Polishing

Once I have written the first draft of the final-looking text, I can focus on improving its language. I straighten the terminological confusions by replacing terms so that I always use the same term in the text. I also focus on writing better sentences that flow nicely. This step usually requires several rounds, but they are fairly stress-free.

Then the text is ready.

Conclusions

The process described in this blog post – especially the recommendation to use bullet points wisely as a vehicle for thinking – are a way to advance in small steps towards a finished article. Although it is not guaranteed, the writer’s block may be better kept away, since there is never a moment when you would need to attempt writing finished-looking from scratch.

Bullet-point style writing has an additional benefit for situations that are common in academic writing: which is collaborative authoring. In many areas or research, academic publications are often written in a team. The writing process is possible to orchestrate using the same bullet point style as individual authoring: one author may take the responsibility for thinking through the paper’s entire storyline, and write bullet points for it. Then different parts can be assigned for different people, and the team members can write their sections independently, with an awareness of how their texts will relate to the whole. This is worth of its own blog post at a later time, but I wanted to mention it here as an opportunity.

Acknowledgments

I want to thank Heidi Paavilainen for informing me about Simon Peyton Jones’s text.

How to get access to articles that are not Open Access

Although there is a strong push within the academic community to make research freely accessible (“open access”) to everyone, many research papers are unfortunately still published and available “behind the paywall”. That means that the publisher of the journal, book or a conference proceedings requires someone to subscribe the content. Typically that payer is a university library, who then can make the content accessible to the university students and employees.

But making use of this subscription can be complicated. Even if the seeker for papers – such as a student or a researcher – would have an affiliation with a university, it is not self-evident how one can get past the paywall and download the paper.

This post presents ways to get through the paywall, but requires that you have a user account in a university.

The focus of this post builds on an assumption that the seeker’s main literature search tool is Google Scholar, due to its simple use and excellent coverage of different research papers.

Google Scholar with basic internet connection: limited access to “full texts”

It is important to notice that Google Scholar provides access to articles differently depending on the type of internet connection. The difference shows in the links that Scholar displays in its search results.

When a user searches papers with Google Scholar with an ordinary internet connection, Google Scholar helps find papers, but cannot always provide the user with the “full text”. By full text, publishing companies mean the PDF that contains the article in its entirety. If access is not full text, then the user can only see the title, abstract, references, and maybe the first page of the paper.

It is sometimes possible to get an access to a full text even with this internet setup. This happens if:

  • The text has been published as Open Access (OA) or with a permission that the authors are allowed to upload a copy of the text to their personal public repository of papers.
  • Someone has uploaded the text somewhere in the Internet even if it is is not OA, thus possibly breaking the copyright of that content. Google has then found that paper.
  • The paper’s preprint or so-called “accepted manuscript” version is available from the authors of the paper. A pre-print is often an earlier version of the paper: similar or even identical in its content with the final one, but not copy-edited to the final layout, and therefore lacks correct page numbers, and may have minor typing errors.

The screenshot below shows an example of what a Google Scholar search can produce as an outcome.

From the screenshot, different kinds of links can be seen. They provide different levels of access to the content:

  • [PDF]: full text can be downloaded from the linked web page
  • [HTML]: full text can be usually downloaded, but not always
  • Getit@Grifols: this is a link whose purpose is not clear to me. No full texts available, anyway.
  • The link of the paper title: takes the user to the publisher’s website. Full text may or may not be downloadable from it.

As a base rule, if the text does not have any links in the right-side column, the full text PDF is not available from any source.

Google Scholar with university’s VPN connection

The access to papers can dramatically improve if you tunnel your Internet connection through a university’s VPN service. With VPN, all the Internet traffic from a computer will travel (i.e., is “tunneled”) through a designated server. Both Google Scholar and the publishing companies then recognize that the user is connecting them from an Internet address that has a right to access also subscribed contents.

It therefore makes sense to use VPN in using Google Scholar. Here is how the same search results look like with VPN. Note that the two previously inaccessible papers are now downloadable via a “sfx@Aalto” link. Sadly, Klein & Weitzenfeld’s text in Educational Psychologist still remains unaccessible: my university does not have a subscription to its contents:

I cannot be sure, but I believe all the universities provide a similar service as Aalto University does, and therefore VPN opens doors to papers (for those who have a university account, of course). Here are the instructions for Aalto users for how you install the VPN client on your computer.

Accessing papers without VPN

It is not necessary to use VPN to access texts, however. An alternative is to navigate to a university library’s search interface and download the paper through it. Every library works slightly differently in service its users. At Aalto University, it is possible to enter e.g., the paper’s title, and the service tries to find a matching piece of content from its digitally subscribed sources. Aalto University’s article search interface is here.

A good idea is to use the text’s DOI (digital object identifier) as the search term instead of a paper’s name, because otherwise you may get lots of unnecessary search hits too. DOI uniquely identifies the paper, and can be always found somewhere from the publisher’s website.

When you find the desired search result, click on the links that promises to take you to the electronic full text. At that stage you will need to provide your user name and password, to prove that you are entitled to access the content.

Another Aalto-specific tip is to use the list of digital paper libraries in libproxy.aalto.fi. If you know the publisher of a paper your are interested in, you can go to that library via a link in the libproxy page. As long as you browse papers in that library, you are surfing within the paywall, and will be able to download full texts.

Paper request from ResearchGate

As the last resort, you may go to ResearchGate or Academia.edu. These are services where researchers can upload their works and create profile pages. If a researcher has uploaded the paper to the service, ResearchGate/Academia.edu provides a feature where you can ask a researcher to send a copy of the paper to you privately.

A word of advice: It is best to use this feature only after you have tried the other possibilities above and they have failed. By my personal experience, it is always irritating to react to ResearchGate’s paper requests if the requested paper is also available as Open Access. A request in such a situation only shows that the requester has not spent even a minimal bit of effort to find it. I get paper requests from ResearchGate approximately once a week. Much more cited researchers therefore probably get several requests each day.

Final words

As can be seen, retrieval of a text can require quite a bit of work. It is better to minimize the amount of work that you need to spend in downloading. Therefore, always download and store the paper on your computer! Also make sure that you can easily find that paper later on from your computer. For example, use a systematic file-naming principle (e.g., authorname year paper title.pdf) for all the papers, or start using reference manager programs such as Mendeley or Zotero. That will save you time next time, and lets you annotate the texts that you read with your own observations.

Acknowledgments

Thanks to Markku Reunanen for informing about the digital library listing at libproxy.aalto.fi.

How the most useful background literature can be found?

Introduction

Knowing and following what others have written about (and around) your own research topic is the basic requirement for any academic project. But how should this be done? Some things are obvious: this task involves, at least 1) searching for possible texts to read; 2) scanning them to filter the good ones out from the mass of less relevant ones; 3) reading the most promising ones; and 4) building syntheses about this mass of texts.

The list is rather simple, but it is clear a lot of complexity underlies these steps. Some of the questions are:

  1. What tools should be used for finding texts?
  2. What types of texts are out there?
  3. What indicators reveal what texts are more credible than others?
  4. How these texts can be downloaded?
  5. How should the article collection be maintained?
  6. How can the synthesis be generated out of a collection of texts?

This blog post focuses only on questions 1–3 and a bit on question 4. I have answered to question 4 also separately in a different post. The others may be looked at later.

1. What tools should be used for finding texts?

At least the following ways exist for finding literature:

1. Standard Google search helps you find a very eclectic mix of texts that have a varying quality and underlying intentions. Very few of the papers found this way are academic papers. Instead they can be, for example, memos written by thinktanks and lobbyist groups, governmental bodies’ reports, press releases, essays written by students. Occasionally, also academic papers can be found with standard Google search, but what one finds is very unpredictable.

While what one finds using this method may be useful, it is usually best to regard contents found this way more as “data” rather than as research knowledge.

2. Google Scholar. This is the best tool for exploratory search for literature: for those situations where you want to find out “what is out there”. Using Google Scholar is like using standard search, but the results are different: they are from academic sources, such as journals, conferences, and books. The search results also contain additional information that help you interpret which papers are better to investigate in more detail

Because Google Scholar is a great tool, its use is covered in more detail below.

3. Snowballing. Academic papers always contain a section called References that lists all the other research that has been cited. By reading the paper and finding out what earlier works are cited, and what the writers say about them, it is possible to get to the sources of knowledge. This helps you find the “must-reads” of the research topic. The problem is that snowballing works only backwards in time: it does not help you find the most recent research.

4. Content alerts. It is possible to ask journals and conferences to send email to you every time they publish a new issue. This is a good way for staying up to date on the most recent research. The problem is that this brings a lot of email to your mailbox, and every issue does not contain articles that you would be interested about. Google Scholar, however, lets the user create keyword-based alerts: send you email every time it finds new research that matches given keywords. You can turn on this feature by clicking on Create alert button in Google Scholar in the left side of the screen (see the image below).

For a PhD students and researchers who need to keep themselves up to date about a research area over a long period of time, this is an essential feature to use.

5. Databases. EBSCOHost, Proquest, ABInform, IEEExplore, Scopus, ACM Digital Library and other databases are great for systematic literature reviews when you know exactly what keywords to use and what journals and conferences to include in your search. But because each database only covers certain journals/conferences, and usually disregards books completely, they are not optimal for exploratory search for knowledge. For that, Google Scholar is much better and nicer to use too.

2. What types of texts are out there?

Given that I recommended Google Scholar as the primary tool for searching texts, I will focus on its use from now on more than on the others.

Academic peer-reviewed papers

In the above, the main characteristic that I mentioned as the difference between the standard Google search and Google Scholar search was that the latter one finds only “academic papers” instead of just any texts or search hits. It is therefore important to define what is unique in academic papers.

The main characteristic of an academic paper is that it is “peer reviewed”. This means that there is a particular editorial process (“review process”) that the paper has undergone before it has been published in a journal or a conference. In this process, It has been examined by a jury of researchers and the authors have had to improve the paper until it has met the necessary quality requirements. Without an exception, the process includes at least one cycle of improvements: the authors have first sent (“submitted”) their paper for review, the members of the jury have evaluated it by writing statements about it, and the authors have been asked improve the paper. Alternatively, the authors have received a “reject” meaning that this process is terminated, and they have to find a different journal/conference that may be willing to publish the paper. They have to start the process again with that other outlet. If the paper, however, was considered promising enough, the second cycle starts when the improved version is received from the authors. The reviewers will evaluate whether the changes are sufficient, and provide further comments. In conferences, one cycle is common; in journals at least two cycles is the norm. In every stage, the possibility of a reject is always possible.

Usually the review process is “double-blind”: the authors do not know who will read their paper, and the members of the jury (i.e., “reviewers”) do not know whose paper they are reading. The communication between the authors and the reviewers is handled by an editor who is a senior researcher in the field, and is responsible for keeping up high standard of this process. The blindness increases the neutrality of the process: even famous academics’ papers can be rejected, and the reviewers do not need to face the consequences of furious authors who are angry at the rejection of their paper. Most papers are rejected; good conferences typically reject 70-75% of the submissions, for example. Top journals reject a larger percentage than that.

The heaviness of the whole process makes paper publishing a slow business. To publish a paper in a good journal often takes at least 2 years, with 3–4 cycles of improvement. But it ensures much better quality for the content, compared to papers that have not had a review process. Thus puts the academic papers apart from other materials that standard Google search can offer.

Books and book chapters

Books are another common type of academic texts. They exist in two kinds: full books that have been written by the same group of people from the beginning to the end, and edited collections where different chapters have been written by different authors. Edited collections have editors who have gathered the texts together and have usually had at least some form of peer review process in the book chapters’ preparation.

Other sources of academic-like texts

There are also semi-academic papers: ones that have been written by researchers, but which have not undergone the review process. These include research institutions’s “white papers” and technical reports, as well as texts that accompany presentations given in research seminars and workshops. These texts are usually published only in a website, instead of in a journal or a conference proceedings.

There are also papers that have been submitted for a review, but which have also been saved in a public repository such as Arxiv, Biorxiv, Citeseer or SSRN. Although doing so breaks the blind review policy, in some fields of science this is accepted and widely used practice. One of the reasons for this practice is the competition within the scientific field: researchers compete for being the first ones to make a certain finding. They do not want to wait the 2 years in the review process before they can tell about the finding. They may also fear that an anonymous reviewer steals their idea, replicates the study, and publishes it as their own. Public archiving protects authors from that.

Finally, papers are also available from ResearchGate and Academia.edu. These are self-archiving repositories where researchers sometimes upload copies of their published works, or where they just publish their research, thereby bypassing the review process. The quality of the content in ResearchGate and Academia.edu varies wildly, and needs to be verified: has the paper been published somewhere, or has it been only uploaded here?

3. What indicators reveal what texts are more credible than others?

It so far seem that just using Google Scholar ensures that every text has the required quality and can be used as a good piece of literature. The truth is not that simple: there are conferences and journals with different levels of quality. Some papers, even if they are per-reviewed, have low quality. Using just any source that one finds may lead to 1) misleading directions; 2) unnecessary amount of work.

There are three simple indicators for finding out which paper is more worthwhile to read than others:

The exact topic of the paper

This is the simple one: it is better to read papers whose titles and abstracts have a good fit with the information that one is looking for. Google Scholar presents the titles of the papers very clearly. In addition, the abstract of the paper can be inspected by clicking at the title. It either shows a popup window or takes the user to the publisher’s website.

The number of citations

Good papers end up usually cited more often than others by other researchers. The number of citations is the total count of all the other papers that cite a given paper. In the following screenshot, for example, Google Scholar tells that the 4th search result has been cited 2325 times by other researchers while other papers have been cited much less. This tells that Dorst and Cross’s paper published in Design Studies is probably more appreciated by researchers than other papers, when it comes to “framing in design process” as the topic.

Screenshot of a Google Scholar search result

Example of a search result in Google Scholar.

Citation count is a good indicator for choosing which papers are “must reads” and give the most relevant information.

The quality of the journal or conference

Although the citation count is a good indicator, it works poorly especially in the evaluation of the importance of very recent research. Recent publications have not had a chance to accumulate citations yet, and seem therefore less relevant. In addition, sometimes there are no publications that would be highly cited, because the research area that the user is interested about is very particular and not much researched.

Then the user should look at the quality of the journal or conference that has published the research. For most journals, it is possible to find what its impact factor is. It is a value that is computed by the number of citations that the papers in the journal gather on average over time. Clarivate Analytics’ JCR (Journal Citation Records) is the most often used impact factor service. It was earlier known as Thomson Reuters. JCR is not publicly accessible: one needs to access it through an university library. Aalto University users can click here to access JCR.

The impact factors for journals range from 0 to several dozens. For example, in the top, impact factors for New England Journal of Medicine, Lancet, Nature and Science are currently 75, 60, 43 and 42, respectively. The problem with the impact factors is that in other fields the best journal may have much lower impact factors. In HCI, Human-Computer Interaction has the highest impact factor, which is currently 4.2. In design research, Design Studies is the leading journal, and its impact factor is 2.8. Design Issues – another good one – is not listed at all, surprisingly. These differences do not mean that design or HCI journals would be of poorer quality than natural science journals – fields cannot be compared based on their journals’ impact factores. Many factors affect the impact factor, including the publication volume in the field, peer competition, the status of conferences or books as reputable publishing outlets, and the centeredness of the field around only a handful of journals, for example.

All this just means that impact factors are meaningful only if one knows already what the range the values is in a given field. In addition, Clarivate Analytics’ JCR does not provide impact factors for conferences, which makes its relevance to HCI much less meaningful.

A better approach, at least in Finland, is to use Finland’s own academic ranking system called “JUFO” – short for “Julkaisufoorumi”. It ranks every journal and conference using 4 levels:

  • 3 = the top journal/conference in its own field
  • 2 = a really good journal or conference
  • 1 = other journals and conferences that have a peer review process
  • 0 = journals and conferences that are known to exist but which cannot prove that they follow the sufficient academic review standards

Generally speaking, any paper published in journal or conference of level 2 or 3 has content that can be considered seriously. Many level 1 outlets are also really good, but there the quality varies a lot. Level 0 conferences and journals should not be used as references. JUFO can be accessed here.

Summary

When you search for literature, you can use the following process:

  1. Try different kinds of search terms in Google Scholar. Often you do not manage to use the best search terms at the first attempt.
  2. When you seem to be getting promising results, look at the titles and the citation counts: they tell which papers are 1) best matches with your interests and 2) most valued by other researchers.
  3. If all citation counts are low, look at the outlets: which conferences and journals have published these works? Prioritise ones whose JUFO ratings are 2 or 3. Consider also ones that have a JUFO 1 rating.
  4. Download every promising paper on your computer.

4. How these texts can be downloaded?

The last step above involves a challenge that will be addressed in the blog post: a vast majority of academic papers is not freely available. Instead they are available from publishers who sell them to universities with a subscription fee. To see, download and read them, one needs to use a university’s authenticated Internet connection.

I have written about this separately too, but my quick advices are to 1) look for links that Google Scholar marks with [PDF] – those are freely accessible papers; 2) tunnel your internet traffic through a university’s VPN service. That makes the publishers open most of the doors for you. Then you can find links with “sfx” in them – they are contents that your university has subscribed; 3) use your university’s article search service: copy the paper title and paste it to the university’s search engine. If the paper can be accessed, you get a link where you can download the paper. Here is the link to Aalto university’s search interface.

See how the same search result page as above has changed when I have used VPN:

Screenshot of Google Scholar's results when using VPN connection

Read also my other blog post to find out how to access and download articles when they are not openly accessible.

How to define a research question or a design problem

Introduction

Many texts state that identifying a good research question (or, equivalently, a design problem) is important for research. Wikipedia, for example, starts (as of writing this text, at least) with the following two sentences:

“A research question is ‘a question that a research project sets out to answer’. Choosing a research question is an essential element of both quantitative and qualitative research.” (Wikipedia, 2020)

However, finding a good research question (RQ) can be a painful experience. It may feel impossible to understand what are the criteria for a good RQ, how a good RQ can be found, and to notice when there are problems with some RQ candidate.

In this text, I will address the pains described above. I start by presenting a scenario of a project that has problems with its RQ. The analysis of that scenario allows me then to describe how to turn the situation described in the scenario for a better research or design project.

Scenario of a problematic project

Let us consider a scenario that you are starting a new research or design project. You have already an idea: your work will be related to communication with instant messaging (IM). Because you are a design-minded person, you are planning to design and develop a new IM feature: a possibility to send predefined replies on a mobile IM app. Your idea is that this feature will allow the user to communicate quickly with others in difficult situations where the they can only connect with others through their mobile phone. Your plan is to supply the mobile IM app with messages like “I’m late by 10 minutes but see you soon”, “I can’t answer back now but will do that later today”, and so on.

Therefore, your plan involves designing such an app, maybe first by sketching it and then illustrating its interaction with a prototyping software like Figma or Adobe XD. You may also decide to make your design functional by programming it and letting a selected number of participants to use it. These kinds of activities will let you demonstrate your skills as a designer-researcher.

Although predefined messages for a mobile IM app can be a topic of a great study, there are some problems with this project that require you to think more about it before you start. As the project is currently defined, it is difficult to provide convincing answers to these challenges:

  • Challenge 1: Why would this be a relevant topic for research or design? Good studies address topics that may interest also other people than the author only. The current research topic, however, does not do that self-evidently yet: it lacks an explanation why it would make sense to equip mobile IM apps with predefined replies. There is only a guess that this could be useful in some situations, but this may not convince the reader about the ingenuity of this project.
  • Challenge 2: How do you demonstrate that your solution is particularly good? For an outsider who will see the project’s outcome, it may not be clear why your final design would be the best one among the other possible designs. If you propose one interaction design for such a feature, what makes that a good one? In other words, the project lacks a yardstick by which its quality should be measured.
  • Challenge 3: How does this project lead to learning or new knowledge? Even if you can show that the topic is relevant (point 1) and that the solution works well (2), the solution may feel too “particularized” – not usable in any other design context. This is an important matter in applied research fields like design and human–computer interaction, because these fields require some form of generalizability from their studies. Findings of a study should result in some kind of knowledge, such as skills, sensitivity to important matters, design solutions or patterns, etc. that could be used also at a later time in other projects, preferably by other people too.

All these problems relate to a problem that this study does not have a RQ yet. Identifying a good research question will help clarify all the above matters, as we will see below.

Adding a research question / design problem

RQs are of many kinds, and they are closely tied to the intended finding of the study: what contribution should the study deliver. A contribution can be, for example, a solution to a problem or creation of novel information or knowledge. Novel information, in turn, can be a new theory, model or hypothesis, analysis that offers deeper understanding, identification of an unattended problem, description about poorly understood phenomenon, a new viewpoint, or many other things.

The researcher or thesis author usually has a lot of freedom in choosing the exact type of contribution that they want to make. This can feel difficult to the author: there may be no-one telling what they should study. In a way, in such a situation, the thesis/article author is the client of their own research: they both define what needs to be done, and then accomplish that work. Some starting points for narrowing down the space of possibilities is offered here.

Most importantly, the RQ needs to be focused on a topic that the author genuinely does not know, and which is important to find out on the path to the intended contribution. In our scenario about a mobile IM app’s predefined replies, there are currently too many alternatives for an intended contribution, and an outsider would not be able to know which one of them to expect:

  1. Demonstration that mobile IM apps will be better to use when they have this new feature.
  2. Report on the ways by which people would use the new feature, if their mobile IM apps would have such a feature.
  3. Requirements analysis for the specific design and detailed features by which the feature should be designed.
  4. Analysis of the situations where the feature would be most needed, and user groups who would most often be in such situations.

All of these are valid contributions, and the author can choose to focus on any one of them. This depends also on the author’s personal interests. This gives a possibility for formulating a RQ for the project. It is important to notice that each one of the possible contributions listed above calls for a different corresponding RQ:

  1. Do predefined replies in mobile IM apps improve their usability?
  2. How will users start using the predefined replies in mobile IM apps?
  3. How should the interaction in the IM app be designed, and what kind of predefined replies need to be offered to the users?
  4. When are predefined replies in IM apps most needed, and by whom?

This list of four RQs, matched with the four possible contributions, shows why the scenario presented in the beginning of this text was problematic. Only after asking these kinds of questions one is able to seek to answer to the earlier-presented three challenges in the end of the previous section. Also, each of the RQs needs a different research or design method, and its own kind of background research.

The choice and fine-tuning of the research question / design problem

Which one of the above RQs should our hypothetical researcher/designer choose? Lists of basic requisites for good RQs have been presented in many websites. They can help identify RQs that will still need refinement. Monash University offers the following kind of helpful list:

  1. Clear and focused. In other words, the question should clearly state what the writer needs to do.
  2. Not too broad and not too narrow. The question should have an appropriate scope. If the question is too broad it will not be possible to answer it thoroughly within the word limit. If it is too narrow you will not have enough to write about and you will struggle to develop a strong argument.
  3. Not too easy to answer. For example, the question should require more than a simple yes or no answer.
  4. Not too difficult to answer. You must be able to answer the question thoroughly within the given timeframe and word limit.
  5. Researchable. You must have access to a suitable amount of quality research materials, such as academic books and refereed journal articles.
  6. Analytical rather than descriptive. In other words, your research question should allow you to produce an analysis of an issue or problem rather than a simple description of it.

If a study meets the above criteria, it has a good chance of avoiding a problem of presenting a “non-contribution”: A laboriously produced finding that nonetheless does not provide new, interesting information. The points 3 and 6 above particularly guard against such studies: they warn the readers from focusing their efforts on something that is already known (3) and only describing what was done or what observations were made, instead of analysing them in more detail (6).

In fine-tuning a possible RQ, it is important to situate it to the right level. For example, the first RQ that one can think of may not be the best one, such as being too broad. RQ4 above (“When are predefined replies in IM apps most needed, and by whom?”), for example, may be a very relevant question, but too broad if one is wishing to do a design-oriented thesis or a project. In such cases it is permitted, and also recommended, to limit the RQ to something more detailed by adding justified assumptions.

In our example scenario, the researcher/designer can decide, for example, that they will consider predefined IM replies from the viewpoint of “face-work” in social interaction. By adopting this viewpoint, the researcher/designer can decide that they will design the IM’s replies with a goal that they help the user to maintain an active, positive image in the eyes of others. When they start designing the reply feature, they can now ask much more specific questions. For example: how could my design help a user in doing face-work in cases where they are in a hurry and can only send a short and blunt message to another person? How could the predefined replies help in situations where the users would not have time to answer but they know they should? Ultimately, would the predefined replies make it easier for users to do face-work in computer-mediated communications (CMC)?

These kinds of follow-up questions that arise from adoption of a particular viewpoint (in this case, face-work) illustrate the benefits of framing the question in a narrower manner, as long as the narrowing can be justified. In this case, the thinking behind the design can be built on sociological research on interpersonal interaction and self-presentation (e.g., Goffman 1967) and its earlier applications to CMC (Nardi et al., 2000; Salovaara et al., 2011). Such a literature provides the starting points for deeper design considerations. Deeper considerations, in turn, increase the contribution of the research, and make it interesting for the readers.

As said, the first RQ that one comes to think of is not necessarily the best and final one. The RQ may need to be adapted (and also can be adapted) over the course of the research. In qualitative research this is very typical, and the same applies to exploratory design projects that proceed through small design experiments (i.e., through their own smaller RQs).

Conclusion

This text promised to address the pains that definition of a RQ or a design problem may pose for a student or a researcher. The main points of the answer may be summarized as follows:

  • The search for a good RQ is a negotiation process between three objectives: what is personally motivating, what is realistically possible to do (e.g., that the work can be built on some earlier literature and there is a method that can answer to the RQ), and what motivates its relevance (i.e., can it lead to interesting findings).
  • The search for a RQ or a design problem is a process and not a task that must be fixed immediately. It is, however, good to get started somewhere, since a RQ gives a lot of focus for future activities: what to read and what methods to choose, for example.

With the presentation of the scenario and its analysis, I sought to demonstrate why and how choosing an additional analytical viewpoint can be a useful strategy. With it, a project whose meaningfulness may be otherwise questionable for an outsider can become interesting when its underpinnings and assumptions are explicated. That helps ensure that the reader will appreciate the work that the author has done with their research.

In the problematization of the scenario, I presented the three challenges related to it. I can now offer possible answers to them, by highlighting why a RQ can serve as a tool for finding them: 

  • Why would this be a relevant topic for research or design? Choice of a RQ often requires some amount of background research that helps the researcher/designer to understand how much about the problem has already been solved by others. This awareness helps shape the RQ to focus on a topic where information is not yet known and more information is needed for a high-quality outcome.
  • How do you demonstrate that your solution is particularly good? By having a question, it is possible to analyse what are the right methods for answering it. The quality of executing these becomes then evaluatable. The focus on a particular question also will permit that the author compromises optimality in other, less central outcomes. For example, if smoothness of interaction is in the focus, then it is easy to explain why long-term robustness and durability of a prototype may not be critical.
  • How does this project lead to learning or new knowledge? Presentation of the results or findings allows the researcher/design to devote their Discussion section (see the IMRaD article format) to topics that would have been impossible to predict before the study. That will demonstrate that the project has generated novel understanding: it has generated knowledge that can be considered insightful.

If and when the researcher/designer pursues further in design and research, the experience of thinking about RQs and design problems accumulates. As one reads literature, the ability to consider different research questions becomes better too. Similarly, as one carries out projects with different RQs and problems, and notices how adjusting them along the way helps shape one’s work, the experience similarly grows. Eventually, one may even learn to enjoy the analytical process of identifying a good research question.

As a suggestion for further reading, Carsten Sørensen’s text (2002) about writing and planning an article in information systems research field is a highly recommended one. It combines the question of choosing the RQ with the question on how to write a paper about it.

References

Goffman, E. (1967). On face-work: An analysis of ritual elements in social interaction. Psychiatry, 18(3), 213–231.  https://doi.org/10.1080/00332747.1955.11023008

Nardi, B. A., Whittaker, S., & Bradner, E. (2000). Interaction and outeraction: Instant messaging in action. In Proceedings of the 2000 ACM Conference on Computer Supported Cooperative Work (CSCW 2000) (pp. 79–88). New York, NY: ACM Press. https://doi.org/10.1145/358916.358975

Salovaara, A., Lindqvist, A., Hasu, T., & Häkkilä, J. (2011). The phone rings but the user doesn’t answer: unavailability in mobile communication. In Proceedings of the 13th International Conference on Human Computer Interaction with Mobile Devices and Services (MobileHCI 2011) (pp. 503–512). New York, NY: ACM Press. https://doi.org/10.1145/2037373.2037448

Sørensen, C. (2002): This is Not an Article — Just Some Food for Thoughts on How to Write One. Working Paper. Department of Information Systems, The London School of Economics and Political Science. No. 121.

Wikipedia (2020). Research question. Retrieved from https://en.wikipedia.org/wiki/Research_question (30 November 2020).

Typology of possible research findings (i.e., “contributions”)

Introduction

A good academic text delivers a clear and interesting message. That is often described as “contribution”. Good contributions teach something to the text’s readers: they change the reader’s way of thinking or acting, and increase their understanding and knowledge about an interesting subject. Thus, “a contribution is made when a manuscript clearly adds, embellishes, or creates something beyond what is already known” (Ladik & Stewart, 2008, p. 157). Such findings therefore present something that the researchers did not know so far; that is what makes the research article interesting.

Also a BA or a MA thesis deliver contributions. But their requirements for significant contribution are lower, as it is not crucial that a finding of a thesis should be a considered as a research contribution that generates novel understanding in the scientific community.

So, what are the possible findings and contributions that academic texts can make? Understanding what a contribution can be becomes easier over time, as one reads more literature and sees more examples of academic publications. But to get started more quickly, this article presents some classifications that I have found from other researchers’ writings, and finally presents a longer list that I have tailored for the fields of design and HCI.

The presentation of contributions in this article is in two parts: first I will discuss academic articles, and then I will add notes about BA/MA theses in HCI and design.

Types of findings and contributions in academic articles

Contributions can be classified along several dimensions. Some of the existing classifications are oriented to theoretical contributions. For example, Ladik and Stewart’s (2008) “contribution continuum”, written for a marketing research audience, divides the possible contributions to 8 classes, organized from minor to fundamental scientific impacts:

  1. Straight replications: studies that verify whether a finding that has been already published can be repeated.
  2. Replication and extension: similar to the one above, but with an adjustment.
  3. Extension of a new theory/method in a new area.
  4. Integrative review (e.g., meta-analysis).
  5. New theories to explain an old phenomenon, possibly also including a comparison between an existing and the new theory against each other to find out which one works better.
  6. Identifications of new phenomena worth of attention.
  7. Grand syntheses that integrate earlier theories together.
  8. New theories that predict new phenomena.

In addition to presenting the continuum, Ladik and Stewart’s (2008) text is great also in emphasizing many other characteristics of good academic texts too, such as a need to think about the target reader audience, need to emphasise surprise, and demonstrate passion and relevance of the topic that has been studied.

In human–computer interaction (HCI), which is more oriented to human-created objects, other kinds of contributions can be recognized. Wobbrock (2012) and Wobbrock and Kientz (2016) do not define the contributions based on their magnitude, but in terms of types of outcomes. As we can see, the theoretical contributions that were listed above are only one possibility in applied fields such as HCI and design:

  • empirical research findings (e.g., what factors and phenomena play an important role in different situations where people use technologies)
  • artefacts (i.e., designs and technologies)
  • methods
  • theories
  • datasets
  • surveys and reviews of existing research
  • opinions

In this classification, Wobbrock and Kientz’s papers themselves could be best classified as survey-like contributions, since their focus is on reviewing the kinds of research contributions in a research field as a whole. In this sence they synthesize together and explicate the practices in the field. In addition to being more directly useful also for HCI/design, Wobbrock’s suggestions are also great because both texts list papers from HCI research that exemplify these contribution types.

What is particular in the list above is the role of artifacts as research contributions. This is particular since it highlight’s HCI’s (and also design’s) nature as a “problem-solving” science (Oulasvirta and Hornbæk, 2016): in addition to producing the traditionally well-acknowledged empirical and conceptual (theoretical) contributions, HCI researchers also make constructive contributions by developing new technologies and designs.

Final distinction between contributions is their level of critical stance towards earlier research and practice. Most contributions are knowledge-increasing: they present new findings, expand the research to new areas, make existing theories and methods more detailed, accurate or more appropriate for some context, for example. These contributions are really common: with my colleagues we found, for example, that 94% of research papers in information systems research are knowledge-increasing (Salovaara et al., 2020). Many of the contribution classes presented in the lists above are like these too.

Other contributions are knowledge-contesting: they identify problems in the existing theories and methods, or in the practices by which they are used (Salovaara et al., 2020). They may also identify limits (“boundary conditions”) to the extent to which earlier contributions can be applied. In the spirit of science and research being a self-correcting process, the purpose of these knowledge-contesting contributions is to correct earlier mistakes in research and keep the research on the right track.

To summarise the considerations above, the following table presents a synthesis of possible contributions in HCI and design. A vast majority of the papers represent one (or sometimes several) of these contributions:

Contribution type Description Example
Boundary condition E.g., finding that there is a hard limit in some theory or a method that cannot be overcome by following the existing strategies. In mobile settings, users are usually not able to attend to their mobile phones more than 4 seconds at a time (Oulasvirta et al., 2005).
Demonstration of novel possibilities E.g., a new technology that has not been possible to build before Machine learning can help web designers develop better user interface layouts (Todi et al., 2016).
Extension to a new field E.g., adapting an approach to a new context What HCI research can offer for sexual wellbeing (Bardzell and Bardzell, 2011).
Falsification Demonstration that an accepted theory/belief is not true or has limited generalizability User interfaces should not be designed by having a premise that human behaviour is always planful. That is because of the situatedness of human action. Communication breakdowns between humans and technologies happen are inevitable, and assuming planful behaviour increases their severity (Suchman, 1987).
Incremental improvement Presentation of a solution that outperforms earlier approaches In mobile maps, it is better to visualize off-screen targets with triangle-like shapes (“wedges”) than with earlier-recommended arrows or radius-based circles (Gustafson et al., 2008).
Introducing another research field to one’s own field’s researchers, and showing that it can help in solving an interesting problem Presentation of a research field whose importance has not been noticed but which offers a lot of value Information foraging theory: that the theories about animals’ food-hunting can be applied to humans’ ways of searching for information in information spaces (Pirolli and Card, 1995).
New method Description of a method that is useful in many situations “Wizard of Oz” method: Futuristic technologies that cannot be built yet may be realistically studied if a human plays the computer’s part without the user’s awareness (Gould, 1983)
Novel concept E.g., an important phenomenon that needs to be remembered in future research and design ”Plausible deniability”: In human-human communication through a technology (e.g., a chat), technology should offer a possibility for humans to remain non-responsive without fear of “losing their face” (Nardi et al., 2000)
Sense-making E.g., an open-ended study that reports and analyses a phenomenon that is so far poorly understood What conversation strategies do online trolls use to derail and harm online conversations (Hardaker, 2013).
Synthesis / meta-analysis / review Summary of existing knowledge in the field Review of literature about the differences in novice and expert designers’ design processes (Cross, 2004).
Recommendations / guidelines Description by an experienced researcher or researcher team on how to use a certain method correctly. How qualitative data should be analysed using thematic analysis (Braun & Clarke, 2006).
Research agenda / manifesto Call for researchers to start addressing a neglected issue Call for attending to the widely spread misunderstanding and misuse of the term ”affordance” in much of design practice (Torenvliet, 2003).

This list is not comprehensive, and some areas have been covered in more detail than others. What is however notable in this list’s items is that papers about these contributions can be written using the same narrative format. That is because most of these contributions require a study: some method by which some material is analysed so that findings can be presented. Such papers can be readily written following the IMRaD-style narrative. Only the last two contributions – recommendations/guidelines and research agendas/manifestos – may need a different kind of a narrative and can therefore be harder to write well.

Examples of non-contributions in academic research

Notably, there are also certain types of papers that are often submitted for publication but which are often rejected and will therefore be rarely found in academic literature. When one is writing an article, it is a good idea to make sure that one is not writing one of those types of papers. Four common non-contributions are following:

  • Presentation of a well-designed system and its design process. These papers present well-designed systems and include evaluations that demonstrate the high quality of the outcome. The problem with these kinds of papers is that for a researcher looking for novel information, such papers offer very little to learn: they “only” describe well-conducted design process that already uses well-known methods. Only if these design processes solve hard problems in some contexts, and that these problems and their solutions generalise to other contexts too, the papers start to have value in terms of an academic contribution. That is because then the academic reader may conclude that the authors have found a way to address a problem that previously has been considered difficult to tackle. This kind of a study can be turned into an academic contribution by identifying a “design problem” that was solved in the process, and explaining why this problem is difficult and in what design situations similar problems can be encountered (i.e., where does the design problem and solution generalise to).
  • Case study report. Papers of this kind present observations or interview-based findings from field studies, and describe carefully methods that were used in these studies. A lot of effort may have been put into gathering all the data and to analyse it. Unfortunately, despite all the effort spent, also in this case, the conclusion by a reader may be that the story is interesting but lacks novelty: papers of this kind may be a well-conducted research projects but which only have applied rigorous methods without yielding novel findings. This kind of a study can be turned into an academic contribution by identifying an interesting and novel finding, and deepening the literature research so that it convinces the reader about the novelty and the need for this finding in the research field (e.g., a “research gap”).
  • Mappings of findings to a framework. Some papers present analyses from a complex settings and map these findings to a well-known theoretical framework (e.g. activity theory). The problem with such a finding is that it counts mostly as a demonstration that the framework can be used to make sense of observational data. This may not be surprising, if the same has been shown in numerous earlier studies too. This can be turned into an academic contribution, for example, by finding out that the framework cannot be used to make sense of some parts of the data, or that the framework needs adaptation because of the novel findings.
  • Landscaping and clustering studies without conclusions. Some automatic data analysis methods nowadays allow researchers to generate elaborate descriptive visualizations and groupings that can summarise complex phenomena in a neat manner. Examples of these methods include social network analysis, clustering methods of multidimensional data (e.g., factor analysis, k-means clustering and topic modeling), and sentiment analysis about natural language. If a paper only presents the outputs of such analyses, without identifying non-obvious patterns or conclusions, the paper easily lacks a clear contribution. An academic contribution would include an actionable message to the research field: a call for changing the research focus, or think about a common phenomenon in a new way. Typically this requires that the researchers interpret their clusters and identify something unexpected from them.

Contributions and findings in BA and MA theses in HCI and design

In BA and MA theses, the requirements are slightly different than in academic articles. The difference lies in the need for presenting a contribution vs another, more modest kind of a finding. A thesis does not need to demonstrate novelty to an entire research field; it only needs to demonstrate the ability to apply the relevant methods, theories and analytical thinking with respect to a meaningful problem of practical importance. Therefore the three last above-presented examples of non-contributions are, in fact, good candidates for excellent BA or MA theses even if they lack an academic contribution.

One may therefore conclude that in BA and MA theses, the goals can be more practically determined: They may orient to finding good designs or solutions for specific design problems. They may be reflections about the nature of a design process, such as explorations whether a certain design approach yields findings that satisfy the designer. They may also be oriented towards a designer or practitioner community than the researchers. Therefore they may deliver a call or message to those communities to start addressing issues or become aware of matters that are being neglected. Such issues do not need to relate to academic activities, but to societal issues, for example.

One or many contributions?

There can be one or many contributions in a paper. Some contribution types also go naturally together. For example, sometimes the most interesting contributions appear in the Discussion, after the answers to the research question(s) have already been presented. Thus a paper about an exploratory study may be sense-making in its Findings (e.g., by identifying an interesting underlying pattern or concept in the findings and by giving a name for it), but a manifesto-like contribution in its Discussion if it then shows how that concept may be crucial to remember in other situations too. Many readers may find that this manifesto-like contribution is actually more important than the text’s original finding.

However, many instructions on academic writing recommend that every text focuses on delivering only one “contribution”. For example, instructions published in Nature’s web page recommend to “Keep your message clear” (Gewin, 2018). There is a good reason for this: To offer a clear and interesting message, different contributions usually require different investigations. If one tries to combine several contributions together, they may require different methods, and these methods may conflict with each other, leading to biased and compromised results. Another problem is the need to reach a high clarity with the paper: if there are several intended contributions, explaining them clearly can be difficult. Jumping from talking about one contribution to another may be necessary, but this may confuse the reader. It is important to remember that it is the author’s responsibility to demonstrate that the findings are significant and interesting (e.g., Ladik and Stewart, 2008). Confusions should be avoided at all cost.

Conclusion

To conclude, to offer a clear contribution or a finding, it is a good idea to identify early on what kind of a story one wants to tell with their text. Following the recommendations of the IMRaD structure, for instance, all the attention of the paper’s argumentation can then be directed to delivering that message as clearly and convincingly as possible. This helps the readers – evaluators, reviewers, and others – appreciate the work that the author has done.

Acknowledgments

Thanks for Oscar Person for tipping me about Ladik & Stewart’s paper on research continuum.

References

Bardzell, J. & Bardzell, S. (2011). Pleasure is your birthright: Digitally enabled designer sex toys as a case of third-wave HCI. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI 2011) (pp. 257–266). New York, NY: ACM Press. https://doi.org/10.1145/1978942.1978979

Braun, V. & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3, 77–101. https://doi.org/10.1191/1478088706qp063oa

Gewin, V. (2018). The write stuff: How to produce a first-class paper that will get published, stand out from the crowd and pull in plenty of readers. Nature, Vol. 555, pp. 129-130. Available at: https://media.nature.com/original/magazine-assets/d41586-018-02404-4/d41586-018-02404-4.pdf. Also available, with a different title, at https://www.nature.com/articles/d41586-018-02404-4 (retrieved 11 November 2020).

Cross, N. (2004). Expertise in design: An overview. Design Studies, 25(5), 427–441. https://doi.org/10.1016/j.destud.2004.06.002

Gould, J. D., Conti, J., & Hovanyecz, T. (1983). Composing letters with a simulated listening typewriter. Communications of the ACM, 26(4), 295–308.  https://doi.org/10.1145/2163.358100

Gustafson, S., Baudisch, P., Gutwin, C., & Irani, P. (2008). Wedge: Clutter-free visualization of off-screen locations. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI 2008) (pp. 787–796). New York, NY: ACM Press.. https://doi.org/10.1145/1357054.1357179

Hardaker, C. (2013). “Uh….not to be nitpicky,,,, but…the past tense of drag is dragged, not drug.” – An overview of trolling strategies. Journal of Language Aggression and Conflict, 1(1), 58–86.  https://doi.org/10.1075/jlac.1.1.04har

Ladik, D. M. & Stewart, D. W. (2008). The contribution continuum. Journal of the Academy of Marketing Science, 36, 157–165.  https://doi.org/10.1007/s11747-008-0087-z

Nardi, B. A., Whittaker, S., & Bradner, E. (2000). Interaction and outeraction: Instant messaging in action. In Proceedings of the 2000 ACM Conference on Computer Supported Cooperative Work (CSCW 2000) (pp. 79–88). New York, NY: ACM Press. https://doi.org/10.1145/358916.358975

Oulasvirta, A., Tamminen, S., Roto, V., & Kuorelahti, J. (2005). Interaction in 4-second bursts: The fragmented nature of attentional resources in mobile HCI. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI 2005) (pp. 919–928). New York, NY: ACM Press. https://doi.org/10.1145/1054972.1055101

Oulasvirta, A. & Hornbæk, K. (2016). HCI research as problem-solving. In J. Kaye, A. Druin, C. Lampe, D. Morris, & J. P. Hourcade (Eds.), Proceedings of the SIGCHI Conference on Human Factors in Computing (CHI 2016) (pp. 4956–4967). New York, NY: ACM Press.  https://doi.org/10.1145/2858036.2858283

Pirolli, P. & Card, S. (1995). Information foraging in information access environments. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI 1995) (pp. 51–58). New York, NY: ACM Press/Addison-Wesley. https://doi.org/10.1145/223904.223911

Salovaara, A., Upreti, B. R., Nykänen, J. I., & Merikivi, J. (2020). Building on shaky foundations? Lack of falsification and knowledge contestation in IS theories, methods, and practices. European Journal of Information Systems, 29(1), 65–83.  https://doi.org/10.1080/0960085X.2019.1685737

Suchman, L. A. (1987). Plans and Situated Actions: The Problem of Human–Machine Communication. Cambridge, UK: Cambridge University Press.

Todi, K., Weir, D., & Oulasvirta, A. (2016). Sketchplore: Sketch and explore with a layout optimiser. In Proceedings of the 2016 ACM Conference on Designing Interactive Systems (CHI 2016) (pp. 543–555). New York, NY: ACM Press. https://doi.org/10.1145/2901790.2901817

Torenvliet, G. (2003). We can’t afford it! The devaluation of a usability term. Interactions, 10(4), 12–17. https://doi.org/10.1145/838830.838857

Wobbrock, J. O. (2012). Seven Research Contributions in HCI. The Information School, DUB Group, University of Washington. http://faculty.washington.edu/wobbrock/pubs/Wobbrock-2012.pdf (retrieved 12 November 2020).

Wobbrock, J. O. & Kientz, J. A. (2016). Research contributions in human–computer interaction. Interactions (May–June), 38–44.  https://doi.org/10.1145/2907069

Writing a good storyline for a thesis or an article

TL;DR: Click here to jump to the table of a detailed storyline contents directly.

Introduction

It is a big undertaking to write a thesis or an article. When the writer finally sends it for examination or review, they can only wish that its reader – whoever that may be – can grasp what they have wanted to say, find it credible, and get a positive vibe about its findings. If that succeeds, it would result in the outcome that the writer is looking for: a favourable evaluation (in the case of a thesis) or a review (in the case of an academic article), leading to a good grade or a publication.

Ensuring that the reviewer or examiner is happy and satisfied is however difficult because of many reasons, at least the following:

  • The reader is often unknown, and yet this person’s opinion is critical for the favourable review. One has to make the text understandable for a rather wide audience, as it cannot be tailored for a particular individual.
  • The author may have little prior experience on writing a text like a thesis or an academic article. To do anything for the first time is difficult, yet in this case this person should manage to perform well even if having very little practice.
  • There are lots of things that one wants to say in the article. It is difficult to say all of that so that it makes sense.

Yet many people succeed in this task. Many factors play a role in whether the examination or review is favourable. However one of the crucial ones is the understandable storyline, or “narrative”. The prerequisite for “good vibes” is that the text is presented in a logical order so that the reader never wonders why they are reading a certain part of it. That is because a logical order removes the effort from the reader to make sense of the research, and makes even the difficult topics easy to digest.

In this article, I will study and answer what a successful storyline of an academic text should be like. I will present a formula that is consistently followed in almost all academic writing. It is well-tried and tested, and most readers have learned to expect that a text will follow it, small deviations notwithstanding. Being truthful to what this article seeks to suggest, also this text has been written following the same principles.

After reviewing some of the options that are available for narrative planning, the suggestion will be presented in the Results section.

Review of possible storylines

Requirements for the narratives of academic texts, such as theses and articles, can be found from several places. Taylor & Francis – one of the leading publishers of academic research – emphasizes in its instructions the following rules: i) stick to the point, ii) create a logical framework, iii) don’t be afraid to explain, iv) clarity is key, v) be aware of the other literature in your field (and reference it), vi) make your references current and relevant and vii) be original (Taylor & Francis, n.d.). Of these, points i–iv are particularly related to the narrative. The same requirements of parsimony (i.e., saying only what is essential), being logical and clear, and providing explanations when appropriate, are echoed also in Springer’s – another big publisher – recommendations for giving a logical flow to the content and making the manuscript consistent and easy to read (Springer, n.d.). Similar recommendations can be found throughout academic community, and they echo the norm that should be followed.

With these requirements in mind, it is possible to analyse some canonical plot structures in order to find out how they serve the above-stated requirements. They are presented in the following.

The classical example –Aristotle’s suggestion to divide every narrative into three sections – meets the requirements well, but is rather unspecific. In Poetics, he suggests that in a tragedy, “a whole is what has a beginning and middle and end” (Aristotle, 335 BCE/1932). Wikipedia (n.d.) describes that these three stages should describe one causal chain of actions, where a stream of events is put into motion in the beginning where the protagonist faces a challenge. The middle part describes the protagonist’s attempts for resolution that also becomes a story of learning and personal growth. The end, finally, reveals how the protagonist fares in their question. While Aristotle described the structure of a classical tragedy, the need to have a causal chain of actions and a clear arch that starts from a challenge and concludes with a resolution provides one example of a structure that is logical and easy to read.

Another literature example is the classic thriller plot structure. It involves three stages where things are set in motion from an event such as murder, and the plot ends with a resolution, such as the revelation of the murderer’s identity. By having a question to be answered already in the beginning, the thriller plot structure suits for academic texts that also usually seek to answer a research question or verify a hypothesis. However, this structure does not obey the principle of clarity. This is because it prolongs the revelation of the solution purposely.

A third example is a news article that usually follows a so-called pyramid structure. Here the most important news is announced already in the heading, and it is incrementally substantiated with more background information in a layered manner. This is an economic method that optimizes the amount of information provided to the level of interest that a reader has for the content. The structure is clear, parsimonious and logical, yet it lacks a means to present a research question. It also lacks a causal narrative: a structure that repeatedly presents the facts in a deepening order does not allow the authors explain their reasoning and how they arrive at their conclusions.

Finally, academic world has its own narrative recommendation, known as IMRaD, referring to Introduction, Methods, Results and Discussion. It is also mentioned as the recommended narrative in Springer’s and Taylor & Francis’s instruction pages that were cited above. IMRaD has the Aristotlean structure where the challenge is presented in the Introduction in a form of a hypothesis or research question. This is followed with an incremental development of an answer, consisting of presentation of a research method and a result that has been obtained. The Aristotlean end is offered in the results section. The discussion, interestingly, is an element that previous structures have not mentioned, and is therefore unique to academic tradition. It resembles an epilogue where the author distances themselves from the immediate plot and reflects the narrative and its resolution “from above”. This allows the author for suggesting improvements and further thoughts about the story.

The recommendations stated above can be summarized as follows. A good academic article is parsimonious by sticking to the point, presents its contents in a logical order and explains issues to ensure clarity. Except for the pyramid structure used in news journalism, all the structures that were presented above emphasise that a good storyline starts with a puzzle or a problem, which is then developed, until a resolution is offered in the end of the paper. The IMRaD structure additionally includes a discussion, where the story is lifted to a meta-level reflection to provide the reader an understanding about the wider perspective of the research work.

How the principles of a good storyline can be identified

Having concluded the main elements of a good storyline based on recommendations in the literature, the question remains how such instructions should be “operationalized”, i.e., changed into actionable guidelines. This question is complicated by the fact that different  research fields may have their own particular practices and preferences.

While for other fields of research, a Google search reveals readymade guidelines for authors (e.g., for health care research, see a nice guideline by Perneger and Hudelson, 2004), for research in the fields of design and human–computer interaction (HCI) – main topics of the Writing about Design blog – I have failed to find a similar guideline. In the absence of written guidelines, in my personal case, the method for learning how to write academic texts has developed through practice. The most useful part of this practice for me has been the constant attempt to publish research papers in journals and conferences, especially when writing together with more experienced co-authors.

Another method is to read other authors’ papers and to review them. Especially reviewing and acting as an associate editor/chair develops the critical but constructive eye for narrative structures that are important in good papers. It is easy to learn to notice, for example, when a paper does not answer to the problem that has been defined in the beginning, or when important matters are missing from the paper, leaving the logical order unexplicated.

In the following section that seeks to answer the original research question of this article – what a successful storyline of an academic text should be like – I try to save the reader’s time from learning through trial and error, and directly apply some of the general but actionable principles.

Results

Based on personal experience and reflection on the different narrative structures in literature, Table 1 below presents a suggestion for a common and recommended storyline that can be identified in most texts in design and HCI research.

Table 1. Detailed pieces of an academic text.

Narrative element Description
1. Introduction
Opening sentence A claim that any reader finds easy to agree with. This brings them “on board” with the topic of the text.
Problematization Presentation of a problem that exists in the matters that the opening sentence described and which would need to be solved.
Motivation Clarification why it is important to devote efforts to address the problem.
Research questions (RQs) or Design problems Crystallization of the most important issue in the problem. Explanation why knowing this answer in particular will be an important element in starting to solving the problem. Research question (or, alternatively, a design problem) is the crux of the entire text: it will become the focus of everything that will follow until the Discussion. It is a good practice to use italics so that it is easy to spot from the text. There can be several RQs in a text, the fewer the better. Having three RQs makes the paper very difficult to write well.
Preview of the answer A short paragraph that describes what kind of answer this text will deliver to the RQ.
Outline (optional) If the text has a more complicated structure than is typical, it may be useful to orient the reader to this. For example, an outline may be helpful if the text presents parallel storylines that will meet in the end of the Results section, that can be pointed out.
2. Related research
Identification of the bodies of knowledge needed to answer the RQ Reflection on the RQ and what kinds of knowledge – often from different research fields – are needed to be able to starting to answer the RQ. This reflection can be based on common sense. For example, design problems are often approachable both by benchmarking existing solutions and by analysing requirements and opportunities that the context (e.g., homes, transportation etc.) of the problem poses on the solution. Identification of 2-4 different viewpoints is a good number.

Write this part right after the main heading, before any subsection.

2.1–2.x Review of the bodies of knowledge, each one in its own numbered sub-section A subsection for each viewpoint presented above. These subsections tell what is known about the RQ from that viewpoint. Each subsection ends with a summarising paragraph that summarises how RQ can now be better understood in light of this information, but also point out the remaining ”research gap”. This further legitimises the need for this research.
2.y. Summary of research A wrap-up of all the knowledge that has been collected in the subsections, and a crystallization of what remains to be found out. The summary may also reiterate the original RQ and present a more focused one for the rest of the text.
3. Method
Methodological reflection Analysis of the best possible research method to answer the more detailed RQ and what remains to be answered. This analysis presents possible methods, mentions prior studies that have used the same method for a similar RQ, and then chooses the most suitable one, given how it matches with the RQ and what is realistic.

Write this part right after the main heading, before any subsection.

3.1. Presentation of the data collection method A neutral description of how some data was gathered to answer the RQ. If the method was a case study, this part includes also a presentation of the case study context and why it suits well for the RQ that was asked. If the study is about a system that was studied in a field study, there needs to be also a subsection that describes this system. If the text describes a design project, this is a description of how prototyping, for example, was used through a series of solution attempts, to arrive at a solution. This section does not present the data itself; it only tells how it was collected.
3.2. Presentation of data analysis method A neutral description of how the data was analysed. This may be also part of the Results section where it is interleaved with findings, especially in two cases: 1) if the analyses were very straightforward, such as statistical tests, and do not require a lot of explanation about details, or 2) if the analysis cannot be described without also presenting the data – this may be the case in exploratory design projects where different stages required a lot of decision-making and reflection.
4. Results
4.1. Descriptive data (optional) If needed, the Results can be started by briefly reporting general data about the research process and about the research context (e.g., overall statistics about participants).
4.2–4.x. Answers to the RQs A subsection for each RQ that the study had. In the spirit of “sticking to the point” (Taylor & Francis, n.d.) each subsection answers only the RQ. If there are also interesting patterns in the data, they can be pointed out, but their details are analysed in a different sub-section (see item 4.z).
4.y Evaluation of the result (if the result is a proposal) If the research is a product design project, such as a development of a service solution, a product concept or a prototype, this section needs to present the resulting design, and contain some form of evaluation of its success. It the project’s goal was to develop something very ambitious, the presentation can be about a “proof of concept” prototype that demonstrates that it it is possible to build such a product. In that case the product may not need to be evaluated with users: the evaluation of the project’s success can be conceptual. Alternatively, if the goal was not to develop something highly ambitious, the product should be evaluated in its context (e.g., with user evaluation), and there results of the evaluation should be presented here, to assess whether it meets the requirements and addresses sufficiently the RQs.
4.z Additional analyses (optional) If the data analysis revealed surprises that are related to the RQ but do not exactly answer to it, they can be presented and analysed here.
5. Discussion
Return to the beginning Repetition of the text’s goal and its RQs, because during the long Findings section the reader may have forgotten the big picture.
Summary of main findings Answers to the RQs in a nutshell.
Expansion A ”switch of gears” in the paper; simply a statement that says that the findings of the study give grounds for several important implications.
5.1-5.x. Implications A subsection for each wider implication from the findings. These are thoughts that no-one would be able to think about without carrying out all the work that has been described.
5.y. Limitations Presentation of the work’s limitations and an analysis how severe they are regarding the validity of the findings.
5.z. Conclusion (can be also its own section) Summary of the main message of the paper, building on the findings and implications, in relation to the problematization in the Introduction. Closure can have an uplifiting spirit, to leave the reader with a spirit that they have read something important.
Acknowledgments
Funding List of sources of financial support that made the research possible.
Intellectual support People who provided supervision, guidance, or gave comments are listed here. It is a good practice to be generous in the thank-yous.
References

This long and detailed structure follows the IMRaD format but is tailored for the needs of design and HCI related writing, and to applied research in general. It emphasizes the following principles in particular:

  • “Holding the reader by the hand”: Throughout the text, the reader feels never being lost. This is because the text contains intermediate summaries and recognizes all the decision points in the path from the motivation to the summary of findings.
  • Funnel-like structure: the narrative structure narrows down the problem to its most important elements in several stages. Some of the most important refocusing points are the presentation of the RQs, the reiteration of the RQs in light of existing knowledge, and the choice of the methods. Through these stages that are well explained in the structure, the reader is able to understand in all the points why the text “dives” into particular topics more deeply than in other ones.

This article itself has been written following the principles stated in the table. It may be a useful learning exercise to compare the contents in the table to the contents in this article overall.

Discussion

This text has addressed an important problem that especially inexperienced writers face when they are writing their first academic text, such as a thesis, or their first academic article, if they are PhD students. Even if the research itself may have been solid and well conducted, and therefore having interesting findings, the unclear narrative may make it impossible for a reader to appreciate this work.

With the proposal for a storyline presented in this paper, writers can adopt some of the rarely explicated principles in design and HCI literature and use them in their own writing. This decreases unnecessary stress in the writing process, which in itself is always a highly contentious experience.

Another implication from knowing what order to use in explaining the research process, and in checking that no parts are omitted from the text is that it helps in carrying out better research. Awareness of components that are expected from an academic text helps taking them into account when the research process is still ongoing.

However, it has to be emphasized that the structure presented in the Results section is not suitable for all texts. It works best in cases where the work involves an empirical part, such as a case study, interviews, an experiment, or an exploratory process. It is less suitable for essays, manifestos and commentaries, all of which are also academic genres. A critical reader may notice that this text has a weak Methods section. This is because I have used IMRaD to structure an essay-like paper where no clear method has been applied. A result of this has been that there have been difficulties to write anything about research methods. The text has also become longer than one would assume, which violates the call for avoiding anything that is not absolutely necessary. On the other hand, at the same time, important matters have been explained whenever needed – therefore meeting another goal of recommended academic writing (Taylor & Francis, n.d.).

With a clear guidance that is properly explained, even a challenging project such as thesis or article writing can become an insightful experience. Academic writing is a creative activity that gives an opportunity for the author to synthesise and make sense of their work, and crystallize it in a format that they can also later apply in future occasions. By putting down such reflections in a written format, they can also convince others about the rigour of their work, leading them to achieve life goals, be recruited to jobs, and pursue explorations that offer high levels of intellectual satisfaction.

Acknowledgments

I want to thank the participants of HelsinCHI clinic on CHI paper writing in 4 September 2020, where I presented the storyline the first time, and I received useful feedback. Heidi Paavilainen and Severi Uusitalo provided several crucial observations – both detailed and general ones – to the first draft of this blog post, which have now been taken into account.

References

Aristotle (335 BCE/1932). Poetics. Cambridge, MA, Harvard University Press.

Perneger T.V. and Hudelson, P.M. (2004). Writing a research article: advice to beginners. International Journal for Quality in Health Care 16(3), 191–192. https://doi.org/10.1093/intqhc/mzh053

Springer (n.d.). https://www.springernature.com/gp/authors/campaigns/writing-a-manuscript/structuring-your-manuscript (retrieved 26 October 2020).

Taylor & Francis (n.d.). Writing a journal article. https://authorservices.taylorandfrancis.com/writing-a-journal-article/# (retrieved 26 October 2020).

Wikipedia (n.d.). Three-act structure. https://en.wikipedia.org/wiki/Three-act_structure (retrieved 26 October 2020).