In this, slightly longer text, I address a topic that I have wished to explain a long time: how good research plans can be created.

Research designs

Which methods should I use in my study? This is a question that one must answer in each and every study, from simple courseworks to complex PhD theses that may take several years to complete. The question about the right methods is often hard to solve. This article aims to give ideas on how to survive in this task.

The correct methods can be difficult to choose because everyone of us has their personal favourite methods that are more inspiring than others. The researcher may be good at some method in particular – be it interviewing, prototype-building, a quantitative survey, or something else – and they would wish to use it also in the upcoming study. However, the method may not fit with the goals of the study. In such a situation, and also universally, it is important to understand that a method has to be chosen based on the research question (RQ) that is asked. That is, the RQ has a higher importance than the method; methods are merely tools for finding answers. (In this text, I will solely talk about RQs, but the term always includes also design problems).

RQ–method combinations are called research designs. The term “research design” describes the meaning of the concept very well: it refers to the way in which one designs their research: how one goes about carrying out the study and achieving findings. Term is commonly used in quantitative studies (see e.g., Trochim, n.d.), but it is applicable to any other kind of research: qualitative, constructive, action research, even to theorising. 

Research designs can be excellent or terribly poor. There would be, for example, severe flaws in a study that would seek to prove that one user interface (UI #1) of a mobile navigation app would be better than another interface (UI #2) by presenting visual layouts to the two UIs to six participants, asking their opinions about them, and analysing which one gets more positive responses. Perhaps most obviously, such a research design would flawed because interviewing is not the right method for evaluating action-oriented UIs such as navigation apps. A better research design would involve navigation tasks that would be performed using the two UIs. Even after that correction, many smaller methodological decisions would need to be determined. A finished, well-planned research design takes some time to create.

Research designing as a cyclical process

Therefore, it is a good idea to consider research designing as a process that involves many attempts and restarts until a good design is found. The following diagram diagram in Figure 1 presents the main elements of a research design – contribution, RQs and the methods – that need to be fit to each other, as well as the main directions of effects that spark changes between these elements.

Figure 1. Research design planning cycle

Let’s examine the above-presented comparison study of two mobile navigation UIs using this diagram. The left side of the diagram presents the ambitions that the researcher has for the study. Starting from the top on that side, the study’s contribution relates to the superiority of one UI over another. Perhaps UI #1 is researcher’s own, new design, and they wish to prove that it si better than UI #2, which is typical (so called “baseline”) in the market at the moment. Moving to the middle, RQ of the study could be then be something like this: “Is UI #1 better than UI #2 when used for navigation?”. Finally, the method would be to show the two UIs as visual layouts to six people, listen to their opinions, and determine the winning UI. As was already pointed out above, navigation can be better studied by asking people to navigate in the physical world rather than by interviewing them about their opinions.

This is a point where ambition meets realism: It will not be possible to get a valid answer to the RQ using this method. The process now moves to the right side of the diagram where either a new method is chosen, or the RQs and contributions are adapted to match with the method. If the researcher wants to stick to interviewing as the research method, then the RQ has to be changed. It may be possible to ask something like this: “In what ways will experts’ opinions about two mobile navigation UIs differ from each other?”

Now the study asks about expert opinions and does not consider actual end users. This decreases the need for participants, since experts are probably able to provide valid analyses even by just looking at visual layouts – end users would not be able to do that. Experts are probably also more likeminded in their assessments. Therefore maybe six participants will be enough.

Also, the RQ is now framed as an inquiry about differences between expert opinions instead of straightforwardly about finding a winning UI. That suits better for interviews. What is however lost is the sharpness of the intended contribution: with the new RQ, the researcher will not be able to claim that one UI would be better for navigation than another. Such a claim can be validly studied only by observing navigation in action. Now the intended contribution will be somewhat softer: to show that a number of experts consider one UI better than another UI, at least in some respects.

As a result of these changes, we have a workable research design. We have reached it by softening our intended contribution. We have also decided that we need to recruit experts and not end-users as our participants. Finding such people will be harder, but will allow us to carry out fewer interviews, possibly only six. Figure 2 illustrates the transformations that have happened in the research design planning cycle.

Figure 2. Research design planning cycle applied to the example in the text.

What the analysis above tries to illustrate is that creation of a research design involves going back and forth between the contribution, RQs and methods. This is normal, and every experienced researcher is used to doing this kind of process before they start their actual research. The process may take a considerable amount of time, because in addition to just thinking, also reading may be needed, since literature may help in making the RQs sharper with the help of theoretical assumptions and existing empirical findings. The end of this article presents some common “tricks” by which dead ends in research designing can be avoided, and it is eventually possible to find a combination of contributions, RQs and methods that satisfies the necessary criteria of validity and the ambitions of the researcher.

How do I know what methods are required by the RQ?

The text above emphasised that it is the RQ that determines what the right methods are. But how does one know, given that one has a suggestion for an RQ, what are the possible methods to consider? It is not possible to give a universal rule for the question, but good examples can be found from prior studies that have been published about similar problems. It is usually a good idea not to start inventing one’s own methods but to use ones that have already been used successfully. That also gives an idea on the complexity and labouriousity of the study.

Most important, however, is to precisely to understand what one is asking in the RQ. The question should be understood word by word, and taken very literally. This helps figure out what kind of methods (for data gathering and its analysis) could possibly be considered proper what kind of a method could be considered as being proper. Here are some examples:

• Is UI #1 better than UI #2 when used for navigation? This is a comparative question (because of the word “better” in it) and requires that two measurements, one from UI #1 and the other from UI #2, can be compared against each other. It is also asked matter-of-factly, in a universal and objectivistic manner: Is UI #1 better than #2, whereby it refers to all app users and assumes that the comparison is carried out from all possible viewpoints (navigation time, number of errors, stress level, etc.), and that it is independent of researchers’ or users’ subjective opinions. Also, the RQ talks about navigation, which points out that the study should consist of a navigation task and that not an interview, for example. Considering all of this together, a controlled experiment and quantitative methods would be considered the right way to answer to this RQ.
• What are the strengths and weaknesses of UI #1 and UI #2? In this RQ, the comparison is removed, and replaced with an analysis of the two UIs independently from each other. That makes the RQ easier to study. Also qualitative methods, such as interviews and subjective opinions, are now applicable. What is sacrificed is the ability to say that one UI is better than another. Some level of qualitative comparison of the differences is however possible. For example, one can say that “it is possible that UI #1 would be less stressful than UI #2”.
• How should a UI intended for stress-free navigation be designed? This question drops the comparison of two UIs altogether, and also focuses on the design process instead of the outcome of the design. It also narrows down the design focus on stress-freeness instead of focusing on UI’s design from every possible viewpoint. With this new orientation, now the focus must be on the ways in which stress-freeness should be taken into account in different design stages. One may think how stressful navigation situations could be studied in user research, how stress-freeness can be evaluated using prototypes, and so on. In the final design after the process, one can explain how different design choices are based on different attempts to stress-free navigation.

The RQ can be changed also in many other ways. These examples show that it is possible to formulate the RQ in different ways, seeking for a study that remains meaningful and interesting while making the methods realistic to carry out. This illustrates how to make use of Figure 1’s research design planning cycle: one starts from some contribution and an RQ that would suit it, then thinks about what they require from the methods. Then the RQ and the methods are reformulated until they match and the study is possible to carry out.

Triangulation: using several methods for one RQ

Most RQs can be answered using more than one method. This is good: If several methods are used to answer a question, their findings, when put together, are more reliable and valid than with any method individually.

In the best case, the methods that answer the same RQ triangulate each other. This means that their are different in such a way that they cancel each other’s weaknesses. The term’s meaning can be understood by considering the word’s original use in close-to-shore marine navigation. When the captain wants to know their ship’s position, it is best obtained by measuring the directions (“bearings”) to landmarks that are at a right angle (i.e., 90 degrees) to each other. In Figure 3, bearings to the Skyscraper and the Manufacturing plant offer such a situation: the errors of measuring the exactly correct directions to these landmarks cancel each other optimally. The second best solution is to use directions that are not at a right angle, such as bearings to the Skyscraper and the Peaky hill. The least favourable situation is to take the bearing to one landmark only and guess what the distance to it might be.

Figure 3. Triangulation in locating a ship’s exact position.

In research designs, triangulation is useful in an analogical manner. The answer to the RQ is matches with the question about the ship’s exact position. The methods that give answers to the RQ match with the bearings to the landmarks. Similarly as the bearings have their errors, also research methods have their individual errors and biases. Interviewing, which is so-called “self-report” method, for example, depends on participants’ truthfulness and verbalisation skills, and cannot be always trusted. The same applies to a diary-keeping and questionnaires – also they a
re self-report methods. Direct observation of people who carry out tasks without knowing that they are being observed, in contrast, does not have such problems on truthfulness or people’s verbalisation skills. On the other hand, it has its own problems as a method. Together, however, observation and interviewing triangulate each other excellently: if both methods offer the same findings, then our results are much more believable than with only either of the methods alone. Figure 4 illustrates the relationships between these methods using a similar diagram as Figure 3.

Figure 4: Triangulation in research design.

Several RQs and several methods

Finally, it is typical that a study has several, maybe 2 or 3, RQs that crystallize the main aspects of the intended contribution. One could think that these RQs triangulate, in their own way, the intended contribution. For each of these RQs, following the principle of triangulation described above, several methods may be considered. Although it may seem that a study with two RQs, each with two suitable methods that triangulate each other, would require 2 x 2 = 4 different methods, this does not need to be the case. Instead, often the same method can be used to answer two different questions. One can interview, for example, about matters related to RQ1 and then about RQ2. Thus, a research design of a full-fledged study about two mobile navigation apps may have the structure shown in Figure 5. In this example, only one method is used for answering the first RQ, which leads to somewhat weak findings. The second RQ, in comparison, is rather well addressed with three questions where two self-report methods and navigation tasks nicely triangulate each other.

Figure 5. Research design diagram for several RQs and methods.

This diagram is visual and easy to read. It is even possible to point out which methods triangulate each other and which ones involve redundancy (and are especially useful in increasing reliability in data collection) but are mostly susceptible for the same biases and weaknesses. Well-triangulating methods’ arrows can be placed in right angles with each other, while methods with redundancies have arrows that are almost parallel with each other.

In a research plan document, on the other hand, a more detailed format for the same diagram can be useful. Table 1 below shows one example for such a format. In the same way as in the diagrams above, it has the RQs in the left and methods in the right. One column has been added between the RQs and the methods: theoretical constructs and concepts. They are helpful contents from literature that are used to sharpen the focus of the study to most informative elements. In mobile navigation, such a concept can be, for example, multiple resources theory (Wickens, 2002) that presents a model for human attention and cognitive processing capacities in multi-tasking situations. It is informative since navigation using a mobile app involves always multitasking: division of attention between the surroundings and the app. Finally, Table 1 also divides methods into data gathering and analysis stages.

Table 1. Research design matrix (incomplete).

When the research design has reached this stage, and if all the RQs have methods that have methods that involve triangulation, it is a good moment to start the practical preparations for the empirical parts of the study!

References

Trochim, W. M. K. (n.d.). Research design. In Research Methods Knowledge Base at https://conjointly.com/kb/ (accessed 17 April 2022).

Wickens, C. W. (2002). Multiple resources and performance prediction. Theoretical Issues in Ergonomics Science, 3(2), 159–177. https://doi.org/10.1080/14639220210123806

Hart, S. G. and Staveland, L. E. (1988). Development of NASA-TLX (task load index): Results of empirical and theoretical research. Human Mental Workload (1988).

Introduction

In ordinary English, “to reflect” and “reflection” refer to deeper-level thinking. Collins Cobuild English Dictionary describes that “when you reflect, you think deeply about something.” Apple’s dictionary, in turn, defines reflection as “serious thought or consideration: he doesn’t get much time for reflection” (emphasis is original).

In an academic research context, the meaning is quite similar. When a researcher reflects, they seek answers to questions on what research questions they should focus on, do they approach the research question with the right methods, what decisions they have to do during the research, what limitations they have to accept in their research, and what conclusions they can draw from their findings. Asking these questions and answering to them are examples of deep thinking.

These questions show that reflection is an important element in all stages of academic practice, such as

• in the identification of a topic for research
• in the selection of appropriate methods to study the topic
• in the empirical part of the study where the collected data may deviate from what was expected, possibly directing attention to findings that may or may not be even more important than what was originally expected
• in the analysis of the data and making sense of its findings
• in the writing about the process, its findings, and what their significance is

Common to all of the examples in this list is the focus on “staying on track” of the research process: making sure all the time that there will be something meaningful – a research contribution – in the end of the process. Reflection is therefore an essential element of finding out how to navigate successfully past the challenges of the research process, and knowing what the findings imply.

In the spirit of this blog, which focuses on writing, this post will focus on how to write reflectively in a thesis or an academic article. This text works well with my earlier writing about an academic text’s storyline, where it is easy to notice, at least after reading this post, how reflection plays an important role in the narrative generation.

General characteristics of reflection

From the introduction above, I want to highlight the following characteristics:

• Depth. Reflection explores matters beyond, above, or underlying the practical matters of research. By doing this, it supplements those parts of writing that focus on plain reporting (e.g., what has been written in literature, or how research methods were carried out, or what data was collected, or what stages a prototyping project included).
• Explanatory style. Reflection gives an explanation for why the research was carried out in a certain way and not some other way. By explaining the decisions about methods, and the reasonings behind the conclusions, it makes the researcher’s process understandable for the reader.
• Higher viewpoint. Reflection looks at matters from a higher vantage point, “from the outside”. This way it can explain the reasons for subjective choices in more objective terms, such as by linking it to other research writings, practical limitations that could not be overcome, or personal preferences that build on the authors’ prior experiences, for example.
• Navigation in an unknown terrain. With all the characteristics above, the reflection helps in managing the unavoidable uncertainties of every research or design process. It prepares for expecting the challenges that lie ahead. During practical research work, such as when one gathers data or builds prototypes, it helps in noticing interesting features, unexpected outcomes, and needs for decision-making between alternative courses of action. In the writing up the findings, it helps in identifying the strengths and weaknesses of the work that was carried out, what the project’s findings are, and what implication and future considerations arise from them.

Reflection during a research or design process

From the last point, where I listed the benefits of reflection before, during and after the practical “execution” of research or design project, it is evident that researchers and designers should preferably reflect all the time. They should repeatedly ask questions such as “What should be the best course of actions now?”, “Are there alternative ways to do the things that I am now doing?”, “What matters can I leave out from my focus, and which ones should I focus particularly on?”, “Am I still going towards the goal that I had in the beginning, or have I shifted to a different goal? If I have, should I shift back, or change my goal?”

The questions above are examples of reflection-in-action: reflective questions that one can ask as a part of their practical work, and thereby make it part of their creative process. This is a viewpoint that design theorist Donald Schön (1983, 1992) has particularly endorsed: creative practice (applies both to research and design) can be described as “reflective conversation with the situation” where the designer/researcher proceeds by making conscious “framing experiments” by which they frame and re-frame the problem that lies in front of them, and learn by observing the results. Schön’s book The Reflective Practitioner (1983) contains a widely cited example of reflection-in-action from architectural practice that is worth reading.

Although Schön does not describe reflection-in-action using the terms “problem space” or “design space”, I find it very useful to think reflection-in-action as an ongoing exploration and re-definition of the problem space and the problem itself. The idea that a design process both explores and constructs the design space has been advocated especially in Kees Dorst and Nigel Cross’s writing (2001), who are other prominent design theorists. One way to conceptualize this reflective process is as follows, based on interaction designer Bill Buxton’s illustration (2007, p. 388) that I have modified in Figure 1 but whose caption I am copying in entirety from Buxton.

Figure 1. “Design is about exploring and comparing the relative merits of alternatives. There is not just one path, and at any given time and for any given reason, there may be numerous different alternatives being considered, only one of which will eventually find itself in a product.” (Buxton, 2007, p. 388).

The exclamation marks (!) in the diagram illustrate learning moments that are particularly important to note when describing the design or research process, such as a choice about a prototype’s shape, or a suitable methodology that should match a research question.

What is important here is that the entire design process can be seen as “branching exploration and comparison”. The same applies to research: also there the different stages of empirical work require their own critical analysis that ascertain that the process is progressing towards a solution. These stages are useful to be presented to the reader when the thesis or article is written from the process. They showcase the author’s skills in being able to work maturely with an open-ended problem and its challenges.

How are the alternatives and their solutions identified

Here are some strategies that help you reflect during the research or design process:

• Analyse whether another researcher or designer could decide to proceed using different methods or approaches. Analyse both alternatives, find reasons why one would be better than another, and select the one that seems better.
• Compare your project to similar ones in the literature. Decide whether you want to use the same approach, or choose a different path, and why.
• Alternatively, use other external reference materials, such as benchmarking to other projects or products, and apply similar reflection as in the second bullet point.
• Explicate your subjective preferences and identify the plausible alternatives (e.g., “I wanted to prioritise human values over utilitarian ones because…”)
• Identify tensions, contradictions or tradeoffs in the design or research problem, and use them as decision-making junctions.

In order to avoid becoming paralysed by the number of possibilities that could be reflected, do not try to be exhaustive in everything that you do. The purpose, as explained by Schön, is to use reflection as your tool that generates meaningful possibilities, and helps you thereby advance further in the design process.

Reflection in a thesis or an article

Now, finally, we can discuss how reflection should be visible in the written text.

Obviously, all the elements, listed in various parts of the sections above, give already an idea what to include as reflections in a text. Thus, following the general characteristics given in this post’s Introduction, reflection should add depth, explanations, a higher viewpoint, and this way navigate the reader through an unknown terrain of your research/design process. It should also describe your answers to the questions that you have been asking during the process, thereby giving an idea for the reader as to what question or design problem you started with in the beginning, and what events possibly led you to change your focus, and what choices you had to do along the way, and how you made those choices.

These examples on what to reflect about in the text are very similar to many elements of my earlier blog post that presents storyline for an academic text. In a nutshell, here are examples from that blog post on reflection points that most academic texts are expected to contain:

If at least most, if not all, of these items are found in your  article or thesis, then it probably satisfies the needs of the readers who expect that the researcher is able to carry out reflection on their own research.

Concluding words for designers

The following words are written especially for designers, but they apply also to applied research fields in general, especially to those studies that have had exploratory, open-ended research questions (e.g., what is quite common in HCI).

Design is a discipline with more open-endedness than many other fields where academic texts are written. The design problems are more open-ended, for example, than in most scientific or engineering disciplines. This means that design processes can involve much more personal judgment and subjective decision-making in design than in these other fields. This is a reason why reflection is a particularly important part of writing about design: you need to be able to explain your design choices because there are no objective criteria on which other disciplines may be able to base their judgments.

One result from the requirement is that although design can be often visually represented (e.g., with pictures), it does not usually speak for itself: it is not guaranteed that the reader can understand what the designer has focused on in the design process, and how they have solved it. Therefore it rarely suffices to present a plain presentation of the starting point, a report about the stages of a project, and the outcome, such as a physical prototype. Instead, these stages require explanation that help the readers (or customers, in a company context), to appreciate the details that the design contains.

Reflection is a skill that can be practiced, and by remaining mindful about the principles and advices above, you will be better and better equipped with an ability to prove that what you have produced is based on solid, critical and productive analysis.

References

Buxton, B. (2007). Sketching User Experiences: Getting the Design Right and the Right Design. San Francisco, CA: Morgan Kaufmann.

Dorst, K. & Cross, N. (2001). Creativity in the design process: co-evolution of problem–solution. Design Studies, 22(5), 425–437.

Schön, D. A. (1983). The Reflective Practitioner: How Professionals Think in Action. New York, NY: Basic Books. 

Schön, D. A. (1992). Designing as reflective conversation with the materials of a design situation. Knowledge-Based Systems, 5(1), 3–14.

Introduction

Writer’s block is a term for a feeling where a writer feels being stuck: unable to get anything written down or any progress made in the writing process. It is a frustrating feeling, and very hard to overcome. There are probably many reasons for the writer’s block. In this blog post I address one of them: the lack of process that would divide writing into meaningful, manageably sized steps.

I offer the following advice based on my personal experience. I have noticed that the steps described in this blog post work for me, and I am also able to explain why that is the case.

While some others may offer different guidance, since there can be many other ways to write and avoid writer’s block too, I am not alone with my ideas. Simon Peyton Jones from Microsoft Research Cambridge (https://www.microsoft.com/en-us/research/academic-program/write-great-research-paper/) illustrates the recommended process like this:

The diagram suggests that research should not be done before the paper has been written. But what if you already have the data and you need to write now? Is there anything useful in this blog post if the requirement is to write before doing the research (i.e., gathering data)? No worries: while Simon Peyton Jones seems to suggest that kind of a process, this blog post helps also in situations where the data already exists. You can find tips from Step 4 onwards below.

The research process (with emphasis on writing)

If I have to simplify, writing of an academic text (e.g., thesis, essay or a publication) is a process consisting of the following stages.

1. Exploration that finds a purpose for the study

This step can be a bit messy, but it has a clear goal: to identify possible contributions (i.e., findings) that my new study might make. This step can be very slow (even years) if there is no deadline, or it may be rushed, if my research has to be ready on a given date and I’m still even lacking all the data. In addition to thinking about possible contributions, this step also involves generation of possible research question(s) (RQs) and searching for background literature that can answer to them. It also includes the identification of research methods that would be suitable for that RQ. I need to adapt the contributions, RQs and methods until they are in alignment and seem to point towards a study that could produce some interesting as a result. This part’s outcomes often look like a collection of diagrams, lists, ideas, questions, etc., – that is, the outcomes do not look like a text at all yet.

I will not focus on Step 1 in this blog text. However, the linked posts above offer recommendations for dealing with this messy and open-ended early stage where the purpose gestates and gives the direction for the later steps.

2. Research plan

Now we get to the first step that resembles writing. In the previous step, I have managed to identify a promising contribution and RQs, as well as the methods that I can use to answer the RQs. I can therefore write a research plan for myself based on the outcomes from the Exploration step. This plan will be the outcome from this step. It has resemblance to the text that Simon Peyton Jones may mean in his “Write paper” step. This plan will have the sam contents as the beginnings of an IMRAD-style academic text has: an Introduction, a Related research section, and presentation of Methods.

In writing this text, I work in the same way as I describe below in Step 4 (Bullet point narrative).

At this point, the text may not be intended for anyone else’s eyes (unless I work as a part of a team). The plan can therefore quite sketchily written. The clarity of ideas is more important than how these ideas are expressed in words.

3. The empirical study

Now I gather the data that I have planned in the research plan. Especially in a qualitative study, this may overlap with the next steps in the process. I may adapt the focus of data collection if I notice needs for particular kinds of data.

4. A bullet point narrative

This step is the culmination point of the writing process. When I reach this point, I have at least some data that I have gathered, and it is time to analyse it and write about the findings.

If I have had a good plan and I have been able to follow it, I should now have data that I can use to answer the RQs, and reach the intended contribution. Usually the data does not have a perfect match with the intended outcomes, so I have to adapt my path, and find a slightly different contribution and RQs for which the data suits better. Maybe I have to change the type of contribution that my research is able to make, for example. Let’s assume that I manage to deal with this problem, and I identify the final, suitable contribution and the RQs.

A lot of uncertainty still exists even if I may have an idea about a finding that I could write about. For example, it is difficult to choose which order I should explain the things. I might wonder how much I should write about some background before I can tell a finding. Would that  sidetrack the text too much from the main plot that I want to tell? What if two things seem to require each other as their background? Which one should I write first?

Because of the puzzles like the ones above, it is important to minimize the time that I may  waste in writing long pieces of text that I may need to delete later. Therefore, the entire narrative of the text – from its very beginning to as far into to end – is better to write using bullet points.

Here is a mini-process of how this is done:

Preparations

If my research contribution has not significantly changed from the Research plan that I produced in Step 3, I can use that as my starting point for the paper. I can then extend that by adding the remaining section headings: Results/Findings and Discussion.

However, more commonly, the case is that my idea of the research contribution has changed because of the data that I collected and what my analysis found out. Then I will start from a new, empty document. I fill it with the main IMRAD section headings: Abstract, Introduction, Related research, Method, Results (or Findings) and Discussion.

Next, I fill the contents of each section with the names of the narrative elements (i.e., opening sentence, problematization, motivation, RQs, …) that most of the applied research follows. If I would not already remember these narrative elements, I could copy its contents from the table in my earlier blog post.

These preparations provide the main “skeleton” for the contents that I will need in different sections. This will help me find the appropriate place for different pieces of text that the text needs.

Writing using full-sentence bullets

After this preparation, I start the most significant part of the writing process. While until now I have been writing in first person, in this part I will switch to an imperative “you” style.

The idea is to work in a way that lets you make changes easily, and lets you focus on the content and logic. Therefore, the bullet points work excellently here. There are some rules that need to be followed, so that I don’t fool myself into thinking that I’m making progress while I’m not. Luckily the rules are quite simple:

Rule 1. Use only full-sentence bullets. These can be directives to yourself (“Explain here why interviewing would this time be the wrong method”) or claims (“Interaction design process involves a lot of trial and error.”). Aim for such a clarity that by reading the bullets, a knowledgeable outsider would be able to describe the logic of your paper.

This is why simple keywords (e.g., “participants”, “content analysis”, “limitations”) would not work: they would be only placeholders for content without the content itself. Writing sentences that express claims has the benefit that you are actually doing the mental effort of planning your logic, and working towards results. Keywords, instead, easily turn out to be to-do lists where actual thinking is postponed to some later time. Writing full sentences is harder, of course, but it is cost-efficient to do the thinking now than at a later time.

Rule 2. Do not leave logical gaps in the narrative. Try to find a solution to every open step: to the narrowing and scoping of your focus, to the selection of fields of literature that you are building on, to the selection of methods, and so on. These choices are things that your final text will need to explain, and now is the right moment to ensure that. Finding a way to justify all the decisions is easiest to do first using bullet points. In particular, pay attention to clear the openings and closings of each section and sub-section.

Rule 3. You can speculate about future results. It is clear that you don’t know everything about every part yet when you write these bullets. But that is not a problem: you can define goals for your analysis, for example, and writing down what you expect to find out or which you would like to write about. You can later see if your analysis lets you do that. You can also “foreshadow” these contents in earlier parts of the text, which makes the narrative stronger.

Therefore, write also speculative sentences. If you still lack information – for example, if you don’t have all the data for Results, or your Results are unfinished and you don’t know yet how to start your Discussion – sketch bullets nonetheless for these sections. These bullets are educated optimistic guesses on what you will tell about your research if things go well in your data collection and analysis. Here my work process and Simon Peyton Jones’s recommendations are again similar.

Rule 4. The narrative does not need to be perfect immediately. Don’t fall in despair! Writing bullets in this way is hard at times. This is because it is often hard to know how one idea should be followed with another one. You can soothe yourself, however, by thinking that these decisions are easiest to make using bullets. They are the least demanding way to find the best narrative for your text. It is easy to reorder them, delete them, and write new ones. The narrative does not need to be perfect – it is sufficient that you find one kind of narrative. Then you already have one plan for how to later write the full text. You can decide to improve the narrative later if you keep it in bullet point format.

When is this stage complete? The main goal in this stage is to generate a full overarching understanding about your entire written document. The resulting narrative should not have contradictions (e.g., starting with one research question but ending with a result that does not answer to it, but answers to something else).

Continue “massaging” your text’s bullet point outline until it meets the requirement above. Adapt the level of detail based on the clarity of your thinking: If you are confident that a certain sub-section will be easy to write, then bullets do not need to go into a detailed level. In contrast, if you are unsure about the narrative of a certain sub-section, use second-level and third-level indented bullets and clarify to yourself how that section can be written.

When this stage is finished, you have a narrative that only waits to be transformed into actual paragraphs. The most important work for the text has been completed!

Example

Below is an authentic example of bullet-style writing from a draft that we eventually developed into a paper that we presented in CHI 2017 conference. You are seeing our plan for a sub-section in the background section from an early stage of the writing process.

Tip

If you use Microsoft Word, make yourself familiar with Ctrl-Shift-arrows (Mac) or Alt-Shift-arrows (Windows) keyboard shortcuts. With up and down arrows, you can reorder entire paragraphs easily. With left and right arrows, in turn, you can quickly change the level of indentation in your bullet points.

5. From bullets to full text

After finishing the bullet point plan, the hard part of writing the paper is over. This is because the bullet point version already outlines for me how to tell the narrative, present the evidence and the findings, and other important matters in the text.

I can now take any bullet point and turn it into approximately one paragraph of final-looking text. I can work on any part I wish because I know what has been said before and what will be said later. Therefore, depending on the mood and inspiration, I can work on different parts of the text on different days, whichever seems best.

The first final-looking paragraphs of text do not need to be perfect. I usually do not mind about detailed terminology on this stage yet, for example. I focus on writing down the text so that it preserves the idea in the bullet point, even if the expressions are clumsy and have repetition.

6. Polishing

Once I have written the first draft of the final-looking text, I can focus on improving its language. I straighten the terminological confusions by replacing terms so that I always use the same term in the text. I also focus on writing better sentences that flow nicely. This step usually requires several rounds, but they are fairly stress-free.

Then the text is ready.

Conclusions

The process described in this blog post – especially the recommendation to use bullet points wisely as a vehicle for thinking – are a way to advance in small steps towards a finished article. Although it is not guaranteed, the writer’s block may be better kept away, since there is never a moment when you would need to attempt writing finished-looking from scratch.

Bullet-point style writing has an additional benefit for situations that are common in academic writing: which is collaborative authoring. In many areas or research, academic publications are often written in a team. The writing process is possible to orchestrate using the same bullet point style as individual authoring: one author may take the responsibility for thinking through the paper’s entire storyline, and write bullet points for it. Then different parts can be assigned for different people, and the team members can write their sections independently, with an awareness of how their texts will relate to the whole. This is worth of its own blog post at a later time, but I wanted to mention it here as an opportunity.

Acknowledgments

I want to thank Heidi Paavilainen for informing me about Simon Peyton Jones’s text.

Although there is a strong push within the academic community to make research freely accessible (“open access”) to everyone, many research papers are unfortunately still published and available “behind the paywall”. That means that the publisher of the journal, book or a conference proceedings requires someone to subscribe the content. Typically that payer is a university library, who then can make the content accessible to the university students and employees.

But making use of this subscription can be complicated. Even if the seeker for papers – such as a student or a researcher – would have an affiliation with a university, it is not self-evident how one can get past the paywall and download the paper.

This post presents ways to get through the paywall, but requires that you have a user account in a university.

The focus of this post builds on an assumption that the seeker’s main literature search tool is Google Scholar, due to its simple use and excellent coverage of different research papers.

Google Scholar with basic internet connection: limited access to “full texts”

It is important to notice that Google Scholar provides access to articles differently depending on the type of internet connection. The difference shows in the links that Scholar displays in its search results.

When a user searches papers with Google Scholar with an ordinary internet connection, Google Scholar helps find papers, but cannot always provide the user with the “full text”. By full text, publishing companies mean the PDF that contains the article in its entirety. If access is not full text, then the user can only see the title, abstract, references, and maybe the first page of the paper.

It is sometimes possible to get an access to a full text even with this internet setup. This happens if:

• The text has been published as Open Access (OA) or with a permission that the authors are allowed to upload a copy of the text to their personal public repository of papers.
• Someone has uploaded the text somewhere in the Internet even if it is is not OA, thus possibly breaking the copyright of that content. Google has then found that paper.
• The paper’s preprint or so-called “accepted manuscript” version is available from the authors of the paper. A pre-print is often an earlier version of the paper: similar or even identical in its content with the final one, but not copy-edited to the final layout, and therefore lacks correct page numbers, and may have minor typing errors.

The screenshot below shows an example of what a Google Scholar search can produce as an outcome.

From the screenshot, different kinds of links can be seen. They provide different levels of access to the content:

• [PDF]: full text can be downloaded from the linked web page
• [HTML]: full text can be usually downloaded, but not always
• Getit@Grifols: this is a link whose purpose is not clear to me. No full texts available, anyway.
• The link of the paper title: takes the user to the publisher’s website. Full text may or may not be downloadable from it.

As a base rule, if the text does not have any links in the right-side column, the full text PDF is not available from any source.

Google Scholar with university’s VPN connection

The access to papers can dramatically improve if you tunnel your Internet connection through a university’s VPN service. With VPN, all the Internet traffic from a computer will travel (i.e., is “tunneled”) through a designated server. Both Google Scholar and the publishing companies then recognize that the user is connecting them from an Internet address that has a right to access also subscribed contents.

It therefore makes sense to use VPN in using Google Scholar. Here is how the same search results look like with VPN. Note that the two previously inaccessible papers are now downloadable via a “sfx@Aalto” link. Sadly, Klein & Weitzenfeld’s text in Educational Psychologist still remains unaccessible: my university does not have a subscription to its contents:

I cannot be sure, but I believe all the universities provide a similar service as Aalto University does, and therefore VPN opens doors to papers (for those who have a university account, of course). Here are the instructions for Aalto users for how you install the VPN client on your computer.

Accessing papers without VPN

It is not necessary to use VPN to access texts, however. An alternative is to navigate to a university library’s search interface and download the paper through it. Every library works slightly differently in service its users. At Aalto University, it is possible to enter e.g., the paper’s title, and the service tries to find a matching piece of content from its digitally subscribed sources. Aalto University’s article search interface is here.

A good idea is to use the text’s DOI (digital object identifier) as the search term instead of a paper’s name, because otherwise you may get lots of unnecessary search hits too. DOI uniquely identifies the paper, and can be always found somewhere from the publisher’s website.

When you find the desired search result, click on the links that promises to take you to the electronic full text. At that stage you will need to provide your user name and password, to prove that you are entitled to access the content.

Another Aalto-specific tip is to use the list of digital paper libraries in libproxy.aalto.fi. If you know the publisher of a paper your are interested in, you can go to that library via a link in the libproxy page. As long as you browse papers in that library, you are surfing within the paywall, and will be able to download full texts.

Paper request from ResearchGate

As the last resort, you may go to ResearchGate or Academia.edu. These are services where researchers can upload their works and create profile pages. If a researcher has uploaded the paper to the service, ResearchGate/Academia.edu provides a feature where you can ask a researcher to send a copy of the paper to you privately.

A word of advice: It is best to use this feature only after you have tried the other possibilities above and they have failed. By my personal experience, it is always irritating to react to ResearchGate’s paper requests if the requested paper is also available as Open Access. A request in such a situation only shows that the requester has not spent even a minimal bit of effort to find it. I get paper requests from ResearchGate approximately once a week. Much more cited researchers therefore probably get several requests each day.

Final words

As can be seen, retrieval of a text can require quite a bit of work. It is better to minimize the amount of work that you need to spend in downloading. Therefore, always download and store the paper on your computer! Also make sure that you can easily find that paper later on from your computer. For example, use a systematic file-naming principle (e.g., authorname year paper title.pdf) for all the papers, or start using reference manager programs such as Mendeley or Zotero. That will save you time next time, and lets you annotate the texts that you read with your own observations.

Acknowledgments

Thanks to Markku Reunanen for informing about the digital library listing at libproxy.aalto.fi.

Introduction

Knowing and following what others have written about (and around) your own research topic is the basic requirement for any academic project. But how should this be done? Some things are obvious: this task involves, at least 1) searching for possible texts to read; 2) scanning them to filter the good ones out from the mass of less relevant ones; 3) reading the most promising ones; and 4) building syntheses about this mass of texts.

The list is rather simple, but it is clear a lot of complexity underlies these steps. Some of the questions are:

1. What tools should be used for finding texts?
2. What types of texts are out there?
3. What indicators reveal what texts are more credible than others?
4. How these texts can be downloaded?
5. How should the article collection be maintained?
6. How can the synthesis be generated out of a collection of texts?

This blog post focuses only on questions 1–3 and a bit on question 4. I have answered to question 4 also separately in a different post. The others may be looked at later.

1. What tools should be used for finding texts?

At least the following ways exist for finding literature:

1. Standard Google search helps you find a very eclectic mix of texts that have a varying quality and underlying intentions. Very few of the papers found this way are academic papers. Instead they can be, for example, memos written by thinktanks and lobbyist groups, governmental bodies’ reports, press releases, essays written by students. Occasionally, also academic papers can be found with standard Google search, but what one finds is very unpredictable.

While what one finds using this method may be useful, it is usually best to regard contents found this way more as “data” rather than as research knowledge.

2. Google Scholar. This is the best tool for exploratory search for literature: for those situations where you want to find out “what is out there”. Using Google Scholar is like using standard search, but the results are different: they are from academic sources, such as journals, conferences, and books. The search results also contain additional information that help you interpret which papers are better to investigate in more detail

Because Google Scholar is a great tool, its use is covered in more detail below.

3. Snowballing. Academic papers always contain a section called References that lists all the other research that has been cited. By reading the paper and finding out what earlier works are cited, and what the writers say about them, it is possible to get to the sources of knowledge. This helps you find the “must-reads” of the research topic. The problem is that snowballing works only backwards in time: it does not help you find the most recent research.

4. Content alerts. It is possible to ask journals and conferences to send email to you every time they publish a new issue. This is a good way for staying up to date on the most recent research. The problem is that this brings a lot of email to your mailbox, and every issue does not contain articles that you would be interested about. Google Scholar, however, lets the user create keyword-based alerts: send you email every time it finds new research that matches given keywords. You can turn on this feature by clicking on in the left side of the screen (see the image below).

For a PhD students and researchers who need to keep themselves up to date about a research area over a long period of time, this is an essential feature to use.

5. Databases. EBSCOHost, Proquest, ABInform, IEEExplore, Scopus, ACM Digital Library and other databases are great for systematic literature reviews when you know exactly what keywords to use and what journals and conferences to include in your search. But because each database only covers certain journals/conferences, and usually disregards books completely, they are not optimal for exploratory search for knowledge. For that, Google Scholar is much better and nicer to use too.

2. What types of texts are out there?

Given that I recommended Google Scholar as the primary tool for searching texts, I will focus on its use from now on more than on the others.

Academic peer-reviewed papers

In the above, the main characteristic that I mentioned as the difference between the standard Google search and Google Scholar search was that the latter one finds only “academic papers” instead of just any texts or search hits. It is therefore important to define what is unique in academic papers.

The main characteristic of an academic paper is that it is “peer reviewed”. This means that there is a particular editorial process (“review process”) that the paper has undergone before it has been published in a journal or a conference. In this process, It has been examined by a jury of researchers and the authors have had to improve the paper until it has met the necessary quality requirements. Without an exception, the process includes at least one cycle of improvements: the authors have first sent (“submitted”) their paper for review, the members of the jury have evaluated it by writing statements about it, and the authors have been asked improve the paper. Alternatively, the authors have received a “reject” meaning that this process is terminated, and they have to find a different journal/conference that may be willing to publish the paper. They have to start the process again with that other outlet. If the paper, however, was considered promising enough, the second cycle starts when the improved version is received from the authors. The reviewers will evaluate whether the changes are sufficient, and provide further comments. In conferences, one cycle is common; in journals at least two cycles is the norm. In every stage, the possibility of a reject is always possible.

Usually the review process is “double-blind”: the authors do not know who will read their paper, and the members of the jury (i.e., “reviewers”) do not know whose paper they are reading. The communication between the authors and the reviewers is handled by an editor who is a senior researcher in the field, and is responsible for keeping up high standard of this process. The blindness increases the neutrality of the process: even famous academics’ papers can be rejected, and the reviewers do not need to face the consequences of furious authors who are angry at the rejection of their paper. Most papers are rejected; good conferences typically reject 70-75% of the submissions, for example. Top journals reject a larger percentage than that.

The heaviness of the whole process makes paper publishing a slow business. To publish a paper in a good journal often takes at least 2 years, with 3–4 cycles of improvement. But it ensures much better quality for the content, compared to papers that have not had a review process. Thus puts the academic papers apart from other materials that standard Google search can offer.

Books and book chapters

Books are another common type of academic texts. They exist in two kinds: full books that have been written by the same group of people from the beginning to the end, and edited collections where different chapters have been written by different authors. Edited collections have editors who have gathered the texts together and have usually had at least some form of peer review process in the book chapters’ preparation.

Other sources of academic-like texts

There are also semi-academic papers: ones that have been written by researchers, but which have not undergone the review process. These include research institutions’s “white papers” and technical reports, as well as texts that accompany presentations given in research seminars and workshops. These texts are usually published only in a website, instead of in a journal or a conference proceedings.

There are also papers that have been submitted for a review, but which have also been saved in a public repository such as Arxiv, Biorxiv, Citeseer or SSRN. Although doing so breaks the blind review policy, in some fields of science this is accepted and widely used practice. One of the reasons for this practice is the competition within the scientific field: researchers compete for being the first ones to make a certain finding. They do not want to wait the 2 years in the review process before they can tell about the finding. They may also fear that an anonymous reviewer steals their idea, replicates the study, and publishes it as their own. Public archiving protects authors from that.

Finally, papers are also available from ResearchGate and Academia.edu. These are self-archiving repositories where researchers sometimes upload copies of their published works, or where they just publish their research, thereby bypassing the review process. The quality of the content in ResearchGate and Academia.edu varies wildly, and needs to be verified: has the paper been published somewhere, or has it been only uploaded here?

3. What indicators reveal what texts are more credible than others?

It so far seem that just using Google Scholar ensures that every text has the required quality and can be used as a good piece of literature. The truth is not that simple: there are conferences and journals with different levels of quality. Some papers, even if they are per-reviewed, have low quality. Using just any source that one finds may lead to 1) misleading directions; 2) unnecessary amount of work.

There are three simple indicators for finding out which paper is more worthwhile to read than others:

The exact topic of the paper

This is the simple one: it is better to read papers whose titles and abstracts have a good fit with the information that one is looking for. Google Scholar presents the titles of the papers very clearly. In addition, the abstract of the paper can be inspected by clicking at the title. It either shows a popup window or takes the user to the publisher’s website.

The number of citations

Good papers end up usually cited more often than others by other researchers. The number of citations is the total count of all the other papers that cite a given paper. In the following screenshot, for example, Google Scholar tells that the 4th search result has been cited 2325 times by other researchers while other papers have been cited much less. This tells that Dorst and Cross’s paper published in Design Studies is probably more appreciated by researchers than other papers, when it comes to “framing in design process” as the topic.

Example of a search result in Google Scholar.

Citation count is a good indicator for choosing which papers are “must reads” and give the most relevant information.

The quality of the journal or conference

Although the citation count is a good indicator, it works poorly especially in the evaluation of the importance of very recent research. Recent publications have not had a chance to accumulate citations yet, and seem therefore less relevant. In addition, sometimes there are no publications that would be highly cited, because the research area that the user is interested about is very particular and not much researched.

Then the user should look at the quality of the journal or conference that has published the research. For most journals, it is possible to find what its impact factor is. It is a value that is computed by the number of citations that the papers in the journal gather on average over time. Clarivate Analytics’ JCR (Journal Citation Records) is the most often used impact factor service. It was earlier known as Thomson Reuters. JCR is not publicly accessible: one needs to access it through an university library. Aalto University users can click here to access JCR.

The impact factors for journals range from 0 to several dozens. For example, in the top, impact factors for New England Journal of Medicine, Lancet, Nature and Science are currently 75, 60, 43 and 42, respectively. The problem with the impact factors is that in other fields the best journal may have much lower impact factors. In HCI, Human-Computer Interaction has the highest impact factor, which is currently 4.2. In design research, Design Studies is the leading journal, and its impact factor is 2.8. Design Issues – another good one – is not listed at all, surprisingly. These differences do not mean that design or HCI journals would be of poorer quality than natural science journals – fields cannot be compared based on their journals’ impact factores. Many factors affect the impact factor, including the publication volume in the field, peer competition, the status of conferences or books as reputable publishing outlets, and the centeredness of the field around only a handful of journals, for example.

All this just means that impact factors are meaningful only if one knows already what the range the values is in a given field. In addition, Clarivate Analytics’ JCR does not provide impact factors for conferences, which makes its relevance to HCI much less meaningful.

A better approach, at least in Finland, is to use Finland’s own academic ranking system called “JUFO” – short for “Julkaisufoorumi”. It ranks every journal and conference using 4 levels:

• 3 = the top journal/conference in its own field
• 2 = a really good journal or conference
• 1 = other journals and conferences that have a peer review process
• 0 = journals and conferences that are known to exist but which cannot prove that they follow the sufficient academic review standards

Generally speaking, any paper published in journal or conference of level 2 or 3 has content that can be considered seriously. Many level 1 outlets are also really good, but there the quality varies a lot. Level 0 conferences and journals should not be used as references. JUFO can be accessed here.

Summary

When you search for literature, you can use the following process:

1. Try different kinds of search terms in Google Scholar. Often you do not manage to use the best search terms at the first attempt.
2. When you seem to be getting promising results, look at the titles and the citation counts: they tell which papers are 1) best matches with your interests and 2) most valued by other researchers.
3. If all citation counts are low, look at the outlets: which conferences and journals have published these works? Prioritise ones whose JUFO ratings are 2 or 3. Consider also ones that have a JUFO 1 rating.
4. Download every promising paper on your computer.

4. How these texts can be downloaded?

The last step above involves a challenge that will be addressed in the blog post: a vast majority of academic papers is not freely available. Instead they are available from publishers who sell them to universities with a subscription fee. To see, download and read them, one needs to use a university’s authenticated Internet connection.

I have written about this separately too, but my quick advices are to 1) look for links that Google Scholar marks with [PDF] – those are freely accessible papers; 2) tunnel your internet traffic through a university’s VPN service. That makes the publishers open most of the doors for you. Then you can find links with “sfx” in them – they are contents that your university has subscribed; 3) use your university’s article search service: copy the paper title and paste it to the university’s search engine. If the paper can be accessed, you get a link where you can download the paper. Here is the link to Aalto university’s search interface.

See how the same search result page as above has changed when I have used VPN:

Read also my other blog post to find out how to access and download articles when they are not openly accessible.