In this, slightly longer text, I address a topic that I have wished to explain a long time: how good research plans can be created.

Research designs

Which methods should I use in my study? This is a question that one must answer in each and every study, from simple courseworks to complex PhD theses that may take several years to complete. The question about the right methods is often hard to solve. This article aims to give ideas on how to survive in this task.

The correct methods can be difficult to choose because everyone of us has their personal favourite methods that are more inspiring than others. The researcher may be good at some method in particular – be it interviewing, prototype-building, a quantitative survey, or something else – and they would wish to use it also in the upcoming study. However, the method may not fit with the goals of the study. In such a situation, and also universally, it is important to understand that a method has to be chosen based on the research question (RQ) that is asked. That is, the RQ has a higher importance than the method; methods are merely tools for finding answers. (In this text, I will solely talk about RQs, but the term always includes also design problems).

RQ–method combinations are called research designs. The term “research design” describes the meaning of the concept very well: it refers to the way in which one designs their research: how one goes about carrying out the study and achieving findings. Term is commonly used in quantitative studies (see e.g., Trochim, n.d.), but it is applicable to any other kind of research: qualitative, constructive, action research, even to theorising. 

Research designs can be excellent or terribly poor. There would be, for example, severe flaws in a study that would seek to prove that one user interface (UI #1) of a mobile navigation app would be better than another interface (UI #2) by presenting visual layouts to the two UIs to six participants, asking their opinions about them, and analysing which one gets more positive responses. Perhaps most obviously, such a research design would flawed because interviewing is not the right method for evaluating action-oriented UIs such as navigation apps. A better research design would involve navigation tasks that would be performed using the two UIs. Even after that correction, many smaller methodological decisions would need to be determined. A finished, well-planned research design takes some time to create.

Research designing as a cyclical process

Therefore, it is a good idea to consider research designing as a process that involves many attempts and restarts until a good design is found. The following diagram diagram in Figure 1 presents the main elements of a research design – contribution, RQs and the methods – that need to be fit to each other, as well as the main directions of effects that spark changes between these elements.

Figure 1. Research design planning cycle

Let’s examine the above-presented comparison study of two mobile navigation UIs using this diagram. The left side of the diagram presents the ambitions that the researcher has for the study. Starting from the top on that side, the study’s contribution relates to the superiority of one UI over another. Perhaps UI #1 is researcher’s own, new design, and they wish to prove that it si better than UI #2, which is typical (so called “baseline”) in the market at the moment. Moving to the middle, RQ of the study could be then be something like this: “Is UI #1 better than UI #2 when used for navigation?”. Finally, the method would be to show the two UIs as visual layouts to six people, listen to their opinions, and determine the winning UI. As was already pointed out above, navigation can be better studied by asking people to navigate in the physical world rather than by interviewing them about their opinions.

This is a point where ambition meets realism: It will not be possible to get a valid answer to the RQ using this method. The process now moves to the right side of the diagram where either a new method is chosen, or the RQs and contributions are adapted to match with the method. If the researcher wants to stick to interviewing as the research method, then the RQ has to be changed. It may be possible to ask something like this: “In what ways will experts’ opinions about two mobile navigation UIs differ from each other?”

Now the study asks about expert opinions and does not consider actual end users. This decreases the need for participants, since experts are probably able to provide valid analyses even by just looking at visual layouts – end users would not be able to do that. Experts are probably also more likeminded in their assessments. Therefore maybe six participants will be enough.

Also, the RQ is now framed as an inquiry about differences between expert opinions instead of straightforwardly about finding a winning UI. That suits better for interviews. What is however lost is the sharpness of the intended contribution: with the new RQ, the researcher will not be able to claim that one UI would be better for navigation than another. Such a claim can be validly studied only by observing navigation in action. Now the intended contribution will be somewhat softer: to show that a number of experts consider one UI better than another UI, at least in some respects.

As a result of these changes, we have a workable research design. We have reached it by softening our intended contribution. We have also decided that we need to recruit experts and not end-users as our participants. Finding such people will be harder, but will allow us to carry out fewer interviews, possibly only six. Figure 2 illustrates the transformations that have happened in the research design planning cycle.

Figure 2. Research design planning cycle applied to the example in the text.

What the analysis above tries to illustrate is that creation of a research design involves going back and forth between the contribution, RQs and methods. This is normal, and every experienced researcher is used to doing this kind of process before they start their actual research. The process may take a considerable amount of time, because in addition to just thinking, also reading may be needed, since literature may help in making the RQs sharper with the help of theoretical assumptions and existing empirical findings. The end of this article presents some common “tricks” by which dead ends in research designing can be avoided, and it is eventually possible to find a combination of contributions, RQs and methods that satisfies the necessary criteria of validity and the ambitions of the researcher.

How do I know what methods are required by the RQ?

The text above emphasised that it is the RQ that determines what the right methods are. But how does one know, given that one has a suggestion for an RQ, what are the possible methods to consider? It is not possible to give a universal rule for the question, but good examples can be found from prior studies that have been published about similar problems. It is usually a good idea not to start inventing one’s own methods but to use ones that have already been used successfully. That also gives an idea on the complexity and labouriousity of the study.

Most important, however, is to precisely to understand what one is asking in the RQ. The question should be understood word by word, and taken very literally. This helps figure out what kind of methods (for data gathering and its analysis) could possibly be considered proper what kind of a method could be considered as being proper. Here are some examples:

• Is UI #1 better than UI #2 when used for navigation? This is a comparative question (because of the word “better” in it) and requires that two measurements, one from UI #1 and the other from UI #2, can be compared against each other. It is also asked matter-of-factly, in a universal and objectivistic manner: Is UI #1 better than #2, whereby it refers to all app users and assumes that the comparison is carried out from all possible viewpoints (navigation time, number of errors, stress level, etc.), and that it is independent of researchers’ or users’ subjective opinions. Also, the RQ talks about navigation, which points out that the study should consist of a navigation task and that not an interview, for example. Considering all of this together, a controlled experiment and quantitative methods would be considered the right way to answer to this RQ.
• What are the strengths and weaknesses of UI #1 and UI #2? In this RQ, the comparison is removed, and replaced with an analysis of the two UIs independently from each other. That makes the RQ easier to study. Also qualitative methods, such as interviews and subjective opinions, are now applicable. What is sacrificed is the ability to say that one UI is better than another. Some level of qualitative comparison of the differences is however possible. For example, one can say that “it is possible that UI #1 would be less stressful than UI #2”.
• How should a UI intended for stress-free navigation be designed? This question drops the comparison of two UIs altogether, and also focuses on the design process instead of the outcome of the design. It also narrows down the design focus on stress-freeness instead of focusing on UI’s design from every possible viewpoint. With this new orientation, now the focus must be on the ways in which stress-freeness should be taken into account in different design stages. One may think how stressful navigation situations could be studied in user research, how stress-freeness can be evaluated using prototypes, and so on. In the final design after the process, one can explain how different design choices are based on different attempts to stress-free navigation.

The RQ can be changed also in many other ways. These examples show that it is possible to formulate the RQ in different ways, seeking for a study that remains meaningful and interesting while making the methods realistic to carry out. This illustrates how to make use of Figure 1’s research design planning cycle: one starts from some contribution and an RQ that would suit it, then thinks about what they require from the methods. Then the RQ and the methods are reformulated until they match and the study is possible to carry out.

Triangulation: using several methods for one RQ

Most RQs can be answered using more than one method. This is good: If several methods are used to answer a question, their findings, when put together, are more reliable and valid than with any method individually.

In the best case, the methods that answer the same RQ triangulate each other. This means that their are different in such a way that they cancel each other’s weaknesses. The term’s meaning can be understood by considering the word’s original use in close-to-shore marine navigation. When the captain wants to know their ship’s position, it is best obtained by measuring the directions (“bearings”) to landmarks that are at a right angle (i.e., 90 degrees) to each other. In Figure 3, bearings to the Skyscraper and the Manufacturing plant offer such a situation: the errors of measuring the exactly correct directions to these landmarks cancel each other optimally. The second best solution is to use directions that are not at a right angle, such as bearings to the Skyscraper and the Peaky hill. The least favourable situation is to take the bearing to one landmark only and guess what the distance to it might be.

Figure 3. Triangulation in locating a ship’s exact position.

In research designs, triangulation is useful in an analogical manner. The answer to the RQ is matches with the question about the ship’s exact position. The methods that give answers to the RQ match with the bearings to the landmarks. Similarly as the bearings have their errors, also research methods have their individual errors and biases. Interviewing, which is so-called “self-report” method, for example, depends on participants’ truthfulness and verbalisation skills, and cannot be always trusted. The same applies to a diary-keeping and questionnaires – also they a
re self-report methods. Direct observation of people who carry out tasks without knowing that they are being observed, in contrast, does not have such problems on truthfulness or people’s verbalisation skills. On the other hand, it has its own problems as a method. Together, however, observation and interviewing triangulate each other excellently: if both methods offer the same findings, then our results are much more believable than with only either of the methods alone. Figure 4 illustrates the relationships between these methods using a similar diagram as Figure 3.

Figure 4: Triangulation in research design.

Several RQs and several methods

Finally, it is typical that a study has several, maybe 2 or 3, RQs that crystallize the main aspects of the intended contribution. One could think that these RQs triangulate, in their own way, the intended contribution. For each of these RQs, following the principle of triangulation described above, several methods may be considered. Although it may seem that a study with two RQs, each with two suitable methods that triangulate each other, would require 2 x 2 = 4 different methods, this does not need to be the case. Instead, often the same method can be used to answer two different questions. One can interview, for example, about matters related to RQ1 and then about RQ2. Thus, a research design of a full-fledged study about two mobile navigation apps may have the structure shown in Figure 5. In this example, only one method is used for answering the first RQ, which leads to somewhat weak findings. The second RQ, in comparison, is rather well addressed with three questions where two self-report methods and navigation tasks nicely triangulate each other.

Figure 5. Research design diagram for several RQs and methods.

This diagram is visual and easy to read. It is even possible to point out which methods triangulate each other and which ones involve redundancy (and are especially useful in increasing reliability in data collection) but are mostly susceptible for the same biases and weaknesses. Well-triangulating methods’ arrows can be placed in right angles with each other, while methods with redundancies have arrows that are almost parallel with each other.

In a research plan document, on the other hand, a more detailed format for the same diagram can be useful. Table 1 below shows one example for such a format. In the same way as in the diagrams above, it has the RQs in the left and methods in the right. One column has been added between the RQs and the methods: theoretical constructs and concepts. They are helpful contents from literature that are used to sharpen the focus of the study to most informative elements. In mobile navigation, such a concept can be, for example, multiple resources theory (Wickens, 2002) that presents a model for human attention and cognitive processing capacities in multi-tasking situations. It is informative since navigation using a mobile app involves always multitasking: division of attention between the surroundings and the app. Finally, Table 1 also divides methods into data gathering and analysis stages.

 

RQs	Theoretical constructs and concepts	Data/material that will be gathered to answer the RQ	How this data/material will be analysed to obtain answer to the RQ
…	…	…	…
Navigation test with end-user who use prototypes of UI #1 and #2	Multiple resources theory	Navigation task where every participant uses both UIs and where they have to continuously switch their attention between the surroundings and the app	Statistical comparison of completion times using the two UIs
(same as above)	Cognitive and physical load	Participants rate their subjective task load using NASA-TLX questionnaire (Hart & Staveland 1988)	Statistical comparison of between questionnaire answers
(same as above)	Human orientation	Interview on whether participants prefer maps that are oriented so that north is always in the top of the screen, or if the map should rotate based on the user’s facing direction	Qualitative analysis of participants’ considerations
…	…	…	…	…

Table 1. Research design matrix (incomplete).

When the research design has reached this stage, and if all the RQs have methods that have methods that involve triangulation, it is a good moment to start the practical preparations for the empirical parts of the study!

References

Trochim, W. M. K. (n.d.). Research design. In Research Methods Knowledge Base at https://conjointly.com/kb/ (accessed 17 April 2022).

Wickens, C. W. (2002). Multiple resources and performance prediction. Theoretical Issues in Ergonomics Science, 3(2), 159–177. https://doi.org/10.1080/14639220210123806

Hart, S. G. and Staveland, L. E. (1988). Development of NASA-TLX (task load index): Results of empirical and theoretical research. Human Mental Workload (1988).