{"id":154,"date":"2022-04-17T21:15:48","date_gmt":"2022-04-17T21:15:48","guid":{"rendered":"https:\/\/blogs.aalto.fi\/writingaboutdesign\/?p=154"},"modified":"2024-08-22T13:07:29","modified_gmt":"2024-08-22T13:07:29","slug":"research-designs-identifying-the-right-methods-for-the-study","status":"publish","type":"post","link":"https:\/\/blogs.aalto.fi\/writingaboutdesign\/2022\/04\/17\/research-designs-identifying-the-right-methods-for-the-study\/","title":{"rendered":"Research designs: Identifying the right methods for the study"},"content":{"rendered":"
\n

Main things about this topic are now also available in a 16-minute video! Thank you Michel Nader for approaching me and carrying out all the recording and editing tasks.<\/p>\n

MA thesis research designs in 16 minutes<\/a><\/p>\n<\/div>\n

In this, slightly longer text, I address a topic that I have wished to explain a long time: how good research plans can be created.<\/span><\/p>\n

Research designs<\/span><\/h1>\n

Which methods should I use in my study? This is a question that one must answer in each and every study, from simple courseworks to complex PhD theses that may take several years to complete. The question about the right methods is often hard to solve. This article aims to give ideas on how to survive in this task.<\/span><\/p>\n

The correct methods can be difficult to choose because everyone of us has their personal favourite methods that are more inspiring than others. The researcher may be good at some method in particular \u2013 be it interviewing, prototype-building, a quantitative survey, or something else \u2013 and they would wish to use it also in the upcoming study. However, the method may not fit with the goals of the study. In such a situation, and also universally, it is important to understand that <\/span>a method has to be chosen based on the research question<\/a> (RQ) that is asked<\/strong>. <\/span><\/i>That is, the RQ has a higher importance than the method; methods are merely tools for finding answers. (In this text, I will solely talk about RQs, but the term always includes also design problems<\/a>).<\/span><\/p>\n

RQ\u2013method combinations are called <\/span>research designs. <\/span><\/i>The term \u201cresearch design\u201d describes the meaning of the concept very well: it refers to the way in which one designs their research: how one goes about carrying out the study and achieving findings. Term is commonly used in quantitative studies (see e.g., Trochim, n.d.<\/a>), but it is applicable to any other kind of research: qualitative, constructive, action research, even to theorising.\u00a0<\/span><\/p>\n

Research designs can be excellent or terribly poor. There would be, for example, severe flaws in a study that would seek to prove that one user interface (UI #1) of a mobile navigation app would be better than another interface (UI #2) by presenting visual layouts to the two UIs to six participants, asking their opinions about them, and analysing which one gets more positive responses. Perhaps most obviously, such a research design would flawed because interviewing is not the right method for evaluating action-oriented UIs such as navigation apps. A better research design would involve navigation tasks that would be performed using the two UIs. Even after that correction, many smaller methodological decisions would need to be determined. A finished, well-planned research design takes some time to create.<\/span><\/p>\n

Research designing as a cyclical process<\/span><\/h1>\n

Therefore, it is a good idea to consider research designing as a process that involves many attempts and restarts until a good design is found. The following diagram diagram in Figure 1 presents the main elements of a research design \u2013\u00a0contribution<\/a>, RQs and the methods \u2013 that need to be fit to each other, as well as the main directions of effects that spark changes between these elements.<\/span><\/p>\n

\"\"<\/p>\n

Figure 1. Research design planning cycle<\/span><\/p>\n

Let\u2019s examine the above-presented comparison study of two mobile navigation UIs using this diagram. The left side of the diagram presents the ambitions that the researcher has for the study. Starting from the top on that side, the study\u2019s contribution relates to the superiority of one UI over another. Perhaps UI #1 is researcher’s own, new design, and they wish to prove that it si better than UI #2, which is typical (so called \u201cbaseline\u201d) in the market at the moment. Moving to the middle, RQ of the study could be then be something like this: \u201cIs UI #1 better than UI #2 when used for navigation?\u201d. Finally, the method would be to show the two UIs as visual layouts to six people, listen to their opinions, and determine the winning UI. As was already pointed out above, navigation can be better studied by asking people to navigate in the physical world rather than by interviewing them about their opinions.<\/span><\/p>\n

This is a point where ambition meets realism: It will not be possible to get a valid answer to the RQ using this method. The process now moves to the right side of the diagram where either a new method is chosen, or the RQs and contributions are adapted to match with the method. If the researcher wants to stick to interviewing as the research method, then the RQ has to be changed. It may be possible to ask something like this: \u201cIn what ways will experts\u2019 opinions about two mobile navigation UIs differ from each other?\u201d <\/span><\/p>\n

Now the study asks about expert opinions and does not consider actual end users. This decreases the need for participants, since experts are probably able to provide valid analyses even by just looking at visual layouts \u2013 end users would not be able to do that. Experts are probably also more likeminded in their assessments. Therefore maybe six participants will be enough.<\/span><\/p>\n

Also, the RQ is now framed as an inquiry about differences between expert opinions instead of straightforwardly about finding a winning UI. That suits better for interviews. What is however lost is the sharpness of the intended contribution: with the new RQ, the researcher will not be able to claim that one UI would be better for navigation than another. Such a claim can be validly studied only by observing navigation in action. Now the intended contribution will be somewhat softer: to show that a number of experts consider one UI better than another UI, at least in some respects. <\/span><\/p>\n

As a result of these changes, we have a workable research design. We have reached it by softening our intended contribution. We have also decided that we need to recruit experts and not end-users as our participants. Finding such people will be harder, but will allow us to carry out fewer interviews, possibly only six. Figure 2 illustrates the transformations that have happened in the research design planning cycle.<\/span><\/p>\n

\"\"<\/p>\n

Figure 2. Research design planning cycle applied to the example in the text.<\/span><\/p>\n

What the analysis above tries to illustrate is that <\/span>creation of a research design involves going back and forth between the contribution, RQs and methods<\/strong>. This is normal, and every experienced researcher is used to doing this kind of process before they start their actual research. The process may take a considerable amount of time, because in addition to just thinking, also reading may be needed, since literature may help in making the RQs sharper with the help of theoretical assumptions and existing empirical findings. The end of this article presents some common \u201ctricks\u201d by which dead ends in research designing can be avoided, and it is eventually possible to find a combination of contributions, RQs and methods that satisfies the necessary criteria of validity and the ambitions of the researcher.<\/span><\/p>\n

How do I know what methods are required by the RQ?<\/span><\/h1>\n

The text above emphasised that it is the RQ that determines what the right methods are. But how does one know, given that one has a suggestion for an RQ, what are the possible methods to consider? It is not possible to give a universal rule for the question, but good examples can be found from prior studies that have been published about similar problems. It is usually a good idea not to start inventing one\u2019s own methods but to use ones that have already been used successfully. That also gives an idea on the complexity and labouriousity of the study.<\/span><\/p>\n

Most important, however, is to precisely to understand what one is asking in the RQ. The question should be understood word by word, and taken very literally. This helps figure out what kind of methods (for data gathering and its analysis) could possibly be considered proper what kind of a method could be considered as being proper. Here are some examples:<\/span><\/p>\n

    \n
  • Is UI #1 better than UI #2 when used for navigation?<\/strong> This is a comparative question (because of the word \u201cbetter\u201d in it) and requires that two measurements, one from UI #1 and the other from UI #2, can be compared against each other. It is also asked matter-of-factly, in a universal and objectivistic manner: Is UI #1 better than #2, whereby it refers to all app users and assumes that the comparison is carried out from all possible viewpoints (navigation time, number of errors, stress level, etc.), and that it is independent of researchers\u2019 or users\u2019 subjective opinions. Also, the RQ talks about navigation, which points out that the study should consist of a navigation task and that not an interview, for example. Considering all of this together, a controlled experiment and quantitative methods would be considered the right way to answer to this RQ.<\/span><\/li>\n
  • What are the strengths and weaknesses of UI #1 and UI #2?<\/strong> In this RQ, the comparison is removed, and replaced with an analysis of the two UIs independently from each other. That makes the RQ easier to study. Also qualitative methods, such as interviews and subjective opinions, are now applicable. What is sacrificed is the ability to say that one UI is better than another. Some level of qualitative comparison of the differences is however possible. For example, one can say that \u201cit is <\/span>possible<\/span><\/i> that UI #1 would be less stressful than UI #2\u201d.<\/span><\/li>\n
  • How should a UI intended for stress-free navigation be designed? <\/strong>This question drops the comparison of two UIs altogether, and also focuses on the design process instead of the outcome of the design. It also narrows down the design focus on stress-freeness instead of focusing on UI\u2019s design from every possible viewpoint. With this new orientation, now the focus must be on the ways in which stress-freeness should be taken into account in different design stages. One may think how stressful navigation situations could be studied in user research, how stress-freeness can be evaluated using prototypes, and so on. In the final design after the process, one can explain how different design choices are based on different attempts to stress-free navigation.<\/span><\/li>\n<\/ul>\n

    The RQ can be changed also in many other ways. These examples show that it is possible to formulate the RQ in different ways, seeking for a study that remains meaningful and interesting while making the methods realistic to carry out. This illustrates how to make use of Figure 1\u2019s research design planning cycle: one starts from some contribution and an RQ that would suit it, then thinks about what they require from the methods. Then the RQ and the methods are reformulated until they match and the study is possible to carry out.<\/span><\/p>\n

    Triangulation: using several methods for one RQ<\/span><\/h2>\n

    Most RQs can be answered using more than one method. This is good: If several methods are used to answer a question, their findings, when put together, are more reliable and valid than with any method individually.<\/span><\/p>\n

    In the best case, the methods that answer the same RQ <\/span>triangulate <\/strong>each other. This means that their are different in such a way that they cancel each other\u2019s weaknesses. The term\u2019s meaning can be understood by considering the word\u2019s original use in close-to-shore marine navigation. When the captain wants to know their ship\u2019s position, it is best obtained by measuring the directions (\u201cbearings\u201d) to landmarks that are at a right angle (i.e., 90 degrees) to each other. In Figure 3, bearings to the Skyscraper and the Manufacturing plant offer such a situation: the errors of measuring the exactly correct directions to these landmarks cancel each other optimally. The second best solution is to use directions that are not at a right angle, such as bearings to the Skyscraper and the Peaky hill. The least favourable situation is to take the bearing to one landmark only and guess what the distance to it might be.<\/span><\/p>\n

    \"\"<\/p>\n

    Figure 3. Triangulation in locating a ship\u2019s exact position.<\/span><\/p>\n

    In research designs, triangulation is useful in an analogical manner. The answer to the RQ is matches with the question about the ship\u2019s exact position. The methods that give answers to the RQ match with the bearings to the landmarks. Similarly as the bearings have their errors, also research methods have their individual errors and biases. Interviewing, which is so-called \u201cself-report\u201d method, for example, depends on participants\u2019 truthfulness and verbalisation skills, and cannot be always trusted. The same applies to a diary-keeping and questionnaires \u2013 also they a
    \nre self-report methods. Direct observation of people who carry out tasks without knowing that they are being observed, in contrast, does not have such problems on truthfulness or people\u2019s verbalisation skills. On the other hand, it has its own problems as a method. Together, however, observation and interviewing triangulate each other excellently: if both methods offer the same findings, then our results are much more believable than with only either of the methods alone. Figure 4 illustrates the relationships between these methods using a similar diagram as Figure 3.<\/span><\/p>\n

    \"\"<\/span><\/p>\n

    Figure 4: Triangulation in research design.<\/span><\/p>\n

    Several RQs and several methods<\/span><\/h1>\n

    Finally, it is typical that a study has several, maybe 2 or 3, RQs that crystallize the main aspects of the intended contribution. One could think that these RQs triangulate, in their own way, the intended contribution. For each of these RQs, following the principle of triangulation described above, several methods may be considered. Although it may seem that a study with two RQs, each with two suitable methods that triangulate each other, would require 2 x 2 = 4 different methods, this does not need to be the case. Instead, often the same method can be used to answer two different questions. One can interview, for example, about matters related to RQ1 and then about RQ2. Thus, a research design of a full-fledged study about two mobile navigation apps may have the structure shown in Figure 5. In this example, only one method is used for answering the first RQ, which leads to somewhat weak findings. The second RQ, in comparison, is rather well addressed with three questions where two self-report methods and navigation tasks nicely triangulate each other.<\/span><\/p>\n

    \"\"<\/p>\n

    Figure 5. Research design diagram for several RQs and methods.<\/span><\/p>\n

    This diagram is visual and easy to read. It is even possible to point out which methods triangulate each other and which ones involve redundancy (and are especially useful in increasing reliability in data collection) but are mostly susceptible for the same biases and weaknesses. Well-triangulating methods\u2019 arrows can be placed in right angles with each other, while methods with redundancies have arrows that are almost parallel with each other.<\/span><\/p>\n

    In a research plan document, on the other hand, a more detailed format for the same diagram can be useful. Table 1 below shows one example for such a format. In the same way as in the diagrams above, it has the RQs in the left and methods in the right. One column has been added between the RQs and the methods: theoretical constructs and concepts. They are helpful contents from literature that are used to sharpen the focus of the study to most informative elements. In mobile navigation, such a concept can be, for example, multiple resources theory (Wickens, 2002) that presents a model for human attention and cognitive processing capacities in multi-tasking situations. It is informative since navigation using a mobile app involves always multitasking: division of attention between the surroundings and the app. Finally, Table 1 also divides methods into data gathering and analysis stages.<\/span><\/p>\n\n\n\n\n\n\n\n\n
    RQs<\/span><\/td>\nTheoretical constructs and concepts<\/span><\/td>\nData\/material that will be gathered to answer the RQ<\/span><\/td>\nHow this data\/material will be analysed to obtain answer to the RQ<\/span><\/td>\n<\/tr>\n
    \u00a0…<\/span><\/td>\n\u00a0…<\/span><\/td>\n\u00a0…<\/span><\/td>\n\u00a0…<\/span><\/td>\n<\/tr>\n
    Navigation test with end-user who use prototypes of UI #1 and #2<\/td>\nMultiple resources theory\u00a0<\/span><\/td>\nNavigation task where every participant uses both UIs and where they have to continuously switch their attention between the surroundings and the app<\/span><\/td>\nStatistical comparison of completion times using the two UIs<\/span><\/td>\n<\/tr>\n
    (same as above)<\/span><\/td>\nCognitive and physical load<\/span><\/td>\nParticipants rate their subjective task load using NASA-TLX questionnaire (Hart & Staveland 1988)<\/span><\/td>\nStatistical comparison of between questionnaire answers<\/span><\/td>\n<\/tr>\n
    (same as above)<\/span><\/td>\nHuman orientation<\/span><\/td>\nInterview on whether participants prefer maps that are oriented so that north is always in the top of the screen, or if the map should rotate based on the user’s facing direction<\/span><\/td>\nQualitative analysis of participants’ considerations<\/span><\/td>\n<\/tr>\n
    …<\/td>\n\u00a0…<\/span><\/td>\n\u00a0…<\/span><\/td>\n\u00a0…<\/span><\/td>\n\u00a0…<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    Table 1. Research design matrix (incomplete).<\/span><\/p>\n

    When the research design has reached this stage, and if all the RQs have methods that have methods that involve triangulation, it is a good moment to start the practical preparations for the empirical parts of the study!<\/span><\/p>\n

    References<\/span><\/h1>\n

    Trochim, W. M. K. (n.d.). Research design. In Research Methods Knowledge Base at https:\/\/conjointly.com\/kb\/<\/a>\u00a0(accessed 17 April 2022).<\/p>\n

    Wickens, C. W. (2002). Multiple resources and performance prediction. Theoretical Issues in Ergonomics Science, 3(2), 159\u2013177. https:\/\/doi.org\/10.1080\/14639220210123806<\/a>
    \n<\/span><\/p>\n

    Hart, S. G. and Staveland, L. E. (1988). Development of NASA-TLX (task load index): Results of empirical and theoretical research. Human Mental Workload (1988).<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"

    Main things about this topic are now also available in a 16-minute video! Thank you Michel Nader for approaching me and carrying out all the recording and editing tasks. MA thesis research designs in 16 minutes In this, slightly longer … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":3589,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[7],"tags":[],"class_list":["post-154","post","type-post","status-publish","format-standard","hentry","category-guidelines"],"_links":{"self":[{"href":"https:\/\/blogs.aalto.fi\/writingaboutdesign\/wp-json\/wp\/v2\/posts\/154","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blogs.aalto.fi\/writingaboutdesign\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.aalto.fi\/writingaboutdesign\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.aalto.fi\/writingaboutdesign\/wp-json\/wp\/v2\/users\/3589"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.aalto.fi\/writingaboutdesign\/wp-json\/wp\/v2\/comments?post=154"}],"version-history":[{"count":9,"href":"https:\/\/blogs.aalto.fi\/writingaboutdesign\/wp-json\/wp\/v2\/posts\/154\/revisions"}],"predecessor-version":[{"id":201,"href":"https:\/\/blogs.aalto.fi\/writingaboutdesign\/wp-json\/wp\/v2\/posts\/154\/revisions\/201"}],"wp:attachment":[{"href":"https:\/\/blogs.aalto.fi\/writingaboutdesign\/wp-json\/wp\/v2\/media?parent=154"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.aalto.fi\/writingaboutdesign\/wp-json\/wp\/v2\/categories?post=154"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.aalto.fi\/writingaboutdesign\/wp-json\/wp\/v2\/tags?post=154"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}