Since some time already, Altmeric has harvested information on how much various policy and guidance documents have been citing research publications. The way they do it is mainly by text-mining the References section of PDF documents, and verifying results against CrossRef and PubMed. As a friendly gesture, Altmetric also provides their data via the public API. All of this is great news, and possibly important when we think about the growing interest in assessing impact.
How much and on which fields has Finnish university research output gained attention in international (here = English) policy sources?
Related to this, Kim Holmberg et al. have already shown evidence in the research project Measuring the societal impact of open science of RUSE, that research profiles of Finnish universities seem to be reflected in altmetrics.
In this exercise, I narrowed the scope of publications to a single year, 2015, for practical reasons. First, it is the latest full year covered by the new Virta REST API (page in Finnish), the to-be Top1 open data source of Finnish research publications. Second, I decided to manually go through all policy cites, to get some understanding on what we are talking about here.
For those interested in gory details, here’s the R source code of getting, cleaning & querying the dataset, and building a small interactive web app on results. The app itself is here. Please be patient, it’s a little slow in its turns.
By default, the app shows all data in two dimensions: the number of Twitter tweets, and the Altmetric score. While at it, try selecting differenct metrics on the vertical axis. It reveals how, in this material, various altmetrics sources are relative to this algorithmic, weighted score.
The size of the plot points (circles) tells about the degree of openness of the publication. The two biggest ones refer to either an OA journal, or an OA repository, respectively. The smaller ones are either not OA, or the status is not known. This information comes from the publication entry proper. In case the article is co-authored by multiple universities, they may have given differing information on the OA status. This is a sign how unclear the terminology still is in the OA field. Note also that when you filter data by university, the colour scale changes from universities to reflect the OA status, but unfortunately the palette isn’t fixed. Anyway, I’m sure you’ll get my point.
Statistics
Out of those 4563 publications that have gathered at least some altmetrics (12.3% out of total 37K) and thus are represented here, 27 (0.6%) have been cited by some policy or guidance documents. To put this in altmetrical perspective, nearly 500 (11%) have been mentioned in some news site, which constitute the most heavy-weight ingredient of the Altmetric score.
In their preprint How many scientific papers are mentioned in policy-related documents? An empirical investigation using Web of Science and Altmetric data (Konkiel) Haunschild and Bornmann found out that, from papers published in 2000-2014 and indexed by WoS, only about 0.6% had got mentions in policy documents. Why so few? Their reasoning feels legit.
Possible reasons for the low percentage of papers mentioned in policy-related documents are: (1) Altmetric quite recently started to analyze policy documents and the coverage of the literature is still low (but will be extended). (2) Maybe only a small part of the literature is really policy relevant and most of the papers are only relevant for scientists. (3) Authors of policy-related documents often are not researchers themselves. Therefore, a scientific citation style should not be expected in policy-related documents in general. Thus, policy-related documents may not mention every important paper on which a policy-related document is based on. (4) There are possible barriers and low interaction levels between researchers and policy makers.
What was the corresponding percentage here, in Finnish 2015 papers? I haven’t checked how many of them is indexed by WoS, but with a very rough estimate of 50% would give us 0.1%
EFSA and food safety
By far the most frequent policy organisation in this Finnish sample is EFSA, European Food Safety Authority. From their How we work:
Most of EFSA’s work is undertaken in response to requests for scientific advice from the European Commission, the European Parliament and EU Member States. We also carry out scientific work on our own initiative, in particular to examine emerging issues and new hazards and to update our assessment methods and approaches. This is known as “self-tasking”.
EFSA’s scientific advice is mostly provided by its Scientific Panels and Scientific Committee, members of which are appointed through an open selection procedure.
A few EFSA Scientific Panels have selected Finnish members in the prevailing period of 2015-2018. They are
- Panel on Dietetic Products, Nutrition and Allergies (NDA) Marina Heinonen, University of Helsinki
- Panel on Animal Health and Welfare (AWAH) Liisa Sihvonen, University of Helsinki
- Panel on Additives and Products or Substances used in Animal Feed (FEEDAP) Maria Saarela, VTT Technical Centre of Finland
- Panel on Plant Protection Products and their Residues (PPR) Olavi Pelkonen, professor emeritus, University of Oulu
In addition, some also had Finns during the previous period, lasting until 2015.
All but one of the EFSA cites that Altmetric adds up for a given publication, come from EFSA Journal which is the publishing platform of the Scientific Opinions of the Panels. So, I guess it is a matter of taste whether you, in this context, like to count these mentions as self-citations or some such. In this 2015 dataset, the only other citing source for EFSA-originated publications is GOV.UK, referring Scientific Opinion on lumpy skin disease (AWAH) in its various monitoring documents.
In the above-mentioned preprint, authors note that there is no way of knowing whether the citing document indeed was a legitimate source, because sites that Altmetric track, include also CV’s.
However, considering the small percentage of WoS publications mentioned in policy-related documents (which we found in this study), we expect that only very few mentions originate from such unintended sites [as CV’s]
In my small sample here, if we consider EFSA Journal to be an unintended source, the percentage was significant. Still, numbers don’t necessarily tell anything about the potential impact of these publications. In fact, given their specialized and instrumental character, EFSA Opinions may indeed be very impact-rich.
Reading the Abstracts of the featured Opinions is fascinating. Proportionally, they are almost equally divided between NDA and AWAH.
NDA is often asked to deliver an opinion on the scientific substantiation of a health claim related to X, where X can be L-tug lycopene; fat-free yogurts and fermented milks; glycaemic carbohydrates and maintenance of normal brain function pursuant; native chicory inulin; refined Buglossoides oil as a novel food ingredient, and other chemical substances and combounds which, if they get scientific green light from the Panel, can eventually emerge on market shelves as functional food products. On the other hand, the span of influence of Scientific Opinion on Dietary Reference Values for vitamin A is potentially the whole human population.
Not all AWAH Opinion Abstracts are for the faint hearted. Topics range from slaughterhouse technologies such as electrical parameters for stunning of small ruminants, and use of carbon dioxide for stunning rabbits to global threats like African swine fever and Oyster mortality.
The only publication related to the broader theme of food and nutrition, and not cited by EFSA above, comes from University of Tampere. The impact of lipid-based nutrient supplement provision to pregnant women on newborn size in rural Malawi: a randomized controlled trial1-4 has been cited by World Bank.
Society at large
Out of the 27 publications in this sample, two fall in the thematic area of society. They come from Aalto University, and University of Jyväskylä, respectively.
Hedge Funds and Stock Market Efficiency is cited by Australia Policy Online (APO), whereas Democracy, poverty and civil society in Mozambique has got a mention from The Royal Institute of International Affairs. What is perhaps noteworthy with the latter is that it has a single author.
Health & diseases
Similarly, only two papers in this theme. The first one is from University of Tampere, and the second one is co-authored by universities of Oulu and Jyväskylä.
The effects of unemployment and perceived job insecurity: a comparison of their association with psychological and somatic complaints, self-rated health and satisfaction is cited by International Labour Organization (ILO), and Genome characterisation of two Ljungan virus isolates from wild bank voles (Myodes glareolus) in Sweden is mentioned by GOV.UK. However, here we have an unintentional source again; three authors of this paper belong to the staff of Animal & Plant Health Agency (APHA), and is listed in Papers published by APHA Staff in 2015.
Wicked problems?
In his kick-off speech of the new academic year 2015, Rector of University of Helsinki, Jukka Kola, is quoted to have said that academic knowledge is in the front-line of solving so-called wicked problems such as
Climate change, Euro-Russian relations, healthcare challenges, strict nationalist movements
Scientific impact on these would be really something, wouldn’t it? Yet, the first giant step for mankind is to miraculously, uniformly, honestly and openly acknowledge the problem.
From the perspective of Aalto University, had I chosen a wider year range for publications to this exercise, I would’ve found multiple citations from policy documents to our research on climate change, notably Global sea level linked to global temperature (2009), and Climate related sea-level variations over the past two millennia (2011). For the record, here are all the papers published 2007-2015 which have policy mentions at Altmetric (again, by DOI). The only extra is number of WoS citations, added manually to these 24 papers.
Future work
The harvesting work of Altmetric helps a great deal, but understandably their focus is on international and UK sources. How about Finland? How much is Finnish academic research cited in local policy documents? Are there differences between publications written by consulting companies and those written by authors coming from the academia itself? Few starting points for questions like these include
- Government’s analysis, assessment and research activities (Reports)
- Institutional Repository for the Government (links to publications of Ministries)
- Demos Helsinki
- Research Institute of the Finnish Economy ETLA
- Åland Islands Peace Institute (Report series)
See you in PDF parsing sessions!
