Digging into altmetrics

One of the talks at the recent Service Goes Accessible symposium was Social media and open access – the future of research? by Dr Toma Susi, Aalto SCI. From the abstract of his slides:

Another recent development is the increasing popularity of social media. It’s adept use is becoming an integral part of science outreach. In addition, it seems that such activity can directly influence the number of article downloads and citations, although more studies are needed to ascertain this effect.

This made me wonder how much our publications are visible and mentioned in various social media in the first place. To start with, I asked for an API key to two of the most well-known aggregator services, Altmetric and ImpactStory. Their APIs can be easily queried thanks to two R libraries provided by rOpenSci: rAltmetric and rImpactStory.

Altmetric and ImpactStory gather, clean and serve altmetrics. Altmetric is a for-profit start-up company, ImpactStory is funded by Alfred P. Sloan Foundation et al.

As a concept, altmetrics is still in its infancy, and a lively discussion is going on whether it aims to offer an alternative or complementary to traditional, citation-based metrics, a toolbox to measure the varied forms of scholarly communication of the digital age, or yet something else or all of this. For a readable introduction to various issues around altmetrics, see Judy Luther’s blog post Altmetrics – Trying to Fill the Gap and its comments. As a side dish: steps in scholarly communication in a handy, chronological table Why ’altmetrics’ now? by Tomoko Tsuchiya.

From all Aalto research output I decided to just take a look at publications with a Digital Objective Identifier (DOI), and more specifically, only those DOIs that the Thomson Reuters Web of Science (WoS) is aware of; in addition to just mere counts, I wanted to have a look whether there would be any signs of similar correlation between Mendeley saves and WoS citations reported by e.g. Altmetrics in the Wild: Using Social Media to Explore Scholarly Impact (referred to by Susi). My time span: from 2007 to present day. Please note though that I am not a scientist, and this blog post should not be taken as scholarly research. I am just curious.

The day when I ran the Altmetric query, late last week, the number of Aalto DOIs according to WoS, published since 2007, was 6879. Out of these, the Altmetric API returned metrics to 388.

rImpactStory returned a rather elaborate nested list, and with my present R skills I need more time to digest it. Unlike in rAltmetric, there is no helper function (yet) that would pull out relevant scores in a data frame. Therefore, I left the ImpactStory query result unprocessed for the moment and instead, made manually two collections at the ImpactStory web site. The first one is on publications that occupy the top 10 in Mendeley saves, according to Altmetric. The other collection is on publications that take the top of WoS citations. Please take a look how ImpactStory visualizes different metrics, the links are given further below. I hope they are persistent. For a list of all metrics that ImpactStory gathers, see their FAQ.

Note that also Altmetric makes visualizations. Moreover, they have introduced an Altmetric score. One example of a page with the Altmetric “donut” and the score: Rebounding Droplet-Droplet Collisions on Superhydrophobic Surfaces: from the Phenomenon to Droplet Logic. This publication has had good media coverage also locally.

So, according to Altmetric, which Aalto publications have been mentioned, shared, recommended and made attention to the most? Legend of the numbered lists below:

  • name of the publication with a live link to the publisher’s site
  • publication year
  • number of mentions in this particular media
  • number of unique paper saves in Mendeley reference management and PDF organizer service
  • number of Web of Science (WoS) citations

Top10 Mendeley saves and Top10 WoS citations are each plotted as a grouped bar chart along with scores of WoS and Mendeley, respectively, and with some – significantly lower – other metrics.

Note that while Facebook doesn’t readily come to mind when whe talk about science, “The Altmetric list will be papers popular with both scientists and members of the public”, as Euan Adie from Altmetric commented in Nature News blog.

Top5 in Twitter

  1. Early Specialization for Voice and Emotion Processing in the Infant Brain (2011) Twitter: 9, Mendeley: 29, WoS: 8
  2. Genome-wide association study identifies multiple loci influencing human serum metabolite levels (2012) Twitter: 9, Mendeley: 21, WoS: 11
  3. Entropy of Dynamical Social Networks (2011) Twitter: 9, Mendeley: 21, WoS: 0
  4. Networks of Emotion Concepts (2012) Twitter: 9, Mendeley: 1, WoS: 2
  5. Correlated Dynamics in Egocentric Communication Networks (2012) Twitter: 9, Mendeley: 2, WoS: 0

Top5 in Google+

  1. Sex differences in intimate relationships (2012) Google+: 9, Mendeley: 66, WoS: 1
  2. The impact of salient advertisements on reading and attention on web pages (2011) Google+: 7, Mendeley: 25, WoS: 4
  3. Global sea level linked to global temperature (2009) Google+: 7, Mendeley: 10, WoS: 127
  4. Reversible switching between superhydrophobic states on a hierarchically structured surface (2012) Google+: 7, Mendeley: 19, WoS: 0
  5. Climate related sea-level variations over the past two millennia (2011) Google+: 5, Mendeley: 51, WoS: 19

Top5 in Facebook

  1. The Existence of a Hypnotic State Revealed by Eye Movements (2011) Facebook: 6, Mendeley: 15, WoS: 2
  2. Clay Nanopaper with Tough Cellulose Nanofiber Matrix for Fire Retardancy and Gas Barrier Functions (2011) Facebook: 2, Mendeley: 14, WoS: 22
  3. Genetic variants in novel pathways influence blood pressure and cardiovascular disease risk (2011) Facebook: 2, Mendeley: 115, WoS: 115
  4. Two-Dimensional Nanostructured Growth of Nanoclusters and Molecules on Insulating Surfaces (2012) Facebook: 1, Mendeley: 7, WoS: 0
  5. Rebounding Droplet-Droplet Collisions on Superhydrophobic Surfaces: from the Phenomenon to Droplet Logic (2012) Facebook: 1, Mendeley: 0, WoS: 0

Top5 in news feeds

  1. Climate related sea-level variations over the past two millennia (2011) Feeds: 5, Mendeley: 51, WoS: 19
  2. A single-pixel wireless contact lens display (2011) Feeds: 5, Mendeley: 14, WoS: 3
  3. The chemical structure of a molecule resolved by atomic force microscopy (2009) Feeds: 3, Mendeley: 145, WoS: 165
  4. The Existence of a Hypnotic State Revealed by Eye Movements (2011) Feeds: 3, Mendeley: 15, WoS: 2
  5. Rebounding Droplet-Droplet Collisions on Superhydrophobic Surfaces: from the Phenomenon to Droplet Logic (2012) Feeds: 2, Mendeley: 0, WoS: 0

Top10 in Mendeley article saves

Aalto top10 in Mendeley saves 2007-2012

  1. Structure and tie strengths in mobile communication networks (2007)
  2. Biological, clinical and population relevance of 95 loci for blood lipids (2010)
  3. The chemical structure of a molecule resolved by atomic force microscopy (2009)
  4. Genetic variants in novel pathways influence blood pressure and cardiovascular disease risk (2011)
  5. Brain Basis of Human Social Interaction: From Concepts to Brain Imaging (2009)
  6. Temporal Networks (2012)
  7. Measuring the charge state of an adatom with noncontact atomic force microscopy (2009)
  8. Single-shot readout of an electron spin in silicon (2010)
  9. The brain in time: insights from neuromagnetic recordings (2010)
  10. Methodology for combined TMS and EEG (2010)

The same articles as an ImpactStory collection (note that some metrics may be missing because of different sources) EDIT 19.12.2013: collections are history, and widgets will cease to work mid-January 2014. ImpactStory will be concentrating in profiles.

Top10 in WoS citations

Note that these are articles with 1) a DOI and 2) data in Altmetric.

Aalto top10 publications in WoS citations  (with DOI and Altmetric data)

  1. Biological, clinical and population relevance of 95 loci for blood lipids (2010)
  2. A unified view of ligand-protected gold clusters as superatom complexes (2008)
  3. Structure and tie strengths in mobile communication networks (2007)
  4. Current-Induced Hydrogen Tautomerization and Conductance Switching of Naphthalocyanine Molecules (2007)
  5. The chemical structure of a molecule resolved by atomic force microscopy (2009)
  6. Maps and streams in the auditory cortex: nonhuman primates illuminate human speech processing (2009)
  7. Global sea level linked to global temperature (2009)
  8. Self-assembly of Janus dendrimers into uniform dendrimersomes and other complex architectures (2010)
  9. Genetic variants in novel pathways influence blood pressure and cardiovascular disease risk (2011)
  10. The inner jet of an active galactic nucleus as revealed by a radio-to-γ-ray outburst (2008)

The same articles as an ImpactStory collection EDIT 19.12.2013: collections are history, and widgets will cease to work mid-January 2014. ImpactStory will be concentrating in profiles.

Few brief observations.

  • traditional (WoS) citations accumulate much slower
  • Mendeley saves and WoS citations are surprisingly close to each other in some publications

The authors of Altmetrics in the Wild write:

Figure 12 reports correlations on a restricted sample: only articles published in 2010. In this case the indicator values were left unnormalized and untransformed, so Spearman correlations were used because of high skew in the untransformed data [...] in the age-restricted sample (Figure 12), the correlations between Mendeley and Web of Science citations rivaled or surpassed those of Scopus, PubMed, and CrossRef citations for all three journals

Again, this blog is no science. That said and while at it, I still wanted to give R a try.

First I subsetted the data to those items published between 2009 and 2011. That would be my restricted sample. Any relation between these two variables, Mendeley and WoS?

Scatterplot of the relationship between the variables Mendeley and WoS

Looking at the scatterplot: perhaps. Then, with the R cor function, the Spearman correlation. The result: 0.614077 (n = 113). To my understanding, the score is moderate. But does it speak about the possible influence of Mendeley saves to the WoS citation score?

I would be more than glad if you’d like to share a comment!

The R code

Posted by Tuija Sonkkila

About Tuija Sonkkila

Data Curator at Aalto University Library. When out of office, in the (rain)forest with binoculars and a travel zoom.
This entry was posted in Coding, Data and tagged , , , , , . Bookmark the permalink.

9 Responses to Digging into altmetrics

  1. seed says:

    Wow, amazing blog structure! How long have you ever been running a blog for?
    you made running a blog look easy. The total
    glance of your site is excellent, let alone the content material!

  2. Superb, what a webpage it is! This web site provides valuable facts to us, keep it up.

  3. Tuija Sonkkila says:

    Good to hear that, and good luck!

  4. Pingback: Digging into altmetrics « Suoritin II | Sous le soleil de la BU...excatement ! | Scoop.it

  5. Pingback: Digging into altmetrics « Suoritin II | Exploring Altmetrics | Scoop.it

  6. Jason Priem says:

    Thanks for your thoughts; Makes sense. Do let us know if you have any ideas for other new features… We’re going to be adding lots of new stuff next few months, and working to get new ideas, always :-)

  7. Pingback: Digging into altmetrics « Suoritin II | AnyMetrics | Scoop.it

  8. Tuija Sonkkila says:

    Thank you Jason, kind words. Perhaps I’ll start calling my postings staff science ;)

    Thanks also for reminding about the other download formats on your site. I should’ve probably written more about why I left the ImpactStory result set out of this exercise, even though I noticed that there’d be a lot of interesting data to be extracted.

    Since some time now, I’ve been practising R, and as you know, at this stage all nails look like I’d need to hit them by the R hammer and that only. Karthik Ram has written both the rAltmetric and the rImpactStory packages, and I have the hunch that even he doesn’t have more than 24 hours in his day. Anyway, I will ask if he has plans to develop rImpactStory any further. I would love to contribute, but doesn’t have the skills quite yet.

    Another thing: I am fully aware (well, at least I’d like to say I am) of the limitations of only looking at data. Again, it is R’s fault!! Seriously, I like the way ImpactStory advocates for telling stories behind data but doing it in a coherent way by trying to normalize them via percentiles. Collections, I like that idea too.

    What is perhaps more challenging is where to tell stories and by whom. I’d guess it depends partly on where scientists feel their research home is in the internet. Some will always be happy with a 100% automated home à la Google Scholar Citations. Others want to have a say in what the home looks like and how accurate it tells about the occupant.

    I don’t really know about the situation in the US but in Europe, those universities without a modern CRIS service (Current Resarch Information System) are busy planning to put it up. Our university is among them. To give a front-end to research, to enhance collaboration etc. To my understanding, all CRIS systems with a good reputation can import data from outer sources. It is their lifeblood really. We’ll see whether our CRIS will, at some stage, incorporate altmetrics in one way or the other. Let’s hope so.

  9. Jason Priem says:

    Nice post! And don’t shortchange yourself–I think this is totally science! A great thing about science is there’s lots of ways to do it, and lots of ways to share what you find.

    Very interesting to see the Mendeley/WoS correlation…your results are concordant with a growing number of studies looking at this.

    To answer your question: yep, ImpactStory URLs are indeed persistant. Also, if you want the raw data behind the collection, you can download it as CSV or JSON using the buttons on the top right.

    Finally, since you’re writing about this and interested in it: any thoughts on how to make ImpactStory more useful for you? We’re still in very early stages, and excited to listen to users as we roll out new features.

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>