Exploring the Boundaries

“Out flew the web and floated wide” – Tennyson, The Lady of Shalott

The ability to detect sharing and recommendation events that enabled the creation of the altmetrics movement also offers to enrich our understanding of how scholarly communication is used in education and governance, and how research outcomes may influence society as a whole. As the trend towards open science and open access publishing continues, it will become critical for funding agencies, publishers, and researchers to understand these communication pathways and how to accommodate and adapt to these increasingly important usage scenarios.

Introduction

In retrospect, the period during which we relied upon formal citation of article-by-article as a measurement of usage, quality, and impact will appear to have been primitive. And the following period when we attempted to enlarge our view by using formal citation of articles in online platforms will be seen to have been a small step forward (Figure 1)—but far from a revolutionary step—in how we measure, appraise, and understand scholarly impact in society.

As with any system that relies upon measurement-by- proxy, conclusions about what those measurements might mean can only be relied on when backed by significant theory and evidence. It took approximately 20 years for bibliographic citation analysis to achieve acceptability as a measure of academic impact [Vaughan and Shaw], and it may well take another 20 years for web analytics to provide an adequate picture of how scholarly research influences society as a whole.

Just as bibliographic citation is the formal referencing of one work by another, so is much of the data in altmetrics the formal referencing of a work. In short, it is reference by hyperlink or DOI, and although some interesting work is being done by Altmetric.com to extend the reference scope, there is considerable effort needed to go beyond reliance on the formal citation link.

The current constituent elements of altmetrics’ scope are varied in their type. Articles may be added to social reading lists, mentioned in the mass media, subject to scrutiny in blogs and open referee platforms, or neutrally shared on Twitter. Reference or re-use can be made of the various constituent elements—the graphics, data, computer code, and methodologies. Conference slides and videos can be repeatedly viewed for years to come. An article has life beyond the journal and these different facets provide us with the possibility of some fascinating insights into that life. Altmetrics is at the first stage of providing us with this insight.

Clearly these different elements have a common feature. They are article-centric and, equally clearly, they can convey very different meanings about how the article is being consumed, used, and re-used.

The Cites Not Counted

Although altmetrics is making its first steps away from retrieving data that isn’t formally linked to the original paper, there is a wealth of data that has yet to be added to the corpus. In part, some of this is for historical reasons. Although scholarly books are largely online and it is technically possible to mine books for citations to journals, it hasn’t hitherto been the practice of the bibliographic experts to include the various book citation figures (i.e., when articles are cited by books, when books are cited by articles, and when books are cited by other sources). This isn’t to say that altmetricians couldn’t add this information to their data sources; although book citations are less well structured than journal references, there is considerable expertise and technology available for automatically identifying and resolving citations.

However, scholarly books are only the start of where this expansion might take us. There are numerous locations where research articles are cited beyond other research articles: government reports, professional institutions’ guidelines for best practice, and press releases, to mention a few (see Table 1). While these cites might convey radically different appraisals of what is meant, they are, at the moment, outside the sphere of either formal bibliometrics or altmetrics, while certainly being—from a technical and access point of view—readily analyzable.

Furthermore, there are many scholarly documents that might reference articles, including massive online open courses (MOOCs), coursepacks, and reading lists. Although it is important to stress that there is no assumption that a citation in a MOOC has an equivalence to a citation in an article, there is clearly room for analytics and interpretation in understanding the role of primary research in education at all levels.

However, not all activity is online—and not all online activity can be accessed. Clearly it is impossible to measure directly the extent of this activity, although we can borrow techniques from e-commerce marketing and we can develop research projects that will shed light on off-line usage.

Encouraging and enabling people to share online content using tools that yield usage data forms a large part of what e-commerce and e-marketing practitioners have been doing for over a decade. Any search on “tracking viral communications” or “encouraging marketing share” will yield millions of search results. And essentially, this is transferrable to the field of scholarly communications. Some may find the metaphor to be distasteful, but if output may be equated to a product, producers and publishers might seek conversion to a similar interim point (pageviews both for scholarly content and e-commerce) and then measure outcomes in definitive terms, albeit in terms of citation (whether formal or informal) rather than sales.

Much of this marketing advice would be to make articles easy to share—and indeed many scholarly platforms have added links and buttons to make citation easier, particularly when it comes to adding documents to specialized platforms such as Mendeley, Zotero, or CiteULike. However, we have a great deal to learn from how e-commerce platforms encourage user engagement, and it is no surprise to see the emergence of consultants who aim to improve social reach and impact.

Additionally, when publishers and researchers are involved in promoting scholarly work that promises to have a high uptake, we should actively encourage formal referencing, particularly in press releases. A generic scholarly system for sharing DOI-based links—perhaps allied with ORCID and CrossMark® for identity and versioning management, respectively—would not only enable tracking and usage statistics, but would hugely enhance the articles, e-mails, or bookmarks in which they were used.

Discovering the Differences: How Do Disciplines Differ in Influence and Reach?

The extent to which disciplines’ formal citation practices vary is well known, and it is assumed that different disciplines will have different social citation patterns. However, different disciplines have different socio-economic and legal environments and these have very different levels of transparency and public discussion and will vary over time as shown in Figure 2.

For example, the connections between research and medical best practice are well linked in the UK, with legal organizations that publish best practice guidelines citing primary research. This provides clear evidence of the social impact of this research; through its use in the guidelines, it may influence many thousand practitioners and millions of patients. (Unfortunately, these guidelines are usually identified by ISBNs—at least in the UK—and are, therefore, usually not included in formal journal citation counts.) In contrast, economists—who may occasionally make statements to the mass media and advise politicians, occasionally in public—wield enormous influence but with very little legal authority and limited governance. There is, of course, a great deal of difficulty in distinguishing the role of published advice. Often recommendations are made and ignored, or “taken under advisement” into other areas of policy; and there may be little to distinguish the status of these documents on government websites.

That said, the power of primary research to influence society is enormous, particularly in medical areas, which have large numbers of practitioners treating whole populations with quasi-legal governance.

Likewise, the degree to which research can achieve influence though education is unknown. The formulation and use of research citations in textbooks is uncertain whereas with monographs and serials, citation is much more journal-like. There is certainly a need for some significant research on how people use these different forms of publication.

For example, some research questions addressed to medical practitioners might be:

• Are revised guidelines circulated throughout the team and executed precisely?

• Are revised guidelines discussed and mediated before execution?

• What role does additional research, team experience, and finance play in the mediation of guidelines?

However, while publications are placed online in reasonably well-known locations, with reasonable provenance, and with citation forms that are predictable (and are therefore readable by computers), it is inevitable (and correct) that the altmetrics platforms will discover them and that these references will start showing up as part of altmetrics.

Altmetrics – An Advantage in a Competitive World?

Research has an unusual set of dynamics. It is not only collaborative—researchers are expected to use, refer, test, and improve on others’ work—but it is also competitive, with researchers competing for attention, publication, and research grants. Additionally, granting agencies may feel a competitive tension between themselves, and journals, publishers, and editors certainly compete for both authors and readership. Clearly the growing movement towards open science and open access publishing will address some of the balance in these competitive relationships.

Under the circumstances of a changing environment with competitive relationships, it seems likely that new elements will be brought into play to gain an advantage. Altmetrics is obviously an important element in these relationships. With all parties having an interest in impact (both scholarly and social) and reach (again, both scholarly and social), the promise of altmetrics is, at the very minimum, to provide some description of the reach of scholarly impact.

In fact, this description, although only a part of what altmetrics hopes to achieve, is an exciting prospect for all people involved in scholarly work. Hitherto, we have had the most crude figures for knowing whether research is being read or used. The increasing strength of altmetrics— particularly as the platforms compete over their relative efficiencies in different areas (e.g., mass media, non-English language platforms, and governmental publications)— will be to increase the detail and scope of the description of research in society. Not only will formal links, recommendations, and re-uses get counted, but linguistic and pattern matching technologies can be leveraged to discover softer citations.

However, the description is only one element of the work of altmetrics, and it is likely to be the simpler of the two parts of the movement.

The pathway from published research to social impact is multi-factorial and complex. As well as the socio-economic and legalistic frameworks in which research achieves its impact, there is cultural variation. For example, humanities research can become politically weighted when nations undergo a period of change [Tongshik] and linguistics and the management of lexical change can achieve quasi- governmental status [Académie française].

At the very least, these observations suggest that in order to begin the task of comprehending social reach in an objective way, it will be necessary to develop a methodology that can accommodate all these variations and to understand the interplay between the different elements that make up altmetrics data, coupled with their influence on the formal citation count. Fortunately, machine learning can provide us with these tools, but this work must be coupled with on-the- ground research to discover how people use, adapt, and translate research. It is possible that, over time, this human- scale work will migrate online and become part of the overall description—but we cannot wait for this to happen. A greater insight into how people work with research and how research reaches its impact at a human level is more within the scope of the humanities than computer science; but without this work—and without the mutual engagement of the humanities and altmetrics—the analytical part of altmetrics will only ever be a limited proxy for social impact.

The background to bibliometrics and the science and business of evaluation and comparison has set the scene for the advent of altmetrics. It is inevitable—given the competitive and dynamic environment—that one of the first ambitions of researchers in this area is to attemptto enhance existing figures in an evolutionary direction. However, the ability to detect sharing, recommendation, and influence is technologically mediated—continuing to grow, both qualitatively and quantitatively—and is the challenge for all fields of research. The potential for what we currently call altmetrics is nothing short of a complete map of scholarly activity and influence, one that is as complicated and multi-disciplinary as any field of study that exists at present. Altmetrics will grow to include all impact—including bibliometric citation—becoming a genuine revolution in scholarly communication.

Mike Taylor (mi.taylor@elsevier.com) is a Publishing Research Specialist at Elsevier in Oxford, UK.

ORCID: http://orcid.org/0000-0002-8534-5985 Twitter: @herrison