It would probably come as a surprise to most researchers, working with assumption that citation counts as the one true measure of research excellence, that there is no accepted theory of what it is we are doing when we reference a work.
As noted in a recent blog post, it is pretty clear that, when authors reference other works, they are doing many different things. To dig further into this and try to understand the implications in the current policy arena, I’ve taken a closer look at these competing theories to try to find out what we do – and don’t – know about the motivation behind the practice of referencing.
References are not citations
One of the most valuable distinctions is between in-text references and citation database entries. Most recently, Paul Wouters has drawn this distinction: an in-text reference is a marker in a text that refers to another text. These markers, occur in the text itself, generally matched with an entry in a bibliographic reference list (or in a footnote), and are the result of actions by authors.
By contrast, the line in a citation database that states that a reference exists is the result of an action by the database provider. It also strips all the author's meaning and intent. This abstraction may seem like hair splitting, each of them being reflections and inversions of each other, but the stripping of meaning is crucial because it is used to make citation database entries countable.
Having countable database entries is critical to constructing quantitative indicators. Even the humble “citation count” for an article is a construction, dependent on the workings of the database, and only indirectly on any decisions by authors.
At each stage we strip away meaning and also start to construct new meaning. Wouters has argued that, rather than needing a theory of referencing or citation, we need a theory of indicators.
Theories of theory
The above distinction in terminology is useful because it helps us highlight a key point: Most of the social theories we have about why authors choose to refer to a specific work are theories of referencing. The argument that citation databases are a reasonable basis on which to evaluate research has looked at correlations between citation counts. These are studies of data from citation databases.
We have little, if any, theory of indicators, either ones that provide a strong framework for building them and understanding their limitations, or of how they actually affect the behaviour of researchers and institutions. We have plenty of anecdotal claims and occasionally a little bit of data, but no robust way to predict how changes in indicator design actually play out in practice.
What about the power of choice?
Another issue to examine is what an author chooses to reference, and how they respond to indicators; these are issues of individual behaviour, while most of the work on citation data looks at correlations and association, not an individual’s choice.
While we often use correlations at the collection level to help us analyse or test a theory about individuals, it is a logical and statistical fallacy to work in the opposite direction. Where variances are low and correlations high – notably absent for citations – such associations might be useful as proxies or even predictors, but they cannot show causation.
Goodheart’s law, that when something is measured it becomes the target, is often cited as the source of our problems in the world of research evaluation. But it can only apply if the measure is only a proxy. If there were a grounded causal link then measurement would not lead to a change in behaviour. These abstract arguments really matter on the ground, both to understand how measurement affects behaviour and also to ask whether changes to incentives can affect behaviour.
Individuals and groups; normative and constructivist motivations
It may seem we’re no closer to an understanding of what is going on in the world of citation metrics, but the observation of differences between group attributes and individuals may offer a way in.
Broadly speaking, the social theories of referencing break into two categories. Normative theories state that scholarship is capable of working towards creating valuable insights because scholars observe a set of norms of behaviour. These include, importantly, the acknowledgement of others' contributions to work and the provision of supporting evidence that allows validation of that work.
Social constructivist theories focus on the process of persuasion and how authors navigate networks of power. They often focus on how references are deployed to persuade, to neutralise criticism and to defend claims, and are critical themselves of the outcomes of scientific endeavour. Neither strand of theory is satisfying in its extreme form, but several scholars have tried to bridge this divide. (Leydesdorrf (1998) and Small (2004)).
In my view, the way to pull these together is to identify how individuals in their own socially constructed context are motivated to identify with the norms of any one group. In turn, these norms are part of a culture that has survived over time because it is successful in producing value. Understanding how our current policy culture draws in the individual – as well as the institution - and its ability to enforce the normative behaviours is worth further examination.
But what does citation really mean?
A more recent strand of work on referencing and citations focuses on the question of meaning. This is what led Paul Wouters to analyse the differences between authors’ acts of referencing and an index’s creation of a citation database. This new strand of theory can help us to understand both where meaning is stripped away, but also where it is being constructed. It also offers us understanding of how stages in processing citation data add or remove meaning.
Perhaps most importantly, these discussions give us a rigorous way of examining how individuals are exchanging signs and meaning with the community. That is, what is the exchange process by which an author is seeking identity with a community and how does the community create a culture of social norms?
Understanding how and where meaning is constructed and whether or not it is underpinned by exchange of information will be crucial to developing an understanding of the varying motivations for referencing and the varying responses to indicators well beyond the implementation of the REF. More importantly, they will affect the choices that researchers and institutions make for years to come.
Cameron Neylon is an advocate for open access and professor of research communications at the Centre for Culture and Technology at Curtin University. You can find out more about his work and get in touch with Cameron via his personal page, Science in the Open.