JISC

Download this document

This report was commissioned by the Research Information Network (RIN) and JISC to gather and analyse evidence about the motivations, incentives and constraints that lead researchers in the UK in different subjects and disciplines to publish and disseminate their work in different ways, how and why researchers cite other researchers’ work, and in particular, how researchers’ decisions on publication and citation are influenced (or not) by considerations arising from research assessment.

Executive Summary

Researchers are driven by a desire to enhance our knowledge and understanding of the world we inhabit, and to communicate their findings to others. But both governments and other funders are increasingly interested in demonstrating the social and economic returns from their investments in research, and in assessing research performance.

The many different criteria for success, and the lack of any consensus on how success should be assessed or measured, however, mean that researchers often find themselves in receipt of confused or conflicting messages. And they are pulled in different directions in deciding which channels of communication they should adopt.

This report is complimented by four supporting papers which provide detailed descriptions of the methods used, a full analysis of the data, and further details of the findings.

How researchers publish and why

Researchers publish and disseminate their work in many different ways: through formal publication in books and in learned and professional journals; through conferences and their proceedings; and through a variety of less formal means, now including web-based tools for social networking. The choices they make are underpinned by a number of interrelated motives beyond the simple desire to pass on their findings to those who may be interested in them. These motivations include the desire not only to maximise dissemination to a target audience, but to register their claim to the work they have done, and to gain peer esteem and the rewards that may flow from that. Specific requirements from funders, or institutional guidelines, or pressure from coauthors or collaborators, are much less influential. In deciding when, where and how to communicate their work, researchers may have to make choices between speedy dissemination to a desired audience, and less speedy publication in a high-status journal. Such choices are made more complex because researchers know that publications serve not only as means of communication. They can be monitored or measured as indicators of quality or impact (in the academic world and more widely). And the difficulty in choosing between different channels of communication is exacerbated because researchers often find the messages they get from different agencies, including universities, conflicting or unclear. But the perception that their work is being monitored and assessed in particular ways, notably by the RAE, has a major influence on how they communicate.

Articles in scholarly journals are more easily ranked and measured using a series of readily-available and increasinglysophisticated metrics; and it is partly because of that – especially in disciplines where they have not predominated in the past – that they have come to dominate all other forms of publication. Yet there is a rich array of other kinds of output employed and valued by researchers, and many feel uncomfortable with the dominance of the article – particularly the article published in a high-status journal. They are concerned that communications through other channels – especially those that are better-suited to applied or practice-based research, and to communicating with non-academic audiences – seem to have low status and prestige in the academic world.

The only major exceptions to the dominance of the journal article are the continuing high status attached to monographs and edited volumes in the humanities, and to practice-based outputs in the arts. Yet even in the humanities, journal articles are now by far the largest publication format by volume; although books continue to be highly valued, including in submissions to the RAE, there are increasing concerns about the decline of the book, attributed variously to shrinking library purchase budgets, publishers’ reluctance, and by some, to the pressures of the RAE.

Many researchers are confused by the mixed messages they are receiving as to how best to communicate their findings. If they are to make optimal use of the various communications channels open to them, it is essential that researchers should receive more consistent and effective guidance on their use of different channels; and that in framing their messages, funders and others should take account of the value researchers themselves attach to the channels appropriate to their work.

Funders and policy-makers must also take account of the various misperceptions of their policies noted in this report. In particular, if they wish to encourage researchers to publish and disseminate their work through channels other than high-status journals, they will need to give stronger and more positive messages about how these channels will be valued when it comes to assessing researchers’ performance.

Disciplinary diversity

The motivations that lead researchers to publish in different formats – particularly in scholarly journals – differ significantly across disciplines. Researchers in the sciences are more likely to see publication in a learned journal as a ‘natural’ means of communication with their desired audience, while their colleagues in engineering, the humanities and the social sciences are more likely to see it as meeting essentially external requirements for research assessment and career advancement.

In these latter disciplines, therefore, the rise of journals is more closely associated with an environment where there is increasing emphasis on measuring, assessing, and evaluating research, its outputs and impact. Yet in the humanities especially, there is a complex, even contradictory, array of perceptions at work: researchers are producing more articles, partly because they believe that is what they are being told to do; but many resent the limitations (especially the brevity) of the format, and when it comes to the RAE, there is a strong tendency to submit books instead.

Many differences between disciplines relate to the speed with which they move, and the nature and scope of their engagement with non-academic audiences. In computer science, for example, the pace of change means that conferences are particularly important, and these may attract higher prestige than journal articles. Speed of development may also be a factor in the take-up of open access. Repositories have achieved less traction in the humanities and social sciences than in many science and engineering subjects.

In areas where applied research is a prominent feature, the choice between publishing in a prestigious journals and effective dissemination to potential users may be especially difficult. Researchers in areas such as cancer studies, nursing, psychology, education and politics all stress the importance of communication and engagement with practitioners and policy-makers. Tensions between effective dissemination and the prestige attached to publishing in a high-status journal seem to be less acute in the physical and life sciences.

Collaboration and co-authorship

The push from research funders for more collaboration across institutional, national and disciplinary boundaries is reflected in the growing number of multi-authored publications. Multiauthorship is the norm in the sciences and engineering, but much less common in the humanities. Its rise has also been accompanied by difficulties over issues including responsibility for the conduct and validity of the research, the inclusion and exclusion of individual authors, and the order in which authors are listed; and by complaints about some senior researchers abusing their position.

There are important differences of practice in the attribution and listing of authors. Listing in order of contribution is the commonest practice except in the humanities, where alphabetical listing is the norm. There are also notable variations in practice within discipline groups: in some areas, for example, the major or senior contributor may be placed last.

It is important that all who are involved in assessing research – whether via bibliometrics or through peer review – should be well-informed about different conventions and their meaning, and how they are changing. Funders, learned societies and publishers may also wish to consider whether they might take more of a lead in helping to devise guidelines on good practice.

What researchers cite and why

Referencing other work is integral to the process of communicating research findings, and citations can be found in virtually all publications. Researchers cite previous work to establish their knowledge of the context and to provide supporting evidence. But the increasing emphasis on citation data as a means of assessing research performance makes it more important that we understand how researchers decide what to cite.

The major influences on researchers are the perceived authority of the publications and the authors, although there are different views on which of these predominates. Our research does not support the suggestion that personal contact is a major factor in deciding to cite an author. Indeed, disagreement with previous findings is among the significant reasons for citing – strongly so in the humanities and social sciences, but in the physical and life sciences too.

Citations are clearly influenced by disciplinary norms. Humanities and social science researchers cite more sources on average, mainly because they write at greater length and cite primary sources as well as the work of peers. They also cite more grey literature and websites, and works with which they disagree. Scientists are more likely to concentrate solely on journal articles. Citation practice is largely self-taught. Few researchers have been trained beyond any guidance they received as young researchers from their supervisors. They also receive advice from reviewers  and co-authors, and they tailor their citations to meet to the real or perceived requirements of specific journals. Advice from reviewers and editors is often received positively, but may be seen as an attempt to promote their own work.

Access to online material has speeded up the process of finding, reading and deciding what to cite. A third of researchers in the life sciences – even more of the younger ones – say that easy accessibility has a major influence on what they cite. In the humanities and social sciences, accessibility has less influence. But citation practice is related as much to researchers’ length of experience as to the disciplines in which they work. Younger researchers are more likely to be influenced by the authority of or familiarity with an author, by the standing of the journal and by ease of access to the article. If such differences persist as younger researchers progress through their careers, funders and others concerned with assessing research performance may need to take account of significant changes in the patterns of citation.

Another increasing influence is the limits some high-status journals impose on the number of references to be included in an article. If such limitations continue to increase, one effect could be to lessen the usefulness of citation data for bibliometric and assessment purposes, even in those fields where they are considered robust at present.

Research assessment and its influence

The influence of the RAE on researchers’ behaviours and attitudes should be set in the broader context of their concerns about what they see as an increasing stress from funders and institutions on assessing and evaluating research and its impact (with impact varyingly defined). The RAE is a major concern for researchers, much more important for most of them than other forms of assessment. There are significant differences, however, between what researchers publish and consider to be important, and what is submitted to the RAE.

Researchers’ perceptions and understanding of RAE requirements are mediated via universities, which develop their own strategies to maximise their RAE performance. Thus what the funding councils say is not necessarily what researchers hear. A common view is that the RAE is a game researchers have to play; and that it may constrain intellectual autonomy. A quarter of researchers believe that important outputs were not submitted to the last RAE; and many more are concerned about pressures they perceive to seek publication only in high-status journals. With the exception of monographs in the humanities and practice-based outputs in the arts, researchers see the RAE, perhaps wrongly, as a disincentive to any other forms of dissemination. Since journal articles are the publications most readily measured, and thus most susceptible to evaluation through any system of performance assessment, there is a risk that their dominance will increase.

Researchers are also concerned about the relationship between the timescales for research and for the RAE. Most believe that it often takes longer than the length of an RAE cycle for the significance and value of research findings to be recognised: they often talk of periods of ten years or more. The proposal that the impact of research beyond the academic and research communities should be a significant feature in the RAE may help to clarify the mixed messages that researchers think they are receiving about the goals they should seek; and the relative priority they should give to criteria for success such as academic quality, speed of dissemination, engagement with non-academic audiences, and wider socio-economic impact. But the timescales for research, recognition and impact differ widely across different disciplines and kinds of research. Research timescales need to be carefully considered in any arrangements for the assessment of performance.

Our research has been undertaken in a climate where there has been considerable debate about the format of the Research Excellence Framework (REF) and the role that bibliometrics might play in it. There has been considerable scope for speculation and misconception. Many researchers say that any move to give greater weight to citation analysis will have a significant effect on their behaviour: they will publish more; they will submit their work more often to journals with high impact factors; and they will make their publications open access. It will also change their citation practice. Many are concerned about the scope for misunderstanding and manipulation of citations, especially in the light of differences in author attribution and citation practice within and across disciplines. Only a small minority say they will cite competitors’ work less often; but even while they deprecate citation clubs and circles, nearly two-fifths of researchers say that they will cite their collaborators’ work more often. Possible changes in practice will need to be carefully monitored as the REF develops.