JISC

Charles Oppenheim and Fytton Rowland, March 2009

Report summary

A study based on interviews with '26 authoritative people, covering a wide range of academic disciplines' to find their views on quality control relating to:

• research communications
• data compilations
• digital learning and teaching materials in higher education
• scholarly communications in Web 2.0 media

The introduction gives a brief history of peer review procedures from the 1660s onwards, noting the emergence of editorial boards after the Second World War and the role of editors in improving works as well as keeping “poor ones out”. The authors also note a few criticisms of the system (For instance, the imbalance arising from the fact that referees know who the authors are, but authors don’t know about the referees). They then focus in on Open Access and its implications for peer review (noting some suggest Open Access could weaken peer review procedures) and Web 2.0. They then discuss the judging the quality of institutions based on their output of journals and mooted changes to the Research Assessment Exercise (including a name change to Research Excellence Framework). There’s also an overview of the need for quality control as regard teaching materials coming from sources like Wikipedia and new Web 2.0 tools that are “increasingly blurring the boundaries between traditional categories of scholarly publication”.

The report then details the interview methods used and the experience and qualifications of the interviewees before moving on to their responses.

These show that although respondents have concerns about traditional peer review (such as a bias towards conservatism and ‘safe’ pieces of research), they still see it as “essential” to maintain academic quality. Many involved in the editing of journals note how the pressure to produce journals (for which recognition is received) is preventing many academics from also having the time to referee them (since no recognition is received for that onerous and time consuming task). Various ways of ‘rewarding’ referees and their departments are discussed.

The report uncovers various dubious practices (some admitted to by academics some ‘anecdotal’) such as journals accepting poor work by good scholars so they don’t lose out on the good scholars’ top material too and discusses alternative forms of refereeing such as “double-blind” (when no one knows who has written the paper and who is reviewing it) and open (when everyone knows)

They discuss open peer review, which isn’t favoured as a replacement for traditional peer review but seen as potentially useful after publication (following traditional peer review) to allow scholars to comment upon and improve papers. They also discuss the comment facilities on papers put on the website of the Atmospheric Chemistry and Physics (ACP) journal.

They then look at different methods of peer review and how much criticism, reconstruction of papers and arguments referees do, before moving on to whether electronic publishing has affected the quality control of journals (broadly, no). The latter question was asked of Open Access journals which also were not seen as a major problem in terms of quality control, although did arouse more concerns, particularly with regard to copy-editing and questions of articles being accepted on the basis of whether the author could pay as much as their quality. Repositories too were not seen as a major problem.

The next section details teaching methods and concerns about what should be cited as reference material (Wikipedia, yes or no?), borrowing course structures, the emergence of Virtual Learning Environments and similar. They conclude that “the main quality-control mechanism for L & T materials is the professionalism of the lecturer using them”.

They then collate the responses of five researchers involved in data collection, noting that “it seems that it is difficult to generalise across disciplines”. Hard scientific and social science data are so different in terms of stability, sourcing, potential flaws and storage that there are large variants in judgements about quality control.

The conclusions go over these findings and take a few suggestions from them such that peer review should both evaluate and improve papers, before recommendations on training and refereeing procedures.

Key points

“It emerged that little change was expected in peer review procedures in the next five years, and that ‘double-blind’ peer review is widespread in the social sciences but almost unknown in the sciences.”

“Neither electronic-only publication nor Open Access is expected to impact substantially on quality control of research communications.”

“Blogs are seen to have a possible usefulness in the scholarly world, but there is doubt about ‘the wisdom of crowds’, with a relatively more positive view of wikis coming from the humanities and social sciences, and a much more negative view of them from the hard sciences.”

“There was near-unanimity across all respondents and all disciplines that the traditional per review system for the refereeing of scholarly articles remains essential to maintain academic quality.”

“…another respondent reported that their society journal was published through a commercial publisher, whose legal advisors had warned that, in the event of any dispute going to court, the anonymity of the referee could probably not be maintained anyway.  A civil servant agreed; in all probability, referee’s reports would have to be released following a Freedom of Information Act request.”

“To the question ‘Assuming OA were to become widespread, would you expect the quality-control system necessarily to change?’ the answer from the majority of our respondents was a clear ‘no’”

Read the full report (Word)