What has the real-time REF review taught us about future research assessment?

Creative Commons attribution information

A focus group discuss their opinions.

©10'000 Hours via Getty

All rights reserved

This was the vision when Research England, on behalf of the four UK higher education funding bodies1, commissioned RAND Europe to conduct a real-time evaluation of attitudes to and perspectives on REF 2021.

Not everyone experiences REF in the same way

If one thing is clear from the survey, it’s that REF continues to divide opinion. Individuals' views are shaped by their personal experiences, as well as their career stage and disciplinary focus.

If we are to find an assessment model that serves the whole research community, then we need to continue to listen to a diverse and distributed set of perspectives.

We held a number of roundtables earlier this year where we spoke to early career researchers, technicians, impact professionals and equality advocates – those who play a key role in the sector but do not always contribute to formal consultations on research assessment.

Their views will feed into the sector-wide consultation that forms a key part of the Future Research Assessment Programme's (FRAP) work.

The positive impact of REF 2021

While it would be a stretch to say that the REF is a beloved part of academic life, the review’s participants noted its positive effect on aspects of the research system. It is heartening to hear that the current exercise is seen to have increased both open research and the wider relevance of research.

Another encouraging finding was that the evolution of the exercise from 2014 to 2021 was broadly seen as positive, particularly those changes made to staff and outputs selection. When we come to design a future system, it will be important to reflect on what has worked well and carry this forward into future exercises.

Does the REF have an image problem?

It’s important to remember that the review set out to study perceptions and understanding of the REF, rather than analyse its direct impact on research and researchers. Looking at the results, we were particularly struck by the discrepancy between researchers’ views of the impact of the REF on their research communities and how they experience the REF in their own professional life.

For instance, we heard widespread concerns about ‘gameplaying’ by universities but, when asked about behaviour at their own institution, participants were less likely to identify these practices.

However, we don’t dismiss these concerns as unfounded or uninformed. Perceptions help perpetuate the misconceptions and myths surrounding the REF, which have a real-life impact on what institutions submit and, in turn, what outputs and impacts researchers are encouraged to produce.

We know that the REF plays a significant role in shaping behaviours, attitudes and culture in the UK’s research community, and that practices not directly experienced by individual researchers can nonetheless be felt at the level of the research system.

It is crucial, then, that the FRAP does not limit itself to evaluating the tangible effects of the REF on individual researchers. We also have to work with the academic community to understand how the exercise – through what it assesses, recognises and rewards – shapes what is valued, encouraged and incentivised in institutions and the wider research community.

The real-time REF review is the first output of the FRAP, launched by the four funding bodies earlier this year. The review’s findings will sit alongside a range of evidence from other evaluation activities that will help us to understand what worked well in REF 2021 and how future research assessment can best contribute to a thriving research system.

Consultation is also crucial to FRAP and here, too, we can learn from the review. This study is innovative in a number of ways. Rather than waiting until the end of REF 2021, we’ve been able to listen to views as they emerged and evolved alongside the exercise.

We’ve also heard from a wide range of voices across disciplines, career stages, professional roles, and institutions. While we are still in the early stages of FRAP and any future exercise is some way away, it is clear to us that the rules of the exercise must be stated early on to ensure the evaluation process is equitable.

The real-time REF review has not only given us a rich pool of evidence at an early stage, but also provides a great foundation to inform other pieces of evaluation going forward. That’s why we’ve launched an online survey to understand how REF 2021 has affected individual researchers.

This survey draws on the findings of the real-time REF review and seeks to understand how widely the views expressed are shared across the research community.

Anyone who wants to contribute to the review can access the survey - the closing date is noon on 26 January 2022. Read the review (pdf).

Footnotes

• 1 The Scottish Funding Council, the Higher Education Funding Council for Wales, and the Department for the Economy, Northern Ireland.