Consultation on the design of the UK’s future research assessment system
To help shape the Future Research Assessment Programme, the funding bodies asked for your views through a consultation.
The consultation survey closed at midday on Friday 6 May 2022.
Research England, Scottish Funding Council, Higher Education Funding Council for Wales, Department for the Economy, Northern Ireland asked for feedback on:
- The purposes of a future exercise
- The principles that should guide its development
- The assessment criteria and processes
These questions had been informed by early discussions with a wide range of individuals, institutions and organisations who worked in and engaged with the research sector. You can read a summary of these discussions and how the consultation responses will be used in the consultation document (docx).
The responses to this consultation will be considered alongside a range of internal and commissioned evaluations, feedback on the current exercise (gathered separately from institutions and individuals), and advice from the programme’s international advisory group.
Consultation responses were invited from any higher education institution, association, organisation or individual with an interest in the conduct, quality, funding or use of research. This included schools, faculties or departments within an institution and also individual staff members.
Consultation events
As part of the consultation, the funding bodies held three online ‘town hall’ events during March and April 2022. These gave insight into the background, scope, and objectives of the FRAP. They also outlined key issues addressed by the consultation, and how the outcomes of the consultation will feed into the development of the next research assessment exercise. Download the presentation from the online town halls (.pptx).
The FRAP team also hosted regular Zoom drop-in sessions for those who intended to respond to the consultation.
Discussion topics
The funding bodies summarised answers to some of the questions and issues raised at the events:
Design options for future exercises
The funding bodies are committed to communicating changes as early as possible. This must be balanced with the need to ensure that any changes are informed by a robust evaluation of REF 2021 and widespread consultation with the sector.
This is established good practice and aligns with the principles of responsible research assessment, which underpin FRAP. The potential impact of any new changes will be considered and factored into timelines for a future exercise.
We are mindful that substantial changes to the exercise will mean that institutions will have to recalibrate their approaches to engaging with the REF or future research assessment exercise. However, the research landscape has evolved since the design of the current REF 2021 exercise and it is right that the exercise should be reviewed to ensure that it remains fit-for-purpose. This is the reason why we are consulting on the design of a future exercise. The timescales for defining clear guidance on the future exercise will obviously depend on the outcomes of the consultation we are currently running.
As the REF is run by the four UK funding bodies, each having their own governance structures, the Boards of the funding bodies, made up of a broad mix of individuals bringing different perspectives and expertise to the table, will examine the recommendations made from the consultation outcomes and of the evaluation work in the middle of 2022.
The funding bodies are committed to communicating changes as early as possible. This must be balanced with the need to ensure that any changes are informed by a robust evaluation of REF 2021 and widespread consultation with the sector.
This is established good practice and aligns with the principles of responsible research assessment, which underpin FRAP. The potential impact of any new changes will be considered and factored into timelines for a future exercise.
We are mindful that substantial changes to the exercise will mean that institutions will have to recalibrate their approaches to engaging with the REF or future research assessment exercise. However, the research landscape has evolved since the design of the current REF 2021 exercise and it is right that the exercise should be reviewed to ensure that it remains fit-for-purpose. This is the reason why we are consulting on the design of a future exercise. The timescales for defining clear guidance on the future exercise will obviously depend on the outcomes of the consultation we are currently running.
As the REF is run by the four UK funding bodies, each having their own governance structures, the Boards of the funding bodies, made up of a broad mix of individuals bringing different perspectives and expertise to the table, will examine the recommendations made from the consultation outcomes and of the evaluation work in the middle of 2022.
Options for the future design system
FRAP is exploring a wide range of options, ranging from the evolution of the current system to more substantial changes to the exercise. At this stage, the funding bodies are not consulting on specific models but want to hear the sector's views on what purposes a future exercise should serve and – at a high level – how research performance should be assessed and against what criteria. There are several factors to consider when developing the next exercise, some of which may need to be tensioned against each other.
It is inappropriate at this stage to comment on details of how the future exercise might be designed. At the moment, we are consulting on these considerations at a high level, and you should comment on the appropriateness of the principles of the various approaches. Many of these issues will be inter-dependent and we will seek to consult further with the sector on the practicalities of adopting any changes, including the systems required to make submissions. Providing a rationale for your views on the high-level design aspects will enable us to anticipate any potential challenges to implementation.
Following the announcement of the high-level design of the next exercise, input into the detailed design of the next exercise will be sought. The timing and extent of further consultation will depend on the outcomes of FRAP.
FRAP is exploring a wide range of options, ranging from the evolution of the current system to more substantial changes to the exercise. At this stage, the funding bodies are not consulting on specific models but want to hear the sector's views on what purposes a future exercise should serve and – at a high level – how research performance should be assessed and against what criteria. There are several factors to consider when developing the next exercise, some of which may need to be tensioned against each other.
It is inappropriate at this stage to comment on details of how the future exercise might be designed. At the moment, we are consulting on these considerations at a high level, and you should comment on the appropriateness of the principles of the various approaches. Many of these issues will be inter-dependent and we will seek to consult further with the sector on the practicalities of adopting any changes, including the systems required to make submissions. Providing a rationale for your views on the high-level design aspects will enable us to anticipate any potential challenges to implementation.
Following the announcement of the high-level design of the next exercise, input into the detailed design of the next exercise will be sought. The timing and extent of further consultation will depend on the outcomes of FRAP.
Mitigating 'game-playing' in future exercises
We're very much aware that the REF exercise can drive behaviours in unanticipated ways. We are open to hearing from stakeholders on how the REF drives such behaviours and the potential for future exercises to have similar consequences.
The consultation is intended to encourage a forward-looking approach, rather than focusing on current and previous exercises. However, evaluation of the current exercise forms another key strand of FRAP. The funding bodies have gathered feedback on the current exercise through various channels, including the real-time REF review and feedback provided by institutions and individuals. It would be helpful, however, if you could suggest in advance where gameplaying might be an unintended consequence of any aspects of the future system set out in the consultation.
We're very much aware that the REF exercise can drive behaviours in unanticipated ways. We are open to hearing from stakeholders on how the REF drives such behaviours and the potential for future exercises to have similar consequences.
The consultation is intended to encourage a forward-looking approach, rather than focusing on current and previous exercises. However, evaluation of the current exercise forms another key strand of FRAP. The funding bodies have gathered feedback on the current exercise through various channels, including the real-time REF review and feedback provided by institutions and individuals. It would be helpful, however, if you could suggest in advance where gameplaying might be an unintended consequence of any aspects of the future system set out in the consultation.
Ensuring a wide range of stakeholders are involved
The consultation is open to all stakeholders with an interest in the design of future research assessment exercises. We would encourage responses from schools, departments, or faculties within all types of higher education institutions and also from individual staff members. All respondents are especially encouraged to express their unique positions and how the design of possible future research assessment exercise might impact on their research strategies. Respondents are also reminded that they do not have to respond to every question.
We have also engaged with wider stakeholder groups, such as research users or funders, and will hold further discussions with them as FRAP progresses.
The consultation is open to all stakeholders with an interest in the design of future research assessment exercises. We would encourage responses from schools, departments, or faculties within all types of higher education institutions and also from individual staff members. All respondents are especially encouraged to express their unique positions and how the design of possible future research assessment exercise might impact on their research strategies. Respondents are also reminded that they do not have to respond to every question.
We have also engaged with wider stakeholder groups, such as research users or funders, and will hold further discussions with them as FRAP progresses.
Open access (OA) policy for the next exercise
It is too early to speculate on the nature of the policy on open access (OA) publication in the next exercise, as this will be influenced by its overall design. We are aware of the developing landscape in this area and of different policy drivers (for example broader UKRI policy on OA).
Wherever possible we will endeavour to align policies to facilitate straightforward engagement with different processes regarding OA. It is the intention that the funding bodies will consider a UKRI open access compliant publication to meet any future national research assessment open access policy without additional action from the author and/or institution.
The funding bodies note that the scope of an open access policy for the future national research assessment exercise is much broader than the UKRI Open Access Policy. This will be considered when developing the future national research assessment position. The funding bodies will not introduce requirements retrospectively.
It is too early to speculate on the nature of the policy on open access (OA) publication in the next exercise, as this will be influenced by its overall design. We are aware of the developing landscape in this area and of different policy drivers (for example broader UKRI policy on OA).
Wherever possible we will endeavour to align policies to facilitate straightforward engagement with different processes regarding OA. It is the intention that the funding bodies will consider a UKRI open access compliant publication to meet any future national research assessment open access policy without additional action from the author and/or institution.
The funding bodies note that the scope of an open access policy for the future national research assessment exercise is much broader than the UKRI Open Access Policy. This will be considered when developing the future national research assessment position. The funding bodies will not introduce requirements retrospectively.
Consulting REF panels
The REF team is seeking feedback from the panels on the operation of the current exercise. In addition, all panels will produce reports on their operation of the assessment. Focus groups with panel members are also planned.
In addition, as stakeholders with an interest in the future research assessment systems in the UK, it would be entirely appropriate for REF panel members to respond to the consultation.
The REF team is seeking feedback from the panels on the operation of the current exercise. In addition, all panels will produce reports on their operation of the assessment. Focus groups with panel members are also planned.
In addition, as stakeholders with an interest in the future research assessment systems in the UK, it would be entirely appropriate for REF panel members to respond to the consultation.
Scope for exercises to support and encourage collaboration
We agree that future research assessment exercises should foster and encourage collaboration within and between higher education institutions.
You should feel free to comment on how design of a future exercise could negatively or positively influence such approaches to research or put forward ideas on how collaboration could be encouraged and rewarded in future exercises.
We agree that future research assessment exercises should foster and encourage collaboration within and between higher education institutions.
You should feel free to comment on how design of a future exercise could negatively or positively influence such approaches to research or put forward ideas on how collaboration could be encouraged and rewarded in future exercises.
The relationship between REF and other forms of assessment
We are aware of other evaluation exercises (such as KEF and research funder processes) and we will do everything we can to align data collection structures, where possible. However, it’s important to note that whilst REF is UK-wide, many other forms of assessment (eg KEF) may not be, so there are some limitations.
We’ve been asked to define early what metrics or data will be required, but also aligned to that is a need to carefully consider the outcomes of the consultation and provide guidance on that basis.
We are aware of other evaluation exercises (such as KEF and research funder processes) and we will do everything we can to align data collection structures, where possible. However, it’s important to note that whilst REF is UK-wide, many other forms of assessment (eg KEF) may not be, so there are some limitations.
We’ve been asked to define early what metrics or data will be required, but also aligned to that is a need to carefully consider the outcomes of the consultation and provide guidance on that basis.
Assessment of collaborative, cross-disciplinary and interdisciplinary research
We are keen that future research assessment exercises should foster and encourage collaboration within and between higher education institutions. You should feel free to comment on how design of a future exercise could negatively or positively influence such approaches to research or put forward ideas on how collaboration could be encouraged and rewarded in future exercises.
Similarly there are several processes in the current exercise (joint submission, cross referral, and interdisciplinary advisors) which are intended to ensure research groups based in schools/departments, but researching in fields more affiliated with other subjects are not disadvantaged by limited comparability. It’s important for us to understand the sector’s perception of their effectiveness, the extent to which they’re achieving the intended purpose, and to understand how cross-disciplinary and interdisciplinary research can be supported in future exercises.
We will be speaking to the panels and considering new things that we might do - this is where the potential for institutional assessment may in fact be helpful, as well as allowing institutions to provide a bit more narrative context about their UOA submissions.
We are keen that future research assessment exercises should foster and encourage collaboration within and between higher education institutions. You should feel free to comment on how design of a future exercise could negatively or positively influence such approaches to research or put forward ideas on how collaboration could be encouraged and rewarded in future exercises.
Similarly there are several processes in the current exercise (joint submission, cross referral, and interdisciplinary advisors) which are intended to ensure research groups based in schools/departments, but researching in fields more affiliated with other subjects are not disadvantaged by limited comparability. It’s important for us to understand the sector’s perception of their effectiveness, the extent to which they’re achieving the intended purpose, and to understand how cross-disciplinary and interdisciplinary research can be supported in future exercises.
We will be speaking to the panels and considering new things that we might do - this is where the potential for institutional assessment may in fact be helpful, as well as allowing institutions to provide a bit more narrative context about their UOA submissions.