To ensure continuation of funding, it is essential that JISC, through its projects, programmes and services, continuously demonstrates its collective value, benefit and impact. JISC has the potential to impact the community on a number of levels through a range of different activities, all of which need to be considered.

Impact

In this section you will find information on:

Why is impact important?
What is impact?
How can impact be 'measured'?
What should project managers do?
Examples of how to demonstrate impact

Why is impact important?

Updated October 2011

To ensure continuation of funding, it is essential that JISC, through its projects, programmes and services, continuously demonstrates its collective value, benefit and impact. JISC has the potential to impact the community on a number of levels through a range of different activities, all of which need to be considered, such activities are:

  1.  project and programme activities (innovative/developmental work)
  2.  services
  3.  communication and marketing
  4.  initiatives and partnerships

JISC’s approach to identifying impact does not seek to introduce a separate set of processes, rather to maximise and enhance the value of existing processes (such as project management) to capture and surface evidence of impact more effectively.

Due to the finite length of JISC funded activity it is essential that evidence is collected during the life-cycle of projects and programmes – not just at the end or at some point after the activity is completed. It is therefore the responsibility of project teams, through their JISC Programme Manager, to capture the appropriate information. This approach has a number of benefits, such as ensuring the capture of short and mid-term impacts, and providing evidence that could be used in any impact studies in the longer-term should they be conducted.

What is impact?

Many definitions of impact can be found with varying degrees of consistency. JISC will not seek to define ‘impact’, rather the term will be seen to encompass any change, benefit or value resulting from our activities.  This includes intended as well as unintended effects, negative as well as positive, and long-term as well as short-term.

For JISC, identifying and demonstrating impact are complex issues. Some of our impacts are immediate while others such as changes in practice or behaviour may take longer to realise. Some will be more tangible and directly measurable than others, e.g. those caused by JISC alone, while others are several stages removed, often in partnership with other organisations, and hence more complicated to measure. Some will operate at the individual level while others more strategically. The magnitude of our impact is not just about how many people we are benefitting but also whom. For example influencing 50 senior managers to change an institutional strategy may carry as much impact as changing 1000 researchers’ behaviour.  

The following diagram shows examples of different types of impact we might expect JISC’s work to be having at different levels on our audiences. It shows that some impacts can be realised sooner than others. For example changing policy is a strategic issue and likely to take much longer to achieve than say delivering a direct cost saving by adopting a new technology. Its impact however might actually be greater.

diagram showing the impact we might expect JISC’s work to be having at different levels on our audiences

How can impact be 'measured'?

Due to the often complex nature of JISCs work, a pragmatic approach is needed. A basic question to ask is whether it is possible to measure impact or if it is better to illustrate it.  This will very much depend on the nature of what your Project or Programme is trying to achieve.  If it is possible to establish indicators or measures and capture quantitative data to demonstrate this then do so.  For some activities this may be more challenging due to the nebulous nature and indirect impact of them and so other qualitative techniques might be more useful. It is therefore a case of selecting the most appropriate methodology or methodologies to achieve this.

JISC demonstrates impact by using a mixture of qualitative and quantitative methodologies such as statistics (e.g. usage, take up, market penetration), metrics, (e.g. value for money or return on investment), case studies, illustrative examples, counter-factual arguments and attributed testimonials.

The JISC-funded Toolkit for the Impact of Digitised Scholarly Resources (TIDSR) is an example of the range of approaches and methodologies that can be adopted to measure the usage and impact of digitised content on teaching, learning and research, and also provides a number of case studies.

Especially powerful are examples of impact in, or value derived by, institutions unrelated to the institution(s) in which the project is taking place.  An example of such a ‘wider benefit’ is an institution using an output created by a project in a different institution, to help with implementing practice or developing a strategy.

Publication


See JISC’s Transformation Through Technology impact report or Supporting Your Institution section of the JISC website for real life examples.

 

What should Project Managers do?

It is important to understand and articulate why we’re doing what we’re doing, not just what we’re doing, or how we’re doing it, as these are the stories that will resonate with JISC’s audiences.  Take time when planning your project to be clear about why you are doing the work and what it is trying to achieve – this will help you identify evidence of impact and capture it in a useful and meaningful way.

Project Managers should:

  1. Consider the ‘why’s
  2. Take a pragmatic approach
  3. Accumulate and store evidence along the way
  4. Attribute testimonials
  5. Complete a Project Closure Survey
  6. Tag impact outputs

1. Consider the ‘why’s’

Why the activity is being undertaken should be considered in the context of how JISC is helping the sector address the need to

  1. maintain research excellence
  2. maintain teaching & learning excellence
  3. be more effective/save money
  4. have a positive impact on wider society
  5. be ready for technology needs in the future

These five areas are known as our Impact Areas.

Three of the impact areas are aligned with key strategic JISC aims (1, 2, 4) and the other two are core principals upon which JISC is based (3, 5).

The final impact area; ‘being ready for future needs’, presents an opportunity for JISC to demonstrate how its thought leadership, horizon scanning ability and innovative activities benefit the sector, for example, by mitigating risk to save the community time and money in technological developments. This broad framework enables JISC to capture impact in a consistent way. 

Two other areas that JISC needs to demonstrate impact in are ‘Internationalisation’ and ‘contributing to UK PLC’. These issues cut across all the five areas listed above, so should be considered within all rather than being separate. 

This framework is shown in the following diagram:

Framework diagram of JISC's impact areas

The ‘why’s’ should be considered at the start of your project to help you to  identify any impact ‘indicators’ and mechanisms through which evidence can  be captured or collated (however sophisticated) during the project life-cycle.  It probably won’t be possible to identify all expected impacts upfront and they may change or become apparent during the course of the project. This is to be expected and any new or different evidence should be captured as it emerges – the key is to be thinking about the need for doing this throughout the project and recognising that a range of methodologies can be used as appropriate.

2. Take a pragmatic approach

Adopt an achievable and flexible approach to capturing impact using different methodologies. e.g. statistics (usage, take up, market penetration), metrics (e.g. value for money or return on investment),  case studies (especially from institutions beyond those funded directly by the project), illustrative examples, counter-factual arguments and attributed testimonials.  All are valid and help to demonstrate the impact of your work.

3. Accumulate and store evidence along the way

Don’t leave it all to the end. Start thinking from the outset of your project (when formulating your Project Plan) what impact you are trying to achieve and how you will gather evidence during the life of your project to demonstrate it. Use the Impact Areas and Impact Types as a structure for this. It will also make completing the Project Closure Survey much easier.

4. Attribute testimonials

Make sure that testimonials are attributed to an individual, and as with case studies ensure the necessary permissions for use (IPR etc) are in place. Ideally evidence should be captured and presented in a format that can be repurposed by JISC and used in any output (eg developing a summary paragraph from a longer form case study).

5. Complete a Project Closure Survey

You will be asked to complete a Project Closure Survey at the end of your project.  Make sure that you draw on all of the impact evidence you have been collecting through the life of the project to complete the impact questions in this survey.

The survey will ask you to identify which of the five the Impact Area(s) your project has had an impact in

  • research excellence
  • teaching and learning excellence
  • increasing effectiveness/saving money
  • positive impact on society
  • helping with future technology needs.

Of course these impact areas are not mutually exclusive so it is possible to impact in more than one area. For example, helping an institution become more effective in the area of research ticks two of the main impact areas; research excellence and being more effective.

You will then need to select the nature of the impact; Impact Type (see examples of how to describe impact)

  1. Informed/developed strategies
  2. Changed practice/behaviour
  3. Influenced policy/thinking
  4. Provided used support and advice
  5. Created new or enhanced tools, services and methodologies
  6. Increased uptake or use
  7. Developed skills/built capacity
  8. Defined the scale and nature of a challenge
  9. Resolved a problem or challenge
  10. Improved institutional processes 
  11. Increased availability/access to content or materials  
  12. Provided leadership
  13. Other

It is essential that you have evidence that supports the impact you are describing. Select what format you have the impact evidence in:

  1. Statistic
  2. Metric 
  3. Case study
  4. Illustrative example
  5. Counter-factual argument
  6. Testimonial
  7. Other

6. Social Bookmarking

The JISC Market Research team is using social bookmarking (currently Delicious.com but we will be changing to another once this service closes) to build an easily accessible catalogue of impact evidence (this is similar to the approach used by JISC RSCs to catalogue good practice case studies across the FE sector). It’s particularly useful for keeping a record of mentions in the media, websites or other publications etc. If you are able to tag any impact outputs using a social bookmarking service, please use the tags jisc-mr and impact as a minimum and we will be able to find them and aggregate them).

Examples of how to demonstrate impact

The different ways of showing impact don’t have to be treated in isolation of one another. For example statistics can be combined with case studies and testimonials to present an all round picture of impact. A good model (in an ideal world) for example would be “x number or y% of  universities have download our toolkit/briefing paper (statistic), a survey said z% found it useful (metric), here’s an example of it has been used to benefit someone (case study), and here’s a testimonial from someone suitably influential.“ For example: 19th Century British Library digitisation project; 3,000,000 pages digitised and freely accessible, 57,000 visitors to the website, 269 further or higher education institutions subscribing, if all institutions currently subscribing had to pay for their subscriptions the cost to the sector would be: £26,000  x 269 institutions = £6,994,000, in a survey of users of digital humanities resources, 77% of UK respondents and 61% of overall respondents were aware of this collection, “19th Century Newspapers Online gives us an invaluable new perspective on....” (Prof. Arthur Burns, Kings College London).

Type? Example
Statistic  Usage, take up, downloads, market penetration (eg x% of universities using service), % of users stating resource valuable, direct financial saving (£)  In the academic year 2007/2008,Digimap, Ordnance Survey was used a total of 240,010 times by 105 different HEIs (equating to an average of 2,286 sessions per institution). In addition, 43 FE colleges used the service, 3,242 times (an average of 75 sessions per college) (p14)

In January 2009 XCRi was approved by the Information Standards Board (ISB) for education, skillsand children’s services. ISB are currently developing an adoption plan so that course brokers and learning providers can start to realise the benefits of the standard, including estimated financial benefits of £6m per annum. (p21)

Metric

Increase in % or number of users before project started to afterwards.

Value for money (return on investment).

With total public funding of just under £3.4m in 2008/2009 the JISC Advance services delivered savings to the value of £41.5m. This represents a saving of £12.25 for every £1. (p13)

Case study Describe what project is about, why important, what it achieved, how JISC project helped, include attributed testimonials, statistics etc where possible Demonstrating the value of shared funding: APT STAIRS project (p9)
Illustrative example Not as formal as a case study but a different way of describing an impact of a project Broadening opportunity by allowing students to study remotely using online learning technologies and techniques such as enabling 300 undergraduate students from Cardiff University previously unable to do so, to view and interact with live surgical procedures through webcams and podcasts while on placement (p60)
Counter-factual argument Useful when there is no alternative to compare with or projects of a more nebulous nature.

JISC-funded R&D work which proved that a technology was not appropriate for sector-wide adoption; smart cards (p10)

Testimonial

Attributed.
Permission for re-use is essential.
More influential/relevant the individual the better. Clearly highlights why the project, activity, initiative has been of impact or made a difference. Eg without JISC….

JISC has played a big part in how we develop our thinking about e-learning.Because the project, through its published resources, was able to provideauthentic evidence of the importance of digital literacy, Oxford Brookes was able to use this evidence to inform the dialogue with Deans and course teams when reviewing the University’s modular scheme.’(Richard Francis, Head of e-Learning)

"I recently came across the Identity Management Toolkit and felt compelled to contact you in order to pass on my heartfelt thanks and gratitude for what is an absolutely fantastic resource". (Jonathan Day, ISS Manager, Lancaster University).

Publication 
Examples taken from JISC Transformation Through Technology report (page reference given in brackets)

Bookmark and Share