The Impact Factor

“We are historians, we’ve never studied people who answer back”. This is how a team member from the Old Bailey Online, a successful resource which provides access to nearly 200,000 trials of London’s central court 1674-1913, summed up the challenge they faced when trying to measure the impact of their digital resource on research, teaching and learning.

This statement is revealing of wider issues institutions face in today’s times of financial constraints: how do we know if a digital resource is having an impact on its target audience? How do we reach and speak to scholars, teachers and students to measure their satisfaction? What metrics should be adopted in the context of digitised scholarly material? How much does a digital resource tell about the institution that created it? And above all, has the investment paid back? These are not easy things to assess and often impact just takes time to materialise.

In order to support content creators, resources managers and information professionals within institutions in the task of assessing the usage and impact of their digital resources, Jisc has supported the development of the Toolkit for the Impact of Digitised Scholarly Resources (TIDSR), by the Oxford Internet Institute (OII).

The toolkit, first developed in 2009 and recently updated, provides a framework for conducting this kind of analysis and offers guidance on a range of qualitative and quantitative methodologies that can be used such as webometrics, content analysis, surveys and focus groups. The TIDSR was used by projects in the Jisc Impact and embedding of digitised resources programme, of which the Old Bailey was one, to conduct an analysis of their collections, identify where resources were working well and what could be done to improve them and better embed their content within teaching and research.

The case studies drawn from the experience of the projects are available in the toolkit and are a useful starting point for beginners in the field. They also provided fertile ground for the programme’s final report, “Splashes and Ripples: Synthesising the Evidence on the Impacts of Digital Resources” (PDF), by Eric Meyer. The report begins to sketch a picture, based on evidence rather than anecdotes, of how digitised resources in the humanities are currently being used and provides a set of recommendations to content creators on how to go about maximising the impact of their resource such as:

[quote from report]

1.     Remember in advance that you will want to contact your users. A number of projects had a difficult time finding users to survey or interview, but users are a key resource that you will want to approach from time to time.

2.     Use the media to your advantage. One of the undeniable advantages that arts and humanities resources have in the United Kingdom is that there is considerable public interest in these topics. […] The Old Bailey Proceedings Online project has benefitted from inclusion in popular BBC programmes (see page 11).

3.     The media and the public are influenced by numbers and metrics. Being able to demonstrate your impact numerically can be a means of convincing others to visit your resource, and thus increase the resource’s future impact. For instance, the amount of traffic and size of iTunesU featured prominently in early press reports (see page 21).

4.     Make your resource easy to find. This can involve a number of strategies, including search engine optimization (SEO), partnerships with more prominent related sources (see page 31), links in related sites, and inclusion in Wikipedia and other sources. A Vision of Britain through Time has been the most proactive resource in this regard, (see page 28),

5.     Give your resource an unambiguous name and acronym/initialism, both to increase the likelihood that your resource turns up at the top of relevant searches, and to make measuring mentions of your resource result in as few false positives as possible.

6.     Create quick wins for new visitors to your collection. By finding things that they can quickly learn, do, see, or contribute, you can increase the stickiness of your site, and increase the likelihood that your resources will be used. Oxford University’s podcasts, for instance, are easy to immediately access and hear (see page 23).

7.     Leverage your wins. Using the most popular aspects of your resource to attract people to other parts of the collection via features such as suggested links and recommendations for further information can increase the time spent with your collection.

8.     Adopt Cool URLs as persistent, consistent, human-readable, and citable links to digital resources. The British History Online collection has used this method to increase the readability of its links (see page 39).

9.     Provide the ability to export citations directly to reference management software such as Zotero and EndNote.

10.     APIs are the future. Linked data, apps, and other ways that enable researchers to access and combine the data in your resource will increase its utility.