JISC

24 February 1999

Monitoring and Evaluating User Behaviour in Information Seeking and Use of Information Technology and Information Services in UK HE

Invitation to Submit Proposals

Introduction

1. This circular invites bids against an allocation of up to £100,000 per cycle, for a framework for monitoring and evaluating the behaviour of users in information seeking and use of information technology and information services. The framework will consist of a range of quantitative and qualitative survey instruments and will commence in August 1999. The deadline for responses to this Circular is 5pm on Wednesday 21^st April 1999.

2. The funding is under the umbrella of JISC's Committee on Awareness, Liaison and Training (CALT). CALT has a remit to promote awareness of the potential of IT, disseminate good practice and lessons learned, and to look at the social considerations which are critical to any understanding of the use of technology within HEIs.

3. JISC's Committee for Electronic Information and CALT are responsible for providing and promoting the use of electronic information services (EIS) in higher education institutions (HEIs). The Committees are concerned that JISC is engaging in extensive but uncoordinated evaluation and monitoring activities, such as end user surveys, which unnecessarily duplicate evaluation efforts, and place growing burdens on the user community, yet which continue to leave a deficit in our understanding of user behaviour and needs.

4. While JISC has led a necessary period of exploration and experimentation with different approaches to evaluating information services in HE, there is now a need for a more systematic and cost effective approach to monitoring usage across the range of JISC and other information services in HE . This can be achieved through putting mechanisms in place which ensure consistent learning and feedback.

5. In order to understand better and address the needs of scholars, teachers and learners in their use of EIS, the JISC is adopting a framework for monitoring and evaluating the behaviour of users in information seeking and use of information technology and information services. This consists of a range of quantitative and qualitative survey instruments.

Background

6. Following discussion papers by Lynne Brindley and by Joan Day, Harry East and Peter Stone a joint CALT/CEI working group on user behaviour commissioned the Tavistock Institute to outline a possible framework for monitoring and evaluating user behaviour. The Tavistock Institute were able to draw on the their experience of working with various CEI initiatives and drawing on the work of the eLib project IMPEL, based at the University of Northumbria and the Database Resources Research Group (DRRG) at City University.

7. The working group determined that the direct quantitative and qualitative monitoring of actual user behaviour and declared user needs should form the basis of any monitoring framework and that these activities be clearly related to a technological forecasting function which would inform JISC about areas for future investment and advise HEIs on effective investment and deployment.

8. A small group of experts (the 'Monitoring Framework Group') was convened to flesh out the specifics of the framework, taking into account the existing roles of JISC's Technical Advisory Unit (TAU) and Monitoring and Advisory Unit (MAU) based at the University of Kent, JISC ASSIST, the JISC secretariat itself, and the financial and other practical implications of establishing such a framework.

Advisor to JISC

9. In the context of the framework, it is intended to appoint a part-time scientific advisor to JISC on monitoring, evaluating and forecasting electronic information service use and needs in HE. This post will be advertised shortly. The advisor will be a senior person on secondment with a wide ranging brief but no separate staff or budget.

10. The advisor to JISC will: co-ordinate the various strands of the evaluation framework; relate the data gathered to broader trends in the use of networked information reported in the literature and elsewhere; provide forecasts and advice to JISC based on this; advise JISC on additional monitoring and evaluation activities as appropriate (including, for example, any 'once-off' studies of specific disciplinary or institutional needs); and advise the JISC on any modifications to the framework for future cycles.

The Framework

11. The framework for monitoring the provision, use and need for electronic information services consists of a number of elements, including those already undertaken by the TAU and the MAU. Whilst these existing units undertake survey work, this is directly related to the services they are contracted to act as JISC's managing agents for and are monitoring the quality of service.

12. The role of the TAU is to provide input to the JISC on issues concerning the contract and work programme of the JISC's Advisory Committee for Networking (ACN), and to undertake value for money exercises on the contracted services; being mostly those services contracted to UKERNA. TAU activities include drawing up and monitoring service level agreements (SLAs), undertaking technical audits, investigating tariffing and other structures, soliciting and listening to user views of the operational JANET network (through questionnaires, site visits and attendance at user group meetings), and performing some direct monitoring of the network in order to complement and verify the statistics reported by the service providers.

13. The role of the MAU is to establish memoranda of understanding for the services funded under the auspices of CEI and to monitor their activities to ensure they provide value for money through cost effective operation. The MAU assesses the quality of services from the users' point of view and the level of customer satisfaction using a wide range of approaches, including attendance at user meetings, focus groups, questionnaire surveys and informal and formal feedback.

14. The framework for monitoring and evaluation outlined below is in addition to the current work of the TAU and MAU at least for the period 1999/00 AY. The MAU will be directing element B of the framework, which essentially provides the additional resource required to extend the MAU's existing survey work. The other three elements have a much wider context. The four new elements of the framework are:

A. A general survey of end users of all electronic information services

A single (but differentiated) broadbased sample survey of patterns of use and non-use of all electronic information services by librarians, academics and students administered through face-to-face or telephone interviews to ensure accurate sampling of all relevant sub-populations.

B. A 'real time' survey of use of JISC services

A combination of Web based questionnaires, session logs, and user password data to undertake 'real time' surveys to profile the users of JISC services.

C. A general survey of EIS provision

A combination of a Web survey of resource access provided by individual HEIs, with telephone polls of purchasing intentions, backed up by a small number of detailed (face to face) interviews with key informants to profile service provision.

D. A qualitative longitudinal monitoring of EIS use

A linked ongoing programme of longitudinal qualitative evaluation centred around a selection of actual and virtual 'sites' within UK HE, and an associated programme of once off studies on the behaviour and needs of specific disciplinary groups.

Timescales

15. The first 'cycle' of the monitoring system envisaged under the framework will take place in the first half of the academic year 1999/00. Following revisions to the methodologies in the light of the experience of this first cycle, a second cycle will take place in the first half of the academic year 2000/2001 (i.e. approximately 12 months later). JISC will then decide on a continuing annual or bi-annual cycle as appropriate.

Areas of Work

16. Bids are invited for some or all elements of work specified below. Proposers are free to suggest re-combinations of the various elements.

A. A general survey of end users of all electronic information services

A mechanism for periodically measuring and evaluating the overall awareness, uptake, usage and usefulness of information technologies and information services in UK HE.

17. A periodic survey of EIS uptake and use based on face to face and telephone interviewing (complementing the existing work of the MAU and the 'real time' survey of use of JISC services, see B below). Face to face/telephone interviews will allow for smaller but more reliable samples than those presently used in postal and on-line questionnaires, will be cost effective, and will avoid the current survey overload on users. Such a survey will investigate both the quantity and the quality of take up (although qualitative judgements will be informed by the linked programme of longitudinal monitoring, see D below). The survey will provide information on the awareness and usage of services and help service providers bridge the gap between their perceptions of user behaviour and the reality of user behaviour.

18. This mechanism should operate on an regular cycle and consist of a single (but differentiated) broadbased sample survey of patterns of use and non-use of information services by librarians, academics and students administered through face-to-face or telephone interviews to ensure accurate sampling of all sub-populations.

19. The scale of the annual fieldwork will be in the range of several hundred telephone interviews with academics, librarians and other ancillary staff, and several hundred face to face interviews with students.

20. It is assumed that for economic reasons fieldwork will be sub-contracted to a commercial market research organisation or will use student labour.

21. The actual content of the data collection will be determined by negotiation between all potentially interested parties including JISC itself, the various data services, and other current projects and programmes. The content, sampling frame and timing of the survey will be approved with the JISC appointed framework co-ordinator.

B. A 'real time' survey of use of JISC services

22. 'Real time' surveys of use of JISC services will use a combination of Web based questionnaires, session logs, and user password data to undertake 'real time' surveys to profile service users, to over come the difficulties created by the absence of automatic monitoring of resource usage.

23. 'Real time' surveys of use of JISC EIS will be undertaken under the direction of the MAU and in collaboration with the individual data centres, and allow a more consistent profiling of actual users and their usage.

24. The 'real time' surveys will sample incidents of use across the year (controlling for 'seasonal' fluctuations), relating user profile data from on-line questionnaires ('who are you?'), to actual behaviour recorded in session logs, and any indications of origin which can be gleaned from user passwords.

25. It is envisaged that once the approach is settled most of data collection and analysis can be undertaken by a relatively junior researcher.

26. The actual content of the data collection will be determined by negotiation between all potentially interested parties including JISC itself and the various data services. The content, sampling frame and timing of the survey will be approved with the JISC appointed framework co-ordinator.

C. A general survey of EIS provision

27. A combination of a Web survey of resource access provided by individual HEIs, with telephone polls of purchasing intentions, backed up by a small number of detailed (face to face) interviews with key informants to profile service provision.

28. Profiles of current and planned service provision can be built from

• on-line reviews of EIS offers documented on HEI web sites,
• telephone polls of libraries/IT centres, and
• in-depth interviews with a smaller number of key informants (mainly senior librarians).

29. Profiling of EIS provision implies, respectively, a periodic review (annually) of HEI Web sites undertaken by a research assistant, a simple structured telephone poll (undertaken by a commercial market research organisation or student labour), and around 20 in-depth interviews.

D. A qualitative longitudinal monitoring of EIS use

A linked programme of ongoing (longitudinal) monitoring of the information behaviour, needs and opportunities for specific academic and student communities and for academics and students in general.

30. A qualitative longitudinal monitoring of EIS use will complement the other elements of the evaluation framework (for example, to provide overlapping coverage of services, to follow a similar annual monitoring and reporting cycle). The focus of qualitative monitoring will change over time as the service environment evolves (in classes of services, in types of institutions, departments and disciplines) but it will be important to also maintain some continuity in monitoring 'sites' to track evolution in detail.

31. An ongoing programme of longitudinal qualitative monitoring will centre around a selection of actual (HEIs, Departments) and virtual (discipline/subject communities/networks) 'sites', with an periodic reporting cycle.

32. The statistical basis of longitudinal monitoring could involve the use of probit analysis and hierarchical linear modelling, which allows the use of very small panels with high expected levels of attrition and replacement of panel members by new 'equivalent' members, by shifting the unit of analysis from samples of individuals to samples of 'occasions'. Data collection can be undertaken through a rolling panel survey or panel based focus groups, tracking people over time, or simply a virtual cohort of focus groups sampling amongst specific disciplines.

33. However, there are a number of possible ways to undertake this strand effectively and proposers to do this work are free to put forward whatever strategy they feel is reliable and cost effective.

Budget

34. An indicative budget for all elements is in the area of 100k per 'cycle'.

35. Proposers for Tasks A to C should put forward budgets for two cycles. Proposers for Task D should put forward budgets for three cycles.

36. Contractors for Tasks A to C are likely to be offered in the first instance contracts for the first cycle only with subsequent rolling contracts based on performance, but JISC expects to contract in the first instance for at least two cycles of Task D (Qualitative Longitudinal Monitoring).

Format of Proposals

37. Proposals should be set out under the following headings:

• Work tasks (or part of tasks) tendered for
• Description of proposing team
• Rationale for approach to work
• Proposed methodology
• Outputs and deliverables
• Detailed workplan, milestones and deadlines
• Project management
• Budget summary
• Details of previous relevant work
• Individual CVs
• Supporting letter from a senior representative of the institution.

Selection Criteria

38. It is intended to undertake the selection process in May 1999, once the JISC Advisor has been appointed. Proposals will be evaluated, in descending order, against the following list of criteria:

1. Technical rigour, coherence and feasibility of methodology
2. Competence and track record of team
3. Understanding of context
4. Value for money.

39. In general, evaluators will be seeking to judge how reliable proposers will be in delivering the specified outputs and fulfilling their overall contract. JISC will not normally make any contribution towards broader institutional overheads.

Submission of bids

40. Applications may be submitted by any individuals or groups in the higher education sector in the UK, or by independent organisations which maintain close links with the HE sector, provided that such applicants can demonstrate a credible track record in blending academic and practical perspectives. Collaborations between institutions are welcomed where this is of clear benefit to the project.

41. Grantholders under this programme must be prepared to assign publication rights (including electronic publication rights) to JISC or its nominee(s). For its part, JISC will ensure that full credit will be given to authors and contributors in any resulting publication(s).

42. 8 copies of each proposal and letter of institutional support must be submitted in hard copy, printed single sided on A4 white paper. Proposals should be up to 10 sides in length per strand. Proposers are free to bid for some or all elements of the framework and/or suggest re-combinations of the various elements.

43. Each proposal must include a cover sheet. The cover sheet must include:

• the strand(s) (A,B,C,D)
• the name and position held of the contact individual, together with telephone, fax and email details;
• the name and address of the lead institution/organisation;
• details of any collaborating organisations;
• an executive summary of the project;
• the total cost of the work;
• the timescale for completion.

44. Bids should be submitted by 5pm on Wednesday 21^st April 1999. Fax, electronic mail or late applications are not acceptable. Bids should be addressed as follows:

Ms Virginia Twist
JISC Secretariat
Northavon House
Coldharbour Lane
BRISTOL BS16 1QD

45. Each bid will be considered by a panel of independent experts, who will recommend to the CALT and CEI which projects should be funded, and who may also recommend that bidding institutions form collaborative groups with other sites.

46. Funding under this programme will, if approved, commence as from 1 August 1999. Successful projects will be expected to start work on or close to that date.

47. JISC funds are allocated annually by the funding councils. Although the JISC intends to continue funding successful proposals which run beyond 1999/00 AY, those funds can only be fully confirmed on an annual basis.

Further Information

48. Further information about this circular is available from the CALT Secretary, Ms Alice Colban, email: A.Colban@jisc.ac.uk tel: 0117 931 7257.

49. Further information about the JISC and a copy of this Circular can be found on the JISC web site.

Alice Colban
Secretary to CALT