e-Learning programme and project evaluation
Project evaluation
Evaluation helps us understand what has happened, adds richness to the story, and makes the evidence clearer for any change.
Evaluation is an important part of any project or programme, and aims to bring out the value of what has been done, and help capture the learning that has taken place during the activity. JISC asks all projects to produce an evaluation plan and carry out some evaluation. This is not to monitor what the project has done or check whether it has been a ‘good’ project, but to help projects and the programme understand what is happening or has happened, to add richness to the project’s story, and make the evidence clearer for any change that has taken place.
Projects are not required to use an external evaluator; in fact, it is very important for the project team to own the evaluation, understand its aims, and feel that it is meets their needs, rather than feeling that it is something being done to them by someone else.
JISC has funded Glenaffric Ltd to produce an evaluation handbook for JISC projects and programmes, outlining six steps to effective evaluation. The six steps start at the inception of a project and are:
- Identify stakeholders
- Describe project and understand programme
- Design evaluation
- Gather evidence
- Analyse results
- Report findings
The handbook offers guidance on each step, and includes useful templates. These templates, a checklist and the handbook are available at the foot of the page.
A number of resources have also been produced to support projects in carrying out learner-centred evaluation which aims to understand learners’ experiences of using technology and find out whether and how a given tool or approach has been useful in a real learning situation. Resources include introductory and more detailed guidance, and example tools such as a learner profile and learner consent form. For more information, please see the page Guidance on learner-centred evaluation.
Programme evaluation framework
The evaluation of the e-learning programme focuses on three key questions:
- What have we done/built/achieved, to what quality, and how efficiently?
- What has been learned or confirmed through development activities?
- How has the learning been acted on or fed back in?
The evidence to answer the first two questions will largely be based on project activities, outputs and outcomes, and drawn from project reports. Examples of the questions the programme is seeking to answer under these two headings are:
- To what extent has our development work led to improved or more efficient practice in learning, teaching and administration?
- What technological developments have been achieved, and to what extent does their function, quality and supporting information make them suitable for uptake by the sector?
- What tangible/measurable benefits have been realised through the work of the programme?
- How effectively have projects under the programme contributed to positive and sustainable change in their institutional strategy, processes and practice?
- What contributions have been made to the e-Framework knowledge base?
- To what extent have the projects and studies contributed to increased knowledge in the programme or activity area? For example:
-
- How can a range of technologies and approaches support learners in FE colleges in the development of HE-level skills?
- What are the issues around integrating student-and practitioner-owned technologies with institutional systems?
- How do learners experience and participate in learning in technology-rich environments?
- What are the drivers and brakes for successful implementation/use of technology?
- What was the impact of innovations on learners? On teachers? On the institution?
- What were the unanticipated outcomes?
Project evaluations would not be expected and should not aim to cover all these aspects, but should be able to contribute to some of them. Some possible project-level evaluation questions, which would contribute to the programme evaluation, are:
- To what extent have the project activities remained relevant to the strategic needs of the institution/partnership, region and sector?
- How effectively has the project contributed to positive and sustainable change in its institution’s strategy, processes and practice?
- How well have project activities learned from and influenced practice?
- What changes in the learner experience have occurred and to what extent can these be described as positive?
- How modular, extendable, flexible, scalable and standards-compliant are any technical innovations?
- How easy it is to find out about, locate, download and use the innovations?
- To what extent have the programme activities remained relevant to the strategic needs of the sector?
- What do we need to do next as a result of programme activity and lessons?