e-Assessment: An Overview of JISC Activities
e-Assessment – our definition: The end-to-end electronic assessment processes where ICT is used for the presentation of assessment and the recording of responses.
Version 2: July 2008 (read Version 1)
Introduction
e-Assessment is a critical issue for post-16, higher education (HE) and the wider education community; it is recognised within the Department for Innovation, Universities and Skills (DIUS) as fundamental to its success and as an important factor in delivering the emerging programmes of learning for 14- to 19-year-olds.
JISC recognised the importance of e-assessment for the UK education and research community as long ago as the late 1990s as part of the groundbreaking work on Managed Learning Environments and on standards. JISC realises that it has an important role to play through its work in this area. As software suppliers and test developers become increasingly involved in producing e-assessment tools and content, so JISC is bringing the issues associated with this increasingly complex area to the attention of the communities that it serves.
This document describes recent work on e-assessment within the JISC e-Learning programme.
JISC’s activities support education and research by promoting innovation in new technologies and by the central support of Information and Communication Technology (ICT) services.
As a relatively mature area of work e-assessment impacts on and is affected by several factors in the current learning and teaching environment and JISC, working with its partners, is currently aiming to identify the interdependencies and connections between these to ensure that developments do not happen in isolation.
Impact drivers and influencers
There are three main areas that impact on and are affected by e-assessment:
- Institutional: strategic, cultural and operational
- Technical: infrastructure, delivery, security, content design, presentation, item banking, standards, interoperability, accessibility
- Pedagogical: new opportunities for learning and teaching
Other key activities that impact on e-assessment:
- Developments in e-portfolios: issues of ownership, short-term/lifelong records
- JISC’s partnership with Becta, the Higher Education Academy (HEA) and other stakeholders to provide a cohesive approach to e-assessment and e-portfolios for schools, post-16 and HE
JISC Funded e-Assessment Developments
Descriptions of the e-assessment projects can be found below. Individual projects rarely focus on one particular e-assessment area such as institutional, pedagogical or technical; most are informed by and related to each other. Several projects cross all three areas, and so are described in three main sections:
- Mapping the terrain: current national, institutional, pedagogical and technical issues affecting developments
- Developing tools: developing and testing software components
- Piloting innovative use of e-assessment: projects experimenting with uses of e-assessment with new learner communities
Mapping the Terrain
e-Assessment roadmap
This project (completed in April 2006) provided an overview of the current drivers and barriers to e-assessment in post-16 education, covering summative (or high stakes) assessment, formative (or low stakes) assessment and diagnostic testing. The survey identified experts’ opinions on the following issues:
- The timings of policy implementation and realisation in HE and further education (FE)
- How e-assessment can help to reduce the burden of undertaking high quality assessment
- How e-assessment can contribute to improving the quality of e-learning
- The implications for the vision set by institutional, regulatory and national policy documents
- Visions for the future
- Barriers to these visions
JISC plans to update the roadmap during 2008.
The FREMA project provides an Assessment Reference Model for systems in the assessment domain of e-learning that are built on top of Service-Oriented Architectures (SOAs); such as Web Services and the Grid, and in particular the JISC e-Framework.
The Assessment Reference Model begins to describe how the assessment domain maps to the e-Framework and thus acts as a driver for implementation and evaluation. FREMA aims to ease the development of further web services and promote the re-use of existing ones and will be fully integrated into the e-Framework knowledge base.
Guide to Effective Practice with e-Assessment
This document combines:
- The e-assessment roadmap
- The e-assessment glossary
- A set of e-assessment case studies of innovative and effective practice involving 55 institutions across the UK and large-scale assessment outside the FE and HE sectors. The studies cover issues such as innovation, effectiveness, maturity, accessibility, confidence-based testing and single champions in an institution
Developing Tools
The AQuRate project has developed an open source, QTI v2 standards compliant, platform independent question authoring tool with API.
The Minibix project has developed an open source item banking system for QTI v2 items capable of supporting both high-stakes private item banks and low-stakes item banks for sharing questions. The tool interoperates with the HEA Physical Sciences item bank. The project team will provide a QTI SUM to the e-Framework.
The ASDEL project has developed an open source QTI v2 assessment delivery engine that can be deployed as a stand-alone web application or as part of a SOA-enabled virtual learning environment (VLE). The delivery engine includes assessment developer tools such as a QTI v2 validator. The tools interoperate with ASDEL and Minibix and provide a Moodle connector.
WebPA is an open source online peer assessment system which has been developing in maturity since 2006. It now supports peer assessment of group work, SOA design and integration with VLEs.
The project produced a simple web service enabling very easy production of a popular subset of QTI questions from any MCQFM consuming application, thus enabling the community of QTI development practitioners to further disseminate the standard.
The Xmarks project has developed components to allow the exchange of assessment information between an institution’s student records system and VLE, using a web services model. The project is aimed at developers within FE and HE institutions who are looking for a way to synchronise assessment and marks data.
This project further developed OpenMentor, a learning support tool for teachers in FE and HE, which helps them by providing reflective comments on their assessment and feedback of student assignments and coursework.
PyAssess provides an open source toolkit for implementing QTI v2-based assessment (particularly marking) services in Python, building on the Python QTI v1.2 to v2.0 migration tool produced earlier. The project also produced a command line demonstrator to illustrate the use of the toolkit and support the testing and debugging of complex questions.
This project scoped and implemented services to enable the packaging of assessment items for storage in an item bank and the search and retrieval services necessary for the use of item banks.
UK-CDR is a software design project to define functional requirements for an item bank tool to support storage, exchange and analysis of high-stakes assessment items for use in UK FE and HE.
Piloting Innovative Use of e-Assessment
This project aims to derive a personalised e-learning assessment and reporting model which meets the needs of work-based learners, local employers and the training provider, based on e-assessment of engineering students employed by Rolls Royce and studying at Derby College.
This project is exploring the role of e-assessment in enhancing the student experience of HE students in an FE college.
JISC Services to Support e-Assessment
JISC Assessment Special Interest Group (SIG)
Established to monitor interoperability initiatives in e-assessment in FE and HE, the Assessment SIG also monitors wider developments in the technology and application of e-assessment. The SIG is run by the JISC-CETIS innovation service and includes a mailing list and regular open meetings.
JISC infoNet offers information on e-assessment through its Effective Use of VLEs infoKit. JISC infoNet is an advisory service promoting good practice and innovation within the education sector.
This service offers help and advice on the plagiarism surrounding e-assessment.
Netskills offers workshops and support materials on assessment and communications for e-learning.
TechDis provides advice on accessibility in relation to e-assessment.
Other Sources of e-Assessment Information
The Higher Education Academy’s mission is to help institutions, discipline groups and all staff to provide the best possible learning experience for their students. Their website includes information about several e-assessment projects.
Becta steers the UK drive to inspire and lead the effective and innovative use of technology throughout learning in schools and colleges. Their website includes resources relating to e-assessment and e-portfolios.
This Qualifications and Curriculum Authority (QCA) website is designed to help centres develop e-assessment strategies and systems fit for the future. It provides a ‘one stop shop’ for everyone involved in e-assessment.
Futurelab develop innovative resources and practices that support new approaches to learning for the 21st century. They have produced a number of reports on formative and summative assessment and the state of the art in e-assessment.
The e-Assessment Question is an annual conference on e-assessment, focusing particularly on summative assessment and the schools and FE sectors. The website hosts an archive of previous presentations.
The CAA Conference is relevant to those with an interest in e-assessment implementation and pedagogy, the more technically minded, and those involved at a strategic level such as policy makers, managers and support services. The website hosts an archive of refereed proceedings.
Future JISC Activities
Outputs from all our projects inform our future e-assessment activities. In particular we anticipate continuing work in the following areas:
Formative e-assessment
Developing requirements for formative assessment systems, components and processes based on proven pedagogical practice, to enable future developments of formative e-assessment systems, services and content to be undertaken with confidence.
Quality of e-assessment
Research to look at qualitative and quantitative measures of fitness for purpose in the content of e-assessments used in the JISC community today. While there is consensus that e-assessment offers potential to enhance assessment quality through both improvements in efficiency and effectiveness, there are also concerns about multiple choice tests leading to impoverishment of assessment, emphasising memorisation and factual recall rather than higher level skills.
Advanced e-assessment
Reviewing innovative technologies and pedagogies which find effective use in e-assessment, for example item presentation and response, test adaptation, free-text and mathematical marking, expert systems for marking support and analysis, to inform future development planning for JISC and the sector.
Standards and interoperability
Supporting interoperability of e-assessment content and results between e-assessment content and delivery systems, and interoperability of these systems with other systems as part of the wider JISC e-Framework developments.
Demonstrator projects
Using Service Oriented Approaches to create composite applications based on existing components produced in previous JISC projects and elsewhere to broaden the uptake and usage of QTI2, supporting the associated emerging developer community.
This document has been edited and produced by Myles Danson (JISC Executive) plus John Winkley (Alpha Consultancy) and the JISC Communications and Marketing Team.
Further Information
More information about JISC e-Assessment
Contact the e-Assessment Programme Manager: Myles Danson
Join the JISC e-Assessment mailing list: email jiscmail@jiscmail.ac.uk from your own account giving the line join e-Assessment [firstname] [lastname] in the main body of the email
This publication is available electronically only.