e-Assessment: An overview of JISC activities
As more and more software suppliers and developers become involved in producing e-assessment products, so JISC is bringing the issues associated with this increasingly complex area to the attention of the communities that it serves.
JISC continues to support a range of development initiatives and studies that have examined, or are currently investigating, various issues around e-assessment.
Contents
Introduction
e-Assessment is a critical issue for post-16, higher education (HE) and the wider education community and is recognised within the Department for Education and Skills (DfES) unified e-learning strategy as fundamental to its success and as an important factor in delivering the new 14–19 proposals. JISC recognised the importance of e-assessment for the UK education and research community as long ago as the late 1990’s as part of the groundbreaking work on Managed Learning Environments and work on standards. JISC realises that it has an important role to play through its work in this area. As more and more software suppliers and developers become involved in producing e-assessment products, so JISC is bringing the issues associated with this increasingly complex area to the attention of the communities that it serves.
e-Assessment – our definition:
The end-to-end electronic assessment processes where ICT is used for the presentation of assessment and the recording of responses.
JISC continues to support a range of development initiatives and studies that have examined, or are currently investigating, various issues around e-assessment. Outcomes include:
- a wide range of tools and software development
- case studies
- exploratory reports
- a reference model
- a glossary and a roadmap
These activities help us identify appropriate ways to take development work forward next year.
JISC funded e-assessment
Individual projects rarely focus on one particular e-assessment area such as institutional pedagogical or technical, most are informed by and related to each other. Several projects cross all three areas, and so are described in two main sections:
- Mapping the terrain current national, institutional, pedagogical and technical issues affecting developments
- Tools development and testing
Impact drivers and influencers
There are three main areas that both impact on and are affected by e-assessment:
- Institutional strategic, cultural and operational
- Technical infrastructure, delivery, security; content design, presentation, item bank use; standards, interoperability, accessibility
- Pedagogical new opportunities for learning and teaching
Other key activities that impact on e-assessment:
- developments in e-portfolios – issues of ownership, short-term/ lifelong records
- JISC partnership with Becta to provide a cohesive approach to e-assessment and e-portfolios for schools, post-16 and HE
As a relatively mature area of work e-assessment impacts on and is affected by several factors in the current learning and teaching environment and JISC, working with its partners, is currently aiming to identify the interdependencies and connections between these to ensure that developments do not happen in isolation.
Mapping the terrain
e-assessment Roadmap
This project aimed to provide an overview of the current drivers and barriers to e-assessment in post-16 education. The Roadmap will be using a broad definition of the term ‘e-assessment’ to cover summative (or high stakes) assessment, formative (or low stakes) assessment and diagnostic testing/assessment. The survey identified experts’ opinions on the following issues:
- The timings of policy implementation, ie their realisation in HE and FE (2009 deadline by QCA, not so in Scotland)
- The way in which e-assessment can make a significant contribution to cutting the burden of quality of assessment
- Ways in which e-assessment will make a significant contribution to improving the quality of e-Learning
- The implications for the vision set by the policy documents (some maybe unforeseen)
- Visions for the future
- Barriers to the visions
This project ran from September 2005–April 2006. More information
FREMA
The goal of FREMA is to develop a reference model for systems in the Assessment domain of e-learning that are built on top of Service-Oriented Architectures, such as Web Services and the Grid, and in particular the JISC e-Learning Framework (ELF).
The Assessment Reference Model will describe how the assessment domain maps to the ELF and thus acts as a driver for implementation and evaluation. Once complete, the Assessment Reference Model will ease the development of further services and promote the re-use of existing ones.
IBIS – Item Banks Infrastructure Study
This project put forward a proposal for a distributed item bank system suitable for use in UK FE and HE.This project ran in 2004. More information
Tools – development and testing
APIS – Assessment Provision through Interoperable Segments
Modular assessment item rendering engine utilising the IMS Question and Test Interoperability version 2.0 specification. This project ran from May–October 2004. More information
APIS-MS: Assessment Provision through Interoperable Segments-Marking Services
APIS-MS extended the functionality offered by the original APIS project, and produced a toolkit for integrating external marking web services into e-learning applications such as virtual learning environments (VLEs) or assessment management systems. It also worked on extending the Remote Question protocol specification developed by the Serving Maths project (see below) and successfully integrated it into Moodle. More information
ASAP: Automated System for the Assessment of Programming
The ASAP project produced a set of application services to automate part of the assessment process for the teaching of computer programming languages, including: unit testing of student code and the provision of feedback; the automated generation of questions; and plagiarism detection and reporting. A significant outcome was the production of tools that can be used within a number of VLEs and interfaces to relevant thirdparty software. This project ran from September 2004–March 2005. More information
ASAP 2: Automatic System for the Assessment of Programming
This project extended work on the original ASAP project by developing the Pilot Software to Production Stage (Assessment Web Service and Blackboard User Agent) and developing Additional Tools and Services to a Pilot Stage (User Agent Tools and Assessment Manager). This project ran from May 2005–March 2006.
Assessment Tools in Mathematics
This project evaluated, piloted and developed further the Serving Mathematics tool. The following elements were built in: an authoring tool for computer-aided assessment in Maths; MathQTI specifications and XML Schema for mathematical question test items; and a rendering and assessment tool for mathematical question tests marked up in MathQTI. This project ran from May 2005–March 2006.
ASSIS – Assessment and Simple Sequencing Integration Services
Tools for teachers to browse, search, preview and select assessment objects from question banks. These can then be incorporated into packages of content, which may also include sequencing instructions. This project ran from September 2004–March 2005. More information
CATS: Constructing Assessments using Tools and Services
CATS aims to create a toolkit of loosely coupled web services, which support the various tasks inherent to automated assessment construction, eg searching for, retrieving and aggregating assessment items held in multiple item banks. This project runs from March–August 2006. More information
ISIS
ISIS produced libraries to enable the production and consumption of content-sequencing services using IMS Simple Sequencing alongside tools for the visualisation and running of sequences. The project’s outputs include an integration guide for IMS Simple Sequencing and QTI. More information
OpenMentor
This project aimed to provide a learning support tool for teachers in FE and HE to help them by providing reflective comments on their assessment and feedback of student assignments and coursework. This project ran from September 2004–March 2005. More information
OpenMentor 2
This project further developed the OpenMentor tool. This project ran from May 2005–March 2006. More information
PyAssess
PyAssess provides an open source toolkit for implementing QTI v2-based assessment (particularly marking) services in Python, building on the Python QTI v1.2 to v2.0 migration tool produced earlier. The project also produced a command line demonstrator to illustrate the use of the toolkit and support the testing and debugging of complex questions. By separating response processing from delivery, PyAssess provides a very secure method of assessment delivery that minimises the possibility of cheating by technically aware candidates. More information
Serving Maths
The project developed open source software tools to address issues around the use of mathematical expressions in assessments. One of the outputs of the project was the Remote Question Protocol, which supports remote processing of assessment items on behalf of assessment systems and therefore maximises the range of item types and formats it is possible to deliver within a single assessment session. More information
SPAID – Storage and Packaging of Assessment Item Data
This project scoped and implemented services to enable the packaging of assessment items for storage in an item bank and the search and retrieval services necessary for the use of item banks. More information
TOIA
This project developed an advanced online assessment management system available free of charge to all UK FE and HE institutions. This project has been running since 2002.
e-Assessment Glossary A glossary of assessment terminology for schools, post-16 and
HE sectors has been developed. This project was jointly funded by the
QCA. The Glossary is supported by all UK regulators and it defines over 700 terms for teachers, awarding bodies, regulators, vendors, institutions and learners. The Glossary was launched in January 2006 and a fully searchable online version will be available from June 2006. The project ran from January 2005–May 2006.
More information
e-Assessment case studies
A set of e-assessment case studies of innovative and effective practice has also been funded. These involve 55 institutions across the UK and also consider largescale assessment outside the FE and HE sectors. The case studies have investigated several e-assessment issues in a variety of institutions and contexts including: innovation, effectiveness, maturity, accessibility, confidence-based testing and single champions in an institution. The final report and case studies will be available from October 2006. More information
REAP (Re-Engineering Assessment Practices)
Funded by the Scottish Funding Council (SFC) as one of their e-Learning Transformation projects, this project is investigating innovative approaches to assessment practices across three HE universities. This project is particularly engaged with institutional issues relating to change in assessment practices. These projects are being managed and supported by the JISC e-Learning team on behalf of the SFC. This project runs from March 2005–July 2007. More information
JISC’s distributed e-Learning projects
The e-Learning Tools projects under the Distributed e-Learning (DeL) projects were completed in March 2005. A number of projects relate to developing specific tools to support e-assessment or e-portfolio creation and are included in the above list.
- ASAP
- ASSIS
- OpenMentor
- Serving Maths
Phase 2 of the DeL projects develop this work further:
- ASAP 2
- Assessment Tools in Mathematics
- OpenMentor 2
Projects from the JISC Managed Learning Environments (MLEs) for Lifelong Learning programme (Phase 2)
Several of the projects in this strand were engaged with e-portfolios and Personal Development Planning (PDP) and included assessment within their remits.
Specifying an e-Portfolio enhanced learner information for flexible admissions and transitions into Higher Education.
Transfer of Portfolios Assisting Lifelong Learning project
Developing Learning and Teaching Aspects of Progress File and The Personal Development Profile Across FE and HE
PROSPERO PROcesses for Support of PErsonal development Records Online
All in one place
The outcomes from the Roadmap study, the Glossary and case studies were synthesised to form the Effective Practice with e-Assessment guide.
JISC services to support e-Assessment
- JISC Cetis Assessment Special Interest Group Established to monitor interoperability initiatives in the area of online assessment for FE and HE practitioners in the UK. JISC Cetis will keep you up to date with activities in standards and specifications development as well as sharing knowledge and experience. JISC Cetis works directly with the IMS Global Learning Consortium to ensure that the emerging interoperability standards meet the requirements of UK FE and HE.
- JISC infoNet Amongst many other activities, JISC infoNet offers information on e-assessment through its 'Effective use of VLEs Infokit'
- JISC Plagiarism Advisory Service can offer help and advice on the plagiarism surrounding e-assessment.
- Netskills offers a workshop and support materials on Assessment and Communications for e-Learning.
- TechDis The JISC funded service provides advice on accessibility in relation to e-assessment. They have produced an e-assessment staff pack
Future JISC activities
Outputs from all our projects inform our future e-assessment activities. In particular we anticipate continuing work in the following areas:
- Linkage of Roadmap of e-assessment to equivalent roadmap for e-assessment in schools
- e-Portfolio roadmap in conjunction with Becta
- Projects around innovative assessment practice – use of mobile technologies and gaming
- Effective practice with e-assessment guide
- Standards
- Interoperability
- Accessibility
- Consistent language
- Advising the education community
- Large-scale demonstrator projects of developed tools
- Partnerships and collaboration – including JISC Services, Becta, HE Academy, and the UK regulatory bodies
You may wish to join the e-assessment mailing list – to do so, send an email message (from your own account) containing the line: (join e-assessment firstname lastname (in the main body of the email) to jiscmail@jiscmail.ac.uk
This document has been edited and produced by Dicky Maidment-Otlet and the JISC Communications and Marketing team.