Review of Advanced e-Assessment Techniques
Download the full report
The aim of this project was to provide JISC and its community with a review of state-of-the-art techniques in e-assessment which should be considered for application in UK Higher Education and Further Education settings.
Executive Summary
There is exciting and effective practice in e-assessment in HEIs in the UK but take-up is patchy. This is due to a number of well-rehearsed factors but there are signs that progress being made in the development of off-the-shelf e-assessment solutions may encourage the take-up of these techniques more widely.
It is apparent from this study that e-assessment programmes have needed champions, both in the sense of the developer (often university-based, usually working in spare hours stolen from family or leisure time) and in the sense of institutional indulgence. Without these champions, potential academic end-users have found it unappealing to try and scale the walls of the new technology.
What is changing is usability. Where previously much of the preparatory work had to be done by third party or other technically expert staff, programs are increasingly providing end-user academics with the tools to implement their own e-assessment. The projects described in this report and six companion case studies show that benefits accrue. High quality, valid assessment can be performed which would not be economic without e-assessment; deadlines can be reached which would be otherwise unattainable. Not only is marking workload decreased, but information flow is increased. Tutors can easily check the efficacy of their teaching programmes; students can receive instant feedback on their learning; plagiarism is detected and discouraged; the impact of curriculum changes can be gauged.
Change is attractive when it meets problems that are already appreciated. One of the obstacles to wider acceptance of e-assessment is that it addresses issues that it has been all too easy to ignore. But increased demands for transparency, for timely feedback to students and for measuring teaching quality have raised the stakes. E-assessment offers the means of establishing students’ misconceptions and of correcting them, of demonstrating and improving teacher effectiveness.
The aim of this project is to provide JISC and its community with a review of state-of-the-art techniques in e-assessment which should be considered for application in UK Higher Education (HE) and Further Education (FE) settings. This review will contribute to other programme work to define and develop a roadmap of e-assessment research and development activities. The objectives of the study are to:
-
develop a picture of the breadth of advanced e-assessment techniques in use around the world, and undertake a technical and educational evaluation of a selection of the most promising
-
present short case studies describing the technical, educational and operational details of a selection of the most significant examples
-
look at wider generic technologies and investigate how and when these might be applied to e-assessment
-
draw conclusions about how the most relevant of these could be applied to UK
FE and
HE settings, and provide advice to JISC and its community about such application could be developed
The Review of Advanced e-Assessment Techniques project began by considering what constituted an advanced technique. “Advanced” refers to techniques that are used in isolated or restricted domains, and which have successfully applied technology to create an assessment tool. “Advanced” does not necessarily imply “newness”.
The first stage within the project was to collate top-level descriptive information about potential candidates for further study in the project. This long-list ultimately contained 96 projects. It was a surprise to the project team how few previously unknown advanced e-assessment projects came to light through the trawls for information. The community of experts and departments using e-assessment is small, and this continues to have implications for scaling e-assessment and for stimulating the growth of additional innovative approaches.
The project team had expected to find secure examples of e-assessment techniques being developed within Web2 environments. Although environments such as Second Life are in use as social and learning environments, this project found little evidence of summative or formative assessment applications. We did identify a number of peer-to-peer assessment tools (such as Caspar at Bournemouth University and CAP at Glamorgan University), but considered that the generalisable issues relating to peer-to-peer have been well documented though the WebPA project. We also collected information about several other projects – such as TRIADS and eSCAPE – which certainly fulfil our criteria for “advanced techniques”, but we did not include further study of these as much has already been published. E-Portfolios generally were not included as this assessment approach has been reviewed by a number of other JISC projects.
During the initial collection of information about advanced e-assessment techniques, the project team sought a range of projects covering as many different aspects of the assessment process as possible. We divided assessment business processes into three broad stages:
| Three Stages |
| Pre-hoc process |
This stage covers all processes prior to students completing a test, including processes such as authoring test items and preparing/printing the test papers. |
| Test administration |
This stage covers all processes involved in students sitting a test or examination on a specific date. |
| Post-hoc processes |
This stage covers all business processes that take place after the student has completed the test and might include, for example, marking and moderation. |
In order to cover as broad a range as possible of these processes, we investigated a number of international projects, and looked at projects from the schools and professional training sectors.
A catalogue of all the Advanced e-Assessment Projects has been published at http://www.dur.ac.uk/smart.centre1/aeac
A total of five case studies have also been researched and written. Each of the case studies was selected and organised either around a specific e-assessment theme, or to investigate and describe a single e-assessment project. The case studies were as follows:
Formative Assessment
The focus of this case study is AsTTle The main function of AsTTle is to provide users (tutors or teachers) with a system which enables them to specify and select test items to configure a test. The selection might be predicated on a focus on a particular curriculum domain, for example, or on the demand of the test items. The benefits of such a system derive from the ability to provide teachers with access to large, high-quality and calibrated item banks whilst also providing them with the flexibility to create test forms which closely reflect their teaching. Interestingly, AsTTle is an application developed in HE (in New Zealand), but providing services to school teachers.
Higher Order Skills
Some advocates of e-assessment point to the potential of computers to support simulation and scenario-based assessment. There are few examples of this category of e-assessment being developed successfully, especially not in high stakes testing contexts. Primum is an exception. It provides an assessment of trainee medical practitioners’ ability in making medical diagnoses when presented with a fictitious patient exhibiting a number of symptoms. This automated assessment has been designed to provide an authentic and reliable assessment at a price that compares favourably with the alternative – human scored evaluation at patients’ bedsides.
Combining human and computer marking
Few e-assessment systems are designed to drive efficient marking by automatically sorting and classifying students’ responses, then directing the sorted listing of responses to human markers. Assessment21 does. The benefit of such an approach is that it delivers more efficient marking (a human marker can browse sorted responses, applying one judgement to multiple, identical responses) as well as supporting greater consistency in marking (a marker views all similarly constructed responses at one time and applies a single, common judgement to all of those similar responses).
Automatic scoring of foreign language textual and spoken response
Computers have only limited ability to analyse and score the grammatical complexity of written language, and accuracy and fluency in the spoken word. Two e-assessment applications which do are LISC and Versant. LISC provides a detailed analysis of the grammatical accuracy of sentences translated into a second language by students, supporting students in their understanding and application of grammatical rules. Versant provides an automated spoken language proficiency assessment with applications as diverse as Dutch government citizenship tests, university entrance fluency tests, and air-traffic controllers’ proficiency tests. By designing assessments of highly focused skill domains, both of these e-assessments deliver effective assessments of skills that are prohibitively expensive to provide in a human-scored system.
Hidden Markov Model Toolkit
Automatic scoring of constructed, short text responses
There are a number of e-assessment products which score short-text responses. This case study provides a survey of current practice, with a particular focus on two systems - Automark and IAT. The strengths and limitations of short-test scoring are well understood and, although not a panacea, short-text scoring is a technically strong area of e-assessment. Short-text scoring systems provide the benefits of using constructed responses items (which can provide valid assessments of students’ understanding and application of knowledge) and provide the technology-driven benefits of efficiency and accuracy.
Centre for Open Learning of Mathematics, Science, Computing and Technology (Open University)
Dundee University
This review of advanced e-assessment techniques has found examples of innovative and well researched e-assessment techniques. However, there is little evidence that e-assessment is having a substantial and positive impact on learning and teaching in UK universities. The examples of advanced e-assessment techniques studies in this project suggest that e-assessment can provide some solutions to concerns regarding assessment practice in higher education in the UK:
- e-assessment can offer opportunities to broaden the range of assessment methods used in universities and colleges, where currently the monoculture of essays continues to prevail
- e-assessment offers opportunities to derive efficiencies in assessment – by automating the scoring of some types of item; by providing better diagnostic feedback; and by providing more flexibility over the timing of assessment to students and their tutors
- e-assessment approaches could also provide routes to delivering greater consistency in assessment methods and standards between institutions
It is beyond the scope of this report to speculate why advanced e-assessment techniques are not more prevalent in HE. The aim has been to find examples of e-assessment from which JISC and individual institutions can gain confidence about the education value and technical robustness of e-assessment systems.