JISC

We define formative e-assessment as the use of ICT to support the iterative process of gathering and analysing information about student learning by teachers as well as learners and of evaluating it in relation to prior achievement and attainment of intended, as well as unintended learning outcomes, in a way that allows the teacher or student to adjust the learning trajectory.

Black and Wiliam (2009) conceptualise formative assessment in terms of five key strategies:

1. engineering effective classroom discussion, questions, and learning tasks that elicit evidence of learning
2. providing feedback that moves learners forward
3. clarifying and sharing learning intentions and criteria for success
4. activating students as owners of their own learning
5. activating students as instructional resources for one another

Figure 1: Key aspects of formative assessment (Black and Wiliam, 2009)

No assessment technology is in itself formative, but almost any technology can be used in a formative way – if the right conditions are set in place. This observation is in line with a socio-technical view of educational systems, which sees the technological dimensions (e.g. speed, storage capacity, processing, communication, construction and representation and mutability) as inseparable from the pedagogical parameters (e.g. verbal/electronic/synchronous/ asynchronous interaction between key players which brings about changes in concepts or skills). It is consistent with a view of learning as ‘conversational’, and this range of combined resources impacts not only on how students act but also informs what teachers do to enhance learning. In the domain of formative assessment, the pivotal factor we wish to propose is the concept of Moments of Contingency: critical points in the teaching and learning process where the flow of instruction cannot be predetermined (Black and Wiliam, 2009). Moments of contingency contain within them the scope for learners’ understanding to be ‘otherwise’. The technology itself does not create these moments; they are dependent on teachers’ and learners’ actions. But for technology to perform formatively, it needs to acknowledge and support these moments.

The report develops an extended vision for formative e-assessment.

Key points from the literature

• There are widely differing theoretical emphases in the literature and, within e-assessment, a tendency to conflate formative and summative assessment, within a view of ‘adaptivity’ as a core component of e-assessment processes
• A core component around which there is much difference is the role of the ‘teacher’ and to what extent their role in formative assessment includes adaptation of pedagogy. To what extent is ‘monitoring’ and ‘managing’ assessment processes formative in terms of transforming the learning environment or pedagogy in response to evidence of learners’ progress?
• Some examples of formative e-assessment can be argued to be serial summative assessment. Formative assessment appears to be equated with ‘low stakes’ assessment, or ‘practice’ assessment in preparation or contributing towards high stakes summative outcomes
• The role of ‘evidence’ is core (how it is used, generated, by whom/what and affecting whom/what). When thinking about assessment as a noun, it is useful to distinguish the event which generates the evidence (e.g. a test as ‘an assessment’) and the evidence itself (the score)
• Learner self-regulation is a core feature, linked to motivation and emotional factors which affect learners’ engagement with feedback

Methodology

Practitioners were prompted to recount their experiences of using formative eassessment as case stories, and discuss these with their peers. The construction and discussion of these narratives were scaffolded by a set of tools and activities to extract transferable and verifiable elements of design knowledge in the form of design patterns. These patterns were then applied to novel problems from real situations by both teachers and software developers to develop use scenarios.

Case Studies

A major section of the report describes a series of exemplars of e-assessment practice currently to be found across the sector. The case studies described are: Academic writing, Audiofiles, Como: mobiles + flickr = co-reflective practice, Open Mentor, String Comparison, Medical students, Shadows, Personal Response System (PRS), Click. These case studies constitute the data from which design patterns for formative e-assessment were developed.

Patterns

Another major section of the report presents ten design patterns (Classroom Display, Feedback on Feedback, Narrative Spaces, Objects To Talk With, Round and Deep, Showcase Learning, Soft Scaffolding, Try Once, Refine Once, Wear Your Skills on Your Shirt, Use my stuff) derived from the case studies. The patterns serve as mediational tools: on the one hand they provide stimuli for practitioners to critically review and innovate their practice; on the other hand they provide a basis for an understanding of key pedagogical issues attendant to formative e-assessment for software developers. These patterns are mapped against the five key learning strategies derived from the literature and shown in Figure 1.

What ‘e’ adds to formative assessment

Speed

• Speed of response is often important in enabling feedback to have an effect
• Supports rapid iteration – the ability to give feedback quickly means that the student’s next problem solving iteration can begin more quickly

Storage capacity

• Ability to access very large amounts of data (so appropriate feedback/additional work/illustrations can be identified).

Processing

• Automation - in some situations the e-assessment system can analyse responses automatically and provide appropriate feedback
• Scalability – can often be the result of some level of automation
• Adaptivity – systems can adapt to students.

Communication

• Often the advantage of the ‘e’ is that it enables rapid communication of ideas across a range of audiences, and the technology allows this range to be controlled, it can be just one person, a group, a class or more
• This communication aspect means that aspects of communication can be captured and given a degree of semi-permanence
• This semi-permanence supports the sharing of intellectual objects

Construction and representation

• Representation – the ability to represent ideas in a variety of ways and to move and translate between these representations
• Technology can support learners in the representation of their own ideas
• Through representation technology enables concepts to be ‘shaped’ and this helps learners develops their meaning
• In representing their ideas in digital artefacts learners open up a window on their thinking

Mutability

• Shared objects are not fixed, they can change/be changed easily

Key recommendations

• Developers wishing to address formative e-assessment, both developers of new assessment tools, and those wishing to introduce elements of formative assessment into existing e-learning systems should consult the pattern collections in this domain
• Existing e-learning tools should be adapted to function in a formative way through the identification of design patterns of formative e-assessment that can be readily applied to these systems
• Developers need to beware of the danger of applying the ‘right pattern to the wrong problem’, as patterns derive their power from being context and problem dependent
• JISC should fund further research to develop a comprehensive language of design patterns for formative e-assessment. and to engage interdisciplinary communities of educators and software developers in iterative participatory pattern-based production of tools for formative e-assessment

Report available electronically only