e-Learning Framework Reference Model for Assessment
The FREMA Project ran for nearly two years, involved a large number of people, and has produced a complex set of outputs. This document reports on the history and evolution of the project, and how the outputs fit into this story (the outputs are highlighted in bold). It then presents each of the outputs in detail in individual sections.
Summary
FREMA was funded as an eighteen month project to produce a Service Oriented Reference Model for the Assessment Domain, in particular the objective was to produce some sort of conceptualisation of services working within the domain that could help guide other projects producing services for the emerging JISC e-Learning Framework (since renamed the e-Framework). Several reference model projects were funded, and the shape of the conceptualisation was left intentionally ambiguous by JISC, in the hope that a more concrete notion would emerge as the projects progressed.
The FREMA team took the view that the necessary shape of a reference model would change according to the nature of the domain being modelled. Assessment can be characterised as a brown field site, in that it is very broad, with a lot of previous activity, but cannot really be considered mature. As a result we felt that a Reference Model for Assessment should attempt to capture the efforts within the domain, extrapolate important common use cases, and show how to build abstract service designs on those use cases. We called the resulting pyramid structure (implementations on designs on use cases on domain resources) a Community Reference Model, as the intention is to provide a growing community resource, with a proper audit trail of decision making, that can be extended as the community sees fit (for example, by extrapolating new use cases, or building service implementations of designs already in the reference model). The important point was that FREMA needed to deliver the whole pyramid, not just a slice (such as a layer of design).
Community Reference Model Layers
To deliver such a complex model would require a dynamic information resource rather than a static document, so we opted to build FREMA as a heavily interlinked web site, where a user could see the connections between the different layers (for example, which projects and standards contributed to which use case. We modelled the information in a knowledge base and delivered it as a dynamic ontological web site. Our ontology was built for the Assessment domain, but may well be of use to other domains, as it contains common concepts such as People, Organisations and Standards. We populated the Knowledge Base through a prolonged paper and person based activity. On paper we surveyed previous JISC projects in the domain, and other significant projects such as SAKAI. In person we visited a number of UK sites in order to prioritise our activity and begin to extrapolate use cases. Once populated the knowledge base allowed us to use automatic tools to build a gap analysis (by looking at the holes between use cases and service designs). Although the ontology was a powerful modelling tool, it was not the most natural way for users to explore the resources within the reference model as they had to know what sort of resource they wanted to find before they began. We developed a concept map tool, which overlaid a concept map on the resources and allowed users to explore more naturally. The concept maps themselves were created in a series of workshops and validated by the Assessment SIG, we created two maps to cover different views of the Assessment domain: a concept map of entities, and a concept map of processes. An interesting side effect of using concept maps, was that it became simple to create a topology of effort for the Assessment domain, by mapping resources such as projects and standards to concepts, this tool enables people to see where most activity has occurred within the domain.
Once we had identified and gathered examples of the most important use cases in the domain (we identified three in particular: End-to-end Summative and Peer Assessment, and External Examination) we were able to work on establishing canonical formal versions. In particular we focused on Summative End-to-End as it was the best represented in terms of technology (such as Item Banks, and the QTI standard). The jump from a formal use case to a service design is not well understood, even by the Service-Oriented Computing Community, so we took an agile software engineering approach to service design, and borrowed Class Responsibility and Collaboration Cards from the Object Oriented world. By factoring use cases into Service Responsibility and Collaboration Cards (SRCs) and refactoring as necessary we were able to create a set of abstract services for Summative End-to-End Assessment. We used UML interaction diagrams, Web Service Interaction diagrams (WS-Is), to show how they could work together. At the top of the pyramid we wanted to create a number of example service implementations, since there are many Item Banks in existence we decided to work on wrapping a number of them with Services from the End-to-End design. There were a number of ways in which the existing tools could be wrapped, and we explore these and formally recorded them as Service Patterns within the Reference Model knowledge base. Once this was done we had service implementations, designs, use cases and interaction diagrams in the knowledge base, we summarised these as a single Service Usage Model (SUM) for End-to-end Summative Assessment. Although the initial project had now ended, we received extra funds to continue the work, and in particular to hand the Reference Model over to the community so that our view of it as an evolving resource could become a reality. Although we continued to add to the layers of the reference model, the most important change was to consider how the community could add to it, and edit what was already inside it. We looked at the possibility of adding editing functionality to our existing web system, but realised that a full move to a Wiki type environment was a realistic alternative. This is because the knowledge base models typed links between resources, and a new generation of Semantic Wikis was capable of supporting typed links. We investigated a number of Semantic Wiki implementations and found that the Semantic Media Wiki (SMW) open source project was the most robust and the fitted our requirements most closely.
We ported the existing knowledge base into an instance of SMW – the FREMA Semantic Wiki – each resource in the database became a new Wiki page, and the ontological connections became typed Wiki links. To recreate the full FREMA functionality we also had to rewrite some of our tools as SMW extensions and scripts.
Once we had completed this process (which was not a perfect mapping) we had a Wiki based system that was capable of modelling the assessment domain, and the development of services and SUMs within it. As it was Wiki-based users were free to add to any layer of the reference model they wanted, and to develop new use cases, services designs and implementations as appropriate. Although we are still hosting the FREMA SMW site, we are currently in the process of handing control of it to the Assessment SIG to maintain and promote its use in the UK community.
Outputs
The following sections detail each of the separate outputs of the project:
- Section 2: The Community Reference Model Approach
- Section 3: FREMA Community Reference Model Implementation
- Version 1 using an Ontological database (KB)
- Version 2 developed as a semantic wiki with authoring, discussion and moderation tools for full community ownership
- Section 4: Concept maps and Concept Map Tool
- Topology of the Assessment Domain
- Section 5: Agile Service Development Methodology (SRI-DM)
- SRCs for Service Definition
- WS-Is for Service Interoperability
- Section 6: Full Overview of the SUM for End to End Summative Assessment
- Use Cases, Service Expressions (SRCs) and Example workflow (WS-Is)
- Section 7: Introduction to other SUMs within FREMA
- Peer Assessment and External Examiner
- Section 8: Design Patterns for Service Interoperability
- Section 9: Evaluations and Reviews of FREMA