JISC

The project explored the use of the DialogPlus toolkit (DPT) and the Learning Activity Management System (LAMS) individually and in tandem for designing, creating and delivering applications which support learning. It investigated the issues surrounding, and the viability of, these toolkits in teaching staff engaging in pedagogic design; the support needed to ensure successful implementation and use of the design tools; and teachers' evaluation of their effectiveness. It identified the success factors for embedding DPT and LAMS into staff development programmes and explored initial specifications for future developments of toolkits as requested by the participants in workshops.

Executive Summary

Aims

The initial aims of the project were to:

• explore the use of the DialogPlus toolkit (DPT) and the Learning Activity Management System (LAMS) individually and in tandem for designing, creating and delivering applications which support learning
• investigate the issues surrounding, and the viability of, teaching staff engaging in pedagogic design
• the support needed to ensure successful implementation and use of the design tools
• teachers' evaluation of their effectiveness

Objectives

The specific objectives were to:

• Develop and deliver design for learning workshops, using DPT and LAMS, for academic and related staff at the Universities of Southampton and Warwick
• Develop and deliver a module on design for learning using DPT and LAMS for a University of Southampton MSc distance learning course aimed at school teachers
• Develop and deliver design for learning workshops using DPT and LAMS for the wider community of teachers and related staff in UK FE/HE
• Evaluate the usability and effectiveness of DPT and LAMS for the staff who design and implement learning activities at these workshops
• Identify success factors for embedding DPT and LAMS into staff development programmes
• Report on the outcomes of the workshops, the MSc module and the evaluation
• Disseminate the outcomes to the wider community

Approaches

These toolkits were evaluated through a series of workshops held at the two collaborating institutions, the University of Southampton and the University of Warwick, shown in table 1. Both of the departments involved in the project at the two separate institutions are staff development units, as well as (at the time the project ran) having a remit to research into new approaches in learning and teaching. The workshops were therefore located within the general professional development programmes delivered at the institutions. The approach was to use the toolkits as a means to introduce staff to the practice of learning design, and then ask them to reflect on the tools, the use to which they could be put within their departments, and the value of learning design in general. The workshops consisted of, therefore, an interactive presentation, followed by a focus group discussion (the standard model for staff development workshops at the institutions). The focus group discussions were recorded and these then doubled as a data-gathering process. Evaluation questionnaires, on the tools and on the workshops, were completed by the participants. Although the response to the tools themselves was mixed, the focus of the workshops on the more generic principles of learning design meant that the feedback about the workshops was mainly positive.

Although the original aims of the project were simply to evaluate the tools, after the pilot workshop in October 2006, it became apparent that participants had come to explore the tools rather then to produce learning designs and applications. The toolkits also fell far short of the participants’ expectations of what a toolkit should provide, and so an additional aim of the project was introduced, that of identifying a set of design specifications for future development of the toolkits that took account of users’ needs.

During the development of the project the following factors became evident, resulting in another revision:

• LAMS and DPT have different roles and uses, and a full understanding of the distinction was necessary for a consistent user needs analysis which was not self-evident to the user groups
• Different user groups have different requirements and data about the nature of the users needed to be captured and correlated with user needs, but in a simple and easily communicable manner
• The introduction of the new pedagogical planner tools, such as the London Planner and Phoebe, was further complicating the picture being collated

Our aims in the final months of the project were therefore expanded to encompass the need to develop a reporting methodology to effectively capture user opinions regarding the toolkits. This reporting methodology was trialled at an end-of-project focus group, and the results disseminated at the ALT-C conference in September, 2007.

Findings

Implementation of toolkits

Users responded well to the use of toolkits as a means to learn about learning design within professional development programmes. The glossary was felt to be helpful, as was the range of different options of learning activities presented by the drop down menus within the toolkits. The heavy pedagogical language was a barrier for the majority of users, although they appreciated the value of the tool as a jargon generator, they also wanted all of the terms translated into more familiar English.

The participants could not see the toolkits being used regularly. The time spent using them, and for most people the cumbersome nature of DPT in particular, was felt to be too onerous to make the use of the tools practical. The opinion was that they could be used as an exercise to when learning about learning design, but they were too impractical for normal use. The exceptions were in the instances of team teaching were the design needed to be shared with colleagues, and in the creation of module approval forms. The jargon generation was felt to be particularly useful in these situations, and considerable interest was shown in integrating DPT with the module approval process.

Many practitioners also expressed concern that the toolkits may be, or be perceived to be seen as a reproach for incorrect practice, rather than an opportunity for receiving support. This barrier may be overcome by disclosing the purposes of the tool and including these in the dissemination and use of the tool. These would include:

• Improving practice
• Enabling reusability of learning designs
• Ensuring quality
• Providing evidence of planning for standards assessment
• Increased productivity

There are larger cultural barriers that need to be addressed within the education sector around issues of quality assurance and the role of pedagogical training, before these tools will be accepted.

Integration with CPD programmes

At both institutions there was considerable confusion about the nature of the workshops from the rest of the programme delivery team, which led to some reluctance to the courses being incorporated within the mainstream programme. Since the toolkits were online, this led to the course teams assuming that the designs were about the delivery of elearning. Typical responses were that there were already elearning workshops being run, and therefore these workshops could be integrated with the other elearning. The statement that these were about generic learning, and the toolkits were simply delivered online were accepted and the forgotten in follow-up discussions. This seemed to indicate an attitude that anything relating to technology was only of interest to elearning practitioners. This is often because the elearning advocates in institutions are those involved in the toolkits’ dissemination. The participants attending the workshops also laboured under this misapprehension. By the end of the project, however, this confusion had been rectified and plans have been made to integrate the learning design toolkits into mainstream sessions about learning design.

Within the remit of a staff development workshop, there is also a limit to the extent to which toolkit evaluation can be carried out, rather than the delivery of content which the participants will find immediately useful. The blend established at the start of the project worked effectively, however, with the first half of the workshops being dedicated to discussing experience of, rationales for and issues concerning learning design in general, followed by a demonstration and experimentation with the toolkits, and discussion of their potential value and usability. In the experimentation part of the workshops, the creation of full learning designs was not justifiable, due to the difficulty of use of the toolkits, the lack of practical use of the final designs (since they were only being created as an exercise in the workshop) and the limited time spent on this part of the workshops as a whole. The feedback received from practitioners indicated that framing the workshop within the rationale of the project, i.e. that the workshop was being run as part of the EDIT4L project, and that they were supporting our research, was met with approval, and encouraged the participants’ engagement with the evaluation.

At the final focus group, the point was raised that the implementation of toolkits cannot be successful if it is left within the responsibility of staff developers. Any change in practice that is widescale enough to constitute "embedding" can only occur if a member of senior management within the institution takes ownership of the process. Persuasion of management at this level must be successfully completed as an early step.

Embedding of the software during this project did not take place for several reasons, and it would have been counterproductive for many users to be exposed to the software, and for the institution to take up the deployment of the software, when it didn’t match their requirements. As a tool from a development project, DPT needs further refinement and development before it could be widely taken up. LAMS has been widely used for a limited range of learning approaches, and individuals need to have a matching requirement before it becomes viable for their institutions to invest in making it available. Previous research (c.f. the WM Share project) indicates that exposing staff members to unreliable or inappropriate technology not only slows down the take-up of that technology, but creates further resistance for any technology that may be introduced in the future.

Usability of toolkits

Reactions to the toolkits varied enormously. Most found DPT very difficult to use, issues were the lack of an overall conceptual understanding of the structure of the designs that DPT was building, lack of ability to jump from one part of the creation process to another, difficulties in identifying simply where one was in the overall creation process. Users that were very linear, and very text-orientated, found the system easy to use. These were the ones who took the steps of the manual one-by-one and were highly structured in the way they approached building their designs. Others who tried to develop their designs in a non-linear manner or needed visual representation of the development of the designs found the toolkit very frustrating, if not almost impossible, to use.

LAMS was felt to be a more intuitively obvious toolkit to use to create designs, however, users were frustrated by the constraints of the system. Because of lack of clarity about LAMS’s place in the development cycle (see later) users expected to be able to use it for developing any learning design, so were disappointed that there was no inclusion of the range of activities that were available within DPT, such as classroom activities, or problem-based learning. The lack of branching was a limitation in the use of it as a design and delivery tool, and the prevention of learners from jumping ahead to read later exercises then returning to complete an earlier activity was particularly annoying.

Disciplinary difference

No discernable difference was detected between the reaction of different practitioners from differing subject disciplines to the usability of the tools, although the data are taken from a small sample size. The difference between the self-identified "linear" and "non-linear" users was significant, but there was no correlation between these types and the faculty they came from. Disciplinary difference in the attitudes to the usefulness of the tools was discernable, but this was because of factors within the departments, such as the Business School using team teaching, or (with the FE lecturers) the requirement to lodge lesson plans with a central (paper-based) repository.

Project design and re-design

The numbers of participants in workshops, and the number of workshops, was less than anticipated in the original project plan. Two national workshops (one at Southampton and one at Warwick) had to be cancelled due to low numbers. The original proposal overestimated the interest of the community as a whole. However a large amount of data was still gathered by conducting deeper and richer analyses with the participants who did take part.

As the data were analysed during the project, it became apparent that there was a lack of consistency in the answers (for example, lack of understanding of the role of LAMS as a design or delivery tool) and a developing realisation that the weaknesses and strengths of the tools were complementary. Mapping the role of the tools in order to see how they may be fitted together was therefore introduced as one of the research aims of the project. This also required a consideration of the production cycle for learning designs (originally referred to as a timeline, but changed in later version in acknowledgment that the process is reiterative in places). A series of preliminary maps were authored and these presented to a selected group of members of the D4L programme and a small number of additional experts. These maps then acted as a focus for a series of discussions. The versions of these maps as they stand at the project end date are included in appendix 2.

The further development of tools

It was observed that the tools currently developed originated from different domains and used the principles of those domains (Phoebe from practitioners; London Planner from administrators; LAMS from schools; DPT from educationalists). What is required is a more integrated approach; but not the creation of an übertool, rather the creation of separate interoperable tools which possess the following features:

• User-configurable So that the different types of users can be catered for
• Progressive disclosure So that new users aren’t confronted with a large range of options. The tool possesses a kernel onto which users can add layers
• Navigability Foregrounding the mental model that the tool uses
• Just-in-time instructions and wizards To allow appropriate learning steps for using the tool
• Engagement Tools that have a larger take-up are those that are sufficiently engaging that they build a community of practice around them. This community can then provide support for new users, which then enable the community to grow further

Three tools are proposed:

The Planning tool

This could be a comprehensive planning tool with the potential to integrate appropriate aspects of course design with institutional planning and quality requirements, for example highlighting ways in which QA requirements are met. However any sense that the tool was creating opportunities for sanction of staff by management would lead to poor take-up and engagement.

The tool provides for the overall planning of a course, composed of sub units with well defined parameters or characteristics, necessary for course monitoring reporting e.g. time for completion of units/sub-units, credits etc. It will compile information from the other tools as required, such as aims, top-level learning outcomes, total times of courses, credits etc. On completion of a course it will collect information on course outputs and feedback, and bring them together with the design parameters recorded earlier. It could provide the facilities for translation of parameters from other tools into the format necessary for institutional reporting

The Design tool provides example of existing designs, and allows designs to be constructed from scratch, or to incorporate exiting designs, in whole or in part. It includes a library of approaches to learning and teaching available through context sensitive help or an index. The Design tool can also be used stand-alone for offline courses, and includes a tutor system to allow self-paced development of pedagogic design skills.

The tool has a number of interfaces, which provide for both structured guidance through a development process, free access to the different stages, and for student design activity. Designs can be annotated during initial and post development. It incorporates a visual design environment in which tags can be incorporated to allow output into (a) tighter structure(s), as available through the structured development interface. Designs need to be able to output appropriate information in formats compatible with the Planning and Aggregator tools, and compatible with institutional administrative requirement or tools. Examples of the latter include Course specification documents, student PDP, PCAP portfolios.

The Aggregator has to be simple to use. It will take output from the Design tool and create a basic structure for development of a learning activity. If possible it will partially populate the activity structure, or provide examples of resources which may fulfil the needs of different phases of the activity. It will contain access to data banks of resources and tools which can be simply incorporated into the activity. For example integration of discussion forums with content so that access is transparent to the user. The Aggregator allows production of a complete learning activity which is ready to use. It does not provide specialist editing facilities for resources such as images, but does provide full editing facilities to change contextual information of imported resources.

Conclusions

Overall, the community we spoke to were not ready for toolkits for learning design, since there were considerable normative values within the community against learning design per se. The practice is seen as onerous and prescriptive adding unnecessarily and unproductively to the workload by those who do not have a strong pedagogical background. Amongst more experienced teachers who do engage with learning principles, the toolkits were felt to be too prescriptive, limiting descriptions of categories to too few choices, and not taking into account that an activity can bridge several categories. An opinion expressed by this group is that, although learning objectives and pedagogical rationales for activities were important, they were sufficiently experienced to hold these learning designs in their heads, they did not need a toolkit to help create them. Sociocultural changes are needed within the higher education sector before toolkits are taken up as means to develop learning designs on a daily basis. Indeed, the rationale for using toolkits needs to be considered in more depth. An implicit assumption in the programme is that toolkits will actually improve practice and this is not proven. There may, however, be alternative uses to which the toolkits can be put, as training tools, a means to share materials and information, or to fulfil administration functions such as module approvals.

The successful outcome of the project, despite the initial low response from the larger community regarding the toolkits, indicates the value of applying flexibility to the aims and methodology of research projects. Identifying a lack of clarity in the field, then creating a new methodology to address this enabled a previously unforeseen but, we anticipate, useful line of research to be investigated. The project drew opportunistically upon the emergent community of practice developing around the D4L programme as a source of expertise for this new line of research, with much success. Practising this form of 'agile development' would be a worthwhile tactic for future projects to consider.

There is an expert community of practice that does have clear ideas about what it requires of planning tools. A more comprehensive review of users’ needs, mapped against the design cycle and the various user groups needs to be conducted in order to develop the correct tools.