Virtual learning environment review
A guide to support organisations wanting to review their virtual learning environment (VLE) use.
Introduction
To fully understand if a VLE platform is meeting the needs of the institution, you should take time to observe and define how people are using the platform. Asking the right questions and listening to the right voices is essential for an effective and successful outcome.
A guide to go further than just going through the motions
It takes time and effort to review any part of an institution’s infrastructure, systems and processes. Whether to overcome issues or because it is timely, an effective review is going further than a box ticking exercise. Each element of this guide is designed to take your review deeper into the practices and culture around your VLE platform(s) to achieve a more effective outcome.
Review stages
The review methods within the guide are tried and tested following over 6 years of running VLE reviews with Jisc customers. We have divided the tools and guidance into different stages of a review process. These stages each have a specific role in getting the most out of a VLE review.
| Review stage | Actions |
|---|---|
| Foundation setting | Setting out why you are reviewing and what you hope to achieve. The purpose and scope of the review |
| Discovery | Immersing the review into the culture of VLE use. Learning about the platform(s) and assuming nothing. Listening to those who use it. Being empathic. Understanding the present. |
| Defining | Understanding all you have learned. Define the details. Identify good practices, challenges and barriers |
| Ideate | What solutions work for your organisation? Blue sky thinking encouraged. What ideas don't fit the need? |
| Actions | Reporting the findings. Who can make change happen? |
In many scoping meetings with Jisc customers, organisations have looked at their VLE from the perspective of IT and infrastructure. While a VLE is a ‘digital platform’ and has criteria that must be met for safe and compliant use, the platform has a major role to play in teaching, learning and assessment. A VLE may also be supporting other functions around an institution. This guide aims to review beyond the digital state of the platform(s) and into the culture of those using it.
What this guide won't say
This guide is designed to help institutions review their VLE platform(s). There will be advice and guidance as well as recommendations for next steps. No suggestion will be made as to which platform is best, or which best suits your needs. Many of the VLE platforms available today are equally capable of supporting and enhancing teaching, learning and assessment. Each organisation will have different needs from a technical and user perspective.
Reviewing a VLE is not always about finding answers, more about deciding on the right questions an institution should ask of its platforms and procedures. Being equitable about providers and brands allows a review to be Impartial and honest. The most honest reviews are the most accurate and effective.
Who is this guide written by?
This guide has been written by subject specialists from the digital practice programme as part of the Jisc advice team.
The Digital Practice programme have been running in-person and online VLE reviews with customer institutions from HE and FE across the UK for more than 6 years. With an emphasis on helping customer institutions develop the role of a VLE, the reviews undertaken have allowed the team to build on their experience and understanding of how VLEs are used and developed.
Getting started
We’ve set out the five stages of the review in this order intentionally. Each one should be informed by the previous phase to allow for a more defined review. As the organisation learns more about its culture and needs for a VLE, the vision, actions and strategy for success are progressively more finely tuned to achieve success.
This guide describes the way we typically use the activities or tools in the reviews Jisc runs. There are often other ways to collect information or learn from others. You might find it more effective to tweak the approach. The most important thing is that you can collect the knowledge and learn about how your VLE is used.
Foundation setting
Set out why you are reviewing and what you hope to achieve.
Defining the purpose and scope of the review
As a first step, it is important to understand why the institution has decided to review. Who has called for it and what were their reasons? Quite often, someone has seen another platform or has come from another institution where a different platform or practices are used. While it may be a good topic of conversation, it should not be a sole reason for review or change.
Define the purpose and scope of the review first. Bringing a group of key stakeholders together to refine and cement the review is key. Each user group (teaching staff, students, professional services etc) will have developed practices around the current platform. They have a huge amount of knowledge and valuable insights. Decide how you can bring their voice into defining the review scope and purpose.
Educators, administrators, managers and leaders are all key users, as well as learners and students. For a few institutions, changing platforms or procedures maybe a necessary step further down the line, but entering into a review with the intention for a change before seeing the results can be a risky process.
The purpose and scope of a review should be to understand the current situation in an unfiltered and unbiased process. The more honest an institution can be in its review, the more easy it is to ask the right questions and make the most informed decisions. Entering into a review without an open mind or making assumptions could mean that key decisions are ill-informed and important issues are overlooked. Changing for the wrong reasons can result in expensive alterations and huge staff disruption for something that could be a poor solution.
What are the risks of not reviewing?
As part of the review, consider what could happen if you were to carry on as you are, without reviewing. For example, what impacts could there be to the ongoing development of teaching, learning and assessment through the VLE? What issues, currently known or unknown, could be obstacles to achieving your strategic goals?
What does a successful review mean to you?
With a clear purpose and scope, you can start to refine what success looks like. It won’t be just one tick box. It shouldn’t be done in one day. Laying out some initial questions can help set some firm foundations on which to build. Try these:
- What do you want to achieve?
- What do you think needs changing?
- How do you explore this?
- How long should a review take?
- What stages should you set?
- Are there any deadlines?
- Who is involved in planning the review and carrying it out?
- Who do we need to speak to?
- Who should the review report to?
- How will you share the findings?
At the end of the review, the organisation will want to have the most informed insights into how different user groups engage with the VLE. These insights should allow the organisation to develop the questions, vision and strategy to achieve a successful and effective VLE.
Activity: form a VLE review working group
Team: key decision makers and users of VLE at institution
Preparation level: medium
Maximum number of people in group: dependent on institution - representation of key institution departments and VLE user groups
Time to complete: the duration of the review process, decision-making and action-taking post-review
An effective review means there should be knowledge gained that can inform decision making around the future of the VLE at the institution. A working group with a remit for the VLE review can take responsibility for running the review as well as assessing the outcomes and taking actions.
When to use this activity
Often there are different groups in an institution that aim to manage and develop different aspects of running a university or college. This can lead to conflict or confusion of where a VLE sits in terms of responsibility. There will be critical processes around security and resilience as well as equally important operations involving teaching, learning and assessment. Creating a group whose sole remit is looking at the VLE review, and VLE use beyond the review, can ensure that the review is observed correctly and is not run as a process to just tick a box.
Who should attend?
Deciding on how the VLE will be developed at the institution will affect all users of the platform. It is important to ensure each user group is represented in the review. IT, managers, teachers and professional services should be a part of developing how the VLE is related to and engaged with. Any actions, new steps and procedures coming from the group will need senior leadership buy-in, so it is important to have their representation.
This group should decide what success looks like, how to engage users and manage the running of the review.
Vision building - working together
When developing something, or creating from new, an idea of what success looks like is important to have a goal to aim for. The problem occurs when so many different people have a varied idea of what that success looks like. It is very hard to judge a review’s outputs if an organisation is unsure of what good, bad, useful or unsuitable looks like.
What does an effective VLE look like?
There will be many types of users in your organisation, including students, teaching staff and professional services. Each one of these user types could have a different idea of what an effective VLE could look like.
Be conscious not to assume that effectiveness can be determined by one person or one group without taking in the opinions of as many other users as possible. The scope and purpose of a review should hear the relevant voices that can help define what successful and effective means for your VLE at your institution.
It is important to identify what the institution needs from a VLE. Every organisation will have its own requirements, teaching styles, infrastructure and culture around digital platforms and VLEs. What works for a nearby college or university may not be a good fit for your own.
A good fit is essential to ensure the platform is used in the most effective and efficient way. Many of our member institutions we have spoken to have previously committed to spending large sums of money for platforms or services with functionality they hardly use or don’t use at all.
Activity: drawing a vision
Team required: two facilitators minimum - one person to drive conversation and one person to capture discussion
Preparation level: medium
Maximum number of attendees: 12
Time to complete: dependent on the number of participants, each session should be long enough for two drawing exercises and time for each picture to be explained with some discussion. Suggested time is 90 minutes.
The exercise
Eliciting colleagues’ impression of a successful VLE can be a difficult task. People may talk in jargon and terminology which can be hard to synthesise accurately. Some people may be unsure of which terms to use or not feel confident enough to speak up.
This activity is designed to break down those language barriers and encourage discussion and sharing of ideas.
Inviting people from different areas of the organisation can bring a mix of opinions that will feed into the discussions around VLE use now and in the future.
Running more than one session allows groups to be broken up or mixed with people that may not normally work together. Key leaders and regularly heard voices will be useful, but opinions from those who may not be as vocal are essential too.
The ‘drawing a vision’ activity is useful when you want to:
- Extract ideas and opinions from users without jargon
- Bring people together who may not work closely regularly
- Give everyone in the session a chance to share an opinion
Drawing a vision: step by step
1. Arrange space(s) and send out invites
- Ensure the rooms allow for group discussions: cafe style tables rather than lecture style.
- Invite participants from across the institution. This is an excellent opportunity to include leadership involvement in the visioning process of the review
- If running multiple sessions, try to separate people from the same working groups or departments
2. Materials for the session
- Marker pens: various colours to encourage creative engagement
- Paper: flip chart paper is the perfect size to allow drawings to grow. This is also a great size to be held up and shared with the room
- Sticky-tac: for mounting pictures on the wall during the session
- Device to capture pictures: a digital camera or a device such as a smart phone with a camera should be used to capture the pictures
3. The session: part one
Ask the participants to draw their impression of the current VLE experience at the institution. Emphasise that artistic talent is not required to complete this task. Abstract metaphors are encouraged as much as straightforward depictions. The essential ingredient is the meaning behind the image.
Generally, 10 mins is enough time to allow the participants to complete their drawings. Encourage discussion on tables whilst drawing.
Once drawing time has finished, go around the tables asking each member to explain their picture to the group. After each picture, ask the room for any comments on the points raised.
If space allows, ask participants to stick their pictures to a wall on one side of the room.
4. The session: part two
As in part one, you are asking the participants to draw again. This time it is a drawing of how they’d like the VLE experience to look in 5 years time. Allow 10 mins again. Remind participants that it may be useful to look across the pictures on the wall drawn in part one.
As you listen to people explain their pictures, look to pull out why participants have decided to draw certain elements:
- What do they see as the main benefits or opportunities for the VLE looking forward?
- Do others agree?
- What strikes a chord with the room?
When all pictures have been shown and discussed. ask participants to stick their 2nd pictures on another wall and look at the difference between the two walls. As a closing question:
- What do participants see as the biggest differences between the wall’s images?
- Do they agree what could be achievable and effective changes?
5. Analysis
Taking photos of the walls in each session, as well as individual drawn pictures, will be useful to capture the thoughts of the VLE users that have participated in the sessions.
- Compare the pictures with the discussion notes and list all common themes and trends that have appeared
- Start to categorise them into groups, for example, there could be themes around user experience, digital confidence or the role of the VLE
- Try not to limit the categories to predetermined issues. The purpose of the activity is to discover what users’ real life VLE experiences are.
At the end of your analysis, you should have an insight of what some people at the institution feel are important issues when thinking about a VLE vision for the future. This could form a robust foundation on which to build upon for a VLE review.
Discovery
Immerse yourself. Learn about your platform(s). Assume nothing. Listen to those who use it.
The focus of this section is taking stock of the current state of the VLE, its users, and finding out what does and doesn't work well.
How does a review ‘take stock’ of a VLE? Just like any digital platform, the effectiveness and success (or lack thereof) isn’t only about the platform itself. A huge part is the people using it. Whether that be their use of the tools and functionality, or the content they upload, the users are the key ingredient to how the platform supports teaching, learning and assessment.
This phase of the review is fact finding. It’s not about what people think is happening, or those tacit assumptions that develop within the culture. Those insights are important, but not in this section. Find the evidence and the data to support it where available.
Understanding the present
It is very difficult to make the most effective decisions about your VLE without looking at the current picture. Understanding the present status and culture around your VLE is an essential part of an effective review.
Often, the people completing the review may not be the people using it. Be aware that how a platform may ‘look’ to an observer/reviewer could be very different to how it performs to someone using it to teach or to learn.
It is worth noting that an effective VLE review cannot be achieved by only completing a desktop review. User voice and their experiences play a vital role in the review.
Essential elements
Every VLE will need the essentials to function as a web-based platform. Users need to navigate it, they need to find information and access their resources, often across multiple devices and screen sizes.
Taking a wider perspective of the platform, how it looks, responds to screen sizes and the experience within, provides the review with a starting point from which the organisation can identify where change may be needed. It also provides an insight into the platform which will become useful when discussing the VLE with the different user groups.
Activity: desktop review
Team required: minimum one person to view the platform, more people to benchmark may be useful
Preparation level: medium
Time to complete: depending on the number of participants, each session should be long enough for two drawing exercises and time for each picture to be explained with some discussion. Suggested time, 90 minutes
How to use this activity
Setting aside time to look specifically at the VLE platform allows the review to capture and record its current state.
While course pages are essential to understanding the experience of teaching and learning, this activity should observe the whole experience.
- What support is visible?
- How does it work on multiple device types?
- Is it accessible for people with additional needs?
Screen grabs
Observing the platform is easily captured using screen grabs from both desktop computers and mobile devices. These are particularly helpful when reporting findings from the review.
However, you should document your findings and ensure you can describe what the screenshot is showing and the impact on users. Screenshots, like images, will need alt text.
Data
Analyse data where available. Most VLE platforms can provide analytical usage data. There may be useful external services that can be applied to your VLE to help your review understand behaviours such as devices used:
- What are busiest times of the day/month/term/semester?
- Which pages are the most heavily used? Which are not used at all?
- What can you learn from the data and the impacts/stories behind the usage?
- Are there characteristics in the pages that impact on the usage behaviours?
Usage data can be the proof, but the review needs to understand what the data means.
Areas to observe
Not every organisation will have the same priorities to review. Be sure your desktop review is going to be thorough enough to capture the priority elements for users at your organisation.
Be sure to view the VLE on different screen sizes and devices. Different access to connectivity will also impact the VLE experience, for example wifi, 3G, 4G and 5G.
Example essential elements could be:
Accessibility
The VLE must be accessible. A desktop review doesn’t need to be a deep dive into how accessible the whole VLE is at this stage, but it may flag that more work and more investigation is needed.
There are two aspects to consider for VLE accessibility:
- The platform itself
- The content uploaded to it
Many VLE platforms will have the functionality to be compliant with accessibility requirements. It is often the management of the platform and/or the content uploaded within it that causes barriers to achieving accessibility compliance.
Process:
- Check navigation links and images for alt text and titles by rolling over them with your cursor
- Use the platform without a mouse by using only the keyboard
- When looking at course pages, how accessible are documents and files that require downloading?
- Find the accessibility statement. This should be visible on the homepage or on every page in somewhere like the footer
- What tools are in place to aid and educate staff on how to create and manage accessible pages and content?
Mobile access/usability
As a continuation of the accessibility checks, it is vital to assess how the VLE performs on both mobile and desktop devices. This ensures it does not become a barrier for students and learners who may lack access to desktop machines or stable WIFI connectivity.
Many pages within a VLE will have images, videos, quizzes and other interactive content. While the VLE platform should respond to different screen sizes, the content within may be unable to.
Checking pages on multiple devices reveals how students and learners could be viewing their teaching and learning content. Many staff might not consider how their content looks on different screen sizes.
Navigation
A prime part of any web-based platform is how users can move around the pages and information within. A VLE is no different. An effective VLE will require easy and quick access to teaching and learning content as well as other areas within the institution.
Is the navigation easy to understand? Not all VLEs that we have reviewed have had logical and coherent navigation. This may be a barrier to an efficient user experience impact on the platform’s effectiveness.
As part of the desktop review, look at how easy the VLE is to navigate from the homepage as well as from within deeper pages.
- Can users quickly move from a course page to somewhere like the library?
- If not, why not?
- Do they need to use the back button?
- Does the VLE use drop-down menus?
- How do these perform in accessibility checks, particularly for non-mouse users?
Homepage
This will often be the most visited page on the VLE. It should be a welcoming and dynamic space that encourages users, particularly learners and students, to want to revisit. Colourful images, videos and links are all welcome in this space.
It sounds obvious, but the institution logo should be clearly visible. Unfortunately, many VLEs that Jisc reviews do not have the institution logo visible at all.
- What news and information is available?
- How often is the page updated? Even the best images and videos can become stale or outdated in a short space of time.
- How does the page represent the role of the VLE?
- Why do users come to the platform? (For example, easy and quick access to teaching and learning content, keeping up to date with news and events, looking for support)
- What links to other areas are visible? Links could include:
- Library
- Support and wellbeing
- News and events
- Employability/careers
- Academic skills
- Screen the response to mobile and tablet devices. Check:
- Do images and navigation adjust accordingly to the different screens?
- How the information is ordered on a smaller screen. Do users need to scroll for a long time for important information?
- Do important elements disappear on mobile screens? For example, the institution logo
Course pages
In our VLE reviews, the biggest frustration raised by learners and students is a lack of consistency and organisation on course pages. Look at your organisation’s VLE course pages and identify where consistency is an issue.
At this stage of the review, make notes about why you think these inconsistencies are occurring. Could it be differences in confidence, knowledge, skills or a lack of guidance?
Another common occurrence on VLE course pages is the appearance of a file repository rather than an engaging learning page. While a repository is not a definition of a poor VLE course page, it may be harder to create engagement and motivation with learners and students without a more appealing experience.
Content
This could be text, images, videos and links on the page.
Explore:
- How inviting does the page look?
- Is there a rich experience on the page?
- This does not include a list of documents for students to download individually
- Are course details and staff introductions included?
- These are essential
Resources
This could be files, documents, slides, downloads, images, and videos to support learning.
Explore:
- Organisation and consistency: can learners or students find what they need quickly?
- Do they work and is it a good experience on mobile devices or devices with touch screens?
- File size: not all learners or students have access to wifi outside of campus. Large files may be ignored if they are too big to download on mobile or pay-as-you-go data connections
- All resources uploaded to the VLE and the VLE itself must meet accessibility standards
- Are all resources up to date?
- Check their content, links and images
- Look for old institution titles or branding
Interactivity
This could be opportunities for staff and learners or students to engage in interactive elements to support or enhance, such as teaching and learning, for example quizzes, forums, wikis games or links.
Explore:
- Recent activity: forums or wikis that have been inactive for months or even weeks may point to a group or teacher who are disengaged
- What sort of conversations are on the page?
- Could they include questions from teaching staff about taught content to engage more learners or students in their learning?
- Ensure only activity from the current cohort is visible
- Are quizzes and games still relevant and engaging?
- Are links still working? Broken links are common on pages that aren’t kept up to date or rolled over each year without updates
- Are there opportunities for other areas of the institution to engage or interact with learners or students relevant to their taught sessions? For example:
- Library
- Academic skills
- Employability
- Support
Accessibility
Review the web accessibility regulations for HE and FE institutions including VLE platforms.
By law, all HE and FE institutions must comply with the 'accessibility requirement' which means the requirement to make a website or mobile application accessible in line with the standards laid out in the Web Content Accessibility Guidelines 2.2 AA.
All content within the VLE page must meet accessibility standards, for example:
- Images should have alt text and links should have titles. This provides learners or students using a screen reader access to additional information about those links or images
- Look for images of text that has been photographed or scanned
- Screen readers are unable to reliably extract the text. These should be typed out to enable all learners or students to use
- Text colour should be high contrast against a plain background
- Avoid text over images or patterns as these are harder to read
- Check if users can alter colours and font sizes. Many browsers can increase or decrease zoom but are there tools within the VLE that allow further adjustments?
- Video use is a common occurrence on VLE pages. Check that transcriptions are available and subtitles are added
Mobile compatibility
Looking at the learners’ or students’ experience of using the VLE course page on a mobile device.
Consider what barriers there could be to using a mobile device to access the course page.
- Course page navigation:
- How do users move about the course page?
- Are there quick links available or do users need to scroll excessively?
- Can users access the whole course page and resources easily on a touchscreen?
- Does the course page fit the width of the device screen?
- Look for text/images/videos going beyond the screen horizontally
- Do resources and course materials need to be downloaded to the device rather than opening in a browser window?
- Slides:
- Many fonts and slide layouts do not render correctly when viewed on a mobile device. Exporting these slides to PDF ensures layouts are maintained
- How quickly does content load on a mobile device? Particularly if on a data connection rather than wifi
- Ultra HD videos and high-resolution images use a huge amount of data to load
Signs of activity
Check how fresh and active the course page looks. Consider how a fresh and dynamic course page will encourage learners or students to revisit and the pages and be encouraged to interact.
- What characteristics are there on the page to encourage learners or students to revisit the page?
- Does the page sit unaltered for long periods of time?
- What updates of news or events are visible on the course page?
- Are learners or students encouraged to engage with the resource through a forum or wiki posts?
Out of date content and broken links
Lots of content on the VLE, both in the page or in files uploaded to the platform, will contain links to further reading or viewing. Is it common to find content that has been moved from year to year at the start of the academic year?
Look for links that no longer work or are ‘broken’, where the destination has been removed or moved to a different link.
Look for uploads dated longer than 3 years go or with old logos or institution title so they can be removed.
Housekeeping
Pages should be kept relevant and up to date. Pages must also be cleaned of old and out of date posts or comments.
Look for irrelevant comments or posts. Ensure all posts are from the current cohort using the page.
- What news and events are visible that have passed?
- Are there staff, learners or students still enrolled on the page from previous cohorts?
- Are course details and events still relevant?
Consistency
- Is there a lack of consistency and organisation from the different teaching staff using the platform, sometimes in the same module?
- Do pages adhere to a consistent approach to the structure of content?
- Are there templates in use? Do the pages use them?
Activity: focus sessions
Team required: minimum two facilitators to manage the conversation and capture the discussion through note taking, not recording the sessions.
Preparation level: medium - minimum three sessions. Learners/students, teaching staff and professional services staff. Book a room big enough for a maximum of 20 attendees for each session.
Time to complete: each focus session should last a minimum of 60 minutes.
An essential part of any VLE review is to listen to the people that use the platform. A VLE will be used by people wanting to teach, by people wanting to learn and by people who are supporting teaching and learning.
It is important to give users a voice in reviewing the platform. What may look adequate or seamless to someone on the outside, may be a very different experience for someone using it every day.
A focus session with senior leaders is also a great opportunity to gauge their vision for success and compare to those from the other focus sessions.
When inviting attendees to the focus sessions, be clear about what the intention of the review is. Often users, particularly staff, interpret the review as an intention to change platforms and attend with an agenda to fight their cause rather than have an open discussion. Valuable time can be lost in the session to discussion around a change of platform that doesn't help the review.
Emphasise the discovery work the institution is doing around VLE use.
When to use this activity
This activity should be part of every VLE review. It is more effective after the desktop observations have been made. This will be helpful to put comments from attendees into context or respond during conversation with some knowledge of their experience.
It could be useful to reference challenges and barriers identified in the desktop review. How do the different users see those issues? This could be useful to the review in understanding where different perceptions about the platform exist.
Who should attend?
The review needs to listen to the voices of those that use the platform or have a role that affects the effectiveness and safety of those using it.
Specific sessions for different user groups enable the conversations to be more ‘focused’ on their experiences. It is also helpful for people to feel they can speak freely. As an example, teaching staff may not feel they can criticise or comment negatively about their experience with content on the VLE that was provided by learning technologists (LTs) who are sat in the same focus session.
Session 1: teaching staff
Session 2: professional services staff (e.g. library, learning technologists, student support, faculty admin etc)
Session 3: learners or students
Session 4 (if possible/required): senior leadership/executive team
Ensure every user group has an equal voice in the review. Ensure notes are taken. Be specific about how the notes will be used and how they impact the review.
It is useful to emphasise that while notes are being taken, they remain anonymous. The review should not state who said what. The most effective and authentic reviews are those that hear the honest and accurate experiences of users.
Questions/conversations topics
On the VLE reviews run by Jisc, the questions used in the focus sessions are intended to prompt conversation from attendees. It is important to allow as much discussion as possible. Where there is disagreement, it is important to ensure all opinions are captured.
While each institution can design their own focus sessions, the questions used in Jisc VLE reviews have been developed to bring personal experiences and opinions to the forefront. This will help to ensure that there is no pre agenda attached to the sessions/review.
Example questions used:
Questions for academic and teaching staff:
- What do you see is the role of the VLE in your teaching?
- Do you feel supported to explore different teaching approaches using the VLE?
- How do you use the VLE for assessment?
- Do you use the VLE for any collaborative activities?
- What is the greatest advantage of using the VLE?
- What are the challenges and barriers to using VLE effectively?
Questions for professional services staff:
- From your perspective what do you see is the role of the VLE?
- How do you use the VLE in your role?
- How do you feel the VLE perceived at your organisation?
- What are challenges and barriers to the organisation using a VLE effectively?
- What are the best uses you see of the VLE at your organisation?
Questions for learners or students:
- What do you see is the role of the VLE in your learning?
- How do you access the content?
- What pathway? (URL, bookmarked, google etc)
- What devices do you use? (When, for what task, why)
- What's missing from the VLE?
- What are the greatest benefits with using the VLE?
- What are the biggest challenges with using the VLE?
Defining
Understand all you have learned and define the details.
VLE reviews are more than just looking at the platform and asking a few questions. The review needs to dissect and understand the observations made.
After observing the platform and talking to users, there will be a range of opinions, experiences and emotions about the VLE collected. Whether they be positive or negative, understanding the overall picture is at the heart of the review. Remembering the initial activities in this resource in defining your institution’s vision of success is crucial. These insights can enable the review to identify good practices, challenges and barriers from a more informed position, that affects how you can reach that success.
Look for any re-occurring themes that can be grouped together. There may be obvious groups such as accessibility or support. Many users may point to a lack of time or wanting more leadership on how to use the platform more effectively.
Can you see patterns in the insights? Some could point to a cause and effect’ scenario. Comments from students or staff about a lack of organisation or consistency may be supported by comments about a lack of guidance or support in using the platform. Identify where connections can be made to enable a better understanding of the user experience.
Insights, data and storytelling
It is useful when looking at the insights from the review to analyse how certain experiences impact on platform effectiveness. Why does it matter if some teaching staff have suggested they don’t feel supported to explore new ways of supporting teaching, learning and assessment through the VLE? Are there reasons why learners or students don’t like accessing the VLE on their mobile devices?
Data is often an important stream of information and insight. Looking at it alongside observations and focus group comments, the review can start to combine data and human elements into one picture.
Creating a narrative through storytelling allows the review to express the impacts, challenges and barriers much easier. This may be vital when communicating the review findings.
User personas
To help understand as much as you can from the review insights, it can be useful to develop personas for different user types and look at scenarios where they use the platform. The different settings, challenges and successes can help the review understand how an effective VLE can be an enabler or distractor to teaching, learning, assessment and support. It is also worth looking at how the platform is managed from a systems or senior leader’s perspective.
- How might a student use the platform the most?
- Could it be to catch-up after missing a session?
- Could it be for revising before an exam or assessment?
- Could a teacher use the VLE to access usage data on recent session slides?
- Why has a particular session/slides had so many visits by students? Could this mean the learners or students are struggling with the session or subject?
By creating personas, the review can clarify how specific situations influence user experiences.
Define the need – identify challenges and barriers
From all the observations and insights, the review should define what the institution needs from a VLE.
- What is the current picture of teaching, learning and assessment through the VLE platform?
- What does the organisation need from a VLE platform?
- Where are the gaps right now stopping the institution from achieving success?
An important part of a VLE review is to understand what the next steps are to achieve success.
Without this, the institution could be making less informed decisions on what actions need to be taken and what the main priorities are. By identifying and understanding the biggest challenges and barriers, the review can recognise their impact on users.
Be specific about what it means for the institution, the platform and the users. Use quotes from the focus sessions or observations from the desktop review to evidence the issues.
Celebrate the positives
Not all VLE use at your institution will be riddled with challenges, barriers or bad practices. The review should go beyond identifying system and process flaws. Be sure to recognise and celebrate practices that evidence good practices or are in line with the institution’s vision for success.
Highlighting positive examples in the review's outcomes motivates users and gives them a sense of achievement.
Ideation - recommendations and next steps
What solutions work for your organisation? Which ideas don't fit the need?
After gathering user feedback, a thorough desktop observation, and defining needs, the review should recommend solutions for immediate and long-term action.
Reviewing your VLE should be about understanding how the platform can fulfil a role at the institution. If it hasn't decided what that role is, then it is very difficult for a review to suggest recommendations, measure impact and find success.
Developing ideas to answer the challenges and barriers is key to running a successful review. These are the actions that allow the institution to move forward in an informed direction and ensure the users’ experiences are a valued voice in teaching, learning and assessment.
When deciding on possible solutions, keep the details that have been defined in the review in mind to ensure the most effective and suitable recommendations can be suggested.
Take each challenge identified in the review and explore further to ensure the institution fully understands what the issue is and what it means for the VLE’s user experience and teaching, learning and assessment.
Making space for innovation
When considering solutions to overcome challenges and barriers, there may be opportunities to think ‘outside the box’ and ‘innovate’ or do some ‘blue sky thinking’. It can be a refreshing exercise to encourage ideas beyond the current thinking or current restrictions.
While some ideas may seem extreme and unrealistic, look at the ‘how and why’ it could offer a solution. Can it be refined into a more realistic option?
Innovation can come from approaching a challenge or barrier from a different direction. It could be a new method, a new idea or even product or service. It is important to gauge this carefully. Innovating doesn’t immediately equal ‘better’. Success will come from the effective adoption of solutions from the review. The most innovative ideas may need time and space to be fully absorbed by the institution.
Remember that many challenges and barriers come from the users rather than the system or platform itself. This may be their confidence, practices or motivations. Always keep the user's perspective in mind to ensure their needs are directing the ideation.
Alpha and beta to grow and enhance
Developing recommendations includes planning next steps.
While creating the space to allow new ideas to grow, it is a very good idea to control that growth. As well as creating the space for ideas to be adopted, those ideas need to be fully realised in the context of VLE use. What looks good on paper might struggle when put into action.
Working with small groups means any issues with the recommendations can be quickly tweaked and ironed out with much less impact. If a whole institution discovers problems, it impacts a lot more people and motivations are quickly reduced.
Having a defined structure for implementing ideas as next steps and beyond is a positive way to allow the institution to fully understand how to implement recommendations from the review. It is much better to learn what doesn’t work quickly, before spending huge amounts of time, money and buy-in on it.
Below is an example structure to implement ideas. For each idea, progress it to the point you think you have a solution. Then test it through the stages below. If it can’t pass a stage, then it maybe isn’t the right solution or needs deeper considerations to make it work.
Stage one: lo-fi prototyping
Prototyping in lo-fi ways enable ideas to be quickly visualised at very low cost.
Using pens/paper, sticky notes or whiteboards, form a user’s step by step journey.
- What is the starting point?
- What is the desired outcome?
- Where do you see dead ends?
- Does it fit the need?
How many iterations can you make? Each time you finish one, capture it by photo or screen grab to refer to it later
Stage two: alpha prototyping
If an idea can be visualised through lo-fi prototyping, test it on a small number of users. Whether that be one class of learners or students, or a few members of staff.
- Look for stumbling blocks and teething problems often seen in new projects
- Be fluid with the ideas to allow the users to shape them to fit their needs better
- Document what works well
- Get feedback from users tested
If an idea has been successfully alpha prototyped and the feedback has been addressed, does it offer real world benefits across the users? If so, move to next stage.
Stage three: beta testing
Beta testing takes the Alpha prototypes and everything you have learned and developed them into more robust ideas that are at a stage where you are testing with a wider group of people.
This stage should test:
- If issues found in Alpha have been successfully addressed
- If a recommendation addresses real user needs
- It can be scaled up and delivered sustainably
If these can be answered successfully, begin the process of turning it into a production service that can be rolled out across the VLE and receive continued support.
Stage four: service launch and growth
Putting an action from the review into service should be something that has been rigorously tested and shaped to fit the need. Once ready, steps should be taken to ensure it can be rolled out smoothly.
Some considerations should be:
- Pre-launch: user awareness of the change coming
- Explain why the change is coming. Examples from review can be helpful. How have users’ opinions shaped the change?
- How could it impact users
- Instructions on how to make the most of the change
- Launch
- Be ready to make adjustments if an issue does present itself
- Monitor impact to report findings
- Ask for feedback from users
- Growth
- Does the VLE perform better with the change added?
- Does the change move the VLE closer to your institution’s vision for success.
- How can you encourage more use, better buy-in and motivation to adopt the change into culture?
Actions: finishing the review and reporting the findings
Reporting the review findings should be about identifying the challenges and communicating the recommendations and next steps to stakeholders.
The review's impact comes from the actions taken after it, not just the review itself. Impartial reporting clarifies the review process and the institution's future direction.
Clearly defined actions to challenges and barriers, and why these actions have been chosen, offer stakeholders more information which can encourage buy-in and motivation around VLE use. Communicating the users’ current struggles and reservations with the VLE provide robust evidence for change to happen.
Whether presenting the VLE review findings to senior leaders or the wider institution, ensure the findings are unbiased, accurate and transparent.
VLE review findings do not always present easy reading. The acknowledgement of challenges and barriers, with recommendations to overcome them, do though offer a route for positive change.
Who makes it happen?
Many organisations have committees or groups that the VLE sits under. Often these groups have a wide view of digital use across the institution. It can be beneficial to create a smaller ‘working group’ with specific remit to manage the review outcomes and report progress. With representation from all user groups, which could include learners or students, teaching staff and professional services. This working group could work through the recommendations, develop ideas for solutions and have ownership of successes and further developments.