How an AI ‘study buddy’ is providing support for the University of London’s students

As generative AI tools continue to develop at a rapid pace, it’s not always easy to know which of the many options will offer what your organisation needs. In the University of London’s case, the aim was to find something that could offer a helping hand to its 45,000 students spread across more than 160 countries.

Tim Hall, who has been with the university for almost 17 years, took on a new role in 2022 with the new product innovation team.

The team started to look at what new offerings the university could provide, and how to take full advantage of technology and innovation to support distance learning students.

Tim could not overstate how much of a team effort the pilot has been. He said:

"We've been hugely fortunate to work with colleagues from across the organisation and beyond who've been so generous with their time. We've learned a tremendous amount that can inform our plans around the use of generative AI going forward."

Study buddies

Feedback from the university’s student experience surveys showed that they wanted more opportunity to be assessed outside of summative assessments and examination periods, so the team set about addressing that challenge.

Tim said:

"We were mindful of Blooms 2 sigma problem, which essentially shows that students tend to perform much better with one to one tutor support than in a traditional classroom setting. Of course, AI tutors aren't necessarily going to be able to do what human tutors do, but the idea was to find out if there was some way we could bridge that gap, find some way of supporting students in a way we hadn't before."

One of the early adopters in the sphere of AI tutors was Noodle Factory, so the team got in touch to talk to them about a potential pilot programme. They looked at the AI teaching assistant tool as less of a tutor, more of a ‘study buddy’, an additional resource to support students.

To begin with, the team reached out to academics to find out their interest in getting involved with a pilot and then started to think about which courses would be the best choice.

“We knew we could choose something like computer science because we know those involved would be keen, but at the same time we wanted to get some really constructive feedback from people who might not have necessarily been the first on our list. So we spoke to colleagues in undergraduate and postgraduate law and they were really receptive, even if they understandably had some reservations about what it could and couldn't do, and about the value it could bring to students.”

ChatGPT was introduced to the AI scene when the university was signing contracts and getting started on testing, so discussions around AI across the education sector picked up speed.

Tim and the team knew that Noodle Factory always had a road map for its products so were confident in their partnership.

However, the pace at which ChatGPT was moving meant Noodle Factory had to move quickly to adapt to emerging technologies, update existing tools, and look into any necessary additional functionality.

Using the tool

One key objective of the pilot was to evaluate the quality of the tool itself by measuring student engagement to identify potential benefits and to consider how it might impact learning in the longer term.

"Our students are really keen to understand the impact of this technology and how it will impact their working life when they graduate. I think it's important for us as a university to really ensure that we know what we're talking about."

The team knew they also needed to be confident that the tool was doing what they wanted an AI study buddy to do.

They also needed to know how much human intervention was required because if it was going to create extra work for already busy tutors then it wouldn’t be feasible.

One of the major benefits the team found was how intuitive the tool was, lacking any need for coding and instead allowing resources to simply be dragged and dropped into it.

With the material then used to generate a knowledge base, the students could essentially chat with the tool and ask it questions.

Another important feature was the security of the material once it’s added to the tool.

Tim said:

"It's very much a walled garden. The way in which the tool works means these materials are not going into a large language model (LLM) outside of the university, so we can put our own copyright material or intellectual property on there and not have to worry about it being used to train ChatGPT or other AI models."

Feedback is key

During the pilot and afterwards, the university carried out a series of surveys to learn about the students experiences.

Staff who helped support the pilot were also interviewed and a report was collated by Noodle Factory. Tim and the team are also currently awaiting a review of student assessments to measure the impact on that.

“I think one of the big things as well for us was growing this kind of institutional awareness because it was really useful to be able to demonstrate the way in which we were proactively engaging with the technology. It also allowed us to raise discussions about the challenges, ethics, risks and opportunities that come with AI.”

As of May 2024, the tool had seen 2400 chat sessions,1434 unique users, and 97% of questions were answered. It was also clear that as examination periods got closer, the tool was used more, highlighting that students were clearly keen to access more support.

Reflecting on the pilot process, Tim shared how helpful the guidance from Jisc’s AI team had been and how the opportunity to engage with AI communities had proved invaluable.

Tim said:

"It's been brilliant having opportunities to talk with colleagues across the sector and we’ve definitely benefitted from having that kind of avenue of discussion. I think we need to look at what else we can do. Is there some sort of sector-wide approach that we can take, can we collaborate and have some something more community-based as far as AI tools are concerned?"

Working with Jisc also allowed the university to share insights from the project which contributed to the ongoing sector discussions around AI and informed the guidance on copyright and concerns arising around generative AI.

The pilot won the 2023 Gartner Eye on Innovation Award for Education (Enhancing teaching and Learning Quality) so the university is keen to build on this success.

The lessons learned will be taken forward and the product innovation team will make recommendations on data privacy, improving the prompts, and digital accessibility concerns.

The team will work on improving AI literacy in both staff and students, and there will be a continued focus on the university’s AI principles.

Ideally, in the longer term, Tim would like to be able to expand the use of the tool so that students can access support from the day of registration right through to the end of their studies.

Tim said:

"If there's a choice between a human and an AI option, students will always go for the human but they appreciate that there is a role for AI in supporting them. Ultimately, it’s about finding a balanced approach to AI tutoring."