Can an artificial intelligence obtain a Master's degree?

Would the committee that assesses applications for the faculty part of the Utrecht Education Incentive Fund (Stimuleringsfonds Onderwijs) have noticed anything unusual? It was given an application to assess from Gerko Vink, Anne Scheel and Oisín Ryan. It looked fine, but the lion’s share of it had not been written by the applicants themselves but by ChatGPT and Perplexity.AI. ChatGPT and Perplexity.AI wrote it.

The application was successful. Its title: Can an AI obtain a master’s degree? An investigation into AI-assisted education in the MSBBSS Research Master’s Program. “You’ll find it fairly easy to work your way through a Master’s programme with AI by your side”, says associate professor Gerko Vink. “Research shows that AI can easily obtain an MBA or pass the bar exam. We act as if it’s a big problem and some people even want to ban the use of artificial intelligence in education. Personally, I think we should be embracing AI-assisted learning. Students and lecturers need to learn how to use this technology to their advantage.”

Form an opinion

Gerko knows that if you give students the freedom to use the internet in a test, they will use AI as well. “So, as lecturers, we should be asking ourselves what we can do to make sure our tests continue to be valid and whether we want to know if we’re looking at a student’s own work or something that a student has taken from ChatGPT. To be able to form an opinion about the use of AI in education, it’s important to know what its implications are.”

AI in the exam room

To identify these implications, Gerko and two fellow lecturers want to put AI through its paces in tests in their own two-year Research Master's programme: Methodology and Statistics for the Behavioural, Biomedical and Social Sciences (MSBBS). Gerko: “We are going to present all the programme’s existing tests to two AI models and then study the answers and programming code this yields. We will then check the AI models’ answers and solutions against the existing rubrics.”

We need to teach students how to use AI properly. We already teach students how to recognise the value of information in literature sources; we now need to do the same for AI models

Information skills

Gerko likes the tension that the question of AI in education creates. He compares the consternation it causes to ‘googling’. “Whenever there’s anything we don’t know, we head to the internet for the answer and read the sources that search engines rank for us. And no-one gives it a second thought. But if you then remove the reading part of the process and replace it with a language model like ChatGPT – by entering a prompt, which generates a written text – everything changes and we all start to see it as a threat. And it is, but using a search engine requires all kinds of information skills too, like the ability to assess the quality of the website you have chosen to take information from. So, we need to teach students how to use AI properly. We already teach students how to recognise the value of information in literature sources; we now need to do the same for AI models.”

Programming

Many skills that Methodology and Statistics lecturers test involve some kind of programming. The following assignment, for example: program [a certain complex model] in R. “AI can do this for students,” says Gerko “but they will need to interact correctly with the robot to get it to do it for them. And then they’ll need to check whether the code actually works. So, you do need to understand what’s happening.”

Information less up-to-date

What's more, AI is just one of the techniques that students can use to formulate answers to questions and it cannot help students with everything. “These language models contain everything that was known in 2021, including everything on GitHub. So, anything written just three months ago won’t be on them. That’s why I always expect my students to use something recent.”

Other test formats

Lecturers who really want to stop students getting a robot to take their tests for them can achieve this by opting for a particular test format. Gerko: “Students can’t use ChatGPT when they’re doing an oral or written test because they’d need to check their phones before being able to give an answer. In the pandemic, I put together knowledge tests that students could easily work through in two hours, but not if they had to look elsewhere for the answers.”

The results of the FSO project will be published in a paper by September 2024 at the latest.