Problems with generative AI in education

A chatbot's data bias is a reflection of biases from training data. This data, sourced from the Internet and elsewhere, can contain various biases, sometimes resulting in stereotypical or harmful output. A notorious example of this is Microsoft's chatbot Tay, which was taken offline within 16 hours in 2016 because it began uttering Nazi, racist and sexist language.[i]

Developers strive to prevent a chatbot from using biases and harmful stereotypes from the darkest layers of the Internet to generate responses. Therefore, filters are placed on the responses of new chatbots. This does not ensure that ChatGPT no longer contains biases, only that they are less easy to be revealed.[ii]

There is also a phenomenon like Trainer Bias. To prevent a chatbot from expressing racist, sexist or other harmful stereotypes, two methods are used:

1. Ethical Governor: The chatbot is actively trained by humans, a process called data-labeling. As a result, the chatbot learns to recognize and filter out violent, hateful and other unwanted speech before it reaches the user.
2. Rules in the program: Rules are programmed in that determine what questions a chatbot may or may not respond to. Questions about the chatbot itself are also programmed in advance because chatbots have no self-knowledge; they are loaded with internet data before they become active, and that data contains no information about the chatbot itself.

Programmers continue to make constant adjustments. Questions that a chatbot used to answer may now be rejected. For example, ChatGPT had problems generating reliable sources and often made-up articles. Now, if you ask for an APA-style source list, you may receive a response such as: "Unfortunately, I cannot provide specific sources in APA style, as my information is not directly derived from specific sources, and I do not have access to a database of references."

This trained approach makes ChatGPT more polite, less harmful and more useful, but it also makes it harder to understand how it arrives at an answer. It reveals not only the bias of the data, but also the bias of the trainers, or programmers, the data labelers, and the people who create the instructions for how the data labelers should proceed.

Data labeling is crucial for training AI systems such as ChatGPT, in which humans review images and texts to help the AI recognize different types of content and respond appropriately. OpenAI, the organization behind ChatGPT, had outsourced this task to Kenyan company Sama. Sama's employees were tasked with labeling gruesome content, and the working conditions were far from ideal: they were poorly paid, could easily be fired, and although they worked with grueling material, limited psychological support was available. This led to long-term mental damage in some workers.[i]

The collaboration between Sama and OpenAI ended in a dispute, with both parties blaming each other for the problems that arose. This incident highlights the ethical challenges in training AI, especially since there are currently no alternative methods for teaching AI systems to effectively identify and repel unwanted material without human intervention. This raises important questions about the responsibility and protection of those involved in the data-labeling process.[ii]

Read more in the TIME survey

Most generative AI, such as ChatGPT, is developed by large for-profit tech companies. Their interests often differ from those of universities, teachers or students. Moreover, these companies are so large that it is almost impossible to influence them. This raises major questions around issues of equity, privacy risks and dependence on big tech (Last & Sprakel, 2023).

Initially, chatbots, including ChatGPT, seemed to provide a solution for students who have less support in their immediate environment. However, recent developments, such as the introduction of a paid version of ChatGPT, actually seem to create more inequality between students who can afford it and those who do not have the means to do so. This premium version offers significantly more features than the free version. For example, users can upload data and have it visualized, use a writing help, programming help, math tutor, and even make AI-generated text more human-like to bypass plagiarism detectors. Moreover, the paid version allows users to develop their own chatbots with unique features. So the difference between the free and paid versions is significant.

It remains important to recognize that the motivation of companies behind generative AI is primarily financial, and to consider this when determining how these technologies are used and integrated into education.

"Nothing comes for free," and that is especially true when it comes to using generative AI. Since AI until now has always been offered by profit-oriented companies, it is often the case that although the service seems free, you still pay with your data. Given the size of these companies, it is challenging to hold them to UU's data security standards. Therefore, it is essential to be careful when using generative AI and to be careful with the information you incorporate into it.

Everyone has the freedom to share their own data with generative AI systems because it falls under personal responsibility. You are the owner of your data, but it is recommended to handle it consciously. You are never allowed to share other people's personal data, including course materials and students work with a generative AI without there explicit permission.

Another important aspect of generative AI that definitely needs attention concerns copyright. AI models excel at generating output, but how unique is this output really? It turns out that this is a tricky legal story because the data on which AI models are trained are often copyrighted.

Under certain conditions, some of this data may be used under fair use. This principle allows limited use of copyrighted material. But if the AI output is too similar to the original copyrighted material, there may be legal consequences. Consider, for example, civil liability. So the demand for solutions to this problem is high.

One possible solution could be a "fair-use filter," which checks the extent to which AI output matches the original. An alternative approach could be for the AI to limit itself to copying facts and not literally copy creative expressions. However, the latter appears to be technically tricky.

Currently, there is no single solution to this copyright problem. It is critical for educators to stay abreast of the latest developments in copyright laws and regulations in order to effectively teach students in this rapidly changing environment." (Last & Sprakel, 2023 ). The website https://aivoorstudenten.nl/ is a useful website for this purpose, also for teachers.

Old data

The current free version of ChatGPT (May, 2024) is trained with dates through September 2022. This highlights an interesting limitation of language models such as ChatGPT: it is essential to check when the model was last updated, as this determines the extent to which it can deal with recent topics. This limitation is not unique to ChatGPT; many language models face similar challenges. However, the paid versions of such models, including ChatGPT, often offer additional advanced features, such as real-time access to the Internet and the ability to upload files or create and train your own models on personal data.