#KirstenAmmon: Generative #AI, #Bias, Hallucinations and #GDPR

When using generative Artificial Intelligence (AI), the issues of bias and hallucinations in particular gain practical importance. These problems can arise both when using external AI tools (such as ChatGPT) and when developing own AI models. This blog post intends to illustrate which data protection issues exist in relation to AI under the General Data Protection Regulation (GDPR) and what options are available to solve them.

1. What is bias in AI?
AI bias(es) are distorting effects in the development and use of AI. Particularly in machine learning (ML), a subcategory of AI, there often manifests a tendency of human bias and prejudice. AI bias occurs when an algorithm produces systematically biased results because of flawed assumptions in the ML process. Algorithms can thus, reveal and reinforce existing biases or even create new biases by placing trust in the distorted datasets.

For example, an algorithm for facial recognition could be trained in a way that it more easily recognizes men than women because this type of data was more commonly used in training (e.g., in the automotive industry, crash tests were long conducted using only dummies modeled on the male body and thus did not adequately take into account the special characteristics of women). Another example involves job applications, where algorithms might reject photos with dark skin color and/or foreign names, even though the professional suitability could be objectively assessed to be equal or better on the basis of the available data. This means, existing human bias can be mirrored or even reinforced by algorithms.

Bias can thus have a negative impact on people from minority groups. Discrimination caused by AI bias then in turn hinders equal opportunity and the biased condition is perpetuated.

In the case of so-called “Unconscious Bias”, unintentional distortion effects, it is difficult to recognize those effects.

2. Black-box problem
Bias in unstructured AI is reinforced by the so-called black-box problem.
This occurs in certain forms of AI when the behavior or decision of an AI system are not transparent or comprehensible to humans. In particular, this often makes it difficult to understand how a specific outcome was generated by the AI. The black box issue can arise in several areas of AI, including ML, neural networks, Large Language Models (LLM)/Natural Language Processing (NLP), and other complex AI models. The black-box problem becomes particularly relevant when AI systems are used in safety-critical sectors, such as autonomous vehicles, medical diagnostic systems, or financial decision-making.

Although significant progress has already been made in solving the black-box problem, it remains a challenge and can ultimately only be solved through cooperation between research, industry and regulatory authorities. The Data Protection Officer should always be involved at an early stage to evaluate existing requirements and possible solutions in the individual case.

3. Hallucinations in generative AI
So-called hallucinations in generative AI occur when the AI alleges or invents
false facts (e.g., fake news), in particular, by presenting sources, contexts, or events that do not correspond to the truth or are contradictory. In this case, the processing of personal data is protected in particular by the GDPR and the right to rectification of data subjects applies (in detail on the fulfillment of data subject rights, our further blog post will be covered in our upcoming blog post).).

4. Privacy implications
Provided that personal data is being processed, opaque AI systems, AI bias, and AI hallucinations pose a threat to the protection of personal data. This is especially true when AI systems are not transparent. As training data for AI often requires large amount of personal data (Big Data), it is often difficult to ensure comprehensibility for data subjects (for details on transparency requirements, see our blog post from June 6th, 2023). If no personal data is being processed, the GDPR does not apply. Therefore, it’s important to first assess the extent to which personal data is necessary for the specific AI application. If this is the case, for example, a legal basis is required and further requirements under the GDPR apply (among others, often, a data protection impact assessment (DPIA) will be necessary; another blog post on this will be published shortly).

Read more