top of page

13th of June 2024

Dangers of GenAI

Generative AI chatbots tend to look very impressive. At first glance, a generative chatbot might look more impressive than non-generative chatbot when it comes to understanding and answering customer queries. Generative AI uses input training data to generate new data, often in the form of text, images or videos. These models learn from identifying patterns and structure, allowing it to generate content based on the input. A number of chatbots make use of generative AI technology, such as ChatGPT (OpenAI), Gemini (Google) and Copilot (Microsoft). In this article we will outline the main risks posed by chatbots using generative technology, namely data security, dependency, accuracy, hallucination and jailbreaking.

Data security

Many companies are using generative technology integrated into a chatbot on their website. This is usually done through an API to one of the bigger providors (OpenAI, Google, Microsoft, etc.), which poses a significant data security risk. If the customer has put personal information, such as their name, email address or post code, into the chat, this information will also be sent to the 3rd party application. Although these 3rd party providors have clauses not to use customer data for training, this is often challenged (Rahman-Jones, 2024). By contrast, non-generative chatbots avoid this risk.


Depending on a 3rd party provider for a generative chatbot brings reliability risks. For one, if the 3rd party application has downtime, we will not have a functioning chatbot for the duration of the downtime. In this scenario, we won’t even be able to tell customers when the issue will be resolved, as we would be reliant on the company servicing our generative application. Additionally, using these third party applications means that we are at the mercy of how these applications are maintained. For example, if OpenAI decides ChatGPT will no longer provide any content related to payments, any chatbot serving in the finance sector will no longer function. This applies to performance too - In a research conducted by Chen et al. (2023), it was showcased that ChatGPT generated more mistakes in June than in March.


Generative chatbots also carry risks to do with the content they provide. For one, we have little to no control over the way messages are formulated in a generative chatbot. This is because these solutions generate content directly prior the output. This output varies greatly based on the customer input, for example, the accuracy of LLMs increase with politeness (Yin et al., 2024). This means that customers that are less polite are less likely to get accurate answers.

The illustrate this, consider the chat below from a chatbot on a retailer website using ChatGPT. Though the question is identical, the answers are not only worded differently, but also provide different information. One customer might get the additional information provided in the first answer, whereas another customer asking exactly the same question may not get this information.

The more instructions we give a generative AI bot, the less reliable it will be in providing the same information consistently. Additionally, generative chatbots don’t do so well with negative instructions. This causes problems when it comes to customers asking out of scope content.


Hallucination occurs when a generative chatbot produces information that is not factual – in other words, when it makes up information to answer a question. This is not the same as a chatbot simply providing the wrong answer; when a non-generative chatbot misunderstands the customer and provides the wrong answer, it is still content that was provided and approved by the client. On the other hand, when a generative chatbot hallucinates, it provides information that is not verified or approved by the client using the chatbot. This is because generative AI always has an answer to a question asked by a customer; the only thing that stops it from providing an answer is the prompt telling it not to, and so the generative chatbot.

Though some examples of AI hallucinations might be funny, a chatbot in customer service making up false information can have serious consequences for both the customer and the company operating the chatbot. A recent example is the Air Canada chatbot that hallucinated a policy for bereavement fares (Garcia, 2024). The affected customer successfully sued Air Canada for providing the misleading information.

It’s very easy to get generative AI to provide false information. To illustrate this, in the exchange below, ChatGPT has been told about a fictional return policy, which states that customers have 14 days to return an item and that it must be unused. However, after only a few questions about topics not covered by that information, ChatGPT has made up additional restrictions that were never mentioned in the original prompt, such as requiring a proof of purchase to return an item

When a non-generative chatbot does not have an answer for a customer’s query, it will likely provide a message saying it did not understand or asking the customer to rephrase. Even when a non-generative chatbot provides a wrong answer, it will never provide information that is not factual or not approved by the client using the chatbot.


Generative chatbots rely on a prompt to keep them from producing out-of-scope content. This means that the quality of the prompt is crucial in keeping the chatbot from generating undesirable content. Unfortunately for generative AI chatbots, it’s usually possible to bypass the prompt. This is known as jailbreaking: bypassing the AI’s safeguards to get it to generate prohibited information (Takemoto, 2024).

Below is an example from the GPT-integrated chatbot on the website of the same retailer as above. When it is asked outright to write a poem, it won’t fulfil the request. However, it is very easy to bypass this. After asking the chatbot to repeat itself a couple of times with different requirements, such as asking it to shorten its message, replace words and make it rhyme, the chatbot produced a poem that is not about the products sold by the retailer. This particular chatbot usually cites the pages on the website it bases its answers on; this answer does not cite anything, suggesting it is not based on any website content and hence completely hallucinated.


To conclude, generative AI is an impressive technology, capable of generating new content based solely on input training data. However, it comes with many risks. Companies integrating generative AI on their website brings significant data security risks, as well as reliance on a 3rd party application to keep their chatbot functioning. Additionally, there is a lack of control over the exact wording of messages provided. More seriously, generative chatbots may hallucinate, generating content not verified or approved by the company using the chatbot. It is also often all too easy to bypass the AI’s safeguards and have it generate undesirable content. These risks affect both customers and clients and should be carefully considered before giving in to the allure of generative AI.


Juliette Van Steensel




Chen, L., Zaharia, M. and Zou, J. (2023). How is ChatGPT’s behavior changing over time? (doi:


Garcia, M. (2024). What Air Canada Lost In ‘Remarkable’ Lying AI Chatbot Case. Forbes. Available at:


Rahman-Jones, I. (2024). ChatGPT: Italy says OpenAI’s chatbot breaches data protection rules. BBC News. Available at:


Takemoto, K. (2024). All in How You Ask for It: Simple Black-Box Method for Jailbreak Attacks. Applied Sciences, 14(9), p.3558. (doi:


Yin, Z., Wang, H., Horio, K., Kawahara, D. and Sekine, S. (2024). Should We Respect LLMs? A Cross-Lingual Study on the Influence of Prompt Politeness on LLM Performance. arXiv (Cornell University). (doi:



bottom of page