Skip to content
dvnt.io

Artificial intelligence24 min read4751 wordsRedakcja dvnt

AI hallucinations — why AI makes things up and how a business can catch it

AI hallucinations are confident but false answers from a language model. Why they happen, where they cost a business in Poland and how a process catches them.

AI hallucinations are answers from a language model that sound plausible and are delivered with confidence, but are false: an invented date, a court ruling that does not exist, a quotation that is nowhere in the document. In "Why language models hallucinate" (5 September 2025) OpenAI defines them as "plausible but false statements generated by language models", and the US National Institute of Standards and Technology (NIST) stresses that the system will "confidently present" such content.

OpenAI's ChatGPT help centre (read 3 October 2026) lists three typical forms of hallucination: "Incorrect definitions, dates, or facts"; "Fabricated quotes, studies, citations or references to non-existent sources"; and "Overconfident answers to ambiguous or complex questions". It adds an important caveat: "Confidence isn't reliability: The model may express high confidence even in incorrect answers."

So for a business the question is not whether AI makes mistakes, because every model does. The question is where a mistake costs money or creates liability, and how to catch it before it reaches a customer, a court or the accounts. You cannot switch hallucinations off. You can, however, organise the work so that the model is allowed to say "I don't know", answers from sources, leaves the numbers to a tool, and is checked by a person wherever an error is expensive. Responsibility for what the AI writes stays with the company. We read the sources on 3 October 2026 and the guide of the European bars quoted below on 19 October 2026; quotes from Polish sources are our own translation.

AI hallucinations — what they are (and what they are not)

A hallucination is not only a false statement about the world. It is also an answer that contradicts the text you gave the model yourself. In its Generative AI Profile (NIST AI 600-1, July 2024) NIST uses the term confabulation and writes: "Confabulations also include generated outputs that diverge from the prompts or other input or that contradict previously generated statements in the same context." In practice a company meets two kinds of error:

  • inconsistency with the facts — the model cites a ruling that does not exist, gives the wrong date or the title of a paper nobody wrote;
  • inconsistency with the document — the model summarises a contract you gave it and adds a clause that is not in it, or contradicts in the third paragraph what it wrote in the first.

The second kind is easy to miss, because intuition says that if the model "has the document", it cannot go wrong. It has less room to guess, but it can still guess.

NIST notes that these phenomena "are colloquially also referred to as “hallucinations” or “fabrications.”" You will see the term in the singular (an AI hallucination) and in narrower forms such as LLM hallucination or ChatGPT hallucinations; NIST itself prefers the term confabulation. They all describe the same thing.

So what is not a hallucination? Not every model error. OpenAI observes that large models rarely make spelling mistakes or mismatch parentheses: "Spelling and parentheses follow consistent patterns, so errors there disappear with scale. But arbitrary low-frequency facts, like a pet’s birthday, cannot be predicted from patterns alone and hence lead to hallucinations." The OWASP GenAI Security Project, part of the application security community, adds in its description of risk LLM09:2025: "While hallucinations are a major source of misinformation, they are not the only cause; biases introduced by the training data and incomplete information can also contribute."

How a model produces text token by token, and why it sometimes picks a sequence that is likely but untrue, is explained in our article on large language models (LLMs). Here we look at which facts models invent most often and what follows from that for a business.

Why does AI hallucinate instead of saying “I don’t know”? (Does AI lie?)

AI does not lie in the human sense, because it has no intention of misleading you. It guesses — and it is rewarded for guessing. So the answer to "does AI lie?" or "does ChatGPT lie?" is this: the model does not know that it does not know, and it has been taught that an answer pays better than silence. The clearest account is the research paper "Why Language Models Hallucinate" (arXiv, 4 September 2025). Three of its four authors give OpenAI as their affiliation, the fourth Georgia Tech; it is a preprint, meaning it was published before peer review. Three points from it matter to a business.

A hallucination is an ordinary classification error. The authors write: "Hallucinations need not be mysterious—they originate simply as errors in binary classification. If incorrect statements cannot be distinguished from facts, then hallucinations in pretrained language models will arise through natural statistical pressures." Crucially, they show that "even if the training data were error-free, the objectives optimized during language model training would lead to errors being generated." According to the authors the analysis applies broadly, "including to reasoning and search-and-retrieval language models".

Models err most often on rare facts. According to the paper, "Large language models seldom err on frequently referenced facts, e.g., Einstein’s birthday or dissertation title." They err on facts that appeared in the data once or not at all. An example from the paper: DeepSeek-V3, asked in its app on 11 May 2025 for the birthday of one of the authors, gave three different incorrect dates in three attempts, "even though a response was requested only if known." On its blog OpenAI describes a similar test with the title of the same researcher's PhD dissertation: a widely used chatbot "confidently produced three different answers—none of them correct."

Our conclusion: your company's facts — the price in the current price list, a contract number, the deadline for a complaint, a customer's name, a product's technical specification — are rare or entirely unknown facts to a model. That is exactly where to expect invention if the model answers from memory rather than from a document.

The model is "always in “test-taking” mode". That is how the authors describe the effect of evaluating models with exams that penalise uncertainty. OpenAI's blog shows it with simple arithmetic: if a model guesses someone's birthday, it has "a 1-in-365 chance of being right", while "Saying “I don’t know” guarantees zero points." Over thousands of questions, the guessing model looks better on leaderboards than the careful one. The bluff is often specific, too: the paper's example is "“September 30” rather than “Sometime in autumn”". Specificity looks like knowledge, even when it is not.

NIST adds one more warning that matters to anyone who checks AI output: "GAI outputs may also include confabulated logic or citations that purport to justify or explain the system’s answer, which may further mislead humans into inappropriately trusting the system’s output." It goes on: "LLMs sometimes provide logical steps for how they arrived at an answer even when the answer itself is incorrect." A careful-looking justification is not proof.

Higher accuracy does not mean fewer errors — the test that shows it

A model that is right more often can also make things up more often. The SimpleQA test that OpenAI cites in its 5 September 2025 article shows this. SimpleQA is a set of short factual questions; the figures come from the GPT-5 system card. The answers fall into three groups: accurate, wrong, and abstentions, where the model gave no specific answer.

Two horizontal bars, each stacked to 100 percent.
Accuracy versus errors in SimpleQA — two OpenAI models from 2025Two horizontal bars, each stacked to 100 percent. OpenAI o4-mini: 24 percent accurate answers, 75 percent wrong answers, 1 percent with no specific answer. gpt-5-thinking-mini: 22 percent accurate, 26 percent wrong, 52 percent with no specific answer, that is “I don’t know”. The older model is right 2 percentage points more often, but wrong almost three times as often. An illustration of the mechanism from one vendor test, not an error rate for today’s models.Source: OpenAI, Why language models hallucinate (5 September 2025), data from the GPT-5 system card; SimpleQA test, short factual questions; read 3 October 2026

The older o4-mini answered 24% of the questions correctly and 75% wrongly, and abstained in only 1%. gpt-5-thinking-mini was right in 22%, wrong in 26%, and gave no specific answer in 52%. Accuracy differs by 2 percentage points in favour of the older model, but it gives almost three times as many wrong answers (75 against 26; by our calculation about 2.9 times). OpenAI's comment: "Strategically guessing when uncertain improves accuracy but increases errors and hallucinations." And elsewhere in the same article: "errors are worse than abstentions."

A caveat: this is one test, two OpenAI models from 2025 and data published by the vendor itself. It illustrates the mechanism; it is not an error rate for today's models or for your chat. We do not give a general figure such as "AI is wrong in X% of answers", because we have not found a source that gives one reliably.

OpenAI also says two things that are easy to confuse. On harder tests and in real use, accuracy will not reach 100%: "on more challenging evaluations and in real use, accuracy is capped below 100% because there are some questions whose answer cannot be determined for a variety of reasons such as unavailable information, limited thinking abilities of small models, or ambiguities that need to be clarified." Hallucinations, on the other hand, are not inevitable: "Claim: Hallucinations are inevitable. Finding: They are not, because language models can abstain when uncertain." So it is not true that "OpenAI admitted hallucinations cannot be avoided" — it argues the opposite.

Applied to a business (our assessment): if you judge a chat or an assistant by the share of questions it answered, you reward guessing just as model leaderboards do. Count three numbers separately: accurate answers, wrong answers, and "I don't know" answers passed on to someone else. A wrong answer is the worst outcome; "I don't know" is a service cost, not a system failure.

The paper's authors propose stating a confidence threshold explicitly in test instructions: "Answer only if you are > t confident, since mistakes are penalized t/(1 − t) points, while correct answers receive 1 point, and an answer of “I don’t know” receives 0 points." At a threshold of 0.5 a mistake costs 1 point; at 0.9 it costs 9 points. The higher the required threshold, the more expensive a mistake becomes compared with "I don't know". The authors write about scoring models; carrying the rule over to a company is our step. You can write the threshold into the instructions for the model and into the acceptance criteria of the deployment — higher for prices and contract terms, lower for a newsletter draft.

Where hallucinations cost a business — three areas

An AI error costs money where somebody acts on it. NIST writes that confabulation is "particularly relevant when it comes to open-ended prompts for long-form responses and in domains which require highly contextual and/or domain expertise." Harm arises "when users believe false content – often due to the confident nature of the response – leading users to act upon or promote the false information." NIST does not estimate the scale: "it is difficult to estimate the downstream scale and impact of confabulations." We do not give statistics either, because we have not found reliable ones.

The three cases below come from the United States, England and Canada. None of them is Polish law. What they do show is where courts place responsibility when AI gets it wrong.

Court filings, contracts and the law

Mata v. Avianca (US, 2023). In its sanctions order of 22 June 2023 (No. 22-cv-1461) the US District Court for the Southern District of New York found that the lawyers "submitted non-existent judicial opinions with fake quotes and citations created by the artificial intelligence tool ChatGPT, then continued to stand by the fake opinions after judicial orders called their existence into question." The court imposed a penalty of $5,000, jointly and severally, on the two lawyers and their firm. The reasoning is what matters: "Technological advances are commonplace and there is nothing inherently improper about using a reliable artificial intelligence tool for assistance. But existing rules impose a gatekeeping role on attorneys to ensure the accuracy of their filings." The court also lists the costs: "The opposing party wastes time and money in exposing the deception. The Court’s time is taken from other important endeavors." The case is also summarised in the appendix to the London judgment described below.

Al-Haroun v Qatar National Bank (England, 2025). In its judgment of 6 June 2025 ([2025] EWHC 1383 (Admin)) the High Court in London described a schedule that "lists forty five citations that had been put before the court. In eighteen instances, the case cited does not exist. In respect of those cases that did exist, in many instances they did not contain the quotations that were attributed to them, did not support the propositions for which they were cited, and did not have any relevance to the subject matter of the application."

A grid of 45 squares, one for each citation put before the court.
Al-Haroun: 45 citations put before the court, 18 cases that do not existA grid of 45 squares, one for each citation put before the court. 18 squares highlighted: the case cited does not exist. 27 squares: the case exists, but in many instances without the quotation attributed to it, not supporting the proposition it was cited for, or not relevant to the application; the court gives no breakdown of these 27. 18 of 45 is 40 percent, dvnt calculation.Source: Al-Haroun v Qatar National Bank, [2025] EWHC 1383 (Admin), para 74; High Court (England and Wales), 6 June 2025; read 3 October 2026

The client stated that "the citations were generated using publicly available artificial intelligence tools, legal search engines and online sources." His solicitor relied on that research "without independently verifying the authorities." The court held that "A lawyer is not entitled to rely on their lay client for the accuracy of citations" and referred the solicitor to the regulator. In the same judgment the court dealt with a second case, Ayinde, in which the grounds contained five fake cases; the barrister denied using AI, and the court did not decide the point.

The court also set out a general principle (paras 6–8): "Freely available generative artificial intelligence tools, trained on a large language model such as ChatGPT are not capable of conducting reliable legal research." They "may cite sources that do not exist" and "may purport to quote passages from a genuine source that do not appear in that source." Those who use them have "a professional duty therefore to check the accuracy of such research by reference to authoritative sources, before using it in the course of their professional work", and this "is no different from the responsibility of a lawyer who relies on the work of a trainee solicitor or a pupil barrister for example, or on information obtained from an internet search."

Europe's lawyers have been warned in the same terms. The Council of Bars and Law Societies of Europe (CCBE), in its guide on the use of generative AI by lawyers (2 October 2025), writes that in legal services the output of generative AI "might produce entirely fictional case law, create non-existent court cases or judicial opinions, falsely attribute quotes to judges or legal scholars or construct seemingly plausible but entirely invented legal arguments." It warns that "Lawyers using AI-generated content without appropriate verification could face sanctions for professional misconduct and/or contempt of court, malpractice lawsuits". This is professional guidance for the legal profession, not law.

In Poland, the National Chamber of Legal Advisers (Krajowa Izba Radców Prawnych, KIRP), in its recommendations on the use of AI (2025; the copyright in the publication is held by Microsoft sp. z o.o.), writes of "first reports" of such tools being used without verification, "in particular by citing non-existent court rulings". We found no Polish ruling in such a case, so we do not cite one. For a company outside the legal sector the lesson is the same (our assessment): a contract, terms and conditions, a letter to an authority or a reply to a complaint prepared with the help of AI is the company's document, not the tool's.

Answers to customers

Moffatt v. Air Canada (Canada, 2024). The Civil Resolution Tribunal of British Columbia, which among other things hears small claims, described this situation in its decision of 14 February 2024 (2024 BCCRT 149): before buying a ticket, a customer used the chatbot on Air Canada's website. The chatbot told him he could submit his ticket for a reduced bereavement rate "within 90 days of the date your ticket was issued", even after travelling. The policy page the chatbot itself linked to said the policy "does not apply to requests for bereavement consideration after travel has been completed". An Air Canada representative later "admitted the chatbot had provided “misleading words.”"

Air Canada argued that it could not be held liable for information provided by its agents, servants or representatives, including a chatbot. The tribunal called this "a remarkable submission. While a chatbot has an interactive component, it is still just a part of Air Canada’s website. It should be obvious to Air Canada that it is responsible for all the information on its website. It makes no difference whether the information comes from a static page or a chatbot." It found that Air Canada "did not take reasonable care to ensure its chatbot was accurate" and that the airline did not "explain why customers should have to double-check information found in one part of its website on another part of its website." It awarded CAD 650.88 in damages, CAD 812.02 including interest and fees.

A caveat: Air Canada gave the tribunal no information about how its chatbot worked, so we do not know whether it was a generative model. The lesson concerns responsibility, not technology: if a tool answers customers on a company's behalf, its answer is the company's answer. How chatbots perform in customer service for online shops is covered in our article on e-commerce customer service. We will describe an AI chat for customers on a company website in a separate article.

Numbers, personal data and code

Numbers. In its help centre OpenAI lists the code interpreter and data analysis as tools that enable "accurate calculations". Our conclusion: totals, VAT, currency conversions, discounts and deadlines counted in working days are calculated by a tool or a spreadsheet, and the model at most describes the result in words. The same rigour applies to technical data in sales copy — we cover separately where generative AI helps with product descriptions and where the specifications must be checked.

Personal data. A false answer about a specific person is a GDPR (in Polish, RODO) matter. In its report on ChatGPT (para 30) the European Data Protection Board (EDPB) notes that "due to the probabilistic nature of the system, the current training approach leads to a model which may also produce biased or made up outputs." It adds that such outputs "are likely to be taken as factually accurate by end users, including information relating to individuals, regardless of their actual accuracy", and concludes: "In any case, the principle of data accuracy must be complied with." KIRP's recommendations use the notion of "limited trust": personal data generated by AI may be partly or entirely "made up", and that may affect every decision taken on the strength of it. KIRP also names the risk of overreliance on AI, where someone "relies excessively or exclusively on the output generated by AI". Our conclusion: check any AI-generated fact about a customer, an employee or a candidate before you act on it.

Code. In its LLM09:2025 risk description OWASP lists the case where "The model suggests insecure or non-existent code libraries". In one of its scenarios, "Attackers experiment with popular coding assistants to find commonly hallucinated package names" and then publish malicious packages under those names in widely used repositories. A developer who installs a dependency suggested by AI without checking it opens the door themselves.

A “the AI may be wrong” warning is not enough

A notice under the chat is the minimum of transparency, not error control. In the report of the ChatGPT taskforce (23 May 2024, para 31) the EDPB writes that "proper information on the probabilistic output creation mechanisms and on their limited level of reliability" should be provided by the controller, "including explicit reference to the fact that the generated text, although syntactically correct, may be biased or made up." It continues: "Although the measures taken in order to comply with the transparency principle are beneficial to avoid misinterpretation of the output of ChatGPT, they are not sufficient to comply with the data accuracy principle".

This is the preliminary view of a taskforce on one service, ChatGPT, and its provider, not a decision addressed to companies that use AI. The Air Canada case points the same way: the tribunal did not expect the customer to check the chatbot's answer on another page. And the High Court in London observed (para 82), about guidance issued by professional regulators, that "promulgating such guidance on its own is insufficient to address the misuse of artificial intelligence."

Our conclusion from these two sources: a "the AI may be wrong" warning under a chat does not relieve the company of responsibility for what its tool tells a customer. If, in the EDPB's view, a warning is not enough for the model provider, do not count on it being enough for you. The EDPB was writing about the accuracy principle for personal data, and the tribunal in the Air Canada case held that a company is responsible for chatbot information just as for the rest of its website; that decision does not discuss warnings under a chat. This is our reading of the positions described, not an interpretation of Polish law — a lawyer assesses the specific case.

Labelling AI content still makes sense: OWASP recommends "clearly labeling AI-generated content and informing users on limitations of reliability and accuracy". But a label tells the customer what they are talking to; it does not make the answer correct. That is the job of the process.

How to design a process that catches hallucinations — six elements

You design control around the cost of an error, not around how much you trust the model. Anthropic's documentation "Reduce hallucinations" (read 3 October 2026) advises: "Always validate critical information, especially for high-stakes decisions." OWASP recommends tools and processes "to automatically validate key outputs, especially output from high-stakes environments." The six elements below put that principle into practice.

Diagram of four levels, from the lowest to the highest cost of an error, no numbers.
How much control, by the cost of an error — four levelsDiagram of four levels, from the lowest to the highest cost of an error, no numbers. Level 1, internal draft: the author checks it while editing. Level 2, document for a client: a second person checks the facts, figures and cited sources; figures come from a tool or spreadsheet. Level 3, legal or financial decision: a specialist checks every citation and quotation against an authoritative source or the document; figures from a tool. Level 4, automatic answer to a customer with no person before sending: answers only from the document base, the right to say “I don’t know” with the conversation handed to a person, ongoing review of answers and an error log.Source: dvnt analysis based on: OpenAI (5 September 2025, Help Center), Anthropic (Reduce hallucinations, Citations), NIST AI 600-1 (2024), Mata v. Avianca (2023), Moffatt v. Air Canada (2024), Ayinde / Al-Haroun (2025); read 3 October 2026

1. A map of the cost of an error. List the uses of AI in the company and for each one ask: what happens if the answer is made up, and who will notice first? An internal draft is corrected by its author. An offer, a contract or a letter to an authority goes out under the company's name. A legal or financial decision has consequences you cannot undo with a correction. The hardest case is an automatic answer to a customer, because nobody reads it before it is sent — control has to be built into the design, not added at the end.

2. Sources instead of the model's memory. If the model is to state a fact about your company, give it the document it should take the fact from. Google's grounding overview (updated 1 October 2026) defines grounding AI this way: "grounding is the ability to connect model output to verifiable sources of information", which "reduces the chances of inventing content." How to build answering from company documents, and what breaks it, is covered in our article on RAG.

3. Citations that are checked, not just supplied. NIST recommends that you "Review and verify sources and citations in GAI system outputs during pre-deployment risk measurement and ongoing monitoring activities" (action MS-2.5-003). That is what AI fact checking really means: open the source and find the sentence the model relies on. There is a trap here. For its Citations feature Anthropic states that returned citations "are guaranteed to contain valid pointers to the provided documents." That guarantees the passage exists, not that the claim is true or that the passage supports it. In Al-Haroun some of the cases did exist; they simply did not contain the quotations attributed to them or had nothing to do with the matter.

4. The right to say "I don't know", and a handover to a person. A model that is allowed to decline guesses less often — that is the lesson of the SimpleQA test. In the instructions, explicitly give it the right to answer "I don't know" and set a confidence threshold that matches the cost of an error. But "I don't know" has to lead somewhere: to an employee, a form or a ticket queue. That is how we are designing the AI chat on our own website: no answer in the knowledge base, or low confidence, is meant to hand the conversation to a member of staff rather than produce a guess. The chat is not live yet and there are no results, so we describe the design, not the effect.

5. Numbers by a tool, consistency by comparison. Hand calculations to a tool (a code interpreter, a spreadsheet, the accounting system), not to the model writing the text. To catch invented content, Anthropic describes three techniques for comparing and re-checking outputs. Best-of-N verification: "Run Claude through the same prompt multiple times and compare the outputs. Inconsistencies across outputs could indicate hallucinations." This fits the point about rare facts (DeepSeek-V3 gave three different dates in three attempts). Chain-of-thought verification: when an answer looks wrong, you review the summarised reasoning, which "can reveal faulty logic or assumptions." Iterative refinement: the model's output goes into a follow-up prompt "asking it to verify or expand on previous statements." When reviewing reasoning, remember NIST's caveat: the justification can be confabulated too.

6. A reviewer who knows how to check, and an error log. The weakest link is often the person who is supposed to check and simply approves. NIST describes this as automation bias, "excessive deference to automated systems", and writes that it "can exacerbate other risks of GAI, such as risks of confabulation". The CCBE guide names the same risk for lawyers: "overreliance on outputs, which can lead to ‘automation complacency’ and substituting human judgment with automated conclusions." OWASP recommends that you "Ensure that human reviewers are properly trained to avoid overreliance on AI-generated content." In its AI literacy questions and answers (updated 27 July 2026) the European Commission writes: "The employees and the human-in-the-loop need the appropriate skills, targeted as to the system they are using." After deployment NIST recommends organisational processes for "post-deployment monitoring of GAI systems, particularly for potential confabulation" (action MG-4.1-002). In a small company a simple log is enough (our example, no figures): the question, the model's answer, what the error was, who caught it, and what was corrected in the documents or the instructions.

The more steps a model takes without a check, the more places an error can slip through (our assessment). That applies above all to an AI agent, which chooses its own tools and takes action after action, but also to automations in which a model is one step, for example in n8n, and to tools connected to a model through MCP. How to design approvals in an agent is covered in our guide on how to build an AI agent.

What not to promise: techniques reduce, they do not remove

No technique brings hallucinations down to zero, and nobody who promises otherwise can point to the vendors' documentation. Anthropic writes: "while these techniques significantly reduce hallucinations, they don't eliminate them entirely." Google says grounding "reduces the chances of inventing content." OpenAI says accuracy is "capped below 100%". The vendors give no success rates for individual techniques, so neither do we. Be careful with promises that a newer model, a longer prompt or "a good prompt" will solve it: each of these may reduce the number of errors, none of them takes it to zero.

The map of error costs also marks a boundary (our assessment): where one mistake costs more than doing the work by hand, AI should not work without a person. It can prepare a draft, find documents or point to the places that need checking, but the decision is taken and signed by someone who has checked it. Rules for employees using AI assistants, including checking the output before sending it, are covered in our comparison of ChatGPT, Copilot and Gemini for business.

Hallucinations, the AI Act and the GDPR — in brief

The AI Act does not set a required level of knowledge about hallucinations, but the European Commission expects employees to know about the risk. In the same questions and answers on Article 4 of the AI Act, the Commission asks whether a company "whose employees are using ChatGPT for, e.g., writing advertisement text or translating text" needs to comply with the AI literacy requirement, and answers: "Yes, they should be informed about the specific risks, for example hallucination." According to the Commission, Article 4 does not require a guaranteed level of that knowledge or any measurement of it. Who Article 4 applies to and from when is explained in our article on the EU AI Act for business.

The GDPR comes into play when the model invents something about a specific person: the data accuracy principle applies, and a warning about possible errors does not satisfy it, as described above. This is a description of published positions, not legal advice. For the wider picture of where to start with AI in a company, see our guide to AI in business.

Frequently asked questions about AI hallucinations

Does ChatGPT lie?

Not in the sense of intent. ChatGPT, like any language model, sometimes guesses and presents the guess with confidence. OpenAI explains that standard training and evaluation procedures reward guessing over acknowledging uncertainty, and its help centre warns that confidence is not reliability. For you the effect is similar to a lie: you get an untruth that sounds convincing, so check important information at the source.

Does a newer or paid model stop hallucinating?

It may hallucinate less often, but it does not stop. OpenAI writes that ChatGPT also hallucinates and that hallucinations remain a fundamental challenge for all large language models, and that accuracy will not reach 100%. Anthropic states that techniques for reducing hallucinations do not eliminate them. The SimpleQA test also shows that a model with higher accuracy can be wrong more often if it guesses instead of saying "I don't know". We do not publish rankings of which model hallucinates least, because we have not found reliable measurements for business use.

Is a "the AI may be wrong" notice under the chat enough?

No. In its report on ChatGPT (para 31) the European Data Protection Board writes that telling users the text may be made up helps avoid misinterpretation, but is not sufficient to comply with the data accuracy principle. In Moffatt v. Air Canada the tribunal held that a company is responsible for its chatbot's information just as for the rest of its website, and did not expect the customer to check it elsewhere. Labelling AI content is worth doing, but only a process provides error control.

Who is liable when a company's chatbot gives a customer wrong information?

In Moffatt v. Air Canada (British Columbia, 2024) it was the company: the tribunal held that the chatbot is part of the airline's website, that the airline is responsible for all the information on it, and awarded CAD 812.02 including interest and fees. That is a Canadian decision, not Polish law; it shows a direction, it does not decide a case in Poland. How liability would look under Polish law in a specific case is for a lawyer to assess.

How can I quickly check whether a source cited by AI exists?

Open the link yourself instead of trusting the description — OpenAI advises checking sources when accuracy matters by visiting the links directly. Find the sentence the model quotes in the document; the document existing is not enough, because in Al-Haroun some of the cases existed but did not contain the quotations attributed to them. For legislation and case law, go to authoritative sources, as the London court requires of lawyers. Even a citations feature in an API only guarantees that the passage comes from the document, not that it supports the claim.

Does AI that answers from company documents hallucinate too?

Yes, though usually less often. NIST counts as confabulation also outputs that diverge from the input, that is, from the documents given to the model. Google writes that grounding in sources reduces the chances of inventing content, and Anthropic that techniques for reducing hallucinations, including answering only from the documents provided, do not eliminate them. So a system that answers from documents still needs citation checks, the right to say "I don't know" and a review of its answers.

Related articles

Where would AI really pay off for you?

Tell us which tasks take up most of your time today. We will check whether an off-the-shelf assistant is enough or whether you need an implementation on your own data. If ChatGPT on a business plan will do, we will say so plainly.