What are AI hallucinations and how do you stop them in business apps?

AI hallucinations are confident answers from an AI model that are false, unsupported or made up, such as a fake court case, an invented refund policy or a product spec that does not exist. They happen because language models predict likely text instead of checking facts. No setting removes them, but grounding, citations, abstention rules, testing and human review bring them to a level a business can manage.

AI hallucinations moved from research papers into courtrooms once chatbots started answering customers and drafting legal filings. The AI Hallucination Cases database maintained by researcher Damien Charlotin listed 2,217 court decisions involving AI-fabricated content as of October 25, 2026, and 1,514 of them came from the US.

On this page

• What are AI hallucinations in plain terms
• Why do AI models hallucinate
• How often does AI hallucinate in 2026
• Real cases: what hallucinations have cost businesses
• Why hallucinations cost more in AI agents
• Who is responsible when an AI chatbot gets it wrong
• Which business uses carry the most hallucination risk
• Prompt-level fixes that reduce hallucinations
• System-level fixes: grounding, tools and validation
• How to detect hallucinations automatically
• How to measure hallucinations in your own app
• What hallucination controls cost
• Rules for staff who use AI tools
• A hallucination checklist before you launch an AI feature

What are AI hallucinations in plain terms

The NIST Generative AI Profile (NIST AI 600-1) uses the term confabulation and defines it as "the production of confidently stated but erroneous or false content," the kind that can mislead or deceive users. The confident tone is what makes it dangerous. A wrong answer looks exactly like a right one.

Hallucinations in business software fall into six types:

• Invented facts, such as a founding date, a statistic or a product dimension that no source contains.
• Fake citations, including court cases, research papers and URLs that do not exist.
• Made-up policies, where a support bot promises a refund window or discount nobody approved.
• Misread sources, where the model has the right document but reverses a condition or mixes up two numbers.
• Calculation slips on totals, percentages and dates across long inputs.
• Fictional software packages suggested in generated code, which attackers can register and fill with malware.

Say a 20-person e-commerce brand in Texas puts an AI assistant on its site. A shopper asks whether a jacket is waterproof. The product page says "water-resistant," but the model has seen thousands of pages where jackets like it are called waterproof, so it says yes. The shopper buys, gets soaked and asks for a refund the model also promised. That is two hallucinations and one lost customer.

Why do AI models hallucinate

A language model writes by predicting the next most likely word, one token at a time. It has no separate step that checks a claim against a database before printing it. The guide to how LLMs work covers that mechanism in detail.

OpenAI researchers Adam Tauman Kalai, Ofir Nachum, Santosh Vempala and Edwin Zhang explained the deeper cause in a September 2025 paper, summarized on OpenAI's research blog. They argue that hallucinations start as "errors in binary classification." If a model cannot reliably tell a true statement from a false one, statistical training pressure will produce false statements some of the time.

Their second point explains why hallucinations persist after training. Models are "optimized to be good test-takers, and guessing when uncertain improves test performance." On a multiple-choice exam, a guess sometimes scores, while "I don't know" never does. Benchmarks that grade only accuracy reward the same behavior, so models learn to answer confidently even when they should abstain.

Four conditions raise the risk in business use. The fact is rare or private, such as your own price list or a niche part number. The question sits past the model's training cutoff. The prompt pressures the model to give a complete answer when the source material is incomplete. Or the relevant detail is buried in the middle of a long input, where models are known to read less carefully.

Each condition has a direct counter. Supply private facts in the prompt or through retrieval. Fetch anything recent with a tool. Tell the model that a partial answer is acceptable. Keep the passages you send short and put the most relevant one first.

How often does AI hallucinate in 2026

Rates depend heavily on the task, so a single number for "AI" is meaningless. The table collects measured rates from named sources.

Two lessons stand out. Summarizing a document you supply is far safer than answering open questions from memory, which is why Vectara's leaderboard rates sit in single digits for many models. And a model allowed to say "I don't know" makes far fewer errors. In the SimpleQA comparison, the newer model answered 22 percent of questions correctly against 24 percent for the older one, but it was wrong on 26 percent against 75.

Real cases: what hallucinations have cost businesses

The best-known US case is Mata v. Avianca. On June 22, 2023, Judge P. Kevin Castel of the Southern District of New York sanctioned two lawyers and their firm $5,000 after they filed a brief citing six non-existent judicial opinions generated by ChatGPT. The judge wrote that there is "nothing inherently improper about using a reliable artificial intelligence tool for assistance," but that existing rules give attorneys "a gatekeeping role" over accuracy.

Customer-facing bots carry their own exposure. In Moffatt v. Air Canada, decided February 14, 2024, British Columbia's Civil Resolution Tribunal ordered the airline to pay C$812.02 after its website chatbot described a bereavement fare refund rule that contradicted the airline's actual policy. The airline argued the chatbot was responsible for its own statements. The tribunal called that "a remarkable submission" and held the company responsible for all information on its website, chatbot included.

Code assistants open a security hole. The package hallucination study by Joseph Spracklen and colleagues found 205,474 unique fake package names across 576,000 generated code samples. An attacker who registers one of those names can get malware installed by any developer who trusts the suggestion.

The fines in these cases were small next to the legal fees, staff time, press coverage and lost trust that came with them.

Why hallucinations cost more in AI agents

In a chatbot, a hallucination is a wrong sentence that a person can read and ignore. In an agent, it becomes a wrong action. The model might invent an order number, pick the wrong customer record or pass a made-up amount to a refund tool, and the system carries it out.

Errors also compound across steps. If a model gets each step right 98 percent of the time, a 10-step task succeeds about 82 percent of the time, because 0.98 to the 10th power is 0.82. A value invented in step 2 can flow through steps 3 to 10 without anyone seeing it, so the final email or refund looks correct on the surface.

Three controls matter most for agents:

• Validate every tool argument against the real system before running it, so an order ID or customer ID must exist and belong to the person asking.
• Require human approval for irreversible actions such as payments, refunds above a set amount, cancellations and messages to clients.
• Keep tasks short, with a checkpoint after the step most likely to go wrong.

A refund agent that checks the order exists, confirms the amount matches the invoice and asks a person to approve anything over $100 turns most hallucinations into harmless errors that never leave the system.

Who is responsible when an AI chatbot gets it wrong

The pattern across these cases is consistent: the business that deploys the AI owns what it says. The airline tried "the chatbot said it" as a defense and lost. Read your AI vendor contract to see which risks, if any, the provider accepts, because the customer will hold the company whose name is on the website responsible.

In regulated work, the bar is higher. Lawyers, accountants, insurers and health providers already carry duties to verify what they send, and AI output does not change those duties. The overview of AI laws that apply to US businesses in 2026 covers the rules that already apply. This is general information, not legal advice, so check your own exposure with a lawyer before you launch a customer-facing AI tool.

Two habits reduce exposure at almost no cost. Keep transcripts of every AI conversation for as long as you keep customer email, because the transcript is the record if someone disputes what the bot promised. And label the assistant as AI, with links to the policy pages it draws from, so customers can check the source before they act on an answer.

Which business uses carry the most hallucination risk

Risk depends on two things: how costly a wrong answer is, and whether a person reviews it before anyone acts. The table ranks common uses.

The single most effective rule fits in one line: any number or policy a customer could rely on should come from your database or your documents, never from the model's memory.

For example, a hypothetical 6-chair dental practice in Colorado lets its website bot answer insurance questions. Should the bot claim the practice is in network for a plan it dropped in March, a patient books, receives a $420 bill and the front desk spends 30 minutes on the dispute. The fix costs nothing: the bot reads the accepted-plans list from the practice software and tells callers to confirm any plan not on it.

Prompt-level fixes that reduce hallucinations

Instructions change behavior more than most teams expect. Anthropic's guide to reducing hallucinations lists three basic techniques that work across models:

• Give the model explicit permission to say it does not know. The guide says this "can drastically reduce false information."
• For long documents, ask the model to pull word-for-word quotes first and then answer from those quotes.
• Require a citation for every claim, and if the model "can't find a quote, it must retract the claim."

A support bot prompt built on those rules might include this instruction:

Answer only from the documents provided. Quote the sentence that supports each fact. If the documents do not answer the question, reply: "That information is not in my sources. A team member will follow up within one business day."

The same guide lists advanced checks: run the same prompt several times and compare the answers, since "inconsistencies across outputs could indicate hallucinations," and restrict the model to the provided documents instead of its general knowledge. It also warns that these techniques "don't eliminate them entirely."

System-level fixes: grounding, tools and validation

Prompts set behavior. System design removes whole categories of error. The OWASP Top 10 for LLM applications lists misinformation as LLM09:2025 and names the main defenses, which map to four design choices:

1. Ground answers in your own content with retrieval-augmented generation, so the model writes from your documents instead of its memory.
2. Fetch live facts with tools. Order status, stock, prices and account balances should come from a direct lookup, the job of the tool connections described in the guides to AI agents and the Model Context Protocol.
3. Validate outputs automatically. If the bot quotes a price, code compares it with the catalog before the message is sent. If it names an order number, code confirms the order exists.
4. Route high-stakes answers to a person. A refund over a set amount, a legal question or a complaint goes to staff with the AI draft attached.

RAG alone is not a cure. A 2024 study of legal research tools by Varun Magesh, Daniel E. Ho and colleagues found that RAG-based products from LexisNexis and Thomson Reuters still hallucinated 17 to 33 percent of the time. Grounding narrows the problem, and validation plus review closes most of what is left.

How to detect hallucinations automatically

Manual review does not scale past a few hundred conversations a month. Four automated checks catch problems before a person needs to look:

• Citation checks. Code confirms that every quoted sentence exists in the source document, and an answer with a missing quote goes to review.
• Consistency sampling. The same question runs three times, and answers that disagree on a fact get flagged.
• A judge model. A second, smaller model scores whether each claim is supported by the retrieved passages. Vectara builds its leaderboard this way, with its own HHEM evaluation model.
• Rule checks. Plain code matches prices, dates, SKUs and order numbers against your database.

Rule checks are the cheapest and the most reliable, because they need no AI at all. Start there for anything numeric, add citation checks for document answers, and use a judge model or consistency sampling for open-ended answers where rules cannot reach.

Each check adds a model call or a database query to the answer, so it adds some delay. Consistency sampling triples the model calls for the questions it covers, which is why it belongs on high-risk topics such as pricing, eligibility and policy questions.

How to measure hallucinations in your own app

Public leaderboards tell you little about your use case, so build a test set from your own questions. Track three numbers, the same ones OpenAI reports for SimpleQA: correct answers, wrong answers and abstentions.

1. Collect 100 to 200 real questions from your inbox, chat logs and sales calls, each with an answer your team agrees is correct.
2. Include 20 to 30 questions your documents cannot answer, to test whether the bot abstains.
3. Run the full set, mark each response correct, wrong or abstained, and note why each wrong answer happened.
4. Fix the cause: a missing document, a vague instruction or a retrieval miss.
5. Re-run the set after every fix, document update or model change.
6. After launch, review a sample of 50 live conversations each week.

Say a 200-question test returns 170 correct, 18 abstentions and 12 wrong answers, a 6 percent error rate. Adding the missing documents and an abstention instruction moves it to 176 correct, 21 abstentions and 3 wrong, or 1.5 percent. A business can set a launch bar in advance, for example under 2 percent wrong and zero wrong prices, and hold to it.

What hallucination controls cost

The controls cost less than one bad incident. Take a hypothetical 12-person insurance agency in Arizona whose website assistant handles 3,000 conversations a month. The token figures use Anthropic's list prices as of October 2026: Claude Sonnet 5.5 at $2 per million input tokens and $10 per million output, and Claude Haiku 5.5 at $0.10 and $0.50.

The automated checks together cost under $25 a month. People cost ten times more, which is the strongest argument for automating every check that can be automated and saving staff time for the cases code cannot judge.

Rules for staff who use AI tools

Mata v. Avianca started with a professional who trusted a chat answer. A one-page staff policy closes that gap:

• Treat every AI answer as a first draft from a new hire.
• Check the source before sending any number, date, quote, citation or legal statement.
• Ask the tool where a claim came from, and treat it as unverified if no document or link backs it.
• Paste the source document in and ask the model to answer only from it.
• Keep legal, tax, medical and HR decisions with a qualified person.
• Report wrong answers to the owner of your AI tools, so the test set grows.

Suppose a sales rep asks a chatbot for a competitor's current pricing before a call. The model answers with exact figures from memory, possibly years out of date. The rule above sends the rep to the competitor's pricing page first, which takes two minutes and avoids quoting a number that a prospect can disprove on the spot.

A hallucination checklist before you launch an AI feature

Run through this list before any AI feature reaches customers:

• Every price, policy and date the AI can state comes from a system of record, not the model.
• The prompt allows "I don't know" and names the handoff to a person.
• Answers show a citation or link the customer can check.
• Automatic checks compare numbers and order IDs against your data before sending.
• High-stakes topics route to staff, and the bot says so plainly.
• A test set of at least 100 questions passes your error bar.
• Weekly reviews of live conversations have a named owner.
• Code assistants use an approved package list.
• Staff know which data they may paste into AI tools, as covered in the guide on putting company data into ChatGPT.

Start with the first line of that checklist this week: list every number and policy your AI could state, and where each one should come from. The AI development team at Techno Mantra, with 5,000+ projects delivered for 3,460+ clients, builds chatbots and assistants on OpenAI, Claude or Gemini with grounding, validation checks, test sets and human handoff included. You own all code and accounts. Send your use case through the contact page or to info@technomantraa.com, and every inquiry gets a reply within one business day.