When the AI hallucinates in production, the failure looks like a confident answer that is wrong in a way the system cannot detect. The model does not know it is wrong. The model does not know what it does not know. The model produces the answer with the same confidence whether the answer is right or wrong, and the same confidence is what the production team has to compensate for, and the compensating work stands as the part the marketing does not talk about.
The failure modes in 2026 fall into a few categories, and the categories are what the production team has to plan for.
Failure mode 1: the fabricated citation
The model produces a URL or a paper title that does not exist. The citation gets passed through to the user as if it were real. The user clicks through to a 404, a parked domain, or a malicious clone. The user trusts the citation because the user trusts the model, and the user does not verify the citation because the user does not have a reason to verify the citation.
The mitigation is citation verification at the application layer, before the citation reaches the user. The mitigation is a fetch request to the URL, a check that the response is a real response, a check that the response matches the citation, and a flag to the user if the verification fails. The mitigation is not free. The mitigation falls on the application team.
Failure mode 2: the wrong number
The model produces a statistic that is close to a real statistic but not the real statistic. The user quotes the wrong statistic in a deck, a paper, a press release, a regulatory filing. The wrong statistic gets caught by a careful reader, and the careful reader points out the error, and the user has to retract the deck, the paper, the press release, the regulatory filing.
The mitigation is number verification against the source data. The mitigation becomes the application passing the source data to the model, the model grounding the answer in the source data, and the application verifying that the numbers in the answer match the numbers in the source data. The mitigation is not free. The mitigation falls on the data team, who have to make the source data available to the model.
Failure mode 3: the invented API
The model produces a function call for an API that does not exist, an API that has a different signature than the model expects, or an API that exists but the model has hallucinated the parameters. The function call fails. The application has to handle the failure, the user has to deal with the failure, and the user has to wonder why the AI feature does not work the way the marketing said it would.
The mitigation becomes the function call validation against the API schema, the API stub for the test environment, and the application graceful degradation when the function call fails. The mitigation is not free. The mitigation falls on the application team, who have to build the validation, the stub, and the graceful degradation.
Failure mode 4: the wrong answer to a hard question
The model produces a confident answer that the user trusts. The answer is wrong. The user has to live with the wrong answer, and the wrong answer stands as the kind of failure that does not get caught until the consequences arrive. The user makes a decision based on the wrong answer, the decision turns out to be wrong, and the user has to live with the consequences of the decision.
This counts as the failure mode the evals cannot catch. The evals can catch the obvious failures, and the obvious failures are roughly 60 to 70 percent of the failures. The evals cannot catch the subtle failures, and the subtle failures are the failures that look like the right answers until the consequences arrive. The evals are getting better, and the evals are not going to get to 100 percent. The evals are the work the eval teams are doing well, and the evals are not going to solve the problem on their own.
What the production team has to do
Treat the AI output as untrusted input. Validate the citations, validate the numbers, validate the function calls, validate the answers. The validation falls on the application team, and the validation runs as the work that turns the AI from a demo into a product.
Build the human review into the high stakes workflows. The human review falls on the human, and the human review runs as the work that catches the failure the eval missed. The human review is expensive. The human review counts as the right answer for the high stakes workflows, and the high stakes workflows are the workflows where the failure matters.
Build the observability into the AI features. The observability falls on the operations team, and the observability sits as the work that catches the failure in production. The observability includes the input, the output, the latency, the cost, the failure rate, the human override rate, and the user feedback. The observability amounts to the work that makes the AI feature debuggable, and the debuggability runs as the work that makes the AI feature improvable.
The AI hallucinates. The AI is going to hallucinate. The question is not whether the AI is going to hallucinate. The question is what the application does when the AI hallucinates, and the question is what the application does when the user trusts the hallucination. The application that handles the hallucination well sits as the application that is going to ship. The application that handles the hallucination poorly amounts to the application that is going to be in the news.
The bottom line
The patterns the post covers have been showing up in production for long enough that the patterns have names, the failures, the mitigations, the gaps. The work the security team and the engineering team and the operations team are quietly doing today sits as the work that decides whether the practice the post names sits as a tool the team uses or a liability the team is paying for.
Sources & Further Reading
All claims in this article are sourced from primary documentation, vendor advisories, and reputable security researchers.
Spotted an error? Email the editor. Corrections are issued with a visible correction note.
Editorial standards. Every article on humanrequired.org is reviewed by a human editor before publication. AI may assist with drafting or research; final editorial control is human. Read the full standards.
