How we stop AI from making things up

If you want AI to write content or summaries inside your product, you cannot stop it from ever getting a fact wrong just by asking it to be careful. What you can do is control what it writes from, check what it wrote with something other than another AI, and hold back anything that fails. In our own product we use five layers: give the model a fixed set of facts, forbid it from going beyond them, check the output with ordinary code, score it with a second pass, and keep anything weak away from readers until a person has looked at it. This article explains each layer in plain terms, what the setup still cannot catch, and what to ask any developer who proposes an AI feature to you.

Why made-up facts are the main risk

AI language models, the kind behind tools like ChatGPT and Claude, write by predicting what text should come next. They are very good at producing sentences that sound right. Sounding right and being right are different things. When a model states something false with full confidence, people call it a “hallucination”.

For some uses a small error is harmless. For content that goes out under your name, it is a problem. A wrong price in a product summary, a wrong date in a customer update, or a quote nobody said can cost you trust that took years to build. Readers rarely know which parts were checked, so one visible error makes them doubt everything else.

Our test case: FFSN

We build and run our own product, FFSN, a sports network for a single fantasy football league. A league connects its ESPN league once, and AI sportswriter characters publish recaps, power rankings and trade analysis about that league’s actual teams and managers. There are six AI writers, each with a beat and a voice, plus a separate interviewer character who asks league members for quotes. The models are Anthropic’s Claude models.

This is close to a worst case for made-up facts. The readers are the people the articles are about. They know who won on Sunday, by how much, and who traded whom. A recap that gets a score, a name or a trade wrong is worse than no recap at all. So accuracy had to be designed in from the start.

The five layers

No single check is enough, so we stack them. Each one catches things the others can miss.

1. Facts first. Before anything is written, the system assembles a structured block of facts from the league’s synced data: scores, standings, rosters and trades. Think of it as a briefing sheet. The AI writer is only allowed to write from that block.

2. Rules for the writer. The instructions given to the model forbid inventing statistics, sources or quotes. This helps. But an instruction to a language model is closer to a request than a guarantee, and a model can still drift. That is why the next layers exist.

3. A checker that is not an AI. After the article is written, ordinary code (a normal program, with no language model involved) goes through the text and compares every name, number, quote and dollar amount against the facts block. It recognizes more than 20 kinds of violation. What happens next depends on how serious the problem is: it may remove a single sentence, block a whole section, or flag a warning.

4. An editor pass. A second AI pass reads the finished piece and scores it for quality, much as an editor would before a story runs.

5. A publish gate. Anything that scores below the threshold is held as a draft for a human to review instead of being published automatically.

On top of the layers, the site tells readers the content is AI-generated. People judge what they read differently when they know how it was made, and they should be told.

Why the fact checker is plain code

A common suggestion is to have a second AI read the first AI’s work and ask it “is this true?”. It is tempting because it is easy to build. The problem is that the second model works the same way as the first. It can miss the same error, or accept a confident wrong sentence, for the same reasons the first one wrote it.

Code that compares text to a list of facts cannot be talked into anything. Say a recap claims a team scored 112 points and the facts block says 98. The check fails. Well-written prose has no way to persuade it otherwise. So we use AI for the parts it is good at (writing, and judging quality in the editor pass) and plain code for the part that has a right answer.

What this cannot catch

No setup like this is perfect, and you should be wary of anyone who says theirs is. These are the limits of ours.

• It checks facts and has no sense of taste. The code can confirm a score is right. It cannot tell whether a joke lands or whether a jab at a manager goes too far. The editor pass and human review help there, but that is judgement, and judgement can be wrong.
• It can only check against data it has. If the synced league data is wrong or missing, the checker has nothing correct to compare against. The facts block is only as good as its source.
• It is strongest on specific claims. Names, numbers, quotes and dollar amounts are easy for code to compare. A sentence can be misleading without containing a single wrong figure, for example by calling a close game a blowout, and that kind of framing is much harder for code to catch.
• Human review only works if someone does it. A held draft stays held until a person looks at it.

Questions to ask a developer proposing an AI feature

If someone offers to add AI-written content, summaries or answers to your product, these questions will show you quickly how seriously they have thought about accuracy.

1. Where do the facts come from? Ideally the model writes only from your real data, handed to it each time, rather than from whatever it happens to remember.
2. What checks the output? Ask whether anything other than the AI itself checks the result, and what exactly it compares against.
3. What happens when a check fails? A good answer is specific: the sentence is removed, the section is blocked, or the piece is held. “We log it” is a weak answer if the content still goes live.
4. Who reviews held content? Someone on your side or theirs, and what happens if nobody gets to it.
5. Are readers told it is AI? They should be.

If the answers are vague, the risk sits with you.

Where to start

If your AI feature is internal, like drafting emails your staff read and edit before sending, you may not need any of this. A person checking every output is a perfectly good system, and off-the-shelf tools will often do the job.

The layers above matter when AI output reaches customers without a person reading each piece first. If that is what you are planning, start by writing down which facts the AI is allowed to use and where they live. That list is the foundation for every check that follows. If you want help designing or building it, our AI integration service covers this kind of work, and you can get in touch to talk it through.