How models work

Why AI makes things up

Why confident answers can be wrong, and the habits that catch it.

6 min readPlain English

Read it in

In one sentence

AI writes what sounds likely rather than what it has checked, so when it doesn’t know, it can give a fluent, confident answer that’s simply wrong.

Think of it like…

Some new hires hate saying “I don’t know.” Ask one for a customer’s delivery date and, rather than admit they’re unsure, they’ll give a sensible-sounding date in a confident voice. Usually they’re close. The trouble is, they sound exactly the same when they’re not.

How it works

People call these made-up answers “hallucinations.” They aren’t a bug about to be patched; they come from the way a model writes.

1It predictsUnless it’s connected to search or your files, the model writes each word from learned patterns, not from a store of checked facts.
2Gaps get filledWhen the patterns are thin (a rare name, a niche rule, your own prices), it still produces the most plausible-sounding thing.
3Confidence is styleThe tone tells you nothing. A made-up figure is written just as smoothly as a real one.
4Questions steer itAsk “why did our supplier raise prices?” and it may supply reasons, because the question assumes there are some.

Search, your own documents and clear prompts reduce these mistakes. Nothing gets them to zero.

Good at

Rewording text you’ve given it
Summarizing a document it can see
Common knowledge that’s written everywhere

Bad at

Exact names, dates, figures and quotes
Sources, links and legal references
Anything recent, niche or about your business

Where you’ll see it at work

The invented policy. An online shop’s draft reply promises a refund window the shop has never offered, because it sounds like a typical policy. Give it the real policy to work from.
The source that doesn’t exist. A design studio’s proposal cites an industry report nobody can find. Click every source before a client sees it.
The total that’s nearly right. A bakery asks for last month’s supplier total from a pile of invoices and gets a figure that’s close, not correct. Check it against the spreadsheet.