Know when not to trust the answer

Why it sounds sure when it’s wrong

An AI model writes by predicting likely words. Most of the time, likely and true line up. When they don’t, it still produces a fluent, confident answer: a date that’s off by a year, a statistic nobody measured, a quote nobody said. This is called a hallucination, and every assistant does it sometimes, paid or free.

Two things make it more likely. Its built-in knowledge stops at a training cutoff (the date its training data ends), so anything newer may be missing or out of date. And unless it searches the web, it isn’t looking anything up: it’s answering from something like memory, and that memory is patchy on small details.

It also tends to give an answer rather than admit it doesn’t know. Asking it to flag what it’s unsure of can help, but it’s no guarantee. Confidence isn’t evidence: a wrong answer reads exactly like a right one.

Ask who supplied the facts

This one question sorts most answers. If you pasted the facts in (your notes, a document, a price list) and asked it to rewrite or summarize them, mistakes are less likely, and you can check the result against your original. Summaries can still drop or twist a detail, so check the ones that matter.

If the facts came from the AI’s own knowledge, treat every specific as unchecked. “Summarize the cancellation terms in this contract,” with the contract pasted in, can be checked line by line. “What are the usual cancellation terms for a contract like mine?” can’t, until you read the real contract.

Web search helps, but it doesn’t end the problem. The assistant is summarizing pages it found, which can be old or unreliable, and it can misread them. Check the page, not the summary.

What it’s fine for, and what it’s bad at

It’s fine for drafting, rewording, summarizing what you give it, explaining well-known ideas and brainstorming: jobs where you can judge the result yourself, and a wrong detail is easy to spot and cheap to fix. Be most careful when you ask for:

• Exact numbers, dates, prices and statistics you didn’t give it
• Anything recent or changing: rules, rates, product features, news
• Local and niche facts: your city’s permit rules, a small supplier, the fine print of your trade
• Sources: references and links can be invented, or real but not say what it claims
• The final word on law, tax, health, safety or money, where it can help you understand but shouldn’t decide

Check in proportion to the risk

Match the checking to the cost of being wrong. Ideas and wording: read and judge. A fact going to a customer: check it against your own records. Money, contracts, law, health or safety: go to the primary source (the regulator, the supplier, the contract itself) or ask a qualified person. For a plumbing firm, the wording of a quote is low risk; the price and the building code it cites are not.

When it gives a source, open it and find the sentence that supports the claim. A real-looking link isn’t proof: it may be dead, or say something else. And asking “are you sure?” doesn’t settle anything, because it may back down from a right answer or dig in on a wrong one.

Warning signs worth slowing down for:

• A very precise number with no source
• A quote, study or case you can’t find with a quick search
• A different answer when you ask again in a new chat
• It simply agrees with whatever you suggest