Decide how you’ll judge it before you build it

Set the bar before you build

Once you’ve built something, it’s hard to judge it fairly. You’ve put in the time, the first few answers look impressive, and it’s tempting to call it done. Deciding what “good” means before you build keeps you honest, and gives you a fair way to compare one version with the next. It also settles arguments: when someone says the tool is great or useless, you can point at the sheet.

The way to do it is a test set: real examples of the task, each with a note on what a good result looks like. People who build AI tools call these evals, short for evaluations. Yours can live in a spreadsheet.

Twenty real examples, not ten perfect ones

Start with 20 real examples from the last month or two, pulled from your sent folder, order history or work orders, not written from memory. Real examples carry the mess that invented ones leave out: missing details, two questions in one, the customer who writes in all caps. Replace names, emails, phone numbers and account numbers (Customer A, Order 1) before any of it goes near an AI tool.

Twenty is a rule of thumb: enough to show patterns, few enough to check by hand. Make about a quarter of them the awkward ones:

• A request with a key detail missing
• One that looks routine but isn’t, like an emergency or a complaint
• Something you don’t offer
• One where the right answer is “a person should handle this”

Write checks, not perfect answers

For each example, write two or three short checks instead of a model answer. A model answer tempts you to judge on style; checks give a clear pass or fail, and anyone on the team can mark them. If you have the reply you actually sent, keep it alongside for reference.

For a plumbing firm’s quote replies, the checks might be: includes the current service-call fee; asks for the details needed to quote; flags anything that sounds like an emergency for a person to call back.

Then write your red lines: two or three things that must never happen in any example, such as a firm price for a job that needs a site visit, a promise you can’t keep or the wrong customer’s name. One red-line failure matters more than several small misses.

Write the bar down

Decide now what result means “ready to pilot.” A sensible first bar: at least 16 of the 20 pass, and none break a red line. Raise it if mistakes are costly. Write it at the top of the sheet with today’s date, so it can’t quietly slide later.

Hold on to the test set. You’ll rerun all of it after every change, because a fix for one example can break another. And if a vendor ever offers to sell you a tool for this job, it’s the fairest test you can give them.