Test an automation before you trust it

Test on the past, where you know the answers

Before an automation touches live work, run it on cases you’ve already handled: last month’s emails, inquiries or reports. Your sent folder and your CRM are full of them, and you know what the right answer was, so you can score it. A plumbing firm testing its inquiry router would take last month’s web inquiries and note where each one should have gone.

A spreadsheet is enough: one row per case, with columns for the input, the right answer, what the automation did, right or wrong, and minor or serious. Write the right answers before you run anything, so the output can’t nudge your judgment.

Start with 20 to 30 cases and include the awkward ones: the angry customer, the email that fits two labels, the inquiry with no budget. Easy cases flatter every automation.

Count, and weigh what went wrong

Reading five outputs and thinking “looks good” isn’t a test. Count how many were right, how many were wrong, and which wrong ones would have hurt. A newsletter filed under Suppliers is a shrug. A complaint filed under General and left for a week is not. For each miss, note the cause: an unclear instruction, missing information, or a case that should have gone to a person.

Set your bar before you look, for example: no serious errors, and no more than two minor ones in 30. Decide afterward and any result can look acceptable. Write the bar at the top of the sheet, where everyone can see it.

Then run the same cases again. AI steps can give different answers to the same input, and answers that flip between runs point to instructions that leave room for interpretation.

Test the three questions too

Most tests only check the happy path, where everything goes right. Using a test copy of the automation, check the safety parts as well:

• Approval: send through a case that must go to a person, and confirm it waited.
• Silence: leave one approval unanswered, and confirm the reminder and the backup both fire.
• Failure: break something on purpose in the copy (disconnect an account, rename a column) and confirm a person finds out, through an alert or the daily summary.

Passing earns a trial, not trust

A good score earns a trial run. Run the automation alongside the manual way for two weeks, compare results daily, and switch over when it meets your bar on live work. That call is a person’s, not the tool’s.

Keep the test sheet, and add every new mistake to it as a case. Rerun all of it after any change to the instructions, the form or the spreadsheet, because a fix for one case can break another. Rerun it, too, when the AI model behind the step changes, since the same instructions can behave differently on a new model.