Pilot it on real work for two weeks

Why the test set isn’t enough

Your test set is last month’s work, cleaned up and read carefully. It tells you whether the tool can do the job. A pilot tells you whether it does: a short, real trial on this month’s work, with a few people and a fixed end date. Only real use shows whether the tool fits: whether people remember to use it, whether the input arrives the way the form expects, and what it does with the case nobody thought of.

Two weeks is long enough to see a normal week and an odd one, and short enough that people keep paying attention. Put the start date, a check-in at the end of week one and the final review in the calendar before day one. At the check-in, read the log together and look hardest at what got thrown away.

Small group, real work

Keep the pilot small enough to watch closely and real enough to learn from:

• A small group: the person who owns the workflow, plus one or two people who do the task every week. Not the whole team.
• Real tasks as they arrive: not a demo, and not only the easy ones.
• The old way stays open. If the tool is down or wrong, the work still gets done by hand.
• One version. Don’t change the instructions mid-pilot unless something serious goes wrong, and if you do, note the date so you can read the results fairly.

A person approves every output

During the pilot, nothing the tool produces is sent, saved or acted on until a named person has read it. They mark each output: used as it was, edited or thrown away. That rule is what makes it safe to try the tool on real work, and those marks become your best evidence. Make it clear that throwing an output away is a useful result, not a failure.

Make the check easy to do properly. Show the approver the original request next to the output, and keep the red lines where they can see them. An approver who has to hunt for the original will soon start clicking yes.

Agree before day one what would make you stop early: for example, a red-line error that reaches a customer, or an approver who no longer has time to check properly. Anyone in the pilot can call a stop.

Log every use in ten seconds

Add a row to a shared sheet each time the tool is used. If logging takes longer than the task, people stop logging, so keep each row to five things:

• Date, who used it and a reference number (not the customer’s name)
• Used as it was, edited or thrown away
• Anything wrong caught at review, and how serious
• Minutes spent, including checking
• One line of notes, if something surprised them