Keep it, fix it or drop it

Exit

Lesson 6 · 8 min

Keep it, fix it or drop it

By the end

A written keep, fix or drop decision, with the numbers behind it and a next step.

  1. 1Learn
  2. 2Try it
  3. 3Human check
  4. 4Quick check

Learn · 3 min

Three honest answers

Put the scorecard next to the bar you set before seeing the results. There are only three answers, and each one is a good result if the evidence supports it:

  • Keep: it cleared the bar, people used it without being reminded, what the approvers caught was minor, and the business number held up.
  • Fix: it’s useful, but something specific gets in the way, such as a kind of request it keeps getting wrong, a missing reference file or a clunky step people skip. Change that one thing, rerun the test set and pilot again for a week. Allow two rounds of fixing at most.
  • Drop: it missed the bar and you can’t name a fix, its errors are the kind you can’t afford, or nobody would miss it. Write down why and keep the test set. You found out in two weeks, not a year.

Watch for the usual traps

The same few things bend these decisions:

  • Keeping it because you built it. Time already spent is not a reason to spend more.
  • Judging from one vivid case, good or bad. Read the whole log.
  • Moving the bar. If the results fall short, say so first, then ask whether the bar was wrong.
  • Averaging away a red line. One serious error that got past review outweighs a good average.
  • Keeping it without an owner. Someone has to rerun the test set when your prices, your policies or the AI model behind it change.

What keep means, and when to get help

Keeping a tool doesn’t mean removing the person. The approval step stays on anything that reaches a customer, involves money, promises or contracts, or can’t be undone. You can lighten the check on low-risk outputs later, once the log shows it has earned that.

If keeping it means real engineering (connecting systems, more volume, more people), brief someone: a freelancer, your software provider or a consultant. Hand over your scope, test set, instructions, pilot log and scorecard, and ask to see your test set pass before you pay for the full build. Judge their version against your bar, not against a demo. AI Distilled runs this same process with clients as paid consulting; nothing in this path depends on it.

Try it on your business

5 min

Make the call in writing. The AI argues each side; you decide.

  1. 1

    Paste in your bar, scorecard and notes, and run the prompt. If your pilot is still running, use the numbers so far as a practice run.

  2. 2

    Read the three cases against your own log, and check every number it quotes against your scorecard.

  3. 3

    Decide, and write one paragraph in your shared doc: keep, fix or drop; the numbers that decided it; the next step, with a date; and who owns it.

  4. 4

    Tell the people who ran the pilot what you decided and why.

I ran a two-week pilot of a small AI tool that [WHAT IT DOES].
The bar I set before seeing the results: [PASTE YOUR BAR].
My scorecard: [PASTE THE NUMBERS].
Notes from the people who used it, names removed: [PASTE THE NOTES].

Using only this information, make the strongest honest case for each option: keep it, fix it, drop it. For fix, name the one change most likely to help. Point out anything in my numbers that is missing, too small to judge or doesn’t add up. Don’t decide for me.
Open ChatGPTOpen ClaudeOpen Gemini

Human check: before you use it

Quick check · question 1 of 2No grades

Your tool cleared the bar, but half the team stopped using it in week two. What’s the honest call?

Ask the Distillery

Answers come from The Distillery’s own pages, with sources.

If the Distillery doesn’t know, it says so and points you to a person.