· 3 min read
Test an AI tool on your own work before you buy
Every AI demo works. Here is a one-afternoon test, using twenty of your own cases, that shows how a tool behaves on a normal Monday.
Every AI demo works. That is not cynicism. It is how demos are made. Someone picked the input, rehearsed the path, and stopped before the interesting failure.
Your work is not a demo. It is the invoice with a missing page, the customer who writes in two languages, the request that means the opposite of what it says. The question is not “can this tool do the task?” It is “what does it do on a normal Monday?”
You can find out in one afternoon, before you sign anything.
Run the twenty-case test
1. Pull twenty real cases from last month. Do not pick the cleanest ones. Choose ten ordinary cases, five messy ones, three exceptions, and two you would be embarrassed to get wrong. Remove anything you are not allowed to share.
2. Write the answer key first. For each case, write down what your best person would do, before you see the tool’s output. Otherwise you will grade the tool on how convincing it sounds instead of whether it is right.
3. Run all twenty yourself. Not the vendor. If they insist on driving, you have learned something.
4. Score each result into one of four buckets.
| Result | What it means |
|---|---|
| Right | You would send it or use it as it is |
| Fixable | Under a minute of editing gets it right |
| Wrong | It missed, and it was obvious |
| Confidently wrong | It missed, and it sounded certain |
5. Look hardest at the last bucket. A tool that is wrong and says so is a tool you can build a process around. A tool that is wrong and sounds certain will eventually cost you something, because nobody checks what sounds certain.
Agree the pass mark before you look
Write the number down before the first result comes in. For example: “At least fifteen of twenty need no change, and none of the three exceptions gets a confident wrong answer.” Those figures are only an example. Choose the ones that fit the cost of a mistake in your work.
The reason is simple. Once you have seen the outputs, every result looks good enough. A pass mark set in advance is the only one you can trust.
Ask what the demo never answers
Accuracy is one question. These six decide whether the tool survives contact with your business.
- What does it do when it does not know? The right answers are “it says so,” “it asks,” or “it hands off to a person.” The wrong answer is “it guesses.”
- Where does our data go, and for how long? Ask whether it is stored, who can see it, and whether it is used to train anything.
- What does it cost at ten times today’s volume? A per-seat price and a per-use price behave very differently as you grow.
- Can we see what it did, and why? If something goes wrong, you need a record, not a shrug.
- Can we take our data and configuration with us? The time to ask about leaving is before you arrive.
- Who fixes it, and how fast, when it breaks on a Tuesday? A support promise is only useful if it has a name and a time attached.
Let someone who did not choose it try it
The person championing a tool will forgive it. They know what it is supposed to do, so they steer around what it cannot.
Ask a busy, skeptical colleague to run five of your twenty cases without any guidance. Their friction is the friction the rest of the company will feel. Ten minutes of their time can save you a year of licenses nobody uses.
Why an afternoon is enough
The test costs a few hours. The contract may cost a year. And the result does more than give you a yes or a no. It gives you a baseline to hold the tool to later. If you want to know how to measure whether it really improved anything, start with the workflow itself.
If a pitch is on your desk right now, our AI Audits & Readiness work checks a plan or a vendor claim before you spend. And if it is a decision you would rather talk through first, Executive AI Coaching is a private place to pressure-test it.