Test your AI before it embarrasses you
A simple test set that tells you whether an AI assistant is ready to talk to customers or touch your data.
- Time
- Two to three hours
- You'll need
- A spreadsheet · The AI assistant you're testing
- You'll end up with
- A reusable test of 30 real cases, a pass mark and a record of how your assistant scores.
The steps
- 01
Collect 30 real cases
Twenty everyday requests, five tricky ones and five that must be refused or handed off. Use real wording from customers or staff.
- 02
Write the right answer for each
Not word for word, but what a good answer must include and must not include.
- 03
Run the test and grade it
Run every case, paste the answers into the sheet and mark pass or fail. You can ask a second AI to grade against your notes, but check its grading yourself the first time.
Prompt Grade this AI answer against the notes. Reply PASS or FAIL and one sentence why. It fails if it includes anything from 'must not include', misses anything from 'must include', or states a fact not supported by the notes. Question: [ ] Must include: [ ] Must not include: [ ] Answer: [ ]
- 04
Set the bar and rerun on every change
Decide the pass mark before you look, for example 28 of 30 with no failures on the must-refuse cases. Rerun the whole set whenever you change the prompt, the tool or your policies.
What breaks
- Testing only the happy path.
- Changing the prompt to fix one case and breaking three others without noticing.
Keep it safe
- Anything that fails a must-refuse case doesn't go live, whatever the overall score.
How you'll know it worked
- Pass rate over time.
- Customer complaints about wrong answers.
Do it with Blueprint
AI strategy and workflow automation, sized for small business.
We find where AI pays off in your business, then automate the workflows that eat your week.
About Blueprint →More Business Playbooks
All playbooks →Strategy
Find your first five AI wins
A one-week audit that turns "we should do something with AI" into a ranked list of five projects worth doing.
Read the playbook →
Front office
Never miss a call: set up an AI front desk
Answer every call and message, book appointments and field common questions, day and night, with a human for anything tricky.
Read the playbook →
Sales
Quotes in minutes, not at midnight
Turn a quote request into an accurate draft you review and send, using your own price list and past jobs.
Read the playbook →
Want this running in your business, without the setup?
Book a call (opens in a new tab)New Business Playbooks by email
Get the next guide the day it's published.