You find out with evals, a fixed set of test questions with known right answers that you rerun after every change, and guardrails, runtime checks that block a bad answer before a customer sees it. Suvysoft builds both into every AI setup. Twenty test questions written by the person who knows the business catch most failures.
Most writing on this subject is aimed at machine learning engineers picking a monitoring vendor. This one is aimed at the owner who already paid for an assistant and now has to decide whether to point it at real customers. You do not need to write code to do this. You need a list of questions, a scoring rule, and a monthly number.
What is the difference between an eval and a guardrail?
An eval is a test you run on purpose. You keep a file of questions and the answers you would accept, you run the assistant against it, and you count how many it got right. It tells you whether last week's change to the knowledge base made things better or worse.
A guardrail runs on live traffic and does not care about trends. It sits between the model and the customer and refuses to let certain things through: an answer that cites no source document, a dollar figure the assistant invented, a refund promise nobody authorized, a customer's card number echoed back in plain text.
You need both, because they fail in opposite directions. Evals catch a slow drift you would never notice in any single conversation. Guardrails catch the one catastrophic answer that a 95 percent pass rate still allows through. An assistant with good evals and no guardrails eventually says something expensive. An assistant with guardrails and no evals quietly gets worse for months.
This is the same split the NIST AI Risk Management Framework draws between its Measure and Manage functions. NIST also published a Generative AI Profile, NIST-AI-600-1, on 26 July 2024, which names confabulation, the polite word for making things up, as a risk category in its own right.
How bad is the problem, really?
Bad enough that the gap between adoption and verification is now the story. Stanford's 2026 AI Index reports organizational AI adoption at 88 percent, while documented AI incidents rose to 362 in 2025 from 233 the year before. The same report notes that nearly every frontier model developer publishes capability benchmark results, but reporting on responsible AI benchmarks stays spotty.
The accuracy ceiling is lower than demos suggest. Artificial Analysis runs AA-Omniscience, a 6,000 question benchmark across 42 topics in six domains including business, health, and law. The strongest models tested answer correctly about 65 percent of the time, and the benchmark exists specifically because raw accuracy hides how often a model states a wrong answer confidently rather than declining to answer.
Your assistant is not answering general knowledge questions, so that number is not your number. It is grounded in your documents, which helps a great deal. But it is the reason "the model is good now" is not a testing plan.
How do you build a test set without an engineer?
Open a spreadsheet. Three columns: the question, the answer you would accept, and the source document that answer should come from. Write 20 rows. This takes about 90 minutes and it is the highest value 90 minutes in the whole project.
Pull the questions from real places, not from imagination. Your last 50 support emails. The five things every caller asks. The two policies people always misread. Then add the ones designed to fail:
- A question your documents genuinely do not answer. The right response is "I do not know, here is how to reach a person." Anything else is a failure.
- A question with a false premise, like asking about a service you do not offer. The assistant should correct the premise instead of playing along.
- A pricing or policy question where the correct answer changed recently. This catches a stale knowledge base faster than anything else.
- A rude or emotional message. You are checking tone and escalation, not facts.
- A message trying to talk the assistant out of its instructions. Prompt injection is the first entry on OWASP's Top 10 for LLM Applications, which also lists Overreliance and Excessive Agency.
Score each answer pass or fail yourself the first time. Not partial credit, not a five point scale. Once the set is stable, an automated scorer can grade the reruns for a few dollars a month, and you spot check the ones it marks as failures.
Which four numbers tell you it is working?
Track these monthly. Three columns, because a dashboard nobody reads is worse than a spreadsheet somebody does.
| Number | Healthy range | What a bad reading means |
|---|---|---|
| Eval pass rate | 90% or better | Knowledge base is stale or wrong |
| Grounding rate | 95% or better | Answers are coming from the model, not your docs |
| Escalation rate | 10% to 30% | Too low means it is guessing, too high means it is useless |
| Unresolved rate | Under 10% | Customers are leaving without an answer |
Grounding rate is the one owners skip and the one that matters most. It measures how often an answer can be traced to a specific passage in a document you control. An answer that sounds right but cites nothing is the exact shape of the failure you are trying to prevent.
Two of these move in opposite directions on purpose. Push escalation rate to zero and the assistant starts inventing answers rather than handing off. A handoff is a success, not a defect, which is why we set a floor on escalation rather than a ceiling.
What should a guardrail actually block?
Keep the list short and absolute. A guardrail that fires on judgment calls generates noise until somebody turns it off.
Block an answer that makes a commitment: a discount, a refund, a delivery date, a legal or medical opinion. Block an answer containing a number that appears in no source document, since invented figures are the most quotable and most damaging failure. Block output that repeats a card number, a full account number, or a password back to the user. Block the assistant taking an action that writes to a system, like cancelling an order, without a human approving it first.
Everything else is better handled by fixing the underlying documents. If the assistant keeps getting the return window wrong, the answer is a clearer returns page, not a filter that catches the wrong answer after the fact. Most accuracy problems we see are data problems, not model problems.
What does this cost to set up and run?
Building an eval set and wiring guardrails into an existing assistant runs $1,800 to $6,000 depending on how many workflows the assistant touches and whether it can take actions or only answer questions. An answer-only website chatbot sits at the low end. Anything with write access to a CRM or a booking system sits at the high end, because every action needs its own approval rule.
Running it afterward costs $150 to $600 a month. That covers the automated scoring passes, a monthly review of the failures, and refreshing the test set as your policies change. The scoring itself is cheap. The review is the part you are actually paying for.
Budget roughly 10 to 15 percent of the build cost per year for this, and treat it as maintenance rather than a project. An assistant that was accurate in March and never retested is not an accurate assistant in September, it is an untested one.
What should you demand from whoever built it?
Ask for four things in writing, before launch. Every one of them is reasonable and any competent builder already has them.
The eval set itself, as a file you own, not something living inside a vendor dashboard you rent. A written list of what the guardrails block, in plain language, so you can tell whether it covers the promises your business cannot afford to have made on its behalf. A monthly report with the four numbers above and the actual text of the failures. And a rollback plan: if a change drops the pass rate, how fast does the previous version come back, and who decides.
If a builder cannot produce a test set, they have not tested it. That is worth knowing before the assistant is answering your customers rather than after. We publish this list because we would rather compete on it than around it, and it is the same checklist our AI studio runs internally. If you want a second opinion on an assistant somebody else built, get in touch and we will run your questions through it.
Frequently asked questions
How many test questions is enough?
Twenty is the working minimum and catches most real failures. Fifty is comfortable for an assistant handling several distinct topics. Past about 100 you get diminishing returns unless the assistant covers genuinely separate domains, in which case treat each domain as its own set. Quality beats volume: five well chosen questions your documents cannot answer are worth more than 50 easy ones.
Can I just ask the AI to check its own answers?
Partly. Having a model grade output against a rubric is a standard and useful technique, and it is how automated scoring runs cheaply enough to check every conversation. But a model that is wrong about a fact is often wrong about whether it was wrong. That is why the grounding check matters more: it verifies an answer against a document rather than against another opinion. Keep a human on the failures.
How often should evals run?
After every change to the knowledge base, the instructions, or the underlying model, plus once a month regardless. Model providers update their systems without asking you, so a monthly baseline catches drift you did not cause. Most clients run the automated pass weekly and do the human review monthly.
Do guardrails slow the assistant down?
A little. Checks that run before an answer is sent add a fraction of a second, which is well inside what a person reading a chat reply notices. Checks that require a human to approve an action are slower by design, and that is the point. If latency becomes a real problem, the fix is running fewer, better targeted checks rather than removing them.
What if the assistant is already live and untested?
Do not turn it off. Write the 20 question set this week and run it against what is live now, so you have a baseline instead of a guess. Add the two highest value guardrails first, blocking invented numbers and blocking commitments, since those are the failures that cost money. Then fix the documents behind whatever the test set flagged.
Is any of this different for a voice agent?
The structure is identical, but the failure cost is higher because a caller cannot scroll back and reread. Test sets for voice need spoken variants of each question, including interruptions and background noise, and the escalation guardrail matters more: a voice agent that cannot answer should transfer to a person quickly rather than looping. Budget more review time in the first month.