Home / Insights / Generative AI and LLMs
Generative AI and LLMs
Evaluating an AI Chatbot before Users Depend on It
A chatbot that answers demo questions well can still fail real users. This guide shows how to build an evaluation dataset, classify failures by severity, combine automated and human scoring, set release gates and monitor the chatbot in production.
TopicsGenerative AI and LLMsQA and Software TestingAI Agents and Automation
Key takeaways
- Evaluate a chatbot against a dataset of real questions with expected behaviour, not against a handful of demo prompts.
- Classify failures by type and severity so you know which ones block a release and which can wait.
- Combine automated checks, model-based grading and human review; each catches different problems and needs calibration.
- Set release gates per failure severity, and rerun the evaluation for every change to prompts, models, retrieval or tools.
- In production, trace conversations, sample them for review and turn failures into new test cases.
On this page
Testing traditional software asks whether a function returns the expected output. Chatbots built on language models are harder: the same question can produce different wording, small prompt changes can shift behaviour, and the answer depends on retrieved documents and tool calls as well as the model. A chatbot can impress in a demo and still give wrong answers, invent policies or escalate too rarely once real users arrive.
This guide sets out an evaluation process for business chatbots: customer support assistants, internal knowledge assistants and booking or service bots. It covers what to measure before launch and how to keep measuring after.
What to evaluate
| Dimension | Question | Example measure |
|---|---|---|
| — | — | — |
| Correctness | Is the answer factually right? | Share of answers matching the reference |
| Groundedness | Is the answer supported by retrieved sources? | Share of claims supported by context |
| Retrieval | Did the system find the right documents? | Recall of expected sources in top k |
| Task completion | Did the user achieve their goal? | Completed bookings, resolved queries |
| Policy compliance | Does it follow business rules and refuse when it should? | Violations per 100 conversations |
| Safety and security | Does it resist misuse and protect data? | Prompt-injection and data-leak test pass rate |
| Escalation | Does it hand over when it should? | Correct escalations, missed escalations |
| Tone and clarity | Is it understandable and appropriate? | Human rating on a rubric |
| Latency and cost | Is it fast and affordable enough? | p95 response time, cost per conversation |
Pick the dimensions that matter for your use case; most business chatbots need at least correctness, groundedness, escalation and policy compliance.
Build an evaluation dataset
The evaluation dataset is the foundation. Without it, every change is judged by impression.
Sources of test cases
- Real questions from support tickets, chat logs, search logs or interviews, anonymised.
- Edge cases from domain experts: ambiguous questions, multi-part questions, outdated information.
- Policy cases: requests the bot must refuse or escalate, such as refunds above a limit or medical advice.
- Adversarial cases: prompt injection, attempts to extract other customers’ data, off-topic abuse.
- Multi-turn cases: conversations where context from earlier turns matters.
What each case records
| Field | Example |
|---|---|
| — | — |
| ID and category | SUP-041, rescheduling |
| User context | Logged-in customer with one upcoming service visit |
| Input or conversation | “Can you move my visit to next week?” |
| Expected behaviour | Offers available slots next week; confirms before changing |
| Must not | Change the booking without confirmation; offer slots outside service area |
| Expected sources or tools | Booking API: list slots, reschedule |
| Severity if failed | High |
Expected behaviour is usually more useful than an exact expected answer, because good answers can be worded many ways.
Size and balance
Start with 100 to 300 cases covering your main intents in rough proportion to real traffic, plus deliberate over-representation of high-risk cases. Grow the set from production failures. Keep a held-back portion that is not used while tuning prompts, so you can check for overfitting to the test set.
A failure taxonomy
Not every failure matters equally. Classify them so release decisions are consistent.
| Failure type | Example | Typical severity |
|---|---|---|
| — | — | — |
| Harmful or unsafe output | Gives dangerous instructions, discriminatory content | Critical |
| Data exposure | Reveals another customer’s details | Critical |
| Wrong action | Cancels or changes a booking incorrectly | Critical |
| Policy violation | Promises a refund outside policy | High |
| Fabrication | Invents a price, policy or feature | High |
| Missed escalation | Keeps answering when a human is needed | High |
| Retrieval miss | Right answer exists but was not found | Medium |
| Incomplete answer | Answers part of a multi-part question | Medium |
| Unnecessary escalation | Hands over a question it could answer | Low to medium |
| Tone or clarity | Too long, confusing, off-brand | Low |
Illustrative taxonomy. Agree severities with the business before evaluation starts.
Scoring: automated, model-graded and human
| Method | Good for | Limitations |
|---|---|---|
| — | — | — |
| Deterministic checks | Format, required fields, tool calls made, forbidden phrases, latency | Cannot judge meaning |
| Model-based grading | Correctness against a reference, groundedness, relevance at scale | Can be biased or inconsistent; must be calibrated against humans |
| Human review | Nuance, tone, policy judgement, ambiguous cases | Slow and costly; needs rubrics and calibration |
Use all three, each where it fits.
To calibrate model-based grading, have two or more people score a sample of the same responses with a written rubric, compare their agreement, then check how often the model grader agrees with them. If agreement is low, refine the rubric or the grading prompt before trusting automated scores.
Release gates
Define pass criteria before running the evaluation:
| Gate | Example criterion |
|---|---|
| — | — |
| Critical failures | Zero in the evaluation set |
| High-severity failures | Below an agreed rate, and no increase from the previous version |
| Groundedness | At or above an agreed threshold on knowledge questions |
| Escalation | Missed escalations below an agreed rate on escalation cases |
| Regression | No category worse than the previous version beyond tolerance |
| Latency and cost | p95 latency and cost per conversation within budget |
Thresholds depend on your use case and risk; the principle is to fix them in advance.
Rerun the evaluation for every change to prompts, models, retrieval settings, documents or tools. Model provider updates count as changes too. Open-source tools such as promptfoo can run evaluation suites and compare versions, and can be wired into a pipeline so evaluations run automatically.
Worked example: a service booking assistant
Consider an illustrative assistant that answers questions about air-conditioning servicing and can reschedule visits through a booking API.
- The team collects 220 cases from past chats and calls: 40% booking changes, 30% service questions, 15% billing, 15% policy and adversarial cases.
- First evaluation: correct on most service questions, but it fabricates prices when the price list does not cover a unit type, and it reschedules without confirming in a few cases.
- Fixes: add a rule and test cases requiring “I will check with our team” when no price is found; require an explicit confirmation step before calling the reschedule tool; add a deterministic check that the reschedule tool is called only after confirmation.
- Second evaluation: no critical failures; fabrication cases now pass; two new retrieval misses found, traced to a poorly chunked PDF.
The numbers are illustrative; the pattern is typical: the most important failures are often policy and action errors, not general knowledge.
Monitoring in production
Evaluation before launch cannot anticipate every question. In production:
- Trace conversations: inputs, retrieved context, tool calls, outputs, latency and cost, with prompt and model versions. Observability tools such as Langfuse, or tracing built on OpenTelemetry, support this.
- Protect privacy: mask personal data in traces, restrict access and set retention periods.
- Sample for review: review a random sample plus conversations with negative feedback, escalations and long exchanges.
- Watch the dashboard: resolution rate, escalation rate, negative feedback rate, groundedness scores on sampled traffic, tool error rate, latency and cost per conversation, each by intent.
- Close the loop: every confirmed failure becomes a new evaluation case, so it cannot recur unnoticed.
Our guide to MLOps and LLMOps covers the wider operating model.
Frequently asked questions
How many test cases do we need?
Start with 100 to 300 well-chosen cases and grow from production failures. Coverage of intents and risks matters more than raw count.
Can we rely on another model to grade answers?
For scale, yes, but only after calibrating it against human reviewers and rechecking periodically.
How often should we evaluate?
On every change to prompts, models, retrieval or tools, and on a schedule in production because data and user behaviour change.
Who should own chatbot quality?
A product owner with QA and domain experts. Engineering owns the pipeline; the business owns policies and severities.
Sources and methodology
Facts in this article were checked against the sources below. Directory data (ratings, review counts, rate bands and team sizes) changes over time and is shown as published on the date accessed. Illustrative examples are labelled as such and are not quotes.
- Netguru: AI Chatbot Evaluation and Observability with Promptfoo and Langfuse — Reviewed as a baseline; covers metrics and two tools without dataset design, severity taxonomy or grader calibration
Related reading
Continue your research
Launching or improving an AI chatbot?
Share your chatbot and its users. We will help you build an evaluation dataset, release gates and monitoring so changes improve it rather than break it.
