Every other part of eChat is about building your agent. Evaluate is about checking your work. It runs your agent's replies against a set of real past conversations and scores each answer, so you're not left guessing whether a change to training actually made the agent better or just made it different.
The comparison point can be a real person's actual reply from your conversation history, or an earlier version of your own agent's replies. Either way, an automatic judge scores each exchange and shows you exactly where the agent's answer lined up with the reference reply and where it drifted.
Use Evaluate any time you're about to trust a change you can't easily undo - a large batch of new Q&A, a full site re-crawl, or a switch to a different AI model. It turns "I think this is better" into a number you can point to.
What Evaluate Measures
What it is
Evaluate replays a set of past conversations through your current agent, turn by turn, and scores each of its replies against a reference answer - either what a human on your team actually said at the time, or what an earlier version of the agent said. It gives you a quality signal that isn't just a hunch.
What it's best for
- Confirming a training change actually improved answers, instead of assuming it did
- Comparing your agent's current replies against your best team members' replies
- Catching a regression before your visitors do
How it works
Evaluate takes each exchange in your evaluation set, sends the visitor's message to your live agent, and compares the reply it generates against the reference reply for that same message. An automatic judge scores how closely the two match in substance - not word for word, but whether the agent covered the same information and got it right. You get a score for every exchange, plus an overall picture of how the run went.
Building an Evaluation Set
What it is
An evaluation set is the batch of past conversations Evaluate replays and scores against. You need one before you can run anything.
How it works
You build an evaluation set the same way you'd bring in past conversations for Training > Conversations: either import directly from your eWebinar chat archive, or upload a conversation export file. Each conversation in the set becomes a series of scored exchanges - the visitor's question and the reply that was actually given, which the judge treats as the reference answer to beat. Conversations used for evaluation are only a yardstick; they don't train your agent.
If a set can't be built - for example because your filters were too strict and no conversation had enough reference replies - Evaluate shows a clear failure reason instead of spinning forever on an empty panel. Use that message to loosen the filters or pick a different source of conversations, then try creating the set again.
If you close the conversation picker before choosing, the set stays listed as waiting for conversations. Open it again with Choose conversations on that set's card — you don't have to start over. Once a set's conversations are chosen, they stay fixed for that set: you can't re-sample them later (that would wipe earlier run results).
Deleting a set or a run asks you to confirm first, and the dialog says what will be removed with it.
Best practices
- Pull in conversations that represent your real mix of questions - pricing, setup, troubleshooting - not just the easy ones
- Favor conversations handled by your strongest team members if you're using human replies as the reference; a weak reference makes every score look better than it should
- Refresh your evaluation set occasionally as your product and common questions change, so you're not scoring against conversations from a year ago
Reading Your Results
What it is
Once a run finishes, Evaluate shows you a score for every exchange in the set, not just an overall grade.
How it works
Each exchange shows the visitor's original message, the reference reply, and what your agent said instead, side by side with a score. Where the agent's answer diverges from the reference, you can see exactly why the judge marked it down - a missed detail, an incorrect claim, or an answer that was technically fine but skipped what the reference reply covered. This is where Evaluate is more useful than a single pass/fail number: it points you straight at the specific gap, so you know whether the fix is a missing Q&A pair, an outdated web page in training, or an instruction that needs tightening.
Conversations the run could not score — for example because they timed out, ran out of credits, failed, or weren't judged — show as Not scored with a plain-language reason, instead of a red 0.00. They are listed after the scored conversations and left out of comparisons, so they don't look like regressions. A run that scored nothing at all shows No score rather than 0.00.
A set built from your eWebinar chat archive is judged the way a live webinar chat should be. Each attendee question is replayed the way eWebinar puts a live chat to the agent, and replies are scored on Correct, Answers what was asked, Fits a live chat, and Tone and language, plus pass/fail behaviour checks such as flagging a person when needed, not repeating a colleague, and holding back a topic the video covers later. The archived reply is shown as What came next: one example of a good reply, not the only right answer.
When you compare two runs on the same set, Evaluate tells you which run is better, or shows No established difference when the gap is too small to tell apart from noise.
The Evaluate tab focuses on your sets and runs (the Live Preview pane stays hidden here) so conversation titles and scores have room to read.
Testing Another Model
What it is
Each run can answer with a different AI model setting than the agent's own, so you can see how a switch would score before you make it.
How it works
Next to Run evaluation, a model picker starts on the agent's own setting and lists the same options as the agent's AI Model card, with the credits each one costs per answer. Pick another one and Evaluate confirms that only this run answers with it; the agent keeps its own setting. The list of runs names the model each run answered with.
Best practices
Run the same set once on the agent's setting and once on the one you're considering, then compare the two runs. See Choosing an AI Model.
What an Evaluation Costs
What it is
Running an evaluation uses credits from the same account balance as your agent's live replies.
How it works
Each run is charged at what it actually cost to generate your agent's replies and judge them for the test cases that succeeded, plus 10%, with a minimum of 1 credit per run. Test cases that fail aren't charged. If the account runs out of credits mid-run, those remaining cases finish as failed with a credits-exhausted reason, the run still completes, and those cases aren't charged. Evaluation charges appear in your usage ledger and count toward auto-recharge on paid plans. See Plans, Credits & Usage.
Best practices
Keep evaluation sets focused - a few dozen representative conversations usually tell you as much as hundreds, for a fraction of the credits. Comparing settings costs more with pricier ones, since the agent's replies are generated with the setting you're testing.
When to Run an Evaluation
What it's best for
- After importing a large batch of new Q&A pairs, to confirm they actually raised answer quality
- After a big website re-crawl, since new or changed pages can shift how the agent answers existing questions
- Before switching your agent to a cheaper or faster AI model, to confirm quality holds up before you commit to the switch
Best practices
Don't treat a training change as done just because it's live. Run Evaluate right after, and again a week or two later once you've collected fresh real conversations, so you're checking against current behavior rather than a stale set. If you're chasing lower credits costs by moving to a lighter model, run the same evaluation set on both settings before switching for good - a small dip in score might be an acceptable trade for cost, but you want that to be your decision, not a surprise your visitors notice first.
Evaluate works best alongside Chat Logs, which shows you real visitor feedback as it comes in, and Q&A training, which is usually the fastest way to close a gap Evaluate turns up.