Your Agent Passed the Demo. Now Prove It Works

Every agent demo works. That's the point of a demo. Someone picked a clean input, ran it through a happy path, and the agent booked the appointment or drafted the email or reconciled the invoice. Everyone in the room nodded.

Then it goes live, and the questions start. Why did it hallucinate a policy number? Why did it escalate 40% of tickets last Tuesday and 4% this Tuesday? Why did the output quality drop after we upgraded the model? Nobody can answer, because nobody built the machinery that would let them answer.

That machinery has a name: evals. Short for evaluations — the test suites, scoring rubrics, regression gates and production monitoring that turn "it seemed to work" into "here is the pass rate, here is the trend, here is what changed."

We've argued before that your AI stack should be built so models are swappable, because the underlying models change constantly. This is the other half of that argument. A swappable stack is only useful if you can tell whether the swap broke anything. Without evals, every model upgrade is a coin flip you can't observe.

Why demos mislead, specifically

The failure mode of AI agents isn't that they break. Traditional software breaks loudly — a service returns a 500, a queue backs up, a page won't load. You get an alert.

Agents fail quietly and probabilistically. The same input can produce a good answer on Monday and a subtly wrong one on Thursday. The agent will confidently complete a task it should have refused. It will call the right tool with the wrong argument. It will summarize a document accurately except for the one number that matters. None of that trips a monitor you already own.

Three properties make this hard:

• Non-determinism. Passing a test once tells you almost nothing. You need pass rates across many runs, not a green checkmark.
• No single correct answer. "Did the agent write a good reply to this customer?" isn't a string comparison. It needs a rubric and a scorer.
• Compounding steps. An agent that chains six tool calls with 95% reliability per step is not 95% reliable end to end. Small per-step error rates multiply into visible business failures.

This is why so many agent projects stall between pilot and production. It usually isn't that the technology can't do the job. It's that nobody can produce evidence that it does the job well enough, often enough, to hand it real customers and real money. The pilot doesn't fail — it just never graduates, because there's no defensible way to say it's ready.

What a real evaluation layer looks like

You don't need a research lab. You need four things, and they're all buildable in the same sprint cycles as the agent itself.

1. A golden test set

A curated collection of real inputs with known-good expected outcomes, drawn from your actual operations — including the ugly cases. Ambiguous requests. Missing fields. Angry customers. Documents that are photographs of documents. The awkward 10% is where value and risk both live, so the test set has to over-represent it, not sanitize it.

2. Automated scoring

Some checks are deterministic: did it extract the right invoice total, call the right API, stay inside the schema? Some need graded judgment: was the tone appropriate, was the reasoning sound, did it stay on policy? Graded checks can be scored by a model against an explicit rubric, spot-audited by humans. What matters is that scoring runs automatically, on every change, without a person volunteering an afternoon.

3. A regression gate

Nothing ships if the eval suite drops below threshold. Not a prompt tweak, not a new tool, not a model version bump, not a retrieval config change. This is the single highest-leverage control in the entire system, and it's the one most teams skip.

4. Production observability

Offline tests tell you about the inputs you thought of. Production tells you about the inputs your customers actually send. You need traces of every agent run — inputs, reasoning steps, tool calls, outputs, cost, latency — plus live quality sampling and alerting on drift. Production traffic then feeds back into the test set, so the suite gets tougher over time.

This is core to how we build agentic AI systems at Levels AI: the eval harness and the agent are the same deliverable, not a phase two nobody funds.

The 10-point pre-production readiness checklist

Whether you're building in-house or hiring a partner, ask for these ten things before anything touches a customer. If they can't be produced, you have a prototype.

• 1. A golden test set of real, representative cases — including edge cases and known failure modes — with documented expected outcomes.
• 2. Current pass rates against that set, reported per task type rather than as one flattering aggregate.
• 3. A regression gate wired into deployment, so every prompt, tool, retrieval or model change is re-scored before release.
• 4. Tool-call success rate — how often the agent picks the correct tool and passes valid arguments, tracked separately from final answer quality.
• 5. Escalation-to-human rate, with a defined target band. Too high and you've automated nothing. Too low and it's overconfident.
• 6. Cost per resolved task, not cost per token. This is the number that decides whether the ROI case survives scale.
• 7. Latency at the 95th percentile, because the slow tail is what users actually complain about.
• 8. A full audit trail — every run reconstructable end to end, retained long enough to satisfy your compliance and legal teams.
• 9. A rollback plan that's been rehearsed, with a kill switch someone in operations can pull without an engineer.
• 10. An owner and a review cadence. A named person reviews eval trends weekly and decides what gets fixed.

The compliance dividend

Here's the part that turns evals from an engineering nicety into a board-level asset: documented testing and auditability are exactly what regulated buyers and regulators ask for.

If you sell into healthcare, financial services, insurance or the public sector, your customers' procurement and risk teams will eventually want to know how your AI features were tested, what the known limitations are, what happens when the system is uncertain, and who reviewed it. Regulatory regimes for higher-risk AI uses are converging on the same expectations — documented risk management, human oversight, logging and traceability.

Teams with an eval suite answer those questionnaires from existing artifacts. Teams without one start a two-month fire drill and reconstruct evidence after the fact. The work is the same size; the difference is whether you do it before deployment on your schedule or after a customer's security review on theirs.

What to do this quarter

You don't need to boil the ocean. Pick your highest-volume AI workflow and do four things:

• Assemble 50 real cases from the last month of actual traffic, weighted toward the ones that went badly.
• Define what "correct" means for each, in writing, with the business owner in the room — not the engineer.
• Score your current system against them. The first number is usually uncomfortable, and that's the value.
• Put the gate in CI so the number can't quietly get worse.

That's a week or two of focused work, and it changes the conversation from opinion to measurement. It also makes every subsequent decision cheaper: model upgrades become tested changes instead of gambles, and scope expansions come with evidence attached.

If you're standing up agents across multiple workflows, the same harness pattern extends across them — which is where an AI automation program starts compounding instead of accumulating one-off pilots that nobody trusts enough to scale.

Talk to Levels AI

We build agents that come with their own report card — a test suite, a dashboard and a rollback plan, handed over with the system. If you have an agent stuck between pilot and production, or you're evaluating a vendor and want to know what to ask them, let's talk. Our team in Tucson has shipped 61+ AI projects, and we're happy to review your readiness checklist before you sign anything.