Deployed Isn't Done: How We Measure Every Voice Agent We Build

Launching an AI voice agent is the easy part. Knowing what it’s actually doing, and fixing what isn’t working, is the work.

The tooling that exists for voice agents measures the conversation — and it measures it well. Was the agent clear, did it collect what it needed, did it close politely. Those are real questions with real answers, and they get answered one call at a time.

That leaves two blind spots.

The first is the view across calls. Which question in your intake loses the most callers. Whether a problem you fixed three weeks ago has actually stopped. The answers are in the transcripts, but they don’t assemble themselves, because finding them means reading every call against the structure of the agent itself, comparing the callers who continued against the ones who didn’t, and doing it across enough calls for the answer to mean something.

The second is everything after the call. A transcript ends the moment the call does. It can’t tell you whether the lead the agent took was ever sent on.

So the default state of a deployed voice agent is this: it works, everyone assumes it’s fine, and the questions that would tell you whether it’s improving go unasked.

We built something different. The agents we run for our clients are audited automatically, every week, by an internal tool no client ever sees. What it produces isn’t a score or a report. It’s our list of what to change, with the evidence attached.

What it actually produces

Anyone can build a dashboard of call counts and conversion ratesConversion rateThe share of visitors or callers who take the step you want, such as booking an estimate.. Those tell you something is happening. They don’t tell you what to do Monday morning.

On agents built as a step-by-step call flow, a finding reads more like this:

• At this exact point in the conversation, this many callers out of this many hung up — measurably more than the agent’s own average.
• This is the instruction the agent is working from at that point.
• Where the same question sits on a second route through the same agent, this is how that copy performs, and whether its wording is identical.
• These are the theories we tested about why, and these are the ones the data didn’t back.

That last item on the list is the unusual part. Much of the analysis exists to test explanations, not propose them.

When an agent is losing people at a particular question, there are always several convincing-sounding reasons. The question comes too late. The agent repeats itself. There’s an awkward pause before it speaks. Each is plausible. Each is testable.

So for every problem point, the system compares callers who left against callers who continued — same question, same period, same agent — across a fixed panel of measurements. In a recent run, 41 theories had enough callers on both sides to compare. The data backed 6 of them, and 5 of those 6 were the kind that can’t be told apart from the hangup itself (more on that below), so they were recorded as open questions rather than findings. The sixth held up as a real finding. The other 35 showed no difference at that sample size.

No difference doesn’t prove a theory wrong. A few more calls could change the picture. What it doesn’t do is justify rewriting a prompt.

None of that makes the problem point itself any less real. The theories are about why callers leave there, not whether they do.

Stopping a wrong fix is often worth more than suggesting a right one. A wrong fix costs the rewrite, the testing, the redeployment, and the weeks before anyone notices it didn’t help — and it leaves the real cause in place the whole time.

All of that assumes a step-by-step call flow. An agent built as a single prompt has no steps to pin a hangup to. For those, the audit still classifies every outcome and follows every lead through to the send, but its suggestions point at a section of the prompt or a tool’s settings rather than at a step.

It runs whether or not anyone remembers to run it

Every week, unattended: the system gathers the period’s calls, reconciles them against the supporting systems’ own records, and produces a structured findings document plus two written summaries — one carrying the full technical evidence, one in plain language for reading without it — and a short email to us with the top three things to change for each client. All of it is internal. Clients don’t receive audit output; they get the fixes it leads to. Both summaries are assembled from the findings document by fixed templates; no model writes them. The only model-worded text in either is the rows a model grouped on the list of things to change, marked as such in the technical summary. Everything is committed to version control, so the history is the record. There’s no dashboard to maintain and no separate database that can quietly drift out of agreement with reality.

And every run is compared against history — carefully, because that comparison is subtler than it looks. The audit examines a rolling multi-week window, which means an issue that stopped occurring still appears for a while simply because older calls remain in view. A naive before-and-after would report nothing at all in the exact week an improvement landed.

So the system asks a sharper question: did this recur in the new time since the last run? A standing issue is reported as still occurring as of a specific date — or as quiet for nineteen days, which is the first evidence of an improvement, and not proof of one.

Two rules keep that honest:

• Something disappearing is not something fixed. An issue only counts as quiet if it didn’t come back and callers actually reached that point in the meantime; if nobody passed that way, the audit says it can’t tell. Even then, quiet isn’t fixed. With no change on record it’s reported as quiet, no fix recorded — “fixed” is reserved for a change we actually published, and even that waits for the quiet to confirm it. An issue whose last occurrence falls out of the window altogether is reported as aged out, and never counted as a fix.
• A check that stops working has not improved. If something that used to be measurable becomes unmeasurable, that’s reported as lost visibility and ranked with the regressions. A check that has gone dark is worse than one that’s failing, because a failure is at least visible.

And the weekly summary leads with what changed, because a report that says the same thing ten weeks running is a report nobody opens.

Measured against the right definition of success

What counts as a good outcome is a business judgment, not a technical one — so it’s written down in plain language that can be argued with, and it’s one definition for every client.

A lead is not the only good outcome. Six count, each with a condition that has to be met:

• A lead taken and sent on to the business.
• A lead that failed to send, where a message for the same caller was sent instead. The caller was still served, and the failed lead stays on the list as a problem of its own.
• A message taken for staff and sent on.
• A call where nothing was sent while it was happening, but the follow-up afterwards emailed the business. That call counts as recovered.
• A caller correctly told they’re outside the service area. They got a fast, honest answer, and the business didn’t waste a site visit.
• A caller correctly told a service isn’t offered.

Failure is narrower and far more useful: a caller who could have been helped and reached none of those outcomes — or reached one that never completed, like a lead the agent took that was never sent.

The headline number is the share of calls handled correctly, out of the calls that could have been handled. Spam, dead air, calls dropped before the agent ran, and callers who hang up before the call connects are left out. It is deliberately not a conversion rate.

Measuring only lead conversion misclassifies a large amount of entirely correct agent behavior as failure — and buries the callers who genuinely got nothing among the ones who were served properly.

The same rule decides where the agent is said to lose callers. A caller who hangs up after being correctly told they’re outside the service area isn’t counted as lost; the call went the way it should. A caller the follow-up recovered after the call still is, because the agent did lose them. That’s deliberate: the headline measures whether the business served the caller, and the drop-off analysis measures whether the agent did. A recovered call passes the first and fails the second.

Two layers, and a hard line between them

The mechanical layer handles everything with a checkable answer: where a caller left the conversation, which stage of intake they reached, whether an action completed. All of it is derived from structured records and the agent’s own configuration — the intake sequence itself isn’t written down anywhere; it’s re-derived every run from the agent’s recorded conversation flow, so it follows every recorded edit. No judgment, no model, the same answer every time it runs. That reproducibility is what makes week-over-week comparison mean anything.

The AI layer does two narrow jobs.

The first is describing the calls the structure can’t explain. On a well-structured step-by-step agent, the mechanical layer resolves everything and this has nothing to do at all. It engages only where no mechanical rule can classify a call — a low single-digit percentage of calls overall, nearly all of them on agents built as a single prompt. Its description explains why a call ended the way it did; it never changes how the call counts. A call no rule can explain stays a failure in the headline number, listed on its own so someone reads it. That errs on the hard side on purpose: a number that quietly counts what it can’t explain as handled is how a real failure hides. Its answers are constrained to a fixed vocabulary so they stay countable across periods, checked against a fixed reference set so drift is detectable, and stamped with the exact version of the instructions that produced them — so a trend is never a silent blend of two different instruments.

It’s also required to be able to say “I don’t know,” and it does, by design. A classifier that always returns a confident answer gives you no signal when it’s wrong.

The second is checking the agent’s own writing. Every call that sent a lead or a message is reviewed once: does what the agent wrote down for the business match what the caller actually said? A statement the conversation doesn’t support — a detail in the project description the caller never gave, say — is flagged with the quotation and the reason. Flags that share a cause are grouped into a single row on our list of things to change. The row’s size is the number of calls flagged, not a number the model chose, and the row is marked as worded by a model.

The rule between the layers is absolute: no model gets the final word. No model sets a pass or a fail, decides an outcome, or rules on whether a lead was delivered. What a model contributes lands on our engineering list, marked as its own, and nothing on that list changes an agent until a person approves it.

That rule governs measurement. On the change side, covered below, it takes a different form: AI proposes, and a person decides.

A few more things the system does that most measurement setups don’t:

• Problem points are ranked by statistical confidence, not raw rate. Nine dropouts out of sixteen is 56% — and it’s also four calls from 31%. Ranking on raw percentage sends you off to fix a coin flip. A conservative lower bound is used instead.
• It flags its own circular reasoning. When a caller hangs up, the agent’s final sentence is cut off mid-word — so “the agent didn’t finish speaking” may be describing the hangup, not explaining it. Where that’s possible, the finding is recorded as an open question with both readings stated, never as a conclusion.
• It follows the chain end to end — from what the agent collected, through the automation behind it, to the step that sends it on to the business — rather than stopping at whether the conversation went well.
• Almost nothing is set up per client. The facts it needs — the call flow, the tools, where each lead is sent, the service area — are read from the records already maintained to run the agent. What’s kept for each client is a short file: extra checks, findings we’ve decided are expected for that business (a blank territory line for a company with no territories, say — still reported, but marked accepted), and a record of the fixes made against the list, so each problem’s status follows from what was actually done.

The audit observes. It never acts.

The scheduled audit is read-only by construction. It examines, reconciles, and reports. It has no ability to modify an agent, alter an automation, or contact a customer — not as a matter of policy, but structurally: the capability isn’t present to be misused. Its job ends at “here is what the evidence shows, and here is what I could not determine.”

That separation is deliberate. The thing that measures must not be the thing that changes what it measures.

Acting on the evidence

This is where the audit’s output stops being a report and becomes a specific change.

When we act on the findings, the work happens inside the governed project repository, where an AI engineering system works with the live agent configurations, the committed baselines of every prompt and flow, the backend automation definitions, the verified record of how the underlying platforms actually behave, and the full history of every previous change and why it was made. It reads the findings and formulates targeted remediations: a specific rewording of a specific step, a configuration adjustment, a correction to how a backend workflow routes or validates data.

This is the part people mean when they say “we use AI to improve the agent” — and it’s worth being precise, because the phrase covers two very different activities.

Asking a chatbot to write a better prompt is guesswork. It has no access to what the agent currently says, no evidence about where callers actually drop off, no record of what’s already been tried, and no way to verify its suggestion survived contact with production.

This is an engineering environment. The proposed change is written against the actual current configuration, justified by measured evidence from real calls, checked for consistency with every documented fact about the business, and validated against what the platform genuinely does rather than what its documentation implies.

And every change is gated, recorded, and reversible

The engineering layer proposes. A person decides. Before anything reaches a live agent or a live automation:

• The current live state is read first. Never assume the stored copy matches what’s actually running; if they differ, that’s reconciled before anything is touched.
• Changes are surgical. What was asked for, and nothing else. No tidying, no rewording of adjacent text, no improvements nobody requested.
• A diff is presented for approval. The exact change is reviewed in context — line by line, before it happens — never a description of the change, the change itself.
• Nothing proceeds without an explicit yes. Required on every production change — prompt, configuration, or backend.
• The change is read back after pushing to confirm it actually applied, rather than assuming the write succeeded.
• Everything is versioned and logged with the date and the reason, so any change can be reversed from a recorded restore point and anyone can reconstruct why it was made months later.
• Quality checks run before a change counts as done. Every test agent is checked mechanically for isolation before it takes a single call, and running tests against live agents or live automations is prohibited outright.

The human decision is binary — yes or no — but it sits on every change, and it’s made against a diff rather than a description.

The bottom line

An AI voice agent that nobody measures isn’t a system. It’s a hope. Our clients’ agents are measured every week, automatically, against an explicit definition of what a good outcome actually is — and when the evidence supports a change, that change is engineered against the real configuration, approved against a diff, and reversible in one step. Our clients don’t get a report to read. They get an agent that keeps getting fixed, with the record of why kept on our side.