
How to Measure AI Agent Performance: KPIs That Actually Matter
An agency sends a client the first monthly report for their new AI agent. The headline says the agent handled 2,140 conversations. The client replies with one question: "Great, but did it get us any customers?" Nobody on the team can answer it.
This happens because most dashboards track activity instead of results. Conversation counts and message totals look impressive, yet they say nothing about whether customers got help or whether leads turned into sales. Some popular metrics even count frustrated customers as wins.
This guide explains which KPIs matter for support, lead and voice agents, how to calculate each one, what good results look like and how to put them into a simple monthly report.
Why Chat Volume and Containment Mislead
Containment rate is the share of conversations that never reach a person. Many tools show it first because it is easy to measure and usually high. The problem is that a customer who gives up and closes the chat also counts as "contained." According to research cited by Ada, up to 20% of contained tickets in one study were drop-offs where the bot had not helped.
A second issue is self-grading. Some platforms let their own AI decide which conversations count as resolved, and that label may also drive the monthly bill. That does not make the number wrong, but it does mean it needs an independent check.
The fix is simple. Pair every main KPI with a guard metric that would expose it if it were being gamed.
Start With the Agent's Job
The right KPIs depend on what the agent was built to do. A support bot, a lead capture bot and a phone agent succeed in different ways.
Pick the row that matches your agent and treat everything else as supporting detail. An agent that does more than one job gets one main KPI per job.
Support Agent KPIs
Support is where most businesses start, and where AI customer support software usually takes the biggest load off a team. These five measures show whether that load is being handled well.
Automated Resolution Rate
Formula: conversations resolved without a person ÷ total conversations
This is the clearest sign that the agent is doing its job. Benchmarks vary widely depending on who publishes them. Intercom's Fin team calls 55% to 70% typical for routine support, while other vendors call anything from 30% to 60% good. Most of the gap comes from how each vendor defines a resolution.
Reopen and Repeat-Contact Rate
Formula: "resolved" conversations where the customer returns about the same issue within 48 hours ÷ all resolved conversations
This guard metric catches answers that sounded complete but did not solve anything. Intercom's Fin team treats a rate below 10% as strong. If resolution rises while reopens rise with it, the agent is closing conversations too early.
Handoff Rate and Handoff Quality
The handoff rate is the share of conversations passed to a person. A falling rate is good news only if the right chats still reach staff. Billing disputes, cancellations and anything involving safety should still reach a person quickly.
Quality matters as much as the number. Check a few handoffs each week and ask one question: did the team member get the full conversation, or did the customer have to start again? Passing full context at handoff is one of the oldest chatbot best practices, and still one of the most often skipped.
Unanswered-Question Rate
Formula: replies where the agent says it does not know ÷ total replies
This number points straight at gaps in the content the agent learns from. Group the unanswered questions by topic each month and add the missing answers through your AI knowledge base software. Review the list again the following month to confirm those topics have dropped off.
Sentiment and CSAT
Post-chat surveys reach only a small share of customers, and the ones who answer tend to be very happy or very upset. Sentiment tracking reads every conversation and tags it as positive, neutral or negative. Use both: CSAT for the customer's own words, sentiment for the full picture.
Lead and Booking KPIs
Sales-focused agents are judged by how many conversations move down the funnel. Each step below is measured against the one before it:
Conversations to qualified leads: qualified leads ÷ conversations. Shows whether the agent asks the right questions.
Qualified leads to bookings: bookings ÷ qualified leads. Shows whether the offer and booking flow work.
Bookings to shows: attended appointments ÷ bookings. This is the guard metric, since an agent can book plenty of calls that never happen.
Shows to customers: new customers ÷ attended appointments. This step depends partly on the sales team, though weak leads show up here too.
Add one more number alongside the funnel: speed to lead, the time between a visitor's first message and the first useful reply. A reply within seconds is one of the main reasons businesses use AI lead capture software, so a slow speed to lead usually points to a setup or integration problem.
Voice Agent KPIs
Phone calls bring problems that chat metrics miss. These measures cover them:
Response delay: the pause between a caller finishing a sentence and the agent replying. Long silences make callers talk over the agent or hang up.
Interruption handling: whether the agent stops speaking when the caller cuts in. Listen to a handful of calls to check this.
Transfer rate: the share of calls sent to a person. Read it alongside the reason for each transfer.
Call abandonment rate: the share of callers who hang up before their issue is handled. This is the guard metric for voice.
Average handle time: how long calls last. Shorter is better only when the issue still gets resolved.
Repeat rate: how often callers have to say the same thing twice, often a sign of poor speech recognition or unclear questions.
Teams using AI voice agent software for inbound calls should review call recordings, not just transcripts. Tone, pauses and crosstalk only show up in the audio.
Cost Per Resolution
Formula: total monthly AI cost ÷ automated resolutions
Take a business with 1,000 conversations a month and a 60% resolution rate, or 600 resolved conversations. If the AI costs $300 that month, each resolution costs $0.50.
To compare that with people, work out your own cost per conversation. A team member paid $30 an hour who handles 10 chats an hour costs $3 per chat. On those numbers, the AI handles routine questions at a sixth of the cost.
If the reopen rate is high, divide the AI cost by resolutions that stayed resolved instead. That gives the true figure, and it is often noticeably higher.
KPI Benchmarks at a Glance
The ranges below come from Chaindesk's 2026 chatbot benchmarks. They are vendor figures without independent studies behind them, so treat them as starting points rather than fixed targets.
Industry changes the picture. Healthcare and finance agents usually hand off more often by design, because some questions should always reach a person.
How to Review Conversations When Volume Is Low
Large companies use AI tools to score every conversation. A small business or a new agency client rarely needs that. A weekly sample works well:
Pick 25 conversations at random from the past week.
Score each one yes or no on four checks: accurate, resolved, handed off when it should have been, and right tone.
Tag every "no" with its cause: missing knowledge, unclear instructions or a broken integration. Unclear instructions are often fixed fastest by starting from proven AI customer service prompt templates.
Fix the most common cause first, then pull a fresh sample next week.
Twenty-five conversations take under an hour to review, and patterns show up within two or three weeks.
A Simple Monthly KPI Report
A useful report fits on one page. These six lines cover what an owner or client needs to know:
Main KPI for each agent job: this month's number
Guard metric for each: this month's number
Change from last month: up, down or flat, with a one-line reason
Top three unanswered questions: and whether they have been fixed
Cost per resolution: this month's figure
One action for next month: a single, specific change
Agencies can send the same report to every client each month, which makes progress easy to compare. Those using white-label AI agent software can deliver it under their own brand from the same dashboard their clients log into.
Frequently Asked Questions
What is a good resolution rate for an AI agent?
It depends on the business and the type of questions. A store answering order and return questions can reach a high rate quickly, while a business with complex or custom requests will settle lower. Your own trend over three to six months matters more than any industry number.
How often should I review AI agent KPIs?
Check the main numbers weekly during the first month, then monthly once they settle. Review sooner after any big change, such as a new product line or policy.
Should I trust the resolution rate my vendor reports?
Use it, but ask three questions first: how a resolution is defined, whether abandoned chats count as resolved, and whether answers are checked for accuracy. Clear answers make the number far more useful.
What tools do I need to track these KPIs?
Most AI agent platforms show conversations, handoffs and unanswered questions out of the box. The lead funnel often needs a CRM or a simple spreadsheet that joins chat data with bookings and sales.
How do I compare AI and human agents fairly?
Score both on the same checklist, and compare them on similar question types. AI usually handles the simple questions, so comparing its numbers with staff who get the hardest cases will mislead.
How long until the numbers settle?
Most agents need four to eight weeks of real conversations before the trends become reliable. Early weeks swing a lot as content gaps get filled.
Final Thoughts
Before the first report goes out, write down a target for each main KPI and agree on it with whoever reads the report. Measuring against a target set in advance keeps the review honest, because a number can only be good or bad compared with what you expected it to be.

