# How to Evaluate AI Agents for Customer Conversations: A Buyer Scorecard
Author: Jatin Shah
Author URL: https://www.zigment.ai/blog/author/jatin-shah
Published: 2026-11-19
Category: Agentic AI
Category URL: https://www.zigment.ai/blog/category/agentic-ai
Meta Title: How to Evaluate AI Agents: A Buyer Scorecard for 2026
Meta Description: How to evaluate AI agents for customer conversations: a six-line buyer scorecard, a cross-channel context test, and the five people who must say yes.
Tags: Enterprise AI, context preservation, ai agents, BFSI
Tag URLs: Enterprise AI (https://www.zigment.ai/blog/tag/enterprise-ai), context preservation (https://www.zigment.ai/blog/tag/context-preservation), ai agents (https://www.zigment.ai/blog/tag/ai-agents), BFSI (https://www.zigment.ai/blog/tag/bfsi)
URL: https://www.zigment.ai/blog/how-to-evaluate-ai-agents-buyer-scorecard

![Bold poster cover on coral red: a giant cobalt checkmark standing on its point beside the headline score the agent, not the demo, for a guide on how to evaluate AI agents.](https://prod.superblogcdn.com/site_cuid_cm7ah6s1d005z13xnfz2oplu9/images/00-hero-1795049896160-compressed.webp)

**TL;DR**

- If you're deciding how to evaluate AI agents, set vendor shortlists aside. A buyer scorecard ranks what the agent did in your pilot, on your data, with your customers.

- The scorecard has six lines. They cover actions in core systems, context across channels, unknown answers, compliance evidence, handoff continuity, and change control.

- The context test carries the most weight. Ask the agent on a second channel about a chat from last week and see what it remembers.

- Five people have to say yes before go-live. Bring all five into the evaluation in week one.


Gartner expects more than 40% of agentic AI projects to be canceled by the end of 2027. The firm made that call in [a June 2025 forecast](https://www.gartner.com/en/newsroom/press-releases/2025-06-25-gartner-predicts-over-40-percent-of-agentic-ai-projects-will-be-canceled-by-end-of-2027), and it named rising costs, unclear business value, and weak risk controls as the causes. You can check all three before you sign.

To evaluate AI agents for a bank, a lender, a hospital group, or a dealer network, run a pilot of your own. Score each vendor on what its agent does there, with your systems, your rules, and your customers. A buyer scorecard records whether the agent finishes real work, remembers the customer across channels, and gives your security team evidence it can sign.

A demo shows the best ten minutes a vendor has. This guide gives you the scorecard, the tests behind each line, and the list of people who need to approve the result.

**40%+** of agentic AI projects Gartner expects to be canceled by the end of 2027 (Gartner, 2025)

**5 to 16** people in a typical B2B buying group (Gartner, 2025)

**97%** of organizations with an AI breach lacked AI access controls (IBM, 2025)

## How to evaluate AI agents when every shortlist ranks its author first

Ranked shortlists mislead buyers because the vendor that wrote the list usually sits at the top of it. The ranking measures features on a page. It can't measure how an agent behaves inside your loan system or your appointment book.

The second problem is the label. In the same June 2025 forecast, Gartner estimated that only about 130 of the thousands of vendors selling agentic AI offer the real thing. It calls the rest agent washing, which means older assistants and automation tools sold under a new name.

A list can't tell you which kind you're looking at. A pilot can. What helps is one sheet that scores every vendor the same way, including the ones that publish lists.

If you want a map of the market before you score anyone, read [our overview of the AI agent landscape](https://www.zigment.ai/blog/the-landscape-of-ai-agents "The Landscape of AI Agents: Finding the Right Platform") first. Then come back with three or four names.

A shortlist tells you who is selling. A scorecard tells you what works in your environment.

## How is AI agent evaluation for buyers different from model evals?

AI agent evaluation means two different jobs, and most search results cover the one you don't need. Engineering teams run model evals. They measure accuracy, tool calls, and response time against a test set while they build an agent.

You're buying a finished agent, so your evaluation asks business questions.

Did the customer get an answer? Did the booking land in the system? Can compliance read what happened afterward?

Ask the vendor for its model eval results, and treat them as background. Your decision rests on the six criteria below.

![Six clay columns show how to evaluate AI agents on a weighted buyer scorecard: context across channels 25 in red-orange, actions in core systems 20, guardrails, compliance evidence and change control 15 each, handoff continuity 10.](https://prod.superblogcdn.com/site_cuid_cm7ah6s1d005z13xnfz2oplu9/images/01-context-carries-the-most-composed-1795049899317-compressed.jpg)

## What goes on a buyer scorecard for AI agents?

The buyer scorecard has six criteria, and each one maps to a test you run during the pilot. Score every vendor from 0 to 5 on each line, multiply by the weight, and add the totals.

The weights below are a starting point. Change them to match your risk, then keep them fixed for every vendor.

CriterionWhat you testWeightA pass looks likeContext across channelsWhether the agent remembers earlier conversations on another channel25The agent picks up last week's chat without asking the customer to repeat itActions in core systemsWhether the agent completes the task in your CRM, loan system, or scheduling tool20The record changes, and you can see who changed it and whenUnknown answers and guardrailsWhat the agent does when it lacks an answer or meets a restricted topic15The agent says it doesn't know, stays inside your rules, and routes the caseCompliance and security evidenceWhether the vendor can document data flows, access, retention, and audit logs15Your security team gets written answers and reports in week oneChange control and time to liveHow long a new use case, script change, or policy update takes to reach production15Your team ships a tested change in days, with a rollback pathHandoff continuityWhat travels with the conversation when it moves to a person or another agent10The receiver sees the full history and the reason for the transfer

Two rules keep the scoring honest. Score from pilot evidence only, never from slides. Have two people score each line separately, then compare.

The scorecard has no line for price on purpose. Commercial terms belong in a separate track with procurement, after you know which agents pass.

![Four clay pedestals on one rail lay out the cross-channel context test: WhatsApp on day 1, web chat on day 3, a voice call on day 8 in red-orange, and the CRM timeline check on day 8.](https://prod.superblogcdn.com/site_cuid_cm7ah6s1d005z13xnfz2oplu9/images/02-one-customer-three-chann-composed-1795049901123-compressed.jpg)

## How do you test whether an AI agent remembers context across channels?

You test context retention by talking to the agent on one channel, waiting a week, and returning on another. It's the test most demos skip, and it carries the most weight on the scorecard.

Picture one customer and three doors. She messages on WhatsApp, calls the contact center, then walks into a branch. In many organizations, she starts from zero at each door.

An agent that can't see the earlier conversation adds a fourth door. To find out before your customers do, run this script during the pilot.

1. **Day 1, WhatsApp.** Ask about a specific product, share one detail such as a loan amount or a preferred clinic, and stop mid-conversation.

2. **Day 3, web chat.** Return signed in as the same customer. Ask, "Where were we?"

3. **Day 8, voice.** Call and ask the agent to change the detail you gave on day 1.

4. **Day 8, your CRM.** Open the customer record and check that all three conversations sit on one timeline.


The agent passes when it recalls the product and the detail on days 3 and 8 without asking again. It fails when it asks for the customer's name, need, or number a second time. A partial pass is common, where chat remembers chat but voice starts cold.

Write down what the agent stored and where.

Memory inside one channel is session history. Memory across channels needs a shared customer record that every agent reads before it replies and updates afterward. [The Conversation Graph](https://www.zigment.ai/blog/what-is-the-conversation-graph "What Is the Conversation Graph and Why Your Stack Needs One") is Zigment's version of that record, and our guide to [keeping context across every channel](https://www.zigment.ai/blog/customer-personalization-strategy "Personalization Strategy Beyond the Basics: Keeping Context Across Every Channel") explains why it matters.

If the agent can't answer "where were we?" on a second channel, your customer will have to.

## Three more tests to run in the pilot

Three more tests cover the rest of the scorecard. They check whether the agent finishes work, how it behaves at the edge of its knowledge, and whether the vendor can prove its controls. Run each one on real traffic where you can.

### Can the agent finish the job in your core systems?

An agent that finishes the job changes a record. It books the test drive, updates the loan application, or reschedules the appointment in the system your team already uses. An agent that only answers questions leaves that work for someone else.

To test this, pick one task with a clear end state and run it 20 times. Count how many runs ended with the right record in the right system and no manual fix. Then ask the vendor to show the log for one run, step by step.

The difference between answering and acting is the core of [agentic AI versus conversational AI](https://www.zigment.ai/blog/agentic-ai-vs-conversational-ai-choosing-the-best-solution "Agentic AI vs. Conversational AI"). Your scorecard should reward the second kind.

### What does the agent do when it doesn't know?

A reliable agent admits what it doesn't know and stays inside your rules. To check, ask ten questions that sit outside its knowledge base. Add five it should refuse, such as a request for a guaranteed approval or for medical advice.

Score three behaviors. The agent says it doesn't have the answer. It avoids inventing a rate, a policy, or a diagnosis.

It also routes the customer to the right next step and records why.

For regulated products, ask how you set those boundaries and who can change them. You should be able to configure them once and read them back in plain language.

### Can the vendor show compliance evidence in week one?

A vendor that's ready for a regulated buyer hands over evidence before you ask twice. The need is real.

In IBM's 2025 Cost of a Data Breach Report, which studied 600 organizations, 13% reported a breach of an AI model or application. [IBM found that 97% of those organizations](https://newsroom.ibm.com/2025-07-30-ibm-report-13-of-organizations-reported-breaches-of-ai-models-or-applications,-97-of-which-reported-lacking-proper-ai-access-controls) lacked proper AI access controls.

Buyers have gaps of their own. In an [April 2026 Grant Thornton survey](https://grantthornton.com/insights/press-releases/2026/april/grant-thornton-survey-on-ai-proof-gap) of about 1,000 business leaders, 78% lacked full confidence that they could pass an independent AI governance audit within 90 days.

To keep security review from becoming the last gate, ask each vendor for these items with the first proposal.

- **A data-flow diagram.** It shows where conversation data goes, where it's stored, and which model providers see it.

- **Access controls.** Who can read conversations, and how the vendor grants and removes that access.

- **Retention and deletion.** How long the vendor keeps data, and how you erase one customer's record.

- **An audit log.** A record of every action the agent took, with the reason.

- **Current security reports.** Independent audit and penetration test reports, dated within the last year.


Our guide on [how to secure agentic AI systems](https://www.zigment.ai/blog/how-to-actually-secure-your-agentic-ai-systems "How to Actually Secure Your Agentic AI Systems In 2026") goes deeper on each control.

![Five clay cylinders on one rail show the five approvers for an AI agent and the scorecard line each owns: economic buyer, infosec in red-orange, IT, the business team, and the metrics owner.](https://prod.superblogcdn.com/site_cuid_cm7ah6s1d005z13xnfz2oplu9/images/03-five-approvers-each-scor-composed-1795049902750-compressed.jpg)

## Who has to say yes before an AI agent goes live?

Five people usually have to say yes before an AI agent goes live in an enterprise. Each one scores a different part of the sheet, and any one of them can stop the project.

That matches the wider pattern. [Gartner's 2025 survey of 632 B2B buyers](https://www.gartner.com/en/newsroom/press-releases/2025-05-07-gartner-sales-survey-finds-74-percent-of-b2b-buyer-teams-demonstrate-unhealthy-conflict-during-the-decision-process) found buying groups of 5 to 16 people across as many as four functions. In that survey, 74% of buyer teams showed what Gartner calls unhealthy conflict during the decision.

- **The economic buyer.** Owns the budget and asks what business number moves.

- **Infosec.** Asks where conversation data lives, who can read it, and how long you keep it.

- **IT.** Asks how the agent connects to core systems and who maintains the connection.

- **The business team.** Runs the process today and asks what changes on Monday morning.

- **The metrics owner.** Defines success and asks how you'll measure it against the current baseline.


To avoid a late veto, give each of the five a line on the scorecard in week one. Infosec scores compliance evidence, IT scores actions and change control, and the business team scores context and handoff.

People who helped set the test tend to accept the result. In a bank, add compliance as a sixth voice from the start. Our piece on [conversational AI in banking journeys](https://www.zigment.ai/blog/conversational-ai-is-changing-customer-journeys-in-banking "How Conversational AI Is Changing Customer Journeys in Banking") shows where that review usually bites.

A pilot scored by the business team alone still has four reviews to pass.

## What should a pilot prove before production?

A pilot should prove that the agent passes your scorecard on one real use case, under real traffic, against numbers you agreed on before it started. Anything less is a longer demo.

The pressure to skip this step is high. In a [Gartner survey of 321 customer service leaders](https://www.gartner.com/en/newsroom/press-releases/2026-02-18-gartner-survey-finds-ninety-one-percent-of-customer-service-leaders-under-pressure-to-implement-ai-in-2026), published in February 2026, 91% said executives were pressing them to implement AI this year. Clear gates let you move fast and still say no.

1. **Choose one use case with a countable outcome.** Loan applications completed, appointments booked, or test drives scheduled all work.

2. **Record the baseline.** Capture the current number for the same period, channel, and customer segment.

3. **Set pass marks before day 1.** Agree on a minimum weighted score and a floor for each line, so one strong area can't hide a failed one.

4. **Run on real customers.** Send a share of live traffic to the agent for at least two full business cycles.

5. **Hold the review with all five approvers.** Each one signs their own line, or names what's missing.


If the vendor passes, expand by one use case at a time and run the sheet again. For the weeks after sign-off, see our [eight-week implementation plan](https://www.zigment.ai/blog/client-assurance-implementation-timeline-slas-training "The Eight-Week Client Assurance Plan for Implementing Agentic AI").

## Where does Zigment fit on this scorecard?

Zigment is a Conversational Revenue Orchestration Platform for GTM teams, and you should score it on the same sheet as everyone else. Its AI agents qualify, convert, and retain customers across WhatsApp, voice, and web.

Zigment is built around the context line. The Conversation Graph keeps one timeline per customer across every channel, so an agent on voice can read what the customer said on WhatsApp last week. The agents also act in the systems you already run, inside boundaries your team configures once.

Two published results show what that looks like in practice. Zigment's [case study with Jana Small Finance Bank](https://www.zigment.ai/case-study/jana-small-finance-bank) reports that the bank disbursed 56% more loans from the same approved cohort after moving borrowers onto WhatsApp conversations. Its [Tata Motors case study](https://www.zigment.ai/case-study/tata-motors) reports 35% more test drives booked once buyers got answers the moment they asked.

Don't take those numbers as your score. Run the day 1, day 3, and day 8 script on us too.

## Score the agent you'll live with

A ranked list ages the day it's published. A scorecard you built with your own approvers keeps working for the next vendor, the next use case, and the next renewal.

So before your next demo, send the vendor one question. What will your agent remember about my customer a week from now, on a different channel?

To run the context test on a live Zigment agent, [book a Zigment demo](https://www.zigment.ai/demo "Book a Zigment demo") and bring the conversation your customers repeat most.
## FAQs
Q: Should you test an AI agent on your own customer conversations or the vendor's demo script?
A: Use your own. A demo is a rehearsed conversation where the agent is on its best behaviour, so ask the vendor to run its agent on a set of your de-identified past conversations, including the messy ones. Where you can, also call or message a live production line the vendor runs for a customer in your industry.

Q: How do you check whether a vendor's AI agent resolution rate is measured honestly?
A: Ask for the written definition of a resolved conversation before you compare numbers. Some vendors count a customer going quiet as a resolution, which hides abandonment. Ask them to split results into conversations the agent closed with a confirmed outcome, conversations deflected to self-service and conversations that reached a person, then apply the same definition to every vendor on your shortlist.

Q: How many test conversations do you need before trusting an AI agent evaluation?
A: Enough to make the rate you care about stable, so do the sample-size arithmetic with your analytics team before the pilot starts. Build two sets: a random sample of real conversations to measure rates, and a hand-picked set of edge cases and high-risk requests to look for failures. Run each scenario more than once, because the same agent can answer the same question differently on a second try.

Q: How do you red-team an AI agent for prompt injection before you sign a contract?
A: Write attack scripts that try to make the agent ignore its instructions, reveal another customer's data, take an action the customer is not entitled to, or follow instructions hidden inside a document or web page it reads. Try them over several turns, since some attacks only work after a few friendly messages. No vendor can promise injection never works, so score them on how they detect, block and log these attempts.

Q: Who should grade the test conversations in an AI agent evaluation?
A: Someone who has no stake in the result, using pass and fail criteria you wrote down before any vendor demo. Grade conversations without the vendor name attached where possible, and have a second reviewer score a share of them to check the two of you agree. If you use an AI model to help grade tone or helpfulness, keep it separate from the agent being tested.

Q: How do you evaluate an AI agent for customers who write in several languages or mix them?
A: Test in every language your customers use, including the code-mixed messages they type in practice, such as Hindi written in Roman script with English terms mixed in. Check that the agent answers in the customer's language, keeps product and policy terms accurate, and collects structured data like names and addresses correctly. For voice, add tests for accents, background noise and customers who interrupt.

Q: What should you ask an AI agent vendor about changes to the underlying language model?
A: Ask which models the agent runs on, how much notice you get before they change, and whether you can hold a version while you retest. Ask the vendor to rerun your test set after swapping the model so you can see what shifts. Put a model-change notice and a retest window into the contract, since a silent model update can change answers your compliance team already approved.

Q: How often should you re-evaluate an AI agent after it goes live?
A: Rerun your saved test set every time the vendor changes a model, you change a prompt, workflow or knowledge source, or a regulation that affects your scripts changes. Between those events, review a sample of live conversations on a fixed schedule and track the same scorecard you used to buy. If any must-pass test drops, pause and fix it before the agent takes more traffic.

Q: Should you evaluate one AI agent or a set of specialist agents for customer conversations?
A: Evaluate whatever will sit in front of the customer as one system, because a customer does not care which agent answered. If the vendor splits work across specialist agents for sales, service or collections, test the moments where one agent passes the conversation to another and check that nothing the customer said gets lost. Zigment, a conversational revenue orchestration platform, handles this with its Conversation Graph, which keeps one context for each customer across agents and channels.

Q: What happens to your conversation data and agent setup if you leave an AI agent vendor?
A: Ask before you sign whether you can export full transcripts, including the agent's replies, along with outcome data and customer attributes, in a standard format and on a schedule. Ask which parts of the setup you can take with you, such as knowledge content, workflows and guardrail rules, and how long you have to retrieve data after the contract ends. Agree exit terms during procurement, because your bargaining position is weakest after go-live.




---
This blog is powered by Superblog. Visit https://superblog.ai to know more.
---

