
TL;DR
If you're deciding how to evaluate AI agents, set vendor shortlists aside. A buyer scorecard ranks what the agent did in your pilot, on your data, with your customers.
The scorecard has six lines. They cover actions in core systems, context across channels, unknown answers, compliance evidence, handoff continuity, and change control.
The context test carries the most weight. Ask the agent on a second channel about a chat from last week and see what it remembers.
Five people have to say yes before go-live. Bring all five into the evaluation in week one.
Gartner expects more than 40% of agentic AI projects to be canceled by the end of 2027. The firm made that call in a June 2025 forecast, and it named rising costs, unclear business value, and weak risk controls as the causes. You can check all three before you sign.
To evaluate AI agents for a bank, a lender, a hospital group, or a dealer network, run a pilot of your own. Score each vendor on what its agent does there, with your systems, your rules, and your customers. A buyer scorecard records whether the agent finishes real work, remembers the customer across channels, and gives your security team evidence it can sign.
A demo shows the best ten minutes a vendor has. This guide gives you the scorecard, the tests behind each line, and the list of people who need to approve the result.
How to evaluate AI agents when every shortlist ranks its author first
Ranked shortlists mislead buyers because the vendor that wrote the list usually sits at the top of it. The ranking measures features on a page. It can't measure how an agent behaves inside your loan system or your appointment book.
The second problem is the label. In the same June 2025 forecast, Gartner estimated that only about 130 of the thousands of vendors selling agentic AI offer the real thing. It calls the rest agent washing, which means older assistants and automation tools sold under a new name.
A list can't tell you which kind you're looking at. A pilot can. What helps is one sheet that scores every vendor the same way, including the ones that publish lists.
If you want a map of the market before you score anyone, read our overview of the AI agent landscape first. Then come back with three or four names.
A shortlist tells you who is selling. A scorecard tells you what works in your environment.
How is AI agent evaluation for buyers different from model evals?
AI agent evaluation means two different jobs, and most search results cover the one you don't need. Engineering teams run model evals. They measure accuracy, tool calls, and response time against a test set while they build an agent.
You're buying a finished agent, so your evaluation asks business questions.
Did the customer get an answer? Did the booking land in the system? Can compliance read what happened afterward?
Ask the vendor for its model eval results, and treat them as background. Your decision rests on the six criteria below.
What goes on a buyer scorecard for AI agents?
The buyer scorecard has six criteria, and each one maps to a test you run during the pilot. Score every vendor from 0 to 5 on each line, multiply by the weight, and add the totals.
The weights below are a starting point. Change them to match your risk, then keep them fixed for every vendor.
| Criterion | What you test | Weight | A pass looks like |
|---|---|---|---|
| Context across channels | Whether the agent remembers earlier conversations on another channel | 25 | The agent picks up last week's chat without asking the customer to repeat it |
| Actions in core systems | Whether the agent completes the task in your CRM, loan system, or scheduling tool | 20 | The record changes, and you can see who changed it and when |
| Unknown answers and guardrails | What the agent does when it lacks an answer or meets a restricted topic | 15 | The agent says it doesn't know, stays inside your rules, and routes the case |
| Compliance and security evidence | Whether the vendor can document data flows, access, retention, and audit logs | 15 | Your security team gets written answers and reports in week one |
| Change control and time to live | How long a new use case, script change, or policy update takes to reach production | 15 | Your team ships a tested change in days, with a rollback path |
| Handoff continuity | What travels with the conversation when it moves to a person or another agent | 10 | The receiver sees the full history and the reason for the transfer |
Two rules keep the scoring honest. Score from pilot evidence only, never from slides. Have two people score each line separately, then compare.
The scorecard has no line for price on purpose. Commercial terms belong in a separate track with procurement, after you know which agents pass.
How do you test whether an AI agent remembers context across channels?
You test context retention by talking to the agent on one channel, waiting a week, and returning on another. It's the test most demos skip, and it carries the most weight on the scorecard.
Picture one customer and three doors. She messages on WhatsApp, calls the contact center, then walks into a branch. In many organizations, she starts from zero at each door.
An agent that can't see the earlier conversation adds a fourth door. To find out before your customers do, run this script during the pilot.
Day 1, WhatsApp. Ask about a specific product, share one detail such as a loan amount or a preferred clinic, and stop mid-conversation.
Day 3, web chat. Return signed in as the same customer. Ask, "Where were we?"
Day 8, voice. Call and ask the agent to change the detail you gave on day 1.
Day 8, your CRM. Open the customer record and check that all three conversations sit on one timeline.
The agent passes when it recalls the product and the detail on days 3 and 8 without asking again. It fails when it asks for the customer's name, need, or number a second time. A partial pass is common, where chat remembers chat but voice starts cold.
Write down what the agent stored and where.
Memory inside one channel is session history. Memory across channels needs a shared customer record that every agent reads before it replies and updates afterward. The Conversation Graph is Zigment's version of that record, and our guide to keeping context across every channel explains why it matters.
If the agent can't answer "where were we?" on a second channel, your customer will have to.
Three more tests to run in the pilot
Three more tests cover the rest of the scorecard. They check whether the agent finishes work, how it behaves at the edge of its knowledge, and whether the vendor can prove its controls. Run each one on real traffic where you can.
Can the agent finish the job in your core systems?
An agent that finishes the job changes a record. It books the test drive, updates the loan application, or reschedules the appointment in the system your team already uses. An agent that only answers questions leaves that work for someone else.
To test this, pick one task with a clear end state and run it 20 times. Count how many runs ended with the right record in the right system and no manual fix. Then ask the vendor to show the log for one run, step by step.
The difference between answering and acting is the core of agentic AI versus conversational AI. Your scorecard should reward the second kind.
What does the agent do when it doesn't know?
A reliable agent admits what it doesn't know and stays inside your rules. To check, ask ten questions that sit outside its knowledge base. Add five it should refuse, such as a request for a guaranteed approval or for medical advice.
Score three behaviors. The agent says it doesn't have the answer. It avoids inventing a rate, a policy, or a diagnosis.
It also routes the customer to the right next step and records why.
For regulated products, ask how you set those boundaries and who can change them. You should be able to configure them once and read them back in plain language.
Can the vendor show compliance evidence in week one?
A vendor that's ready for a regulated buyer hands over evidence before you ask twice. The need is real.
In IBM's 2025 Cost of a Data Breach Report, which studied 600 organizations, 13% reported a breach of an AI model or application. IBM found that 97% of those organizations lacked proper AI access controls.
Buyers have gaps of their own. In an April 2026 Grant Thornton survey of about 1,000 business leaders, 78% lacked full confidence that they could pass an independent AI governance audit within 90 days.
To keep security review from becoming the last gate, ask each vendor for these items with the first proposal.
A data-flow diagram. It shows where conversation data goes, where it's stored, and which model providers see it.
Access controls. Who can read conversations, and how the vendor grants and removes that access.
Retention and deletion. How long the vendor keeps data, and how you erase one customer's record.
An audit log. A record of every action the agent took, with the reason.
Current security reports. Independent audit and penetration test reports, dated within the last year.
Our guide on how to secure agentic AI systems goes deeper on each control.
Who has to say yes before an AI agent goes live?
Five people usually have to say yes before an AI agent goes live in an enterprise. Each one scores a different part of the sheet, and any one of them can stop the project.
That matches the wider pattern. Gartner's 2025 survey of 632 B2B buyers found buying groups of 5 to 16 people across as many as four functions. In that survey, 74% of buyer teams showed what Gartner calls unhealthy conflict during the decision.
The economic buyer. Owns the budget and asks what business number moves.
Infosec. Asks where conversation data lives, who can read it, and how long you keep it.
IT. Asks how the agent connects to core systems and who maintains the connection.
The business team. Runs the process today and asks what changes on Monday morning.
The metrics owner. Defines success and asks how you'll measure it against the current baseline.
To avoid a late veto, give each of the five a line on the scorecard in week one. Infosec scores compliance evidence, IT scores actions and change control, and the business team scores context and handoff.
People who helped set the test tend to accept the result. In a bank, add compliance as a sixth voice from the start. Our piece on conversational AI in banking journeys shows where that review usually bites.
A pilot scored by the business team alone still has four reviews to pass.
What should a pilot prove before production?
A pilot should prove that the agent passes your scorecard on one real use case, under real traffic, against numbers you agreed on before it started. Anything less is a longer demo.
The pressure to skip this step is high. In a Gartner survey of 321 customer service leaders, published in February 2026, 91% said executives were pressing them to implement AI this year. Clear gates let you move fast and still say no.
Choose one use case with a countable outcome. Loan applications completed, appointments booked, or test drives scheduled all work.
Record the baseline. Capture the current number for the same period, channel, and customer segment.
Set pass marks before day 1. Agree on a minimum weighted score and a floor for each line, so one strong area can't hide a failed one.
Run on real customers. Send a share of live traffic to the agent for at least two full business cycles.
Hold the review with all five approvers. Each one signs their own line, or names what's missing.
If the vendor passes, expand by one use case at a time and run the sheet again. For the weeks after sign-off, see our eight-week implementation plan.
Where does Zigment fit on this scorecard?
Zigment is a Conversational Revenue Orchestration Platform for GTM teams, and you should score it on the same sheet as everyone else. Its AI agents qualify, convert, and retain customers across WhatsApp, voice, and web.
Zigment is built around the context line. The Conversation Graph keeps one timeline per customer across every channel, so an agent on voice can read what the customer said on WhatsApp last week. The agents also act in the systems you already run, inside boundaries your team configures once.
Two published results show what that looks like in practice. Zigment's case study with Jana Small Finance Bank reports that the bank disbursed 56% more loans from the same approved cohort after moving borrowers onto WhatsApp conversations. Its Tata Motors case study reports 35% more test drives booked once buyers got answers the moment they asked.
Don't take those numbers as your score. Run the day 1, day 3, and day 8 script on us too.
Score the agent you'll live with
A ranked list ages the day it's published. A scorecard you built with your own approvers keeps working for the next vendor, the next use case, and the next renewal.
So before your next demo, send the vendor one question. What will your agent remember about my customer a week from now, on a different channel?
To run the context test on a live Zigment agent, book a Zigment demo and bring the conversation your customers repeat most.
