Salesforce Built a Benchmark for AI Agents. We Scored 87.7%.
Why AI agents struggle inside Salesforce, and what changed when we gave ours one simple rule.
The short version
SCUBA is a benchmark from Salesforce AI Research. It tests whether AI agents can do real work inside Salesforce.
In Salesforce's own paper, the best agent finished 50% of tasks, and that was with a human showing it how.
Given a written test spec, the TestZeus agent finished 87.7% (107 of 122 tasks), graded by SCUBA's own checker.
First, what is a benchmark?
A benchmark is a standard exam for AI. Everyone takes the same test, under the same rules, and gets marked the same way. That makes it possible to compare tools on facts rather than on demos.
Most AI benchmarks test things like maths, coding or browsing simple websites. Almost none test the software that big companies actually run on.
So what is SCUBA?
SCUBA stands for Salesforce Computer Use Benchmark. Salesforce AI Research published it in 2025 to answer one question: can an AI agent use Salesforce the way a person does?
Here is what makes it a tough exam:
300 real tasks. They come from interviews with real Salesforce users: admins, sales reps and service agents.
A real Salesforce org. Every task runs in a live Salesforce environment, wiped clean before each attempt.
No partial credit. A grader looks at what actually changed in Salesforce. Did the record get created? Is the field right? If anything is off, the task fails.
A typical sales task sounds simple: create an account for a new company, add a named person as the contact, and save their mobile number. For a sales rep, that's a minute's work. For an AI agent, it's three different records that all have to link up correctly.
Why does this matter?
AI agents are moving into the systems that run revenue and customer service. Before you let an agent test, or touch, your Salesforce org, you want proof that it gets things right.
SCUBA is useful because nobody can grade their own homework. Salesforce wrote the tasks. Salesforce wrote the grader. The org itself decides pass or fail.
Why AI agents fail on Salesforce
Salesforce isn't a simple web page. It's a platform with its own rules:
Two worlds in one org. Modern Lightning pages sit next to older Setup pages, and they work differently.
Deep tasks. One admin change can cross four or five screens before anything is saved.
Silent failures. A green "Saved" message can vanish in a second. A record can look saved while a related record never got created.
This is where most agents go wrong. They don't crash. They think they are done when they are not, and move on. We call these agentic hallucinations. In testing, that's the worst outcome: a test that says "pass" when something is actually broken.
Who tried SCUBA before us?
The Salesforce team tested a range of AI agents, from open-source models to the best commercial ones. Here's what they found:
Agent | Tasks passed |
Open-source agents, on their own | Under 5% |
Best agent, on its own | 39% |
Best agent, after a human demo of a similar task | 50% |
The lesson from their paper was clear. Agents do much better when they're told how the work is done.
What happened when we ran it
We ran the TestZeus Salesforce testing agent on SCUBA for nine weeks: 3,008 graded runs in live Salesforce orgs.
Given a written test spec, it passed 107 of 122 tasks: 87.7%.
By type of work:
Area | Passed | Score |
Sales (leads, accounts, opportunities, quotes) | 69 of 74 | 93.2% |
Service (cases, escalations, queues) | 28 of 31 | 90.3% |
Admin (deep Setup work) | 10 of 17 | 58.8% |
Sales and service are the flows most teams can't afford to break, and that's where the agent is strongest today. Admin is the hardest part of Salesforce for any agent. We've written specs for only 17 of SCUBA's 171 admin tasks so far, and finishing them is our next job. Our target is the same 90% range there too.
How we did it: no proof, no next step
We didn't use a bigger model. We changed what the agent is given, and what it's allowed to believe.
1. It starts from a written test. QA teams already write down what should happen. So our agent takes a plain-English test, like this:
When I open the Accounts tab and click "New"
And I enter "Acme Robotics" in "Account Name" and save
And I create a contact "Jordan Patel" with mobile "555-0142"
Then the account lists "Jordan Patel" as a contact
A business analyst can read it. A release manager can review it. And it runs the same way every release.
2. It proves every step. A step only counts as done when the agent can see the proof on the page or in the data. A vanished pop-up isn't proof. A saved record is.
3. It fails honestly. If a step stalls, the agent retries a fixed number of times, then reports a failure. It never guesses its way to a green tick.
It's what a careful tester does on a busy day: slow down, check the screen, and never tick a box you didn't see.
The fine print (we'd rather you hear it from us)
The settings differ. Paper agents got only the request, or a human demo. We gave ours a written test spec. That's how testing teams work, but it isn't the same exam, and we don't claim it is.
It's a subset. 87.7% covers the 122 tasks with written specs. Across all 300 tasks, including those still on auto-generated specs, the agent passes 36.7% today. Most of that gap is admin work.
Today's build only. We counted a task only if it passed on the current version of the agent. We dropped 25 tasks that had passed on older builds.
You can check our work. All 122 specs are public on GitHub. We also found 12 bugs in the grader and two tasks no agent can finish, and we've shared them with the SCUBA team.
What this means for Salesforce teams
If you run Salesforce, you can now test your most important flows in plain English, with an agent that has to prove every step. Start where results are proven today: leads, opportunities, quotes, cases and escalations.
And when any AI testing vendor shows you a number, including us, ask five questions:
Which benchmark, and how many of its tasks?
Is it from today's build, or the best run ever?
Who graded it?
Can I rerun it myself?
What's the score on the hardest tasks?
What's next
We're finishing the admin specs, measuring how often the agent passes over repeated runs, and re-running all 300 tasks on one fixed build. We'll publish those results either way.
Accuracy isn't a number you hit once. It's a standard you hold every release.
Want to see the agent run your own Salesforce flows? Book a demo ·
// Start testing //
