Code Isn’t the Bottleneck Anymore. Testing Is.

Anthropic just put out a long playbook on building software with AI agents. One line in it stopped me: “Code is no longer the bottleneck.”

Coming from the people who build Claude, that line matters.

I’ve worked in testing for over 15 years. I wrote a book on Selenium. Now I run TestZeus. So my first thought was simple. If code isn’t the bottleneck, what is?

Testing is. Checking the work. Proving it actually works.

This post is for the people who sign off on AI budgets and own the risk: CTOs, VPs of engineering, heads of QA and IT. I’ll cover what Anthropic said, what the industry data says, and what we’ve seen at TestZeus. Then the moves I’d make if I were in your seat.

The slow part moved

AI agents write code in hours now. But everything around the code stayed the same. Planning, review, testing and release still move at people speed.

Anthropic says it straight. When agents write more code, one of two things happens. The review pile grows, or code ships without enough checks. Neither is okay, especially if you’re in a regulated business.

Think of a restaurant that hires a robot chef. The kitchen gets much faster. But there’s still one person tasting every plate before it goes out.

Guess where the line forms.

This isn’t a hunch. The data agrees.

Anthropic isn’t the only one seeing this. Here’s what the big industry studies found.

Put those together and the picture is clear. Almost everyone uses AI. Output is up. Stability, security and trust are not keeping pace.

DORA’s own summary nails it: “AI doesn’t fix a team; it amplifies what’s already there.” The teams that gain are the ones with automated testing and fast feedback loops. The rest just ship their problems faster.

One more detail from Stack Overflow stuck with me. Nearly half of developers say they trust AI output when they can check it easily. That’s the whole game. Easy checking is what turns AI output into something you can ship.

Every step needs a check now

Anthropic’s answer is to treat software as a loop, not a straight line. Each step leaves a note for the next one: what we want, the spec, the plan, the code, the review, the incident report. When something breaks in production, it starts the next loop.

I like it. But look at what keeps the loop moving. At every step, someone has to say “yes, this is good.”

That “yes” is a test. Take the tests away and the loop still runs. It just runs faster in the wrong direction.

So testing can’t wait at the end anymore. It has to show up at every step.

What we’ve seen at TestZeus

Four ideas in the playbook match what we’ve learned the hard way.

1. Clear specs make better tests

Anthropic tells teams to write down what they want before an agent starts. And to give the agent a goal it can check on its own.

We saw the same thing in SCUBA, our Salesforce benchmark. When our testing agent had written specs, it got 87.7% of tasks right. That’s 107 out of 122.

Vague instructions don’t slow an agent down. They make it wrong, fast. If you can’t write the test, you don’t understand the feature yet.

2. Don’t let the builder grade its own work

Ask an agent to make the tests pass, and it will. Sometimes by fixing the code. Sometimes by changing the test.

Anthropic’s advice is clear. Check the work with fresh eyes. Don’t let the agent touch the tests while it fixes a bug. Never let the same agent approve its own change.

That’s why a testing agent should sit outside the coding agent. TestZeus has its own context, its own test data and its own browsers. When we test a Salesforce agent, we don’t just read its answer. We check the query and the data behind it.

A nice answer on top of the wrong data is still a bug. It’s just better dressed.

3. Use AI for thinking, code for repeating

Anthropic draws a clear line. Use plain scripts for checks that must run the same way every time. Call the AI only when you need judgment.

We learned this one twice.

First, cost. Writing every test step in plain English is easy. But the AI has to re-read and re-think each step on every run, and that adds up fast. So we built Test Blocks. You click through your app once, and we turn it into fixed, repeatable steps. The AI helps you write and debug. It doesn’t relearn where the login button is every night.

Second, speed. When we pushed SCUBA hard on accuracy, the runs got slow. Great answers. Late.

4. AI agents need someone to push back

Normal software does the same thing every time. AI agents don’t.

Anthropic suggests a living set of real test cases. Rerun them every time the agent’s setup changes. Turn every production incident into a new test.

Most agent testing today is a fixed list of questions with expected answers. Your support bot passes all of them on Monday. On Tuesday, a customer asks a little differently, and it makes up a refund policy.

So our Agent Harness plays the tough customer. It talks to your agent over many turns and pushes harder as it goes. It checks which tools the agent used, not just what it said. And it runs each test many times, because one good run of an AI proves very little.

Before launch, you get a health check. After launch, we test against real chats and tickets. A problem comes in, a new test comes out. Same loop Anthropic describes.

What this means if you lead a team

If you own the budget or the risk, three things are worth sitting with.

Your dashboards may be flattering you. The developers in the METR study felt faster and were slower. So if you track lines of code, PRs merged or tickets closed, AI will look like a win long before it is one. Track what reached production and stayed working instead: change failure rate, time to restore, defects that escaped. Those are the DORA metrics, and they’re harder to fool.

Risk grows with output. Every extra line an agent writes is one more line somebody has to trust. If code volume climbs and testing capacity stays flat, you’re not moving faster. You’re building a backlog of unchecked risk. And the bill for bad software was already huge before agents showed up. CISQ put it at $2.41 trillion in the US in 2022.

Most teams have no rules yet. In Stack Overflow’s 2026 survey, only 24% of developers said their employer had published approved AI tools or formal guidance. Three in ten said AI use is left to each person. Anthropic’s playbook is one of the few clear maps out there. But its gates only work if someone owns the checks.

Five moves I’d make in your seat

None of these need a reorg. They need a decision.

1. Fund checking at the same pace as building. If your AI spend doubled code output, ask what happened to your testing capacity. If the answer is “nothing,” you found your bottleneck.

2. Make written specs the entry ticket. No acceptance criteria, no agent. Our SCUBA results are why I’d hold this line.

3. Keep the checker separate from the builder. A different agent, a different context, and evidence from the tools. Never the agent’s own summary of how it went.

4. Use AI for judgment, scripts for repetition. Watch the token bill on test runs the way you watch cloud spend. Replaying a recorded flow shouldn’t need a model to think.

5. Test AI agents like a new hire. Before launch and after. Many conversations, tough customers, repeated runs, scores tracked across versions.

Questions to ask your team on Monday

If you want a quick health check, start here:

• How much of last month’s code did AI write, and did our testing keep up?

• Has our change failure rate moved since we rolled out coding agents?

• Can any agent edit the tests for its own change, or approve its own work?

• Which customer-facing AI agents are live right now, and how were they tested before launch?

• When something breaks in production, does it become a permanent test?

If the answers are fuzzy, that’s not a failure. It’s a starting point. Most teams are there right now.

Where TestZeus fits

Your AI writes the code. TestZeus checks that it works, and shows you the proof.

• Write tests in plain English, straight from your specs.

• Record a flow once with Test Blocks and replay it reliably.

• Run inside your CI/CD and Jira, on certified, secure browsers.

• Test Agentforce and website agents with Agent Harness.

• Bring your own AI model, mask your data, or run on-prem.

Like open source? Try Hercules, our open-source testing agent.

The bottom line

Anthropic ends by saying people still make the final call. I agree. But people need proof to make good calls.

AI made code cheap. Proof is the expensive part now. The teams that win won’t be the ones writing the most code. They’ll be the ones who can prove it works, fastest. So the question for leaders isn’t “how much code can our agents write?” It’s “how fast can we prove it’s safe to ship?”

Want to see it on your own app? Talk to us at TestZeus.

Sources

• Anthropic, The AI-native SDLC playbook, August 2026

• Google Cloud, Announcing the 2025 DORA report

• IT Brief UK, Stack Overflow survey shows developers trust AI with caveats, October 2026

• Veracode, 2026 GenAI Code Security Report, July 2026

• eWeek, Developers using AI take 19% longer on tasks (METR study), July 2025

• Synopsys / CISQ, Cost of Poor Software Quality in the US: 2022