Reliable AI Browser Testing Needs a Determinism Layer

An AI browser agent clicks Submit. The button is visible, but a spinner still covers it. The model tries again, decides it may be stuck, refreshes the page, and erases a half-completed workflow.

The final report says the test failed. It does not say whether the application failed, the browser tool failed, or the agent made a bad recovery decision.

That ambiguity is the real reliability problem in AI browser testing.

The stale belief is that a more autonomous agent is automatically a more capable tester. Browser tests do need judgment: pages change, labels vary, and the next useful action is not always obvious. But they also have a known shape. They begin in a defined state, perform permitted actions, collect evidence, make assertions, and stop.

Reliability comes from giving the model freedom inside that shape, not freedom from it.

Short answer: Why did Hercules move from AG2 to LangGraph?

Hercules moved from AG2 group chat to LangGraph because its AI browser-testing workflow had matured into a bounded plan-act-observe-assert loop. AG2 was valuable for fast, flexible experimentation. As Hercules evolved, explicit state, deterministic routing, strict contracts, bounded loops, fewer unnecessary model calls, browser-state protection, and node-level observability became more important. LangGraph matched that workflow more directly; it was a fit decision, not a verdict on AG2.

The problem was not AG2. It was workflow fit.

AG2 group chat is useful when several agents genuinely need to discuss an open-ended problem. It provides a flexible conversation pattern and even supports constrained speaker transitions. Its own documentation is candid that group chat can become hard to control as the number of agents grows (AG2 GroupChat documentation).

That is a design tradeoff, not an indictment.

AG2 was also a good starting point for Hercules. A general multi-agent framework let an open-source team stand up planner, executor, and helper behavior quickly, observe what happened, and learn where flexibility mattered.

Then the workflow stopped being open-ended.

A browser test usually moves through predictable phases: understand the step, select a permitted tool, act, observe, judge the evidence, and either continue, repair, fail, or stop.

In the AG2 implementation used by Hercules, many small messages accumulated in shared history, coordination created additional model calls, and execution slowed. The team measured calls with LiteLLM and concluded that the orchestration was spending model judgment on transitions the runtime already knew how to make.

In a bounded workflow, unnecessary agent conversation is not collaboration. It is control-plane overhead.

This is why the move to LangGraph should not be read as “state graphs beat group chat.” AG2 can constrain routes, and LangGraph can still loop badly if its stop conditions are wrong.

The useful question is narrower: Does the orchestration model match the topology of the work?

For Hercules, a state graph fit a mature browser-test loop better than a continuing conversation.

That need for control is not unique to Hercules. In an AutoGen community issue, a builder asked how to force an exact multi-agent sequence instead of leaving the flow to speaker selection (AutoGen issue #584).

The request does not prove group chat is unreliable. It shows a recurring maturation point: once a team knows the required route, it wants that route expressed as system behavior rather than prompt intent.

Browser testing is already a state machine

Browser automation is structured, but it is not simple.

The structure is familiar. Gherkin describes an initial context, an event, and an expected outcome through Given, When, and Then. Cucumber executes those steps sequentially and treats the scenario as an executable specification (Cucumber Gherkin reference).

Playwright adds runtime rules of its own. Before a click, it checks that the target resolves, is visible and stable, receives events, and is enabled. Assertions retry until their conditions pass or time out (Playwright actionability).

The complexity lives in the state between those steps.

A browser session carries authentication, cookies, storage, open modals, form contents, navigation history, in-flight requests, DOM mutations, and the peculiar state of a single-page application. A correct next action depends on more than the latest screenshot or the last model message.

Research environments make the gap visible. In WebArena, a benchmark designed around realistic, long-horizon web tasks, the best reported GPT-4-based baseline completed 14.41% of tasks end to end, compared with 78.24% for humans (WebArena paper).

BrowserGym later compared six models across six browser-agent benchmarks and still concluded that robust, efficient web agents remain difficult because real web environments and current models are both complex (BrowserGym paper).

These are research results, not production test-agent benchmarks. But they puncture the idea that a capable model plus a browser tool is enough.

The browser is not a blank tool surface. It is the evolving test state.

The Determinism Layer

The Determinism Layer is the engineering shell around an AI browser-testing agent. It confines probabilistic judgment inside predictable runtime controls.

The model can judge uncertainty. The system should control the route.

That split is the core design decision:

In practice, the Determinism Layer contains graph-based routing, typed shared state, separate planner, executor, and assertion roles, strict JSON contracts, validation and repair loops, planning and navigation budgets, browser-state guards, role-specific model selection, and traces that join model decisions to browser consequences.

This is bounded autonomy.

The model can choose among permitted outcomes. The graph decides what each outcome means operationally.

Separate planning, execution, and assertion so failure becomes legible

When one conversational agent plans, acts, observes, and declares success in the same stream, a failure can smear across the entire transcript.

The agent may have misunderstood the goal, selected the wrong control, clicked correctly while the tool misreported, or asserted against weak evidence. The final answer hides the category of failure.

A graph with explicit planner, executor, and assertion nodes creates inspection points:

1. Plan: What did the system believe the next step required?

2. Execute: Which tool was selected, with which validated arguments?

3. Observe: What changed in the browser, network, DOM, or application state?

4. Assert: Which evidence supports the verdict?

5. Route: Should the run continue, repair, fail, block, or stop?

That separation produces a useful failure taxonomy: planning failure, grounding failure, tool failure, observation failure, assertion failure, contract failure, or boundary exhaustion.

A red test is no longer one undifferentiated event. It becomes a diagnosable system outcome.

This matters to QA teams because a flaky test and a real product defect demand different work. It matters to framework builders because each node can be evaluated independently. It matters to engineering leaders because “agent reliability” becomes a set of measurable components rather than a mood.

Cutting LLM calls is architecture work, not prompt cleanup

Teams often respond to token cost by shortening prompts. That helps at the margin. It does not remove calls that should never have existed.

If state says the executor completed an action successfully and the next required operation is observation, asking a model which agent should speak next is waste.

If an assertion node needs only the expected outcome, observed value, and evidence references, replaying the full conversation is waste.

If a memory operation is simple compression, giving it the same expensive reasoning model as the planner may be waste.

The stronger optimization is structural:

• Replace known transitions with code.

• Pass typed fields instead of entire transcripts.

• Give each node only the state it needs.

• Use role-appropriate models, then validate them with role-level evaluations.

• Track calls, tokens, latency, and cost per node and per successful test.

Hercules evaluated orchestration alternatives and found LangGraph gave the team granular routing, minimal memory primitives, strong documentation, and the lowest observed LLM-call count in its internal experiments.

There is no public percentage attached to that claim, and there should not be one without a reproducible benchmark.

The lesson is still general: the largest token saving may come from removing an unnecessary decision, not compressing its prompt.

Loop limits are runtime contracts, not hacks

An agent that cannot find a control may scroll, inspect, retry, re-plan, or refresh. Some recovery is useful. Unlimited recovery is an outage with a token bill.

The Determinism Layer gives planning rounds, navigation attempts, and wall-clock execution explicit budgets.

LangGraph exposes a recursion limit, but its documentation and community issue tracker also show the important caveat: a graph can reach that limit when it fails to hit a stop condition (LangGraph recursion-limit documentation, community issue #6731).

A framework provides the mechanism. The application still has to define correct termination behavior.

A loop limit is not merely a safety cutoff. It is a diagnostic sensor.

Budget exhaustion may mean the page is not visible to the agent, the chosen tool is weak, the tool is failing, the planner is oscillating, or the application is genuinely slow.

A mature runtime should report those possibilities as distinct outcomes: failed, blocked, tool error, invalid contract, budget exhausted, or inconclusive.

“Maximum iterations reached” is an implementation detail, not a useful test report.

Assertions need JSON contracts, not markdown vibes

Markdown is presentation. JSON is a contract.

An assertion result should not be a persuasive paragraph that another component must interpret. It should be a validated object with fields such as verdict, expected, observed, evidence_refs, confidence, and failure_reason.

Pydantic can generate JSON Schema from typed models and validate returned structures (Pydantic JSON Schema documentation). Invalid output can enter a bounded repair loop; valid output can route deterministically.

Structured output does not make the assertion true.

A model can return perfect JSON and cite the wrong element. Schema validity, referential validity, and semantic correctness are different layers. The contract solves parsing and routing. Evidence rules and evaluations still have to solve truth.

That distinction is easy to miss. A green schema validator proves that the message is machine-readable. It does not prove that the checkout total was correct.

Protect browser state from reflexive recovery

Refresh is a useful browser action. It is also destructive when used as a reflex.

In a single-page application, refresh can clear transient UI state, reset an unsaved form, close a modal, change a session-dependent route, or destroy the exact conditions the test was meant to inspect.

Agents tend to refresh when they feel stuck because it is an obvious recovery move. A reliable test runtime must distinguish a planned refresh, required by the scenario, from an improvisational refresh that erases evidence.

Hercules does not refresh the live Playwright URL before every step. Its engineering direction is the opposite: prevent random refreshes, inject observation logic to detect DOM changes, and rely on Playwright stability mechanisms where they apply.

Playwright’s actionability checks already encode useful browser-level invariants, but they cannot decide whether resetting the business workflow is semantically acceptable.

A browser-state guard should ask:

• Is refresh allowed for this step?

• What state would it destroy?

• Has the DOM actually stopped changing?

• Did navigation occur?

• Is the action idempotent?

Those questions belong in runtime policy, not in an agent’s momentary instinct.

One model for every role is an architectural smell

The planner, navigator, memory component, and visual comparison helper do different work. Treating them as one generic “agent” hides the differences.

Planning benefits from deliberate reasoning and structured decomposition. Navigation benefits from speed, reliable tool calling, and strong grounding. Memory may need inexpensive summarization. A visual helper needs multimodal ability and should be evaluated on visual assertions, not planning quality.

Role-specific routing can improve cost, speed, and accuracy, but it is not automatically better. More model configurations create more evaluation surface and operational complexity.

The correct practice is to define a role contract, build a small evaluation set for it, and choose the least expensive model that meets that contract.

Model selection should follow the node’s job.

Observe the decision and the browser consequence

An LLM trace alone cannot tell you why a click failed. A Playwright trace alone cannot tell you why the agent chose that click.

Reliable AI browser testing needs both planes:

• Model plane: prompts, structured outputs, tool choices, token usage, latency, cost, validation errors, and repair attempts.

• Browser plane: actions, screenshots, DOM snapshots, console logs, network requests, storage state, timeouts, and assertion evidence.

The two planes need a shared run ID and node or step identity.

Tools such as LiteLLM, AgentOps, and LangSmith can help trace model behavior. Playwright Trace Viewer exposes action timing, errors, console output, and network activity (Playwright Trace Viewer).

The public Hercules runtime also emits reports and proof artifacts, with planner and helper traces documented in its repository (Hercules on GitHub).

Observability does not create reliability by itself. It makes reliability engineering possible because it lets a team connect a decision to its consequence and test a fix against the same failure class.

MCP expands the tool boundary, not the trust boundary

Browser tests rarely live entirely inside the browser.

A login may need an OTP from email. A workflow may depend on Salesforce, SAP, Oracle, an API, a database, or a CI job. MCP gives an agent a consistent client-server architecture for reaching tools and context across those systems (MCP architecture).

Hercules can act as an MCP client to consume external capabilities and can run as an MCP server so coding agents, harnesses, or CI/CD systems can trigger tests.

That makes the testing agent composable. It does not make the connection trustworthy by default.

Every MCP boundary still needs authentication, least-privilege permissions, explicit schemas, timeouts, traceability, and data-handling rules.

The protocol expands what the agent can reach. The Determinism Layer defines what it may do.

Plain language to Gherkin is an intermediate contract, not magic

Plain English is useful for capturing business intent, but intent is not executable until the system can map it to available actions and observable outcomes.

Hercules uses two reasoning cycles. First, the model combines the instruction and supplied test data into a workflow expressed as Gherkin. Second, the planner turns those steps into executable operations using the tools available at runtime.

Gherkin is valuable here because it provides a compact Given-When-Then contract between business intent and execution.

The conversion remains partly inferential. Open-source Hercules does not automatically know the application’s routes, domain objects, permissions, or configuration.

A sentence such as “approve the discount” is under-specified if three approval mechanisms exist. Test data, application context, and tool descriptions determine whether the generated scenario preserves intent.

The practical rule is simple: treat generated Gherkin as a reviewable intermediate representation. It is more structured than prose, but it is not ground truth.

What Hercules learned as an open-source test agent matured

The current public Hercules repository documents a LangGraph StateGraph with planner, executor, and assertion nodes. The planner returns strict JSON; the executor routes to bounded navigation helpers; model configuration can vary by role; evidence is written to reports and proof artifacts; and MCP is available on both the client and server sides (Hercules architecture and usage).

The important part is not the framework name. It is the maturation pattern:

1. Start with enough flexibility to discover the real workflow.

2. Measure where calls, latency, ambiguity, and failure accumulate.

3. Identify the topology that stays stable across runs.

4. Move those invariants into typed state, routes, policies, and contracts.

5. Keep model judgment only where it earns its cost.

TestZeus describes the larger shift as moving from script maintenance to agent supervision.

The Determinism Layer is what makes that supervision operational. It gives a human something inspectable to supervise: plans, routes, evidence, limits, and failure categories, not merely a transcript and a verdict.

Practical takeaways for AI testing teams

Before adding another agent, inspect the decisions your current runtime asks a model to make.

• If the next route follows directly from state, encode it.

• If a message crosses a component boundary, type and validate it.

• If a loop can continue, give it a budget and a diagnostic exit.

• If an action can destroy browser state, guard it explicitly.

• If a verdict matters, require evidence references.

• If two roles have different latency or reasoning needs, evaluate them separately.

• If a test can fail, trace both the model decision and the browser consequence.

The contrarian conclusion is that reliable agentic testing may require less agent behavior.

The best AI test agent is not the freest one. It is the one with the right boundaries.

Frequently Asked Questions

Why did Hercules move from AG2 to LangGraph?

Hercules moved because its browser-testing workflow became a stable plan-act-observe-assert graph. AG2 group chat helped the project experiment quickly, but explicit state, routes, contracts, loop limits, browser guards, and node-level traces became a better fit as the system matured.

Is LangGraph always better than AG2 for AI browser testing?

No. AG2 is well suited to flexible, conversational multi-agent collaboration and can constrain speaker transitions. LangGraph is a strong fit when a workflow has explicit states and predictable routes. The right choice depends on the topology of the work, not a universal framework ranking.

What is the Determinism Layer in AI browser testing?

It is the runtime shell that places probabilistic model judgment inside deterministic controls: typed state, graph routes, schemas, validation, loop budgets, timeouts, browser-state policies, evidence requirements, permissions, and traces.

Why do AI test agents need strict JSON contracts?

Structured contracts make outputs parseable, validatable, and routable. They let the runtime distinguish a verdict from its evidence and repair malformed responses. They do not guarantee that the model’s conclusion is semantically correct, so evidence checks and evaluations remain necessary.

How does MCP help Hercules integrate with coding agents and CI/CD?

As an MCP client, Hercules can call external tools or retrieve context needed during a test. As an MCP server, it can expose test execution to coding agents and automation harnesses. Those integrations still require authentication, permissions, schemas, timeouts, and traceability.

Reliable AI browser testing will not be won by the system with the most autonomy. It will be won by systems that know where autonomy belongs and where determinism must take over.

Explore the open-source Hercules repository, inspect the graph, and bring your hardest browser-state edge case to the project.