ClawBenchBrowser agents look much better in controlled environments than they do on real websites. That is not surprising. Controlled environments are built to be reproducible. Real websites are built to serve users, defend themselves, and change without asking your benchmark for permission.
If your product depends on web agents, you need to evaluate them on the kind of websites they will actually touch.
Sandbox web tasks are useful
It is worth saying this clearly: sandbox web tasks are useful. They give researchers stable environments, repeatable tasks, and clean comparisons. Without them, progress would be harder to measure.
But a sandbox result is a lower bound on messiness. It removes many of the things that make production workflows difficult.
Real websites add friction
Real websites introduce problems that have nothing to do with whether the model "understands" the task. A modal appears. The DOM updates after the agent reads it. A session expires. A form validation rule changes. The site blocks suspicious traffic. A button label changes from "Continue" to "Next".
Humans handle this friction because we carry context and adapt quickly. Agents often overfit the current page state. When it changes, they keep trying the old plan.
The failures compound
A browser workflow is rarely one action. It is a chain. Navigate, authenticate, search, filter, inspect, edit, confirm, verify. Each step can fail. Worse, each step can partially succeed and leave the agent in an unexpected state.
That is why pass rate alone is not enough. You need to know where in the chain the agent failed and whether the failure is recoverable.
What to measure
For real websites, measure the final outcome and the path. Did the agent complete the task? Did it use the right account state? Did it avoid destructive actions? Did it recover from page changes? Did it verify the final state?
Also measure cost. Browser tasks can burn tokens and tool calls quickly. A run that eventually passes after fifty uncertain actions may be too expensive or too risky to use.
| Signal | Question | Evidence |
|---|---|---|
| Completion | Did the workflow finish? | Run result |
| State awareness | Did the agent know where it was? | Trace steps |
| Recovery | Did it adapt after failure? | Retries and branch points |
| Safety | Did it avoid irreversible mistakes? | Action log |
| Verification | Did it confirm the final state? | Final trace evidence |
The live-web rule
If your agent needs to operate on real websites, do not ship from sandbox scores alone. Use sandbox benchmarks to shortlist, then run live-web validation before trusting the workflow.
Practical rule
Static web pages test navigation. Real websites test judgment, recovery, and state management. You need both before production.
Continue the evaluation
Use this guide with the benchmark entity pages, leaderboard context, and trace evidence so the query intent can move straight from explanation to product proof.