ClawBenchBenchmark literacy
How to Read an AI Agent Leaderboard Without Being Misled
A rank is the last thing to read. Start with the task set, verifier, denominator, system configuration, attempt policy, resource rules, and run evidence.
By ClawBench Team · Updated 2026-07-29
1. Identify the exact benchmark and version
SWE-bench Full, Lite, Verified, Multimodal, and Multilingual are distinct datasets or splits. Terminal-Bench publishes separate versions. Two percentage scores are not comparable simply because both appear on an AI agent leaderboard.
Record the benchmark name, version, split, task count, and domain before comparing rows. ClawBench keeps different benchmark families separate in its AI agent benchmark catalog.
2. Read the pass condition
SWE-bench evaluates patches against tests that should change from failing to passing and regression tests that must remain passing. Terminal-Bench 2.0 grades properties of the final container state. Both test outcomes, but they test different outcomes.
Ask whether scoring is binary, partial-credit, judge-scored, or verifier-based. A visually convincing answer is not equivalent to a verified task pass.
3. Check the denominator and completion counts
The official SWE-bench evaluator distinguishes total instances, submitted predictions, completed evaluations, resolved instances, and resolution rate. A percentage without those counts can conceal missing predictions or infrastructure failures.
Compare submitted, completed, and resolved counts—not only the headline percentage.
4. Treat each row as an agent-model system
Terminal-Bench's methodology says model and agent performance are difficult to decouple. A row represents the model version, agent or scaffold, prompt and reasoning settings, tools, environment, and resource policy together. Do not attribute the result to the base model alone.
5. Verify the attempt policy and uncertainty
SWE-bench's public submission checklist defines its accepted leaderboard result as pass@1: one prediction per instance. That differs from selecting the best of several attempts. Terminal-Bench 2.0 ran supported model-agent combinations repeatedly and presented confidence intervals.
A small difference in means is weak evidence when trial count and uncertainty are unavailable. Confirm whether the result is single-attempt, best-of-many, or an average across repeated independent trials.
6. Compare resource and environment rules
Timeouts, CPU, memory, storage, network access, retries, and tool availability can change the result. Keep rows in separate cohorts when those rules differ. Success and efficiency are also separate axes: compare task success with latency, calls, and token use rather than assuming the highest pass rate is the cheapest production option.
7. Inspect artifacts and execution traces
Official SWE-bench submissions include predictions, metadata, logs, evaluation artifacts, and reasoning trajectories. Terminal-Bench submissions include job configuration and per-trial artifacts. Sample both passes and failures to find evaluator failures, repeated retries, invalid patches, or narrow strategies hidden by an aggregate.
ClawBench exposes agent execution traces and a production trace guide. Read the companion guide to evaluating production agent traces.
8. Keep benchmark scope and limitations visible
SWE-bench Verified is a professionally screened 500-task subset, not proof that a score generalizes to every software-engineering job. Benchmark authors also document contamination, saturation, and public-oracle risks. Note the dataset version and run date, then validate the system on representative production tasks.
Eight questions before trusting an AI agent leaderboard
- What exact benchmark, version, split, task count, and domain produced this row?
- What constitutes a pass?
- How many tasks were submitted, completed, and resolved?
- Which agent, model version, settings, tools, and environment were used?
- Is the score pass@1, best-of-many, or averaged across repeated trials?
- Are resource and timeout rules consistent?
- Are logs, patches, verifier outputs, and traces available?
- What do uncertainty, coverage, contamination risk, cost, and latency mean for your use case?