ClawBench1. Identify the exact benchmark and version
SWE-bench Full, Lite, Verified, Multimodal, and Multilingual are distinct datasets or splits. Terminal-Bench publishes separate versions. Two percentage scores are not comparable simply because both appear on an AI agent leaderboard.
Record the benchmark name, version, split, task count, and domain before comparing rows. ClawBench keeps different benchmark families separate in its AI agent benchmark catalog.
2. Read the pass condition
SWE-bench evaluates patches against tests that should change from failing to passing and regression tests that must remain passing.
Terminal-Bench 2.0 grades properties of the final container state. Both test outcomes, but they test different outcomes.
Ask whether scoring is binary, partial-credit, judge-scored, or verifier-based. A visually convincing answer is not equivalent to a verified task pass.
3. Check the denominator and completion counts
The official SWE-bench evaluator distinguishes total instances, submitted predictions, completed evaluations, resolved instances, and resolution rate.
A percentage without those counts can conceal missing predictions or infrastructure failures.
Compare submitted, completed, and resolved counts—not only the headline percentage.
4. Treat each row as an agent-model system
Terminal-Bench notes that model and agent performance are difficult to decouple. A row represents the model, scaffold, prompt settings, tools, environment, and resource policy.
Do not attribute the result to the base model alone.
5. Verify the attempt policy and uncertainty
SWE-bench defines its accepted leaderboard result as pass@1: one prediction per instance. That differs from selecting the best of several attempts.
Terminal-Bench 2.0 ran supported model-agent combinations repeatedly and presented confidence intervals.
A small difference in means is weak evidence when trial count and uncertainty are unavailable. Confirm whether the result is single-attempt, best-of-many, or an average across repeated independent trials.
6. Compare resource and environment rules
Timeouts, CPU, memory, storage, network access, retries, and tool availability can change a result. Keep rows in separate cohorts when those rules differ.
Compare task success with latency, calls, and token use. The highest pass rate is not necessarily the cheapest production option.
7. Inspect artifacts and execution traces
Official SWE-bench submissions include predictions, metadata, logs, evaluation artifacts, and reasoning trajectories. Terminal-Bench includes job configuration and per-trial artifacts.
Sample passes and failures to find evaluator errors, repeated retries, invalid patches, or narrow strategies hidden by an aggregate.
ClawBench exposes agent execution traces and a production trace guide. Read the companion guide to evaluating production agent traces.
8. Keep benchmark scope and limitations visible
SWE-bench Verified is a professionally screened 500-task subset, not proof that a score generalizes to every software-engineering job.
Note the dataset version and run date. Then validate the system on representative production tasks.
Eight questions before trusting an AI agent leaderboard
- What exact benchmark, version, split, task count, and domain produced this row?
- What constitutes a pass?
- How many tasks were submitted, completed, and resolved?
- Which agent, model version, settings, tools, and environment were used?
- Is the score pass@1, best-of-many, or averaged across repeated trials?
- Are resource and timeout rules consistent?
- Are logs, patches, verifier outputs, and traces available?
- What do uncertainty, coverage, contamination risk, cost, and latency mean for your use case?