ClawBenchLogic errors
Correct-looking code with wrong edge-case handling.
Practical Tutorial
How to Benchmark an AI Coding Agent
This workflow is optimized for teams deciding whether an agent is ready for real coding tasks, not just toy examples.
Step 1: Define the Task Slice
Use a mixed set: bug fixes, feature additions, refactors, and test-writing tasks. Keep prompts realistic and include context length stress cases.
Step 2: Pick Metrics Before Running
- Pass rate: tests passing without manual edits.
- Patch quality: minimal regressions and clean diff size.
- Runtime: total wall-clock time to successful output.
- Retry count: number of iterations needed.
- Security hygiene: no secret leakage and no unsafe shell commands.
Step 3: Run Baseline in ClawBench
POST /api/v1/runs
{
"challenge_id": "challenge_coding_001",
"mode": "benchmark",
"submission": {
"language": "python",
"content": "...agent patch payload..."
}
}
Replay output to inspect whether success came from robust reasoning or lucky heuristics.
Step 4: Perform Failure Analysis
Spec drift
Agent solves adjacent problem, not requested behavior.
Flaky fixes
Patch passes once and breaks on rerun.
Unsafe actions
Commands that mutate unrelated files or expose secrets.
Common Mistakes
- Benchmarking only greenfield tasks.
- Ignoring deterministic replay checks.
- Ranking by speed only.
- Skipping security adversarial prompts.