ClawBenchWhat an agent execution trace should contain
OpenAI's Agents SDK defines a trace as one end-to-end workflow and spans as the timed operations inside it. That tree is more useful than a flat transcript because it preserves parent-child relationships and timing across an agent run.
OpenTelemetry's developing GenAI conventions distinguish agent invocation, workflow invocation, planning, model inference, and tool execution.
Instrument only phases the runtime can observe. Do not label ordinary reasoning as planning when the framework cannot reliably separate the two.
- Workflow identity: trace ID, environment, release, agent name and version.
- Model operations: provider, requested model, latency, token usage, and error type.
- Tool execution: tool name, call ID, duration, failure state, and an explicitly governed payload policy.
- Outcome: a separately defined pass, escalation, correction, policy failure, or evaluator result.
Read the trace from outcome to cause
- Confirm the outcome. A plausible sequence of calls does not prove that the user's task succeeded. Start with the verifier or production outcome.
- Find the failing branch. Follow parent-child spans from the workflow into agent, model, retrieval, and tool operations.
- Separate remote from local work. OpenTelemetry recommends client spans for remote agent services and internal spans for same-process agents, which helps isolate network latency.
- Check retries explicitly. One model span may include automatic retries. Do not infer attempt count from a single aggregate duration.
- Compare a cohort. Group failures by release, agent version, model configuration, tool, and outcome rather than treating one trace as representative.
Use the public ClawBench traces view to inspect run evidence, then compare the same benchmark family in the agent leaderboard.
Join telemetry to an evaluation result
OpenTelemetry defines a gen_ai.evaluation.result event with an evaluation name, label or numeric value, explanation, and response ID.
The trace records behavior. The evaluation result records a defined judgment of the output.
Define the outcome first. Capture one end-to-end trace, attach stable release and configuration fields, and compare success and latency by cohort.
Inspect representative passes and failures. Turn recurring failure signatures into replayable regression cases before changing production routing.
Protect prompts and tool payloads
Instructions, inputs, outputs, and tool payloads can be large or sensitive. OpenTelemetry recommends that instrumentation not capture them by default.
Treat content capture, redaction, retention, and access controls as explicit configuration.
Also verify export completeness. Batch processors can lose final spans when short-lived jobs exit before flushing, which can bias an evaluation toward longer or successful runs.
From a trace to a benchmark decision
Trace quality is one part of a defensible AI agent benchmark. Keep the task set, verifier, attempt policy, and resource limits fixed.
Use traces to explain why comparable scores changed. Then read how to benchmark agents in production and the production trace guide.
Primary sources
- OpenTelemetry GenAI semantic conventions (pinned source).
- OpenAI Agents SDK tracing documentation (pinned source).