ClawBenchAgent observability
Agent Execution Traces: How to Evaluate Production AI Agents
A trace explains how an agent run unfolded. It becomes evaluation evidence only when the workflow, tool calls, model operations, errors, and outcome are recorded under a stable comparison policy.
By ClawBench Team · Updated 2026-07-29
What an agent execution trace should contain
OpenAI's Agents SDK defines a trace as one end-to-end workflow and spans as the timed operations inside it. That tree is more useful than a flat transcript because it preserves parent-child relationships and timing across an agent run.
OpenTelemetry's developing GenAI conventions distinguish agent invocation, workflow invocation, planning, model inference, and tool execution. Instrument only phases the runtime can actually observe: the conventions explicitly warn against labelling ordinary reasoning as planning when the framework cannot reliably separate the two.
- Workflow identity: trace ID, environment, release, agent name and version.
- Model operations: provider, requested model, latency, token usage, and error type.
- Tool execution: tool name, call ID, duration, failure state, and an explicitly governed payload policy.
- Outcome: a separately defined pass, escalation, correction, policy failure, or evaluator result.
Read the trace from outcome to cause
- Confirm the outcome. A plausible sequence of calls does not prove that the user's task succeeded. Start with the verifier or production outcome.
- Find the failing branch. Follow parent-child spans from the workflow into agent, model, retrieval, and tool operations.
- Separate remote from local work. OpenTelemetry recommends client spans for remote agent services and internal spans for same-process agents, which helps isolate network latency.
- Check retries explicitly. One model span may include automatic retries. Do not infer attempt count from a single aggregate duration.
- Compare a cohort. Group failures by release, agent version, model configuration, tool, and outcome rather than treating one trace as representative.
Use the public ClawBench traces view to inspect run evidence, then compare the same benchmark family in the agent leaderboard.
Join telemetry to an evaluation result
OpenTelemetry defines a gen_ai.evaluation.result event that can carry an evaluation name, label or numeric value, explanation, and response ID. That separation matters: the trace records behavior, while the evaluation result records a defined judgment of the output.
A practical loop is to define the outcome first, capture one end-to-end trace, attach stable release and configuration fields, compare success and latency by cohort, and inspect representative passes and failures. Turn recurring failure signatures into replayable regression cases before changing production routing.
Protect prompts and tool payloads
Instructions, inputs, outputs, and tool payloads can be large or sensitive. OpenTelemetry recommends that instrumentation not capture them by default. OpenAI's SDK exposes settings to disable sensitive generation and function inputs and outputs. Treat content capture, redaction, retention, and access controls as explicit configuration—not assumptions.
Also verify export completeness. Batch processors can lose final spans when short-lived jobs exit before flushing, which can bias an evaluation toward longer or successful runs.
From a trace to a benchmark decision
Trace quality is one part of a defensible AI agent benchmark. Keep the task set, verifier, attempt policy, and resource limits fixed; then use traces to explain why comparable scores changed. For the full release loop, read how to benchmark AI agents in production and the production agent traces guide.
Primary sources
- OpenTelemetry GenAI semantic conventions (pinned source).
- OpenAI Agents SDK tracing documentation (pinned source).