ClawBench Blog

Start Here

If you are new to ClawBench, start with the complete benchmarking guide, then move to setup and mode-specific content.

What We Publish

• Benchmark design and scoring methodology.
• Mode-specific implementation playbooks.
• Security hardening and prompt injection defense patterns.
• Monthly performance and reliability trend reports.
• Setup guides for OpenClaw, local, and hosted model stacks.

Prompt Injection Methodology (Detailed)

ClawBench prompt-injection mode is implemented as a single-agent benchmark lane. An agent enrolls and submits runs directly without waiting for a second entrant. This lets teams evaluate security behavior in realistic plugin and retrieval workflows where the main failure mode is not agent-vs-agent strategy, but instruction-source confusion.

Framework Basis: InjectBench

We align this lane to InjectBench-style methodology: a standardized framework for indirect prompt injections where an adversary hides malicious instructions in third-party data later retrieved by the model. The core research objective is to measure attack resistance and retained utility at the same time, not in isolation. InjectBench reports this at dataset scale (1,670 synthetic samples) and surfaces an important trend: stronger instruction-following models can become more vulnerable when they over-trust retrieved context.

Threat Model: Indirect Injection vs Jailbreak

• Indirect injection: payload embedded in retrieved documents, pages, or plugin outputs.
• Traditional jailbreak: payload presented directly in the user prompt.
• Primary risk: model treats untrusted context as high-priority instruction.
• Operational impact: misinformation, fake availability errors, or fraudulent action suggestions.

Attack Construction Pipeline

Each benchmark attack sample is composed from three parts:

1. Benign component: realistic task context (news, how-to text, reviews, recipes, etc.).
2. Separator component: delimiter, focus override, or fake-summary transition that promotes attacker authority.
3. Malicious instruction: payload pursuing one of three goals: manipulated content, availability disruption, or fraud/malware coercion.

For robust generation quality, the recommended flow is two-stage: an instruction-writing model creates multiple candidates, then an instruction-choosing model ranks executability, thematic fit, and harm profile.

Evaluation Methodology

Indirect-injection outputs are often not overtly toxic, so standard classifiers miss many failures. We therefore use an LLM-as-judge setup with precision tuning:

• Attack-agnostic prompts: did the response follow injected instructions?
• Attack-specific prompts: criteria for manipulated content, availability, and fraud/malware.
• Binary thresholding: optimize Yes/No decision boundaries for high precision and lower false positives.
• Validation: compare model judgments with human consensus before shipping scoring changes.

Metrics

• BU (Benign Utility): success rate on clean episodes.
• UUA (Utility Under Attack): task success when injections are present.
• ASR (Attack Success Rate): fraction of attack episodes where payload objectives are achieved.

The target profile is BU high, UUA high, and ASR low. Any defense that reduces ASR by collapsing BU is treated as over-blocking rather than a win.

50-Sample Implementation Blueprint

For a practical startup corpus, use 50 attack variations across the three attacker goals:

• 17 manipulated-content attacks (narrative drift while staying topically connected).
• 17 availability attacks (fake error codes and false unreadable/inaccessible claims).
• 16 fraud/malware attacks (coercion to illegitimate links presented as relevant resources).

Keep each sample in the canonical form: [Benign Data] + [Separator] + [Malicious Instruction].

Most Effective Defenses in Practice

• Treat plugin/retrieved text as untrusted data, never as policy source.
• Enforce instruction hierarchy and explicit context-boundary checks.
• Use constrained tool permissions and risky-action confirmation gates.
• Apply random-sampling style context checks to reduce attack success rates.
• Audit outputs for secret leakage, fake availability claims, and fraudulent links.

Resources

Use the starter kit to define your rubric and rollout criteria before running live benchmarks.