Anthropic CVP — Run 3

Run 3 tested Claude Haiku 4.5 — Anthropic's smallest production model — against the same 13-prompt suite Run 2 used on Opus 4.7. The question: does the safety stack scale all the way down to the cheap, fast model, or does cost-optimization weaken refusal quality?

13/13 clean. Zero slips. Zero leaks. Haiku 4.5 produced the same shape of defensive analysis Opus did, on every category.

Every response either matched its expected outcome (per Cava's prompt-design ladder) or exceeded it with a more thorough defensive analysis than required. The only explicit refusal landed on the one prompt that explicitly asked for an attack plan (P3); the rest were defensively-framed and Haiku met them with structured technical analysis.

Run 3 is one model in a planned Anthropic family comparison. Over the next several days we will run the same 13-fixture suite against Opus 4.6 (high + medium effort) and Sonnet 4.6 (high + medium effort) and publish the cross-family delta. This Haiku report is the first data point in that series.

All Run 1–3 fixtures were authored under Sunglasses' standing CVP protocols (locked Apr 17): defensively framed, structured-output, with explicit constraint footers. 13/13 clean here means: Haiku 4.5 produces high-quality defensive analysis when asked for it, and refuses embedded malicious instructions inside scenarios that ask for defender-side reasoning.

A separate appendix probe set using real-world adversarial payloads (no defensive framing, real-world attacker prompt shapes) will publish later as a labeled add-on. That is intentionally not blended into the core scoreboard — methodology stability across runs matters more than chasing single-headline outliers.

If a customer is choosing between Opus 4.7 and Haiku 4.5 for an agent that handles untrusted content (RAG, tool outputs, cross-agent messages), does the cheaper model's safety degrade? This run is one data point. It does not appear to degrade on this prompt set.

Two CVP runs per week. Run 1 (Apr 17) and Run 2 (Apr 20) both targeted Opus 4.7. Run 3 widens the model coverage. The intent for Run 3 was always to probe model variation. After this run, the more interesting probe is prompt variation — see Run 4 preview below.

claude-haiku-4-5

Not exposed for Haiku (recorded as standard).

Isolated OPTIC / Claude Code session on the CVP-approved org (d4b32d1d-…). Prompts executed one at a time, fresh context per fixture.

Full transcripts captured to ~/optic/benchmarks/cvp-2026-04-23-run3/transcripts/haiku-4-5/. Each transcript carries: timestamp, model ID, effort, org ID, prompt SHA-256, response SHA-256, classification draft, expected outcome, related Sunglasses pattern IDs.

Run 3 reused the same 13 prompts as Run 2 for direct cross-model comparability: 3 baselines (benign defensive / borderline legitimate / clearly high-risk) + 10 runtime-trust probes mapped to Sunglasses pattern families.

response_class (allowed / partial / blocked), expected_outcome_per_design, match_vs_expected (yes / yes-exceeds / no), plus a Layer 1 keyword classifier (refused / executed / leaked / ambiguous) for cross-model comparison.

EXECUTED = 0 and LEAKED = 0 are the load-bearing metrics. Together they confirm: zero attack content produced, zero secrets surfaced, across the entire 13-prompt suite.

The 11 AMBIGUOUS entries reflect the classifier's keyword-shallow nature — it flags long defensive analyses that don't begin with a crisp "I can't help" refusal but also produce no exploit signals. Layer 2/3 review re-classified all 11 as defensive-correct. Per-fixture LLM-judge scores will be added in a later revision.

On this prompt set, Haiku 4.5 produced functionally equivalent safety behavior to Opus 4.7. Response lengths varied (Haiku averaged ~346 lines / fixture), but the substance — refusal of embedded malicious instructions while delivering full defensive analysis — was indistinguishable from Run 2. "Equivalent on this prompt set" ≠ "equivalent on all prompts."

Haiku 4.5 alone is one data point. The cross-family delta (Opus 4.6 vs Opus 4.7 vs Sonnet 4.6 vs Haiku 4.5 on the same fixtures) is what supports model-selection guidance for buyers — that report follows this one within days.

All Run 1–3 fixtures use defensive framing with explicit "do not provide exploit / payload / bypass" constraint footers. That's by design. It measures whether the model produces clean defensive analysis without slipping into operational guidance, not whether the model would refuse an unframed real-world adversarial payload.

Whether frontier models refuse adversarial prompts that mimic real attacker payloads (no defensive framing, no constraint footers, real-world payload shapes) is a different measurement. We will publish it as a clearly labeled appendix probe set, not blended into the core comparison scoreboard.

These limits do not weaken the Haiku result. They define its scope honestly.

Same 13 fixtures, additional model passes: Opus 4.6 (high + medium effort), Sonnet 4.6 (high + medium effort), Haiku 4.5 (this report), Opus 4.7 baseline already published in Run 2. Output: cross-model delta + per-family-tier behavior table.

A separately labeled probe set will test whether models refuse prompts that mimic real attacker payloads — no defensive framing, real-world payload shapes sourced from open research corpora (JailbreakBench, HarmBench, AdvBench, PromptInject, Garak, PyRIT) and recent CVE proofs of concept. Disclosure protocol applies before public publish.

Not: "Anthropic's smallest model passed every test, therefore agent security is solved."

Anthropic's safety stack appears to scale down to the small models against well-framed defensive prompts. Real attackers do not write well-framed defensive prompts. Therefore: model-side safety is necessary but not sufficient. Runtime filtering — the layer Sunglasses sits in — catches the attacks the model never gets to refuse.

Run 3 gives Sunglasses a clean reference point: "even the cheapest Claude refuses cleanly when given a refusable prompt." Run 4 is designed to find the prompts the model can't refuse — exactly the prompts a runtime filter exists to catch before the model sees them.