python3 tests/benchmark/precision_recall.py reproduces every number on this page, including the one miss we publish on purpose.Most scanners publish a pattern count. We publish precision and recall — measured on a labeled dataset that ships inside the repository, sealed by a SHA-256, reproducible in one command. Including the miss.
A pattern count is marketing. Precision and recall are accountability.
Security scanners advertise how many patterns they ship, and almost never what fraction of real attacks those patterns actually catch — or what they wrongly flag. A number nobody can check is a slogan.
The dataset, the runner, and the results all live in the public repository. python3 tests/benchmark/precision_recall.py regenerates every number on this page. If we're wrong, you can prove it.
This is a self-published benchmark, not a third-party audit — and it says so. What makes it worth your trust is that it's deterministic, SHA-sealed, and it publishes the miss, not just the catches.
We were wrong once building this, and we kept the correction. An earlier draft claimed novel-semantic attacks were "0% — the honest limit of static scanning." Then we actually read the eight cases and found six had crisp structural invariants. The real number is 87.5%. Publishing a limit you haven't tested is how a benchmark becomes marketing.
v0.3.1 added the mechanism layer: 11 rules that match attack shape, not wording.
On the identical dataset and command, the engine went from 64.3% precision / 47.4% recall / 0.545 F1 to 78.7% / 97.4% / 0.871 — while the false-positive count on 73 famous READMEs did not move.
Attacks whose structure the engine has seen before: injection phrasings, exfil-to-URL, authority inversion, zero-click image exfiltration. Every one caught.
Attacks deliberately rephrased so that no pattern in the database matches the wording. The mechanism layer catches 7 of 8 by their structural invariants. The eighth is the known gap below.
The reason a big pattern database still dies to a paraphrase — and what we did about it.
Literal phrasings and their obfuscated variants across 23 languages — fast, precise, auditable. Its honest weakness: it knows "ignore all previous instructions" and has no idea what "forget what your operator told you earlier" means. exact wording
Matches the attack's invariant. Exfiltration, in any phrasing, must name something sensitive and somewhere to send it — drop either half and it stops being the attack. Direction matters: sending a secret to a URL is exfiltration; reading a credential from one is half the HTTP clients on PyPI. structural invariant
A mechanism stands down when a carrier of the same category already fired — no double reporting. And sentence-scoped defensive-framing detection downgrades prose that merely describes an attack, which is what security READMEs (including ours) are made of. Net new false positives from the layer: zero. restraint, enforced by tests
One attack in the dataset is deliberately not flagged. Here is the arithmetic.
deno, ollama, and grype all ship curl … | bash as their legitimate install line. No text-level rule separates the honest one from the malicious one — the difference is what the script does at runtime, which is runtime-control territory, not a text scanner's. Flagging it buys 1 catch and 5 false positives on famous projects. A test asserts we do not flag it, so nobody fixes it by accident: deleting that test means having read why it exists. If a text scanner claims to catch this, ask what its false-positive rate on real READMEs is.
Everything you need to check us, in one place.
tests/benchmark/attacks.json — 38 labeled real agent-input attacks. An attack counts as caught when the decision is block or quarantine.
tests/fp_real_world_corpus/*.md — 73 real famous-repo READMEs (react, kubernetes, vue, numpy, pandas, ollama, transformers…) that must stay clean. Any flag on one is a false positive. Our own README is in the corpus as a mirror test — it documents real attacks and blocks, by design.
No randomness, no network, no LLM judge. Same clone + same command → byte-identical output, sealed by a SHA-256 of the metrics block: d3200d2a3344e068…
The engine is imported from the local package — the benchmark measures exactly what ships on PyPI, not a lab build.
git clone https://github.com/sunglasses-dev/sunglasses && cd sunglasses && python3 tests/benchmark/precision_recall.py (add --json for machine-readable output)
What this does not prove: it is our dataset, and a benchmark only measures the misses its authors thought to write down. Your traffic is not our corpus. For the full honest scope — what the scanner catches and what it structurally cannot — read What We Catch vs What We Don't.
The benchmark, the dataset, and the engine are all in the open repository. Reproduce the numbers, probe past them, and if you find a miss we didn't list — report it. It becomes a test, and the ratchet only turns one way.
At v0.3.1: 78.7% precision, 97.4% recall (37/38), 0.871 F1 on a labeled dataset shipped in-repo — 38 real agent-input attacks as positives, 73 famous open-source READMEs as negatives. By segment: 30/30 known-shape, 7/8 novel-semantic paraphrases the pattern database has never seen.
Clone the repo and run python3 tests/benchmark/precision_recall.py. Deterministic — no randomness, no network, no LLM judge. Byte-identical results, sealed by a SHA-256 of the metrics block, measured against the engine that actually ships.
10 of the 73 clean READMEs still flag — all pre-existing carrier-pattern false positives (one is our own README, which blocks by design). They're tracked in a one-way ratchet that only shrinks. The mechanism layer added zero: 10/73 before, 10/73 after.
The one miss is curl … | bash. Five of the 73 clean READMEs (deno, ollama, grype) ship that exact line legitimately — flagging it buys 1 catch for 5 false positives. It belongs to a runtime control, and a test asserts we do not flag it.
No — self-published, and we say so. What makes it trustworthy is that it's falsifiable: the dataset and runner are public, the output is deterministic and SHA-sealed. Independent replication is welcome; misses we didn't list become tests.
Because published numbers from other tools were measured on their datasets, not this one — and cross-dataset comparisons are how benchmarks turn into marketing. Run ours; run theirs. Same-dataset comparisons are the only ones that mean anything.