How it works
Defenses
Attack Patterns MCP Attack Atlas What we catch Hardening manual OWASP LLM Top 10 MITRE ATLAS
Learn
Encyclopedia (new) Agent Security 101 Blog Reports CVP runs Thesis
Resources
Docs GitHub Action (live) vs Lakera vs Promptfoo Team
Theme
Honest limits

Known Version Gaps

Current as of v0.5.7, measured 2026-09-11 PT
Why this page exists

The failure this project cares most about is a tool that reports a clean result for something it never actually inspected. Staying quiet about a limitation is its own version of that, so the limitations live here.

Read this before you trust a clean result

This page mirrors KNOWN_VERSION_GAPS.md in the repository so a reader on the website sees the same corrections a reader on GitHub sees. Everything below is either reproducible from the shipped package or names the test that asserts it. Where a number is measured, the measurement date is on it.

1) Release history

An audit on May 6, 2026 surfaced historical gaps in this project's version history. We fixed what was fixable and documented the rest.

sunglasses://gaps/release-history
What we found

9 PyPI versions were never published, 7 of them pre launch plus 0.2.27 and 0.2.30 which failed as silent uploads. 8 git tags were missing for versions that did ship.

Tags

All 8 missing tags were created retroactively pointing at the right commits, then pushed.

PyPI gaps

Left alone. pip install sunglasses works fine, and retroactive uploads carrying stale code would create more confusion than they resolve.

Going forward

A preflight gate blocks any new ship that would skip a patch number, and a daily integrity check audits PyPI against git tags, the changelog and the live site so drift surfaces before it builds up.

2) Language coverage is English first

The README used to advertise a language count of 23 in four places. That number counted every language named anywhere in the ruleset as though it were covered. Measured against the shipped pattern database at this release, the real shape is four tiers.

sunglasses://gaps/languages
TierLanguagesWhat actually exists
Full ruleset (1)EnglishAll 1,540 patterns.
Dedicated patterns (13)Spanish, Portuguese, French, German, Russian, Turkish, Arabic, Chinese, Japanese, Korean, Hindi, Indonesian, VietnameseExactly 2 patterns each. One "ignore previous instructions" injection and one credential exfiltration shape.
Keyword level only (7)Italian, Dutch, Ukrainian, Polish, Czech, Azerbaijani, HebrewKeywords appear inside English scoped patterns. No dedicated pattern.
Named only (2)Persian, BengaliNo dedicated pattern and no keyword. Listed as covered, present nowhere.

Two patterns is a seed, not coverage. It catches the single most literal phrasing of one attack and nothing else, so a non English deployment should not be assumed to have English parity. The normalization layer, meaning romanization, Unicode confusables and 17 other obfuscation techniques, is language independent and does apply everywhere. Nothing about detection changed when this was corrected. No pattern was added, removed or edited. Only the claim was.

Deepening real coverage needs per language false positive corpora first, because a language whose false positive rate we cannot measure separately is a language we cannot honestly advertise.

3) What the keyword numbers mean

Four different keyword counts are all correct and all measure different things, which is exactly how a number ends up looking wrong. Measured on the v0.5.7 release tree.

sunglasses://gaps/keyword-counts
CountNumberWhat it is
keyword_entries7,683Every keyword listed on every pattern, summed. Duplicates counted once per pattern.
keywords_declared6,931Unique keywords across the whole database.
keywords6,642What the pre screen index actually holds. This is the number published on this site.
Difference289Generic keywords deliberately excluded from the index because they would pre screen almost everything.

All three are reported by engine.info(), so none of this has to be taken on trust. Pattern count is 1,540 and category count is 118, both verified at this release.

4) The firewall is best effort on very long unbroken tokens

sunglasses://gaps/unbroken-token-cost
What

Matcher cost is roughly linear in input length for ordinary whitespace separated text, and roughly quadratic for a single unbroken token such as a base64 blob, a data: URI, a minified bundle line, a long hex string or a JWT.

Why it matters

The PreToolUse firewall hook has a 10 second timeout, and Claude Code does not treat a timed out hook as a block. The tool call continues through the normal permission flow. So an input of this shape can push the hook past its timeout and the call then proceeds unscanned. That is reachable from attacker controlled input, which makes it a bypass of the enforcement surface rather than a performance complaint. The same curve stalls a repository scan in CI.

The input cap

MAX_SCAN_BYTES is 1 MB. It bounds the length of the input. It does not bound the cost, because for this input shape the cost is quadratic in that length.

Why no test caught it

The test that exists to prove exactly this builds its payload as "a " * n, which is the whitespace separated shape, meaning the fast one. The gate stayed green while the quadratic shape was unbounded underneath it.

Status

Open. A ratio based regression test documents the curve without asserting a wall clock ceiling. The repair belongs in the matcher and lands in v0.6 along with a bounded mitigation so an unscannable input shape fails visibly instead of failing open. Exact byte counts and timings for the quadratic case stay in internal release evidence, so this states the boundary without also handing over a tuned recipe.

Until then

Treat the firewall as best effort on inputs containing very long unbroken tokens. The static scanner still does not execute scanned content. This is about what gets inspected, not about what runs.

5) Scan time depends on input shape, not just size

Measured 2026-09-07 on the release head. Six payloads, each just over the 1 MiB cap so each scans exactly the capped 1,048,576 bytes, warm engine, same machine. This is the ordinary shape band and it does not include the unbroken token case above, which is the quadratic one.

sunglasses://gaps/shape-sensitivity
Payload shapeWall clock
" " * n, a run of spaces51.8 s
"." * n52.9 s
"benign filler. " * n, ordinary prose66.6 s
"filler line\n" * n73.8 s
"a " * n111.0 s
"\n" * n, a run of newlines139.8 s

Same byte count, a 2.7 times spread. It is recorded because the README previously stated a linear rate without qualification, and because a reader sizing a CI budget from that rate would be wrong by a factor of three before ever meeting an adversarial input.

6) A boundary assertion defect leaves some pattern branches unreachable

sunglasses://gaps/unreachable-branches
What

A \b written immediately before a literal that is not a word character, meaning -, . or /, can never assert after whitespace, so the alternative behind it never matches. A static sweep of the pattern database finds 198 unreachable alternatives across 89 patterns, of which 175 are high, 13 medium and 10 critical severity.

Repaired

Four were repaired, including GLS-EX-007 where every alternative began with -. That pattern matched nothing at all from 2026-04-08 until the repair.

Still open

No pattern is wholly unreachable any more. The remaining 89 still fire on their other branches, so the effect is that those rules are quietly narrower than they read. A full sweep, plus a permanent gate that makes this class unshippable, is a v0.6 item.

7) A repository scan and a file scan do not read the same bytes

Both surfaces report their coverage honestly and neither reports a false clean. They simply do not read the same bytes, and there are two separate reasons for that.

sunglasses://gaps/repo-vs-file
Oversized files

scan --file big.txt reads the first 1 MiB, reports truncated: true and inspection_complete: false, and exits 1, so a finding in the first KB of a 1.1 MB file is found alongside an explicit statement that the rest was not read. scan --repo skips any member over the 1 MB walker limit entirely, names it in skipped, and exits 3. Stated plainly, an attacker who pads a poisoned file past 1 MB is invisible to a repository scan while remaining visible to a direct file scan of that same file.

Image and document formats

The walker declines members by extension. The file path routes by content. Crossing the two lists, .bmp .gif .jpeg .jpg .pdf .png .webp are skipped and named in repository mode, while .tif and .tiff are scanned because they are simply absent from the walker's skip list. So an instruction hidden in a committed PNG's EXIF, a GIF's later frame or a PDF's annotations is reported by scan --repo as uninspected scope and by scan --file as a finding. The same instruction in a committed TIFF is found by both.

Why it is a gap and not a false clean

Every skipped member is named, in every output format, and the scan exits 3 rather than 0. Nothing claims to have read them. The skip line also says what closes it, namely scanning that path directly. A consumer that reads skipped is not misled, though a consumer that reads only the exit code learns "incomplete" rather than "there is an injection in here".

Why it is deferred

Reconciling the lists means the walker runs OCR over every committed image, which changes what a repository scan costs, seconds per image and per frame, and what it promises. That is a scope decision with a performance budget attached, taken deliberately in v0.6.

8) Smaller open items

sunglasses://gaps/open-items
Evidence database lags

attack-db/ is generated from the pattern source and has not been regenerated in some time. Re running the exporter rewrites 51 existing files and creates 409 that were never exported. The engine is the source of truth, so treat attack-db/ as a lagging mirror until v0.6 re syncs and gates it.

Pin consent gate location

sunglasses pin asks before starting your MCP servers, but that gate lives in the CLI. A Python caller that imports the firewall module and calls the probe or build functions directly still spawns the configured servers with no prompt. The CLI is the only shipped surface that reaches those functions, so nothing we distribute launches a server unasked. The boundary is still real and anyone embedding the library should know where the gate is before assuming they inherited it. Moving it changes a public API signature, so it is a v0.6 item.

Three silent skips in diagnostic commands

Outside the scan path, sunglasses check skips a settings file it cannot parse and a hook command it cannot lex, and sunglasses receipts skips a receipt line that is not valid JSON. They are the same shape as the defects this release fixed, a thing we could not read passed over without saying so, and in the first two cases that means check can report a healthy firewall while sitting next to a settings file it never managed to read. Recorded rather than fixed, so the next round starts with them on the list.

9) What "every document carries the three axes" is scoped to

A scan document carries threat_found, inspection_complete and is_clean. That sentence is true of every surface enumerated in the acceptance matrix, 28 surfaces across 12 input states, each asserted by a test. It is scoped to those rows deliberately, because an external review refused the unqualified version of the claim and was right to. At that point several convenience functions had no rows and two were returning documents with no axes at all. They have rows now and the defects are fixed, but the honest form of the claim names its scope, because the argument that made it false once is available again the moment somebody adds a surface without adding a row.

sunglasses://gaps/three-axes-scope
Outside the claim

Anything a caller assembles by hand from ScanResult attributes without going through normalize(). The firewall hook verdicts, which answer allow, deny or defer for a tool call rather than a coverage question. And info(), check, version and the other non scan commands.

How to check it rather than take it

The matrix test module runs one test per asserted cell, and the table generator reads the same module the tests parametrize over, so the table cannot claim a cell the suite does not assert. Note what that does not prove. A generated table proves the table and the tests read one source, not that the tests assert anything. That is what the 18 mutation cases in the same file are for, each feeding a response with its coverage evidence stripped and requiring it to fail.

10) Three input states that answer differently

Behaviour changes rather than bug fixes, and each one is a matrix state.

sunglasses://gaps/input-states
Non regular inputs are refused

A FIFO, socket, device node or directory at the input path is an operational error, meaning CLI exit 2. Previously the readability probe proved a path readable by opening it, and opening a FIFO with no writer blocks in the kernel, so an MCP scan of a named pipe never returned at all. The type check now happens on metadata before any file object exists. Note the consequence, scan --file /dev/null is now exit 2 where an empty regular file is exit 0. A device is not a thing we can scan. An empty file is.

Undecodable bytes are incomplete, never clean

Every text read used to ignore decoding errors, which silently drops undecodable bytes. A 256 byte file of non UTF-8 pairs scanned 128 bytes and returned complete and clean. We still scan what decodes, because throwing the file away would lose real coverage, but the answer is exit 3 and the warning names how many bytes went unread. On stdin the answer differs because the transport does. Undecodable stdin is an operational error, since decoding with replacement there would mean reporting on a substitution the caller never sent.

Empty input is complete and clean

An empty text argument, empty stdin, an empty file and an empty MCP string all return exit 0 with 0 bytes scanned. Nothing went unread, so nothing was hidden. Both human renderings say the input was empty rather than reporting no threats detected, so a reader can tell a clean scan of a document from a clean scan of nothing.

11) Open limit as of v0.5.7

sunglasses://gaps/bash-path-policy
Closed

blocked_paths in the firewall policy asks what a call touches. For Write, Edit, MultiEdit, NotebookEdit and Read that question is now answered from the documented path fields, so prose that merely names a protected path is no longer treated as touching it.

Still open, on purpose

For Bash the same false positive remains. Subtracting quoted heredoc bodies from a command before asking the path question was tried twice and was unsafe both times. An independent review executed nine commands where the parser removed text the shell really runs, and seven of them survived the hardened second attempt. The failure underneath is that a fallback for a parser which fails does nothing for a parser which is confidently wrong.

Consequence

A Bash command is judged on all of its text, so writing about a blocked path inside a shell heredoc is refused. The repair needs a real shell grammar rather than another special case, and a named test asserts the limit is still present so it cannot quietly disappear.

Found something we have not listed

A gap we have not written down is worth more to us than a gap we have. Send it to [email protected] or open an issue on GitHub, and if it is real it lands on this page.