We attack our own product, and publish what breaks.
Three internal red-team campaigns have run against Aegis before any customer traffic: 286 probes, 48 findings. Some were complete bypasses of controls we believed worked. Every fixed finding is a permanent regression test; every accepted limit is stated here and in the product. Everything we ever disclosed to a partner as open has since been fixed — and stays on this page, dated, rather than being quietly deleted. Nobody in this field publishes their own failures — we think a security product that won't show you its bypasses is asking for trust it hasn't earned.
A deterministic engine can be tested honestly — so it should be.
Aegis decides in code, which means an attack either bypasses a rule or it does not, and the answer is reproducible. That makes real disclosure possible: what you read below is not a marketing summary of a pentest, it is the actual ledger — the probe counts, the bypasses at first contact, and the state of every finding today. The same corpus these campaigns built now ships in the product, where you can run it against your own policy.
What we ran, what broke, and where it stands.
All three were white-box exercises under an adversary-with-model-control threat model: assume prompt injection succeeds and the model is hostile — does the rail still hold?
| Campaign | Scope | Probes | Findings | Status today |
|---|---|---|---|---|
| 01 — Execution rail | Tool & MCP interception: name policy, constraints, streaming, decision API | 79 | 17 | 16 fixed · 1 accepted |
| 02 — Beta scenario pack | 21 end-to-end attack scenarios across the whole product, plus 28 tenant-isolation probes | 161 | 20 | 16 fixed · 3 accepted · 1 referred |
| 03 — Parity-sprint cross-fire | Thirteen new capabilities exercised against the shipped engine across 46 targeted probes on the new surfaces | 46 | 11 | 11 fixed · 5 in already-shipped code |
"Fixed" means fixed and locked: each finding became a permanent adversarial test case, so a regression fails the build. Three of campaign 02's findings were disclosed to design partners in writing rather than fixed at the time — no data-deletion capability, decision history outside the hash chain, and per-replica rate limits. All three are now closed and counted in its fixed column: the first went with the right-to-erasure work, and the other two are dated below.
The bypasses we are least proud of, in detail.
These are the ones a competitor's brochure would omit. Each one bypassed a control we had already shipped and believed in — which is exactly why the campaigns exist.
One flag skipped the whole rail
Setting stream: true bypassed tool-call interception entirely — streamed
tool fragments were simply dropped. Fixed by buffering and inspecting assembled calls
before any flush. A phase later, campaign 02 found the same gap's twin on tool
results; the transports now share one implementation.
Four spellings of the same tool name
Homoglyphs, fullwidth characters, zero-width joiners, and leading whitespace each slipped the deny-list — and all four render identically to a human reading the audit log. Names are now NFKC-normalized, control characters stripped, and confusables folded before any policy comparison.
Revoking a key made the caller stronger
On a keyless gateway, a revoked credential fell through to the anonymous public tenant — which had no policy, and therefore no deny-list. Incident response widened the incident. A presented-but-invalid credential is now rejected outright in either posture.
Percent-encoding walked past path constraints
An encoded traversal carried no literal path separator, so the argument was not classified as a path and every constraint was skipped before decoding could run. Found by our own customer red-team runner pointed at our own policy language. Paths are now decoded and normalized before classification, and relative traversals are refused under allow-lists.
Our normalizer broke our own detection
Confusable folding built for tool names was applied to prose, turning Cyrillic text into mixed-script strings no detector matched — silently disabling injection detection for every Cyrillic- and Greek-writing customer. Caught only when the multilingual packs' Russian probes failed against demonstrably correct patterns.
The vocabulary miss in our own headline payload
"Ignore previous policies" walked past a detector whose noun list said "instructions". Two capabilities the scenario pack assumed also did not exist at all — destination controls and tool-definition inspection — and were built as a direct result. The pack's assumptions were better than our inventory, which is the finding.
The limits we keep, because the honest answer is a boundary.
Not every finding gets a fix. These are deliberate limits of a deterministic, in-process engine — each one documented in the product rather than discovered by a customer.
Encoded payloads inside arguments
Base64-wrapped instructions are not decoded: retrieved documents legitimately carry base64, and the tool call that results is still inspected on its own merits. Decoding everything would trade a stated limit for a false-positive engine.
Injections with no override vocabulary
A politely-phrased instruction that never uses override language is not detected at the prompt rail. This is the honest limit of deterministic text detection — and the reason the guarantee rides on the execution rail, where the resulting action is inspected regardless of how politely it was induced.
Per-replica behavioural scoring
Cross-turn anomaly scores are computed in-process, so a load-balanced session's score is a floor, not a total. Identity keying blunts session rotation, but only for a caller that sends no session header of its own — the header wins by design, so a caller setting both is not covered. We narrowed that claim on 31 August rather than leave the wider version standing. Scoring state is a fixed-size per-tenant map: interference between tenants is closed, and an authenticated caller can still evict their own older session by minting more than a thousand of them. Stated in the module, in the config surface, and here.
Erasure has two residuals
Audit entries that never got sealed hold cleartext no key destruction reaches — counted and attributed to the subject where the failure can be traced, disclosed as unresolved where it cannot — and a backup taken before an erasure still contains the key until rotation clears it. Both are returned by the erasure API and shown in the console before an operator confirms.
The four findings this page carried as open are all fixed.
They sat here, and in writing with every design partner, for as long as they were open. They stay here now with their dates and what shipped, because a disclosure deleted once it stops being embarrassing was never a disclosure. Nothing disclosed to a partner is open today.
Decision history is committed to the hash chain
What was open: guardrail decisions lived in high-speed telemetry tables that favour throughput over immutability, while the hash-chained audit log covered only control-plane changes — so anyone with database access could delete the row recording that an exfiltration attempt was refused, and the product would report a clean chain while doing it.
What shipped: exactly the recommendation that was on file. A contiguous run of decision rows is hashed into a segment root, chained to the previous one, and that root anchored as an append-only audit entry — one entry per segment rather than per request, so the latency budget is untouched. An erasure re-anchors and records the old root beside the new one with its reason.
And it needed hardening twice, by us, within days: verification was comparing against the mutable root stored beside the rows instead of the one the audit log holds (fixed 30 August — for two days the control did not enforce what it claimed, and that window is recorded rather than smoothed over), and a re-anchor could be made to launder a field edit unless it proves which column moved (fixed 31 August).
Rate limits divide a shared budget
What was open: buckets were process-local, so a key configured for N requests per minute could sustain a multiple of N across replicas — and a flood that triggered autoscaling added replicas that each arrived with a full bucket. The control relaxed under load, which is the opposite of what a limiter is for. Found by reading code, not by a probe: a single-process test cannot observe per-process state.
What shipped: each replica heartbeats a registry once per interval, off the request path entirely, and enforces the configured budget divided by the number of live replicas. Scaling up now tightens each share instead of multiplying the total, and the hot path still reads a local integer and touches no network.
Stated honestly: this is fair-share, not a global token bucket. Under the round-robin balancing production runs, a caller sees approximately the configured budget; pinned to one replica by a sticky balancer, they see less than it. Stricter than configured is the safe direction for a security control, and it belongs in the documentation rather than in a support ticket.
The anchor is ours, not a notary's
What was open: the audit log refuses UPDATE and
DELETE, but INSERT is unrestricted by design and the
entry hash is unkeyed SHA-256, so an actor with database write alone appends a
chain-consistent forged anchor rather than being unable to forge one. The audit chain
verifies clean against a forgery nobody can distinguish from a real append.
What shipped: the anchor payload is now signed with a customer-held
Ed25519 key (ADR 059). Signature lives in the audit entry (cannot be stripped by
UPDATE). A segment that ever carried a valid signature must keep carrying one — a
forged unsigned append carrying a new root cannot reach ok = true.
Stated honestly: signature replay (same payload re-appended verbatim)
verifies under the same key, mitigated by duplicate (root, sig) detection to roughly
101 forged appends, not closed. Binding the audit entry's own
seq into the preimage is the structurally complete fix and is circular to
implement. Pre-signing segments cannot be protected retroactively. The ~100-append
truncation window (same as _weakest_proof) means exhaustion costs ~100
appends in the SIEM log. Both residuals are named in the ADR rather than implied by
"signed."
One alarm clears only through an erasure
What was open: if the attestation table itself is edited — or a
re-anchor dies between writing its audit entry and moving the root — verification
correctly reports anchor-mismatch. The alarm works and is visible. But the
operator's only route to clearing it was a data-subject erasure, which is the wrong
instrument for a row that nobody erased.
What shipped: two new ADMIN-rank routes (ADR 061). repair()
walks every segment, recomputes against the anchored prev_root, and where
rows match the anchor, heals a drifted decision_segments row or clears an
un-absorbed refusal — never moves a root, never writes where the computed root diverged
from the anchor. Idempotent. acknowledge() appends an audited entry carrying
actor, note, current row state and what the audit holds; segment status unchanged.
Idempotent.
The distinction: repair heals what the anchor proves and does nothing to a genuinely tampered segment. Acknowledge records an investigation. An operator with an anomalous segment now asks: "is this from an erasure failure I can repair, or from tampering I need to investigate?" and has the right tool for each answer.
What is open in their place.
These are residuals of the external anchor design, named rather than implied. None is a bypass we know how to run; both are stated in ADR 059 with their mitigations and costs. Published here because a page that lists only solved problems is a brochure, and because closing two findings while silently acquiring others would be the same failure with the sign reversed.
Signature replay: a signed payload re-appended verbatim still verifies
Every field in the preimage is content-derived — segment id, stream, seq, rows, root, prev_root, root_ex_identity, from, to. None is a per-append freshness value. So the bytes signed for one audit entry are byte-identical to the bytes for the same payload re-appended as a new entry, and the original signature still verifies. No forgery is involved — that is what makes it a replay rather than a break.
What it buys is not hiding a fresh tamper: the replayed anchor carries an old
root, which will not match tampered rows. It buys restoring an older
state — concretely, silently reversing a right-to-erasure: restore the
pre-scrub identity on the live row, replay the pre-erasure anchor, and
the segment reports intact / signed / ok while
carrying data that was legally erased.
The precondition genuinely narrows it: the attacker needs the pre-erasure
identity value, which ADR 056 deliberately never persists, so they must
have retained it out of band. Duplicate detection on (root, sig) pairs
raises the cost to roughly 101 forged appends in a single segment.
The structurally complete fix is to bind the audit entry's own seq into
the preimage, but that is circular: seq is assigned by the
INSERT, and UPDATE is blocked, so the value cannot be computed before the
row exists. Named as the next step rather than attempted.
The ~100-append truncation window: signature evidence ages off the end
_anchoring_entries reads at most 100 entries per segment (the same
entries_for_target limit _weakest_proof documents), and the
carried signed_once flag drains as they age off. An attacker who appends
~100 forged, correctly-chained anchors for one segment exhausts the read window; a
segment whose carried flag never reached the oldest 100 entries degrades from "ever
signed" to "never signed," a legacy case. Cost: ~100 appends in a never-pruned, SIEM-exported
log, visible and dated. Same residual as _weakest_proof, applied to the
monotonic carry-forward mechanism this design uses instead of a scan. Both are stated
in ADR 059 rather than implied by "signed."
Findings become tests. The corpus becomes yours.
Every fixed finding is a permanent adversarial test case — the suite that gates each release carries the full history of everything that ever got through. And the corpus these campaigns built ships in the operator console as a red-team runner: point it at your own policy and see which attacks it fails to stop, with the constraint to add — in-process, offline, generating no traffic against your gateway.
Found something we missed? [email protected] — reports are answered by an engineer, and fixed findings join this page.