Content Safety
AIControls scans LLM prompts and MCP tool calls inline, before they're forwarded anywhere. The scanner produces findings — credential, PII, and prompt-injection/jailbreak detections — that feed the built-in content safety policies; a request a policy denies or holds because of one also appears in the audit log and Security Events (see the classifier note below for what's — and isn't — durably recorded). One path is a partial exception: the cloud proxy (Bedrock, Claude-on-Vertex) only extracts a prompt to pre-scan for Anthropic- and Gemini-shaped requests — other model families it fronts (Amazon Nova/Titan, Meta Llama, Mistral, Cohere) and Bedrock's Converse API are forwarded without a request-side scan.
Detectors
Two kinds of detector run today, and you can enable both at once:
- Pattern-based detectors — on by default, for a deployment with its own persistent store (the standard configuration). Regex and heuristic checks for credentials (private keys, cloud/API tokens, connection strings), PII (emails, phone numbers, government IDs), and direct prompt-injection phrasing (instruction-override language, fake system-prompt delimiters, encoded payloads, Unicode obfuscation). These are the detectors
content-safety-credentials,content-safety-pii, andcontent-safety-injectionenforce on by default. A deployment that explicitly disables its own store (storage.enabled: false) has no pattern detector active either, unless Session Intelligence content safety is separately enabled. - ML classifier (optional) — a self-hosted sidecar that scores prompts, tool arguments, and most LLM responses for injection/jailbreak intent on a continuous 0–1 scale, instead of matching fixed phrases. Both request- and response-side coverage have known gaps. On the request side, the cloud proxy (Bedrock, Claude-on-Vertex) only extracts a prompt to score for Anthropic- and Gemini-shaped requests; other Bedrock model families it fronts (Amazon Nova/Titan, Meta Llama, Mistral, Cohere) are forwarded without ever populating a prompt for the classifier, so no request-side classifier score exists for them. On the response side, cloud-proxy streaming responses (Bedrock, Claude-on-Vertex) are forwarded without classifier scoring, and a handful of non-Claude Bedrock model response shapes aren't parsed for scanning at all. On a regular (non-cloud-proxy) streaming response, the classifier also only sees the fully assembled reply when
llmProxy.bufferForDLP: true— left at its default (false), each streamed chunk is scanned separately, which means far more sidecar calls than one per response and a real chance of missing an attack split across a chunk boundary. Buffering itself is also capped (llmProxy.bufferMaxBytes, default 512 KB): a response larger than that has its excess excluded from the full-response scan, falling back to the same per-chunk-only coverage as the unbuffered case for whatever's beyond the cap. See Enable the injection classifier for the buffering trade-off before enabling the classifier on production streaming traffic, and for how to enable it in your Helm values. Off by default.
Evidence highlights and redaction
When a content safety detector fires, the finding records which part of the text triggered it. The finding's evidence shows that text with the trigger highlighted and a Why this was flagged list above it. Each entry names the detector and rule, then shows what was found:
| Detection | What the highlight shows |
|---|---|
| Prompt injection, jailbreak, obfuscation | The matched text itself, for example ignore all previous instructions. |
| Credentials | A typed placeholder with the last four characters when the value is long enough, for example [REDACTED:AWS_KEY:****MPLE]. The value is never stored. |
| PII | A typed placeholder, with the last four characters for identifiers such as cards, SSNs and Aadhaar numbers. Names and addresses show the type only. |
| A detector that reports no location, such as Llama Guard | Nothing is highlighted, because the detector judges the whole text. The evidence says no location was recorded. |
Redaction removes only the sensitive value inside a span. The words around it stay, so the finding still reads as the prompt that was scanned. The stored excerpt is a short window around each match, not the whole prompt, and a different secret or PII value inside that window is redacted as well. Excerpts are stored only when sampleRetention is on, and for at most 32 findings of one scan. Past that limit the finding stores no text at all. Otherwise the evidence says that no excerpt was stored.
Findings recorded before highlighting existed show their text as before, with a note that the reason is not marked.
The Aadhaar detector reports a 12-digit number only when it passes the Verhoeff checksum and starts with a digit from 2 to 9. It also ignores a bare 12-digit run that is part of a file name, path or URL segment, such as stg-123456789012.log or /orders/123456789012/. Card and phone detectors ignore only file names (stg-4155550134.log); a number in a path is still detected, because a path can carry a real card or phone number. Only bare digit runs are skipped, and a number joined to a name on only one side is not skipped. A value after = or :, set off by spaces or brackets, or written with separators (such as 123-45-6789 or 1234 5678 9012) is still detected, even inside a longer name. A group of digits that is part of a longer separated number, such as the first 12 digits of a 16-digit card number, is not reported as an Aadhaar number.
The ML classifier is provisional, not production-validated
Unlike the pattern-based detectors, the shipped classifier has not cleared internal validation against real production traffic — it's currently measured at roughly a 23.4% false-positive rate. Treat every classifier finding as a signal to review, not a verdict to act on automatically, until you've calibrated it against your own tenant's traffic.
This is why enabling the classifier does not, by itself, start holding or denying requests:
- Findings from the classifier alone (with no accompanying pattern-based detection on the same request) are excluded from
content-safety-injection's default disposition — the request is allowed. The scanner still evaluates the finding inline for that decision, but there is currently no durable, per-finding record of the classifier's own score anywhere — for any surface (prompt, tool argument, or response) or any decision. The one table that stores individual finding detail (plugin,score,entity) is written by the separate, opt-in Session Intelligence content safety scanner, and its plugin list deliberately excludes the classifier (to avoid a second, duplicate call to the sidecar for content the primary scan already scored). A request some other policy denies or holds still gets a coarse Security Events row for that decision — it just never carries the classifier's own score. - A request that trips both the classifier and a pattern-based detector is still held for approval as usual — the carve-out only applies to classifier-only findings. That hold is visible in Security Events as a
content-safety-injectiondecision, same as any other hold; it still won't show the classifier's specific score, only that the pattern-based detector's category fired. - To enforce directly on the classifier's confidence score once you've reviewed enough of your own traffic to trust it, write a policy against the score field it populates (ask your platform team for the template, or start from the built-in disabled template shipped for this purpose) and switch it from Audit to Warn/Deny yourself — since there's no built-in review surface for the raw score today, plan to capture and review it through your own external logging in the meantime if you need to calibrate a threshold.
Calibrating before enforcing
The pattern-based detectors and the classifier calibrate differently, since only one of them has a durable per-finding record today:
- Run the detector in Audit mode against real traffic for at least a couple of weeks — for the pattern-based detectors, findings accumulate in Security Events automatically; for the classifier, there is no durable per-finding record (see above), so set up your own external logging of its score/label from the start, before this period begins, or there will be nothing to review afterward.
- Review the findings — for the pattern-based detectors, in Security Events; for the classifier, through your own external logging (see the note above — there is no built-in score-distribution view yet). Either way, a cluster of high scores on traffic you know is benign means the threshold needs to move, not that the detector is broken.
- Use Replay (see policies) to dry-run a candidate Warn/Deny policy against a selected audit-history range before switching it live — for policies that don't reference content-safety fields. Replay re-evaluates a policy's CEL against each event's recorded tool/model/argument fields; it does not re-run content-safety scanning, so
object.request.*fields (PII, credential, injection, and classifier findings alike) are always absent during Replay. A candidate content-safety policy will show zero matches in Replay regardless of real traffic — validate one by reviewing real findings (the step above) instead. - Graduate gradually — Audit → Warn Approve → Deny — rather than jumping straight to blocking.