Skip to main content

Runtime Controls Reference

Reference

Lookup material for Runtime Controls — endpoints, the capability-state contract, reconciliation row and destination fields, suppression reasons, and destination identity rungs. For what these mean and how to read them, see Runtime Controls.

Self-managed installations only

Workload Trust is available only in private, self-managed AIControls installations, on premises or in a cloud cluster that you run. It needs a dedicated cluster that you run, with Kyverno, Kyverno Runtime, and cluster OIDC and RBAC access in place. Nirmata-hosted (SaaS) workspaces don't include it. See Feature availability.

Helm chart values​

The AIControls Helm chart bundles the Kyverno runtime sensor as a subchart, installed by default — see Enable Runtime Controls for the install flow and the upgrade caveat on the collector's certificate. These are the runtimeControls values that field.

For the gRPC push-sink collector specifically — its ports, its self-generated PKI, the bring-your-own-certificate override, and how it composes with the OpenReports ingest path — see Runtime Push-Sink Collector.

ValueDefaultMeaning
runtimeControls.enabledtrueTurns on both the AIControls-side integration and the bundled sensor subchart. Setting this to false disables both together.
runtimeControls.posture.enabledtrueEnables the separate Kyverno posture detection and report reader, with its posture ClusterRole, on eligible self-hosted Kubernetes installations. Setting this to false disables the posture reader while leaving Runtime Controls and its OpenReports ingester enabled. Omitting this key from your override file keeps the merged chart default true. Explicit false or null disables it; only a key absent from the final template values uses the defensive false fallback. See Kubernetes Posture.
kyverno-runtime.enabled(unset — falls through to runtimeControls.enabled)Opts out of the bundled subchart specifically, while leaving runtimeControls.enabled on — use this to manage the sensor as your own separate release.
runtimeControls.collector.enabledtrueTurns on the gRPC push-sink listener the bundled sensor pushes findings to. Explicitly setting this to false while the bundled subchart is still installed fails the chart render, since the sensor would then have nowhere to push.
runtimeControls.sensorNamespace(unset)Overrides which namespace the capability probe and ingest-lag detector look for the sensor's pods in. The chart sets this to the release's own namespace automatically when the bundled subchart is installed; leave it unset for a customer-managed sensor in the conventional kyverno-runtime namespace, or set it explicitly if yours runs somewhere else.
runtimeControls.networkObserveExcludeNamespaces[]See Scope the network observation.
runtimeControls.sensorPathGrammardirectory-prefixHow the deployed sensor matches exec and open path values: directory-prefix for kyverno-runtime v0.1.3 or later (the chart bundles v0.1.4), exact for v0.1.2 or earlier. It decides whether a files: or exec: tier with a directory value compiles. See Path values and the sensor path grammar.
runtimeControls.imageReads.enabledtrueLets AIControls read the config and symlinks of the images of registered workloads from their registries, anonymously, so an exec rule names the program the kernel reports. Set to false for an air-gapped or rate-limited cluster. See Image entrypoint and symlinks.
runtimeControls.categories.credentialFilestrueToggles the credential-open observation shape — see Finding categories below.
runtimeControls.maxFindingsPerWorkload1000Caps the distinct finding series AIControls stores per workload. 0 means unlimited. See Retention and limits.
runtimeControls.podEvents.enabledtrueTurns on the Pod-scheduled event watch: one watch per replica on core/v1 Events, filtered on the API server to involvedObject.kind=Pod,reason=Scheduled, so a new Pod is attributed to its workload within seconds. Adds list and watch on events to the runtime controls ClusterRole; with false, the chart doesn't request them. See How new Pods are attributed.
runtimeControls.podEvents.maxConcurrentLookups4The most targeted Pod lookups one replica runs at once, shared by the event watch and the on-demand lookup. From 1 to 32; 0 uses the default.
runtimeControls.snapshotRelistInterval15mHow often each replica relists every Pod, ReplicaSet, and Job to rebuild its attribution snapshot. A duration from 1m to 24h; any other value fails the chart render or startup.
runtimeControls.podOwnerCacheTTL60sHow long the request-time Pod lookup reuses a successful Pod-to-workload resolution. A shorter value notices a deleted Pod sooner but spends more of the lookup budget. A duration from 1s to 10m; any other value fails the chart render or startup.
runtimeControls.findingLabels[]Pod label keys AIControls records on every runtime finding, such as ebay-app-id. See Workload labels on findings.
runtimeControls.credentialFilePaths[]Overrides the default path-suffix set the credential-open shape watches for. Empty uses the built-in default: /.aws/credentials, /.env, /.aws/config, /.kube/config, /.ssh/id_rsa, /.ssh/id_ed25519, /.docker/config.json, /.npmrc, /.pypirc, /.git-credentials. Monitor-only, like every other observation shape, and requires the open/exec hooks — see Observation families.

On the API/config side, runtimeControls.sensorNamespace maps to config.RuntimeControlsConfig.SensorNamespace, whose own default (used outside the Helm chart) is kyverno-runtime — the conventional namespace a separately-installed sensor release uses.

Endpoints​

Read-only endpoints. All require an admin or viewer role; a developer (builder) token cannot read cluster-wide runtime findings.

EndpointReturns
GET /api/v1/runtime/statusCapability snapshot: CRD presence, per-node sensor and open/exec hook state, RBAC gaps, ingest lag
GET /api/v1/runtime/reconciliationOne row per workload for the window, sorted by bypassedDestinations descending
GET /api/v1/runtime/workloads/by-ref/{namespace}/{kind}/{name}/observationsRaw observations for one workload, grouped by signal family. Use _ as {namespace} for a cluster-scoped resource.
GET /api/v1/runtime/workloads/{key}/observationsDeprecated. Use the by-ref route. The {key} is the percent-encoded <namespace>/<Kind>/<name>, which Envoy Gateway redirects by default.
GET /api/v1/runtime/workloads/by-ref/{namespace}/{kind}/{name}/sequence-matchesSequence-detector matches for one workload. Use _ as {namespace} for a cluster-scoped resource.
GET /api/v1/runtime/workloads/{key}/sequence-matchesDeprecated. Use the by-ref route. The {key} is the percent-encoded <namespace>/<Kind>/<name>.
GET /api/v1/runtime/observations/{fingerprint}Hydrates one observation row by fingerprint, for the unified Events panel's runtime-plane drill-down
GET /api/v1/runtime/findingsThe workload/category rollup, filterable by namespace, category, and workload — see Findings below

One write endpoint excludes a namespace from runtime data. It requires the admin role; see Exclude a namespace from runtime data.

curl "$YOUR_WORKSPACE_URL/api/v1/runtime/reconciliation?window=6h" \
-H "Authorization: Bearer $ADMIN_TOKEN"

The window parameter​

/reconciliation and /observations accept an optional window duration (for example 15m, 1h, 6h, 24h).

  • Default: 1h
  • Clamped to the range 5m – 24h
  • An unparseable value falls back to the default rather than erroring

The key path parameter​

Workload routes address a workload with three path segments: {namespace}/{kind}/{name} after /workloads/by-ref/. For a cluster-scoped subject, use _ as the namespace. A request that omits a segment doesn't match the route and returns 404. A segment that does not decode, or a decoded segment that contains /, returns 400.

The older /workloads/{key}/... routes take the canonical <namespace>/<kind>/<name> string percent-encoded as a single segment. They're deprecated. Use the by-ref routes instead.

curl "$YOUR_WORKSPACE_URL/api/v1/runtime/workloads/by-ref/search/Deployment/rag-indexer/observations" \
-H "Authorization: Bearer $ADMIN_TOKEN"

Exclude a namespace from runtime data​

POST /api/v1/runtime/exclusions/namespaces adds a namespace to the excluded namespaces in Settings > Workloads > Cluster Discovery and deletes the runtime events already stored for it. That list is the single exclusion list: discovery, the observation policies, and runtime ingest all skip the namespaces on it.

curl -X POST "$YOUR_WORKSPACE_URL/api/v1/runtime/exclusions/namespaces" \
-H "Authorization: Bearer $ADMIN_TOKEN" \
-H "Content-Type: application/json" \
-d '{"namespace":"load-test"}'
{"namespace":"load-test","excludeNamespaces":["kube-public","kube-system","load-test"],"deletedEvents":42}
FieldMeaning
namespaceThe namespace you excluded
excludeNamespacesThe full excluded-namespace list after the change
deletedEventsThe number of stored runtime event rows deleted for the namespace: finding series and their occurrence history

The request is idempotent. If the namespace is already excluded, the list stays the same and AIControls deletes any events stored since. The list change and the deletion commit together, and the configuration audit log records both. The endpoint returns 400 for a value that is not a valid Kubernetes namespace name, 409 when concurrent changes to the list keep conflicting, and 503 when the change cannot be stored.

The replica that serves the request applies the change at once. Other replicas apply it within 30 seconds, and the sensor stops observing the namespace after the next policy reconcile.

Rescan workloads​

POST /api/v1/workloads/rescan relists the Pod attribution snapshot on every replica and runs a full discovery scan. It takes no body. It's the API behind Rescan on Inventory > Workloads, and it's admin-only: a complete discovery scan marks workloads it didn't see as gone.

AIControls records the request in the configuration audit log as workload_rescan.requested before it relists or scans. If the audit record can't be written, the endpoint returns 500 and nothing runs. The replica that serves the request relists at once; the other replicas relist within 10 seconds.

The response reports both steps:

FieldMeaning
snapshotRefreshedtrue when the serving replica relisted its Pod snapshot
snapshotReasonWhy the relist didn't run, for example when runtime controls aren't running
discoveryThe discovery scan result: seen, new, gone, complete, and skippedKinds
discoveryReasonWhy the scan didn't run or failed

Pod attribution metrics​

Each replica exports these Prometheus counters for the event watch and the on-demand lookup:

MetricLabelsCounts
aicontrols_runtime_pod_events_totaloutcome: queued, known, excluded, dropped, invalidPod-scheduled events received. dropped means the lookup queue was full; the next relist covers those Pods.
aicontrols_runtime_pod_lookups_totalsource: watch, request, reconcile; result: inserted, not_found, uid_mismatch, errorTargeted Pod GETs
aicontrols_runtime_pod_event_watch_restarts_totalreason: closed, expired, errorWatch restarts. expired is a 410 Gone; the watch lists the events it missed, then resumes.

The disabled shape is a contract, not an error​

When the feature is off, the tenant is Nirmata-hosted, or the controller could not start, every endpoint above returns HTTP 200 with enabled: false and a human-readable reason. None of them ever return a 4xx or 5xx for this case.

The enabled and reason fields are common to all of them. The rest of each payload is that endpoint's own data fields, present but empty:

EndpointEmpty-data fields in the disabled shape
/statusnodes: [], rbacGaps: []
/reconciliationrows: []
/workloads/by-ref/{namespace}/{kind}/{name}/observations (and the deprecated /workloads/{key}/observations)groups: [], plus workloadKey echoing the key you requested
/observations/{fingerprint}found: false, plus fingerprint echoing the value you requested
/findingsfindings: []

For example, /reconciliation:

{
"enabled": false,
"reason": "unavailable on Nirmata-hosted (SaaS) tenants: Workload Trust requires a self-managed installation",
"rows": []
}

This is deliberate: a client — and the admin console page — must be able to state why the surface is dark rather than render an empty view that looks like a clean result. The reasons are:

reasonMeaning
not running in KubernetesThe environment predicate failed; the config flag is irrelevant
unavailable on Nirmata-hosted (SaaS) tenants: Workload Trust requires a self-managed installationHosted deployment (deployment.mode: hosted, or auth.handoff configured); fails closed even with runtimeControls.enabled: true
disabled by configEnvironment is eligible, but runtimeControls.enabled is false
runtime controls is enabled but inactive on this replica (no Kubernetes API access or no shared store)Configured and eligible, but this replica did not start the controller
runtime controls is waiting for the canonical Kubernetes cluster identity ...Configured and eligible, but this replica can't read the kube-system namespace UID yet. Runtime controls start when it becomes readable
runtime controls did not start after the cluster identity was recovered; retrying ...The UID is readable, but the latest start failed. The reason names the failed attempt; the proxy log has the cause. AIControls retries with backoff
runtime controls is not configuredNo configuration is present at all

Kubernetes posture endpoints​

These endpoints read Kyverno policy-report evidence and group it by policy category. They are separate from the runtime sensor observations above. In the table, Portal permits admin or viewer roles; Admin permits only admins. See Kubernetes Posture for report selection, coverage states, and setup.

EndpointAccessReturns
GET /api/v1/posture/statusPortalDetection state, engine/distribution/version evidence, policy inventory, selected report API, latest access-probe rbacGaps, mirrored fact/workload counts, and last observation time.
GET /api/v1/posture/summary?namespace=PortalPer-category result counts, policy counts, findings by severity, and failing workloads, with the active selection. The optional namespace narrows the rollup.
GET /api/v1/posture/workloads/by-ref/{namespace}/{kind}/{name}PortalSame response as the {key} route, with the namespace, kind, and name as separate path segments. Use _ as {namespace} for a cluster-scoped resource.
GET /api/v1/posture/workloads/{key}PortalDeprecated: use the by-ref route, which Envoy Gateway doesn't redirect. One workload’s counted verdicts grouped by category, including state, reason, counts, result details, the active selection, notScanned, and last observation time. The key is percent-encoded <namespace>/<Kind>/<name>, or /<Kind>/<name> for a cluster-scoped resource.
GET /api/v1/posture/policy-selectionPortalThe stored selection (mode and policies) and the available installed policies.
PUT /api/v1/posture/policy-selectionAdminReplaces the selection wholesale. mode is all or selected, and selected needs at least one policy. Invalid selections return 400.
GET /api/v1/posture/workloads?search=&page=&pageSize=PortalSearch and paginate workload keys. Returns enabled, items (each with workloadKey), and the matching total.
GET /api/v1/posture/workloads/by-ref/{namespace}/{kind}/{name}/assessmentPortalSame response, and the same factRef and standard parameters, as the {key} route.
GET /api/v1/posture/workloads/{key}/assessmentPortalDeprecated: use the by-ref route. The deterministic baseline, stored advisory assessment (including typed evidence and independent gaps), stale status, optional enrichment, and evidenceTotal / evidenceTruncated preview metadata. Counts cover all collected facts; result previews contain at most 100 entries.
GET /api/v1/posture/workloads/{key}/assessment?factRef=&standard=PortalOne current citation as {item: WorkloadResult}, including facts outside the preview. Requires a 64-character hexadecimal fact reference and a nonempty UTF-8 standard value of at most 128 bytes. The parameter keeps the name standard for compatibility, but its value is the Kyverno policy category. Invalid parameters return 400, an absent fact or a fact outside that category 404, and workloads exceeding the 10,000-fact limit 422. Does not call an LLM.
GET /api/v1/posture/assessment/settingsPortalThe current assessment configuration directly, or default settings when the integration is inactive. Its enabled field controls assessment generation, rather than reporting integration availability.
PUT /api/v1/posture/assessment/settingsAdminReplaces the assessment configuration and returns it. Invalid settings or unknown fields return 400; an inactive integration returns 503.

When posture is disabled, unavailable in the deployment environment, or inactive on the replica, evidence, inventory, policy selection, and assessment GET requests return HTTP 200 with enabled: false and a human-readable reason. This is an availability response, not evidence that the workload passed. Assessment settings reads are different: they return the configuration directly, or defaults when the integration is inactive; their enabled field controls assessment generation. The policy selection PUT returns 503 when posture is unavailable rather than accepting an update.

assessment.evidence contains stored context snapshots with ref, kind, summary, optional JSON details, and Unix-second observedAt (zero means unknown). Kinds are availability, policy, configuration, runtime, and declaration. Findings with standard: "context" cite these entries through context: references and carry informational severity; they are not report verdicts. Resolve them from the assessment snapshot, not the factRef / standard report endpoint. assessment.gaps lists deterministic collection and coverage limitations independently of the generated rating. Neither the rating nor generated findings enter policy evaluation or workload verification.

Common posture fieldMeaning
enabledWhether the posture integration is available for this request. If false, read reason.
reasonWhy the integration is disabled or inactive. The posture flag’s disabled reason names runtimeControls.posture.enabled: false.
rbacGapsOn posture status, missing grants or incomplete-probe diagnostics from the latest access probe at startup, reader creation/restart, or lease handoff. Each authorization pass has a 30-second deadline; incomplete API discovery means authorization coverage is unknown, while positively absent CRDs are not gaps. Later permission failures affect detection or reader health until the next probe refreshes this shared snapshot.
rbacGaps[].reasonOptional explanation for a gap or diagnostic. An api-discovery diagnostic uses API discovery incomplete; authorization coverage unknown; it is not a missing grant to add.
userStateEngine and policy-inventory state, such as no-engine, engine-policies-unknown, or engine-with-policies. It does not by itself establish report-reader health.
standards[].stateevaluated or unknown. Missing report access, incomplete policy inventory, and unhealthy report mirroring keep standards Unknown even if policies are installed. A per-workload standard is also Unknown when it has no classified verdicts, even if cluster-level coverage is available.
standards[].findingsBySeverityFail/warn counts by normalized severity: critical, high, medium, low, info, or unknown. Values are trimmed and lowercased; unsupported values become unknown. A supported installed policy annotation takes precedence over report severity.
standards[].reasonWhy the standard is Unknown; zero findings must not be used in place of this coverage state.

Posture reader health is renewed every 5 seconds and expires after 15 seconds without renewal. Expired records and older records without an expiry produce Unknown until renewed; this is bounded freshness, not instantaneous crash detection. Heartbeats preserve pending/Unknown states. Each full relist starts pending and becomes healthy only after all pages are mirrored and the watch is established. List pages and mirror writes have 30-second deadlines; one-minute watch deadlines reconnect from the last successfully mirrored resource version. Full relists remain on a 10-minute deadline across reconnects; watch errors, expired resource versions, and persistence failures trigger an earlier relist. Reconnects are pending until the watch resumes from a valid snapshot cursor.

Capability states​

Every capability fact is three-valued, and unknown is never conflated with unavailable.

StateMeaningRead it as
availableThe capability was positively establishedEvidence of this class can be produced
unavailableAbsence was positively establishedEvidence of this class cannot be produced here
unknownThe evidence to decide is not presentCannot tell — not clean, and not absent

unknown arises from, among others: no capability probe has completed yet; a node's status shard is older than the freshness window (two minutes — four missed 30-second flushes); or no positive hook evidence source is wired.

Open/exec hook evidence comes from the sensor's raw event counter, nirmata_runtime_events_ingested_total. A node is available once it reports events from an open/exec hook source: openexec-observe on current sensors, or lsm-observe on older ones. Both hook types (BPF-LSM and the security_file_open program) report through these sources. The exec-trace source does not count, because it comes from a tracepoint that works without BPF-LSM.

The distinction is load-bearing rather than pedantic. A node that has simply been idle since the sensor started looks identical to a node whose hook never loaded, so idleness must yield unknown. Reporting it as unavailable would be a false negative; reporting it as available would be a fail-open — and a fail-open of exactly this kind shipped once and had to be fixed, when a hosted runner with no hook evidence was reported as capable.

status fields​

FieldTypeNotes
enabledboolFalse means read reason; all other fields are empty
crd / crdReasonstate / stringWhether the RuntimePolicy CRD is served. Re-probed every 5 minutes, so a later sensor install is picked up without restarting AIControls
bpfLsm / bpfLsmReasonstate / stringThe policy-level open/exec hooks answer across the fleet. The field keeps its bpfLsm name, but it covers both hook types
nodes[]arrayPer-node node, sensorAlive, bpfLsm, reason
rbacGaps[]arrayPermissions the ServiceAccount lacks, from the startup SelfSubjectAccessReview: group, resource, verb, namespace
ingestLagobjectSee below
probedAtUnixintWhen the capability probe last ran. 0 means never — state is unknown, not clean

ingestLag​

Reconciles the sensor's own counters against what AIControls stored.

FieldMeaning
lagKernel observations the sensor recorded that are not reflected in stored findings. A figure > 0 indicates possible loss
perNodeThe same shortfall, per node
scrapedAtUnixWhen the sensor counters were last read. 0 means never scraped — which is unknown, not clean
scrapeErrorSet when either side of the comparison could not be read. When present, any lag figure is untrustworthy and completeness is unknown

A small transient lag is usually just the sensor's roughly-10-second report flush. A figure that persists across refreshes means findings were dropped — kernel map pressure, or the sensor's 500-result-per-report cap, after which later findings are discarded permanently.

On a busy cluster the common cause is network volume: that policy carries no allow list by design, so in-cluster service-to-service traffic is reported alongside external traffic and competes for the same cap, and a genuine provider destination can be crowded out. This is the operator-visible symptom of an incomplete table — loss surfaces as a persistent lag figure, not as a low count with nothing to explain it.

Per-workload traffic routing​

GET /api/v1/workloads/{namespace}/{kind}/{name}/routing is a read-only Portal endpoint available to administrators and viewers. Builder tokens cannot access it. It reads shared-store observations without running cluster or network probes.

Use cluster to select an exact cluster; omission selects the configured cluster. window defaults to 1h, accepts duration values such as 10m, and is clamped to 5m–24h. The window is half-open: start inclusive, end exclusive. The start advances to the current controller instance's first-seen or recreation boundary when needed.

The drawer uses the selected workload row's cluster. The full detail page uses the cluster of its served workload record; existing full-page navigation remains cluster-local. A routing-only cluster override does not change the scope of that page's other sections.

FieldMeaning
enabled, reasonWhether this deployment can serve routing evidence and, if not, why
found, cluster, namespace, kind, name, workloadUidAccepted workload observation and its exact instance scope
windowStartUnix, windowEndUnix, snapshotAtUnixEffective observation window and shared snapshot timestamp, in Unix seconds
gatewayWindowStartUnixLater positive-call boundary: the completed exact binding snapshot rounded up to a whole second, bounded by the effective window. Earlier calls cannot be reassigned after Pod adoption
historyStartUnixEarliest retained runtime occurrence-history boundary
routing.llm, routing.mcp, routing.tlsIndependent routing evidence for each plane
routing.historyIncomplete, routing.coverageReasonEvidence coverage explanation, separate from protection and routing status. In this response, history is incomplete when the window starts before historyStartUnix. The continuity rules in this row apply only to the R6 readiness read, which uses a separate routing read and the sensorOutage field, not this endpoint. There, history is complete only when the sensor reported healthy on every listed node with no possible ingest loss, without a gap longer than about 12 minutes. A sensor outage or a gap inside the window makes the R6 rule fail with a reason that names the outage, across the whole window (clipped to the start of the current healthy run)
routing.bindingReasonExplains why positive gateway evidence starts later than the runtime observation window and why quiet workloads need fresh calls
routing.notReadyReasonWhy routing evidence was unavailable, such as no sensor data yet. Present only when evidence could not be read; absent when evidence was read and no traffic was seen

Each plane contains status (verified-via-proxy, bypass-observed, or unknown), reason, proxyCalls, bypassObservations, splitTraffic, and optional lastProxyUnix and lastBypassUnix source timestamps. TLS cannot be verified-via-proxy and never exposes a positive proxy-call count. Bypass observations take precedence over proxy calls, including when runtime occurrence history is incomplete. Without complete window history, proxy-only evidence remains unknown.

Positive LLM and MCP evidence requires a verified audit identity, an exact event Pod UID, and matching cluster, workload, namespace, and client identity. A ServiceAccount name or last-seen workload tuple alone is insufficient. Fresh current-controller and Pod UID snapshots are required for every plane. Missing, stale, or incomplete bindings and classification facts return unknown, not an empty success claim.

Snapshots are limited to 10,000 cluster pods and 2 MiB. A workload can bind up to 1,000 pods; a query can classify up to 5,000 aggregated evidence series. Exceeding a limit returns unknown with an explanation. Snapshot freshness is limited to 30 minutes. These limits do not truncate protection facets or change policy enforcement.

See Traffic routing for interpretation and coverage limitations.

Reconciliation payload fields​

Top-level fields on /reconciliation, alongside enabled and reason.

FieldTypeMeaning
windowStartUnix / windowEndUnixintThe window actually used, after clamping
historyStartUnixintEarliest available delta-history boundary: installation or retention cutoff, whichever is later. History is retained for 90 days, independently of audit retention. This does not guarantee complete sensor coverage after that time
historyIncompleteboolThe selected window starts before historyStartUnix. Show a warning alongside counts: earlier lifetime counters are not backfilled, and a low count is not evidence of no traffic
rows[]arrayOne row per workload — see below
networkDetectorDisabledboolTrue when the network-based detector stood down for this window. Two causes produce it — the egress-NAT auto-detect, and unusable cluster facts — so the flag alone does not say which. Network-sourced evidence is suppressed with the matching reason (egress-nat-detector-disabled or cluster-facts-unavailable); DNS-based findings are unaffected in either case
disabledReasonstringWhy, in prose, when networkDetectorDisabled is true. This is what distinguishes the two causes — present so a client can state the reason rather than render a thinner table with no explanation
ingestLagobjectRepeats /status's lag figure so the table can show its banner without a second request
networkDetectorDisabled: true means the window is thinner, not cleaner

Connections are the dominant evidence class. A window where the network detector stood down is backed by DNS evidence alone, so a low bypassedDestinations in that window carries much less weight than the same number in an ordinary one. Surface disabledReason — never present the count on its own, and never assume the egress-NAT cause: read the reason to find out which stand-down fired.

Reconciliation row fields​

FieldTypeMeaning
workloadKeystringCanonical <namespace>/<kind>/<name>
namespace, kind, namestringThe same, split out
bypassedDestinationsintThe headline. Count of distinct unsuppressed provider destinations this workload demonstrably reached outside governance — by kernel-observed connection, or by resolving the provider name itself. Both evidence classes feed this one set; there is no separate connection count. A set count: unit-free, unaffected by DNS caching or connection reuse. 0 always means the set was empty
mintRaw evidence. Positive occurrence deltas reported in the window for unsuppressed external provider destinations, not lifetime totals. Cumulative deltas use the report's latest observation time; pushed findings use their occurrence time
nintRaw evidence. Gateway-recorded calls joined to this workload — every audited request type (MCP tool calls, LLM requests, and egress), not only provider traffic
dnsOnlybooltrue means the row's entire unsuppressed evidence is DNS resolutions, with no observed connection behind any of it — a weaker finding than a connection-backed one. false means either at least one destination was connection-backed or the row has no unsuppressed evidence at all. It must be read together with m / bypassedDestinations, never alone
joinMethodstringAlways name-based: a heuristic join of gateway identities to workloads, never a cryptographic binding
attributionConfirmedboolFalse means the pod could not be resolved to an owning workload and the row fell back to pod-level attribution. The row keeps its full headline count — the evidence is real; only who is uncertain. Rendered as "attribution unconfirmed"
attributionNotestringpod-level (multi-container) when a known mesh or telemetry sidecar was detected: the sensor attributes to the pod cgroup, so sidecar egress cannot be separated from the application's
destinations[]arrayPer-destination evidence, suppressed entries included
m and n are not comparable

m counts kernel events; n counts gateway requests; the populations do not overlap. HTTP/2 multiplexing means one connection carries many requests, and a governed call produces no workload-side kernel evidence at all. Never compute m - n — it is not a quantity. The headline is bypassedDestinations, and it is derived from kernel evidence alone.

dnsOnly: false is not by itself a connection-backed finding

dnsOnly is true only when the row has unsuppressed evidence and all of it is resolutions. A row with no unsuppressed evidence — m: 0, bypassedDestinations: 0 — is therefore false as well, with no connection behind it. The network observation gives nearly every pod in the cluster a row, so most rows in a healthy cluster are exactly that shape. The pair to test is dnsOnly: false and a non-zero m; the admin console likewise does not show a network evidence badge on a row with no evidence.

Rows are sorted by bypassedDestinations descending, then by m descending, then by workload key.

destinations[] fields​

FieldTypeMeaning
destNamestringResolved display name, or ip:<address> (unattributed) when no identity could be established
destIPstringDestination address; empty for DNS-only evidence that never produced a connection
sourcenetwork | dnsThe strongest evidence that fired. network when any connection to this destination was observed, otherwise dns — the destination was resolved but no connection to it was seen. A resolution is not a connection; both are observed, and this field is which one you have
behaviors[]arrayEvery sensor behavior that fired for this destination
countintWindow-scoped positive deltas: connections when source is network; resolutions when dns. A new or reset cumulative series contributes its incoming count; unchanged and stale snapshots contribute nothing
governedCallsintGateway-recorded calls this workload made to this same host. Rendered as "also governed (n)". On an unsuppressed finding this means split traffic and strengthens the finding; it never clears it. Sourced from egress-plane audit events only — see the note below
rung / confidencestring / floatDestination identity ladder result — see below
kindprovider | infra | imds | internalFinding classification
suppressed / suppressReasonbool / stringWhether this destination is excluded from the count, and why. Suppressed destinations remain in the list
governedCalls requires the egress plane

Only audit events with a request type of egress carry a destination host, so governedCalls is derived from network egress filtering alone. Where that plane is not enabled, governedCalls is 0 on every destination and the "also governed" tag never appears — which is not evidence that traffic was unsplit. n is unaffected: it counts every audited request type.

Finding kinds​

kindMeaning
providerAn external destination. Counts toward bypassedDestinations when unsuppressed
infraPackage registries and similar. Suppressed from the count, retained in the list
imdsThe cloud instance metadata service (169.254.169.254). Never suppressed — it gets its own finding kind rather than being swept up with link-local traffic — and not counted as a provider connection
internalCluster or node infrastructure suppressed by a deterministic rule

Suppression reasons​

suppressReasonWhat it excludes
corednsCluster DNS
kube-apiserverThe Kubernetes API server
in-clusterIn-cluster addresses and *.svc.cluster.local names, derived from the API server rather than guessed from CIDR configuration
node-or-link-localNode and link-local addresses — except 169.254.169.254
aicontrols-gatewayAIControls' own Service, by IP or by resolved name. This is the expected shape for a correctly governed workload
reverse-dnsin-addr.arpa PTR questions — resolver plumbing, not destinations
search-domain-expansionVariants generated by the Kubernetes default ndots:5 resolver configuration, which expands every multi-label question several-fold
sidecar-control-planeService-mesh and telemetry control planes, which are never AI providers
package-registryPackage registries (kind: infra)
egress-nat-detector-disabledNetwork-sourced evidence in a window where the egress-NAT auto-detect disabled the network detector. DNS-based findings are unaffected
cluster-facts-unavailableNetwork-sourced evidence in a window where the API-server-derived cluster facts were missing or stale, so an address could not be told apart from an in-cluster one. DNS-based findings are unaffected — they are classified by name

The last two are stand-down reasons rather than per-destination rules: they apply to network-sourced evidence across the whole window and always come with networkDetectorDisabled: true and a disabledReason. Every other rule in the table matches on a name, which is why unusable cluster facts leave an address-only connection with nothing to classify it and force the stand-down.

Separately from suppression, findings attributed to AIControls' own pods — including a previous AIControls pod — are dropped before any counting. The proxy is an in-cluster hop, so a governed workload's provider calls leave the AIControls pod; counting them would inflate the number in the customer's favour.

Destination identity rungs​

An observed connection yields an address, not a name: the kernel supplies IPv4 addresses and TLS SNI is invisible to the sensor. Destination names are therefore resolved through a confidence-graded ladder. Resolution is monotonic-upgrade-only within a window.

rungSourceconfidence
R0The pod's own DNS answer was observed0.95
R2Cluster facts from the API server — Service ClusterIPs, EndpointSlice addresses, Pod IPs0.98
R6No identity could be established. destName is rendered ip:<address> (unattributed) rather than guessed0

Rungs R1, R3, R4, and R5 are reserved for later work and are not produced today; they would carry 0 confidence if they appeared.

R2 is what names the in-cluster half of a connection. The non-cluster half lands on R6 whenever the sensor did not observe the pod's own DNS answer for it — encrypted DNS, a cached answer, or a hardcoded IP. An R6 destination is a real, counted finding whose name is missing, not a discarded one. Closing that gap needs the reserved rungs, which are not implemented.

kyverno-runtime v0.1.3 and earlier are IPv4-only, so with them an IPv6 destination produces no row at all — not an unattributed one, not a flagged one. No field, rung, or reason records that IPv6 traffic was outside the sensor's reach. kyverno-runtime v0.1.4 and later, the version the chart bundles, also observe IPv6 destinations, but how AIControls attributes them isn't verified yet (#3452). Read an empty result as evidence about IPv4 alone: it cannot distinguish "no IPv6 traffic occurred" from "IPv6 traffic occurred and was never observed".

Observation families​

/observations groups raw findings by signal family. AIControls authors one monitor-mode observation policy per family below (except other), covering every namespace but kube-system and kube-public. Families available depend on node capability — open and exec require the open/exec hooks on the nodes (bpffs mounted at /sys/fs/bpf).

familyAdmin console headingRequires open/exec hooksFeeds bypassedDestinations
dnsNames resolved (DNS)NoYes
networkDestinations reached (network)NoYes
openPaths opened (file)YesNo
execBinaries executed (exec)YesNo
otherOther observations—No

The network policy carries a deny-all match with no allow list, so every connection a governed pod opens is reported, in-cluster traffic included. Allow-listing provider destinations would hide the traffic the feature exists to surface; uninteresting destinations are removed after the fact by suppression rules, which stay visible with a reason attached. The cost is volume: on a busy cluster in-cluster chatter competes with external traffic for the sensor's 500-result-per-report-per-node cap, and the symptom of losing that race is a persistent ingestLag figure, never a clean-looking result.

The open family carries two independently-toggleable policies, both reported under the same open rule: the ungovernedLLM category's AI SDK / model / agent-credential filter, and the credentialFiles category's broader infrastructure-credential-file filter (credential-open, above). Disabling one leaves the other's observations unaffected.

The response also carries capped: true when the workload has reached runtimeControls.maxFindingsPerWorkload. Its new behavior then no longer starts new series, so the groups can be missing it. See Retention and limits.

Each item carries rule, behavior, subject, destIP, destName, count, node, podName, namespace, result, mode, observedAt (Unix seconds), and workloadMetadata when labels were recorded (see Workload labels on findings).

Workload labels on findings​

The sensor drops pod labels by design, so a finding arrives without your own application identity. AIControls adds it at ingest: for every runtime finding, from the push collector or from OpenReports, and from any policy, including the cluster-wide observation policies, it reads the observed pod from the Kubernetes API and records the pod labels listed in runtimeControls.findingLabels.

runtimeControls:
findingLabels:
- ebay-app-id
- team
  • Labels come only from AIControls' own Kubernetes reads. A label in the sensor's payload is never used, and a label that is not listed is never recorded.
  • The pod must match. AIControls checks the finding's pod UID against the pod it read, so a recreated pod with the same name never lends its labels to an older finding.
  • The first value is kept. A finding series is scoped to one pod, and it keeps the labels it first recorded. A series first seen before AIControls could read the pod picks the labels up on a later occurrence.
  • Findings recorded before you set the value carry no labels. There is no backfill.

Each entry must be a valid Kubernetes label key, such as ebay-app-id or example.com/cost-center. AIControls ignores an invalid entry and logs a startup warning.

The labels use the same workloadMetadata shape as trusted workload metadata on audit events:

FieldValue
sourcekubernetes
versionThe pod's resourceVersion when AIControls read it
labelsThe allow-listed labels, as a key-value object

workloadMetadata appears on /observations items, on the item of GET /api/v1/runtime/observations/{fingerprint}, on runtime rows of GET /api/v1/security-events and GET /api/v1/activity, in Activity JSON and CSV exports, and on sequence_matched webhooks for a workload subject. To filter runtime findings by a label, pass label=<key>=<value> to /api/v1/security-events, /api/v1/activity, or /api/v1/activity/export. See Security Events.

Findings​

GET /api/v1/runtime/findings returns the workload/category rollup: one row per distinct (workload, category, subject) combination observed in the window, aggregated across however many individual kernel observations produced it — the count and first/last-seen span a Deployment's whole lifetime, not one pod's.

curl "$YOUR_WORKSPACE_URL/api/v1/runtime/findings?namespace=finance&category=ungoverned-llm&window=6h" \
-H "Authorization: Bearer $ADMIN_TOKEN"

Query parameters​

All optional; an omitted parameter applies no constraint.

ParameterMeaning
namespaceExact-match namespace filter
categoryOne of the five values in Finding categories below, including uncategorized
workloadExact match on the canonical <namespace>/<kind>/<name> workload key
windowSame convention as /reconciliation: default 1h, clamped to 5m–24h, an unparseable value falls back to the default

Response fields​

FieldMeaning
workloadKeyCanonical <namespace>/<kind>/<name>
namespaceThe workload's namespace
categoryOne of the five finding categories
subjectWhat was observed — a resolved name, a reached destination, an opened path, or an executed binary, depending on category
countCumulative occurrences summed across every day-bucket the window filter includes
firstSeenUnix / lastSeenUnixUnix seconds, earliest and latest occurrence within those day-buckets

The response also carries cappedWorkloads, the workloads among the findings that have reached runtimeControls.maxFindingsPerWorkload. It is omitted when none has.

The rollup buckets by whole UTC day, not by minute, and window is capped at 24h. window is converted to a day-bucket lower bound, so a short window (15m, 1h) still returns everything recorded today in UTC, not just the last 15 minutes or hour — the count can include occurrences from earlier in the same day that a precise time-box would have excluded. Because bucketing is by calendar day rather than by elapsed time, a window that crosses UTC midnight (for example window=24h requested a few hours after 00:00 UTC) also pulls in all of yesterday's bucket, not just the last 24 hours of activity. The 24h cap means you cannot reach further back than the previous UTC day this way — there is no "last week" view.

A finding row survives a pod restart: attribution resolves to the owning workload (Deployment, StatefulSet, DaemonSet, or Job/CronJob) rather than the pod, so a Deployment's count keeps accumulating across its pods' lifetimes instead of resetting to a fresh series each time a pod is replaced. See How evidence is attributed to a workload.

Finding categories​

AIControls authors four finding categories, each independently toggleable via its own runtimeControls.categories.* config key. A fifth value, uncategorized, is not one of the four and has no toggle — it's where a finding lands when it can't be honestly attributed to one of them (see below). See Runtime Controls finding categories for what each authored category means and what happens when you turn one off.

CategoryConfig keyObservation shape(s)Default
ungoverned-llmruntimeControls.categories.ungovernedLLMopentrue
agent-execruntimeControls.categories.agentExecagent-exec (agent CLI / inference-server)true
mcp-execruntimeControls.categories.mcpExecmcp-exec (MCP server)true
ungoverned-egressruntimeControls.categories.ungovernedEgressdns, networktrue

Each key is a boolean. An omitted key reads as true (enabled) — set it explicitly to false to turn a category off. ungovernedEgress gates both the dns and network shapes together; there is no separate key for each.

A fifth key, runtimeControls.categories.credentialFiles, independently toggles the credential-open observation shape — a second open-behavior policy, alongside ungovernedLLM's, narrowed to a broader set of infrastructure credential files (~/.aws/credentials, .env, ~/.kube/config, SSH keys, and similar — see Helm chart values above for the full default list and the override). It shares the ungoverned-llm category — both observe unauthorized model/credential access — rather than introducing a fifth category value, so it has its own toggle key but is not a new row in the table above. Like ungovernedLLM's open observation, it requires the open/exec hooks; on nodes without them, this evidence class is structurally unavailable rather than merely quiet. It is monitor-only, the same as every other observation shape — it can only report a credential file being opened, never block the open.

exec observations split into agent-exec and mcp-exec as of this release. A finding recorded by an older AIControls version, or by a policy AIControls did not author, is categorized uncategorized when it cannot be honestly attributed to one of the four.

Renamed from the shadow-* categories​

Both the category values and the config keys changed in this release. "Shadow AI" now means an ungoverned AI workload, so these categories were renamed to say what the sensor observed instead.

WasNowConfig key wasConfig key now
shadow-llmungoverned-llmshadowLLMungovernedLLM
shadow-agentagent-execshadowAgentagentExec
shadow-mcpmcp-execshadowMCPmcpExec
shadow-egressungoverned-egressshadowEgressungovernedEgress

Two things to do when you upgrade:

  • Rename the keys in your values file. The old keys are no longer read, and the chart fails the render with the new name if it finds one, so a stale key is an install-time error rather than a silently ignored setting.
  • Update anything that queries the API by category — dashboards, scripts, or saved filters that pass category=shadow-llm match nothing after the upgrade.

Findings already recorded under the old values are renamed in place the first time AIControls starts on the new version, so history and counts carry over unchanged. The migration rewrites stored rows only — it does not translate an old category value supplied in a request, so a query that still names one returns an empty result rather than the rows it used to.

Retention and limits​

AIControls bounds the runtime data it stores:

  • Retention. Occurrence history is kept for 90 days. A finding series not observed for 90 days is deleted with it. If the sensor reports that behavior again, it starts as a new series. Ingest drops a finding observed more than 90 days ago, so a report that still carries old findings doesn't bring a deleted series back.
  • Per-workload cap. runtimeControls.maxFindingsPerWorkload (default 1000, 0 for unlimited) caps the distinct finding series stored per workload. At the cap, ingest drops findings that would start a new series; series already stored keep updating. An enforced denial can still start a new series until the workload reaches twice the cap, so a blocked action is recorded even for a workload at the cap. Each replica refreshes the per-workload counts every 30 seconds and counts the new series it admits in between, so a burst of new series stops at the cap. With several replicas ingesting, a workload can exceed the cap by the new series the other replicas admit in one interval. AIControls logs a warning for a capped workload at most once every 10 minutes, and the /observations and /findings responses report the cap.
  • Excluded namespaces. Ingest drops findings from the namespaces excluded in Settings > Workloads > Cluster Discovery and from AIControls' own namespace, for both the push collector and OpenReports. Every finding is dropped, including findings from a registered workload's own compiled policy and from enforce-mode policies: the sensor still enforces those policies, but AIControls doesn't record their events. Every 15 minutes, AIControls also deletes the stored events of excluded namespaces, so saving the list removes a namespace's existing events within 15 minutes. To delete them at once, exclude the namespace through the API.
  • kubectl exec noise. Each kubectl exec into a pod makes the container runtime open /, /<pid>/setgroups, and /<pid>/task/<tid>/fd. The sensor reports these procfs paths without their /proc prefix. AIControls filters the two numbered paths out of a workload's monitor-mode file policy and drops them at ingest, so an operator's exec does not read as workload behavior. An open of / is kept, because it looks the same as a workload opening the root directory; it appears once per workload, not once per exec. Other paths are kept.

The aicontrols_runtime_findings_dropped_total metric counts the findings ingest drops, by reason: excluded_namespace, workload_cap, exec_setup_noise, or past_retention.

See also​