Enable Runtime Controls

Tasks

These steps turn on kernel-level observation of AI workloads in a cluster you operate, and show you how to read the result. For what Runtime Controls is, what it can and cannot see, and how to interpret its headline number, read Runtime Controls first — the table is easy to over-read without it.

Self-managed installations only

Workload Trust is available only in private, self-managed AIControls installations, on premises or in a cloud cluster that you run. It needs a dedicated cluster that you run, with Kyverno, Kyverno Runtime, and cluster OIDC and RBAC access in place. Nirmata-hosted (SaaS) workspaces don't include it. See Feature availability.

Self-hosted Kubernetes only

Runtime Controls cannot run on a Nirmata-hosted (SaaS) tenant. Your AIControls control plane must be running inside the cluster you want to observe. See Step 1.

Step 1 — Confirm your deployment is eligible

Runtime Controls reads Report objects from an API server and authors observation policy in a cluster. Both have to be your cluster.

1. Confirm you are self-hosted. AIControls must be installed in your own Kubernetes cluster via the Helm chart. On a Nirmata-hosted tenant the control plane runs in Nirmata's cluster — there is no API server of yours for it to read, and no cluster of yours for it to author policy in — so the feature refuses to start regardless of configuration. The chart declares a self-managed installation with deployment.mode: selfManaged, its default; leave it set.

2. Check that nodes mount bpffs if you want file and exec evidence. The network, protocol, and dns behaviors don't need it. The open (file) and exec (binary) behaviors need the open/exec hooks, which need bpffs mounted at /sys/fs/bpf on each node. Most distributions mount it by default; kind and some minimal node images don't. Check with mount | grep /sys/fs/bpf. This is not a blocker — the bypassed-destination headline is built entirely from network and dns evidence — but on nodes without the hooks file and exec findings cannot be produced at all, and the Coverage section will say so per node.

3. Give AI workloads their own ServiceAccounts if you have not already. Gateway-recorded calls are joined to workloads by name (workload name, ServiceAccount name, or generated pod-name pattern). Several workloads sharing default cannot be told apart, which degrades the N column and the governed-overlap annotation. It does not affect the bypassed-destination count, which comes from kernel evidence alone.

Step 2 — Install AIControls

The Kyverno runtime sensor — Nirmata Runtime, a per-node eBPF DaemonSet with one cluster-scoped RuntimePolicy CRD — ships as an optional subchart of the AIControls Helm chart, installed by default alongside AIControls itself. A single helm install gives you both:

helm install aicontrols \
  oci://ghcr.io/nirmata/charts/aicontrols \
  --namespace aicontrols --create-namespace \
  --set admin.email=you@example.com

Runtime Controls' own integration (runtimeControls, Step 3) also defaults to on, so this one command is the entire install flow — you do not need to install a separate chart, configure a push target, or copy a certificate between releases. The sensor's DaemonSet lands in the same namespace as AIControls, and AIControls re-probes for its RuntimePolicy CRD every five minutes, so a sensor that takes a moment longer to become ready needs no restart.

Manage the sensor yourself instead

If your organization already runs Nirmata Runtime as its own release — a separate namespace, its own upgrade cadence, RBAC reviewed independently of AIControls — opt out of the bundled subchart and keep your existing install:

kyverno-runtime:
  enabled: false

With this flag set, AIControls' own runtimeControls integration (Step 3) stays on, but the chart requests none of the sensor's cluster permissions and renders none of its resources. Point your separately-managed sensor at AIControls the same way you always have — the chart's helm install/helm upgrade output (NOTES) prints the exact push.target and push.tls.secretName values and the certificate-copy command for this case.

Version requirement

Runtime Controls depends on the sensor's behaviors/dns schema (runtime.nirmata.io/v1alpha1). If the bundled subchart's published version predates it, build the sensor from source following the instructions in the nirmata/runtime repository, and set kyverno-runtime.enabled: false to manage that build as your own release instead. The admin console shows the current install command inline when it detects the CRD is absent.

Whichever path you use, the sensor must end up with:

• its DaemonSet running on every node you want observed, and
• the runtime.nirmata.io RuntimePolicy CRD served by the API server.

Step 3 — Confirm runtimeControls is enabled

runtimeControls.enabled defaults to true, so a default install already has Runtime Controls' own integration running — there is nothing to add to values.yaml for the common case. Setting it to false disables both the AIControls-side integration and the bundled sensor subchart together (they share one flag, Step 2):

# Self-hosted Kubernetes only. Ignored (with a logged reason) when not
# running in Kubernetes or on a Nirmata-hosted tenant.
runtimeControls:
  enabled: false

The flag does two things. It starts the integration in the process, and it renders a declinable ClusterRole and ClusterRoleBinding for the chart's ServiceAccount. When the flag is false, the chart never even requests those cluster permissions — so a cluster that has opted out has no AIControls ClusterRole to audit.

The role grants, all cluster-scoped:

API group	Resources	Verbs	Why
runtime.nirmata.io	runtimepolicies	full CRUD	AIControls authors the capability-probe and observation policies
runtime.nirmata.io	runtimepolicies/status	read	Per-node applied state, used for capability detection
openreports.io	reports, clusterreports	read	The sensor's findings
core	pods, services, endpoints, nodes, namespaces	read	Attributing evidence to workloads; deriving in-cluster addresses from the API server rather than guessing them
apps	deployments, statefulsets, daemonsets, replicasets	read	Owner-reference walk from pod to workload
batch	jobs, cronjobs	read	Same walk for Job-backed workloads
discovery.k8s.io	endpointslices	read	Resolving cluster destination identity
apiextensions.k8s.io	customresourcedefinitions	read	Detecting whether the sensor's CRDs are installed at all
core	events	list, watch	The Pod-scheduled event watch that attributes a new Pod within seconds. Rendered only when runtimeControls.podEvents.enabled is true (the default). See How new Pods are attributed

Every rule is probed with a SelfSubjectAccessReview at startup. If your cluster's admission or policy stack strips any of them, the missing grant appears as a structured gap on Health → Runtime rather than as a log line nobody reads.

What appears in your cluster

Once enabled, AIControls authors its own RuntimePolicy objects — a capability probe, plus one observation policy per signal: dns (names resolved), network (destinations connected to), open (files opened), and exec (binaries executed). They cover every namespace except kube-system and kube-public.

Policy names say what each policy governs:

Policy	Name
Cluster-wide observation	aic-obs-cluster-<signal>, for example aic-obs-cluster-dns
A registered workload's enforceable policy	aic-enf-<namespace>-<workload>-<hash>, for example aic-enf-runtime-global-runtime-probe-1a2b3c4d
A registered workload's agent-exec observation	aic-obs-<namespace>-<workload>-<hash>
A registered workload's other observations	aic-obs-<namespace>-<workload>-<signal>-<hash>, for example aic-obs-runtime-global-runtime-probe-dns-5e6f7a8b
Capability probe, kill switch	aic-capability-probe-<hash>, aic-killswitch-<hash>

The workload is the Deployment's name. For any other kind, the kind is part of the name too, for example aic-enf-agents-statefulset-worker-<hash>. The 8-character hash comes from the cluster, namespace, kind, and workload name, so two workloads never share a policy, even when their readable parts match. When a name would exceed 63 characters, AIControls shortens the readable part and keeps the hash. When you upgrade from a release that used aic-observe-<hash> and aic-restrict-<hash>, AIControls creates each policy under its new name and then removes the old one.

kubectl get runtimepolicies -l app.kubernetes.io/managed-by=aicontrols

All authored policy is monitor mode

Every one of these observes and reports. None of them denies anything — a network policy from AIControls will not block a connection, and an open or exec policy will not stop a process. Enabling Runtime Controls cannot break a running workload.

A second open policy — credential-open — watches for a broader set of infrastructure credential files being opened: ~/.aws/credentials, ~/.aws/config, .env, ~/.kube/config, ~/.ssh/id_rsa, ~/.ssh/id_ed25519, ~/.docker/config.json, ~/.npmrc, ~/.pypirc, and ~/.git-credentials by default. It is toggled independently with runtimeControls.categories.credentialFiles (default true), and the path list itself is overridable with runtimeControls.credentialFilePaths. Like every authored policy it is monitor-only, and like the rest of the open family it needs the open/exec hooks, which need bpffs mounted at /sys/fs/bpf — on nodes without them, this evidence class cannot be produced at all, not merely observed as clean.

The network policy carries no allow list. That is deliberate: allow-listing provider destinations would hide exactly the traffic the feature exists to surface. Uninteresting destinations are removed afterwards by suppression rules that record their reason, so they stay visible to you in the expanded row.

Scope the network observation away from a noisy namespace

Carrying no allow list is what makes the network observation trustworthy, and it is also what makes it expensive. The sensor writes one report object per namespace and node, capped at 500 results, and once that cap is reached later findings for that namespace and node are dropped permanently. All four observations share that one budget — and three of them narrow what they report: dns by its in-cluster allow list, open and exec by their detection filters. network does not, so it reports one series per pod and destination, in-cluster service-to-service traffic included. In a large or chatty namespace that volume can crowd the namespace's own dns findings out of the report, which costs you a signal that was there before connection evidence existed.

The cap belongs to the sensor and AIControls cannot raise it. Your lever is to scope the network observation away from the namespace producing the volume:

runtimeControls:
  enabled: true
  # Scopes the `network` observation only. kube-system and kube-public are
  # always excluded from every observation and do not need to be listed.
  networkObserveExcludeNamespaces:
    - batch-jobs
    - mesh-system

Only network is scoped. The dns, open, and exec observations continue to cover the listed namespaces unchanged, and excluding a namespace hands its report budget back to those three.

The cost is real, so do not set this preemptively. An excluded namespace produces no connection evidence: its rows go back to being DNS-only, with every weakness that carries — a resolution proves a name was looked up, not that traffic followed, and encrypted DNS, a cached answer, or a hardcoded IP produce nothing there at all. The symptom that justifies the trade is a persistent ingest-lag figure; see Troubleshooting.

Each entry must be a valid namespace name (a DNS-1123 label). A malformed entry fails policy rendering loudly at startup rather than quietly mis-scoping a security control.

Record your workload labels on findings

To route, filter, and report runtime findings by your own application identity, list the pod label keys to record:

runtimeControls:
  enabled: true
  findingLabels:
    - ebay-app-id

AIControls reads the labels from the Kubernetes API when it ingests each finding; the sensor's payload never supplies them. After you upgrade, open a runtime event in Security → Activity → Security and check Workload labels, or type ebay-app-id=checkout in the Workload label box. See Workload labels on findings for the rules.

The gRPC push-sink collector

AIControls can read findings two ways: by listing and watching the sensor's Report objects from the Kubernetes API server, or by receiving them as a live gRPC stream pushed directly from the sensor — lower latency, at the cost of no per-message replay (a finding in flight when the daemon pod dies or the stream drops can be silently lost — the sensor's own EventsDropped{source="pushsink"} metric tracks this on its side). The bundled sensor uses the push path only; a customer-managed sensor uses OpenReports unless you configure its push client.

runtimeControls.collector.enabled defaults to true alongside the bundled subchart (Step 2): the bundled sensor pushes its findings to AIControls, so the listener has to be there to receive them. With the default install from Step 2, this is already wired end to end — no values to add, no certificate to copy.

Do not enable both paths for the same sensor

The collector and the OpenReports path use independent series identities, so a DaemonSet reporting through both at once has every finding counted twice — inflating reconciliation and ingest totals. A customer-managed sensor that enables the push client must also set that release's daemon.reports.enabled: false so exactly one path is active.

The listener is mutual-TLS-only — there is no plaintext mode, matching the sensor's push client, which never sends unauthenticated — so AIControls refuses to start the listener at all if any of the certificate, key, or client CA is missing. Turning it off explicitly (runtimeControls.collector.enabled: false) while the bundled subchart is still installed fails the Helm render outright, rather than silently shipping a sensor with nowhere to push its findings:

runtimeControls:
  collector:
    enabled: false   # fails to render while kyverno-runtime.enabled is true

If you manage the sensor yourself instead, the chart's helm install/helm upgrade NOTES output prints the exact push.target and push.tls.secretName values to set on your separate kyverno-runtime release, plus the command to copy the generated client certificate into that release's namespace.

Upgrade caveat: enabling the bundled subchart on an existing release

The collector's server certificate is minted once and reused on every later helm upgrade (the chart looks up and keeps the existing Secret rather than regenerating it, so an already-trusted certificate does not roll out from under connected daemons). If you already ran the collector against a customer-managed sensor and are now enabling the bundled subchart (kyverno-runtime.enabled: true) on that same release, the persisted certificate predates the bundled sensor's Service name and will not carry the SAN it needs — its TLS handshake fails hostname verification. Delete the existing server-certificate Secret first, so the next helm upgrade mints a fresh one with the current SAN list:

kubectl delete secret <release-name>-aicontrols-runtimecontrols-collector-server -n <namespace>
helm upgrade <release-name> oci://ghcr.io/nirmata/charts/aicontrols -n <namespace> -f values.yaml

Step 4 — Verify coverage

Open Health → Runtime. The tab only appears when the integration is actually available, so if it is missing, revisit Steps 1 and 3.

The tiles and the Nodes table are the first thing to read, and they answer three questions independently:

1. Is the sensor installed? If the CRD is absent you get an explicit alert with the install command, not an empty table.

2. Which nodes have a live sensor? A per-node state of available, unavailable, or unknown. Read unknown as "cannot tell" — a node whose status shard is merely stale reports unknown, never unavailable.

3. Which nodes can produce file and exec findings? The open/exec hooks column, same three states. Nodes without bpffs mounted at /sys/fs/bpf show unavailable here while still showing available for network, protocol, and DNS.

You can also check it directly:

curl "$YOUR_WORKSPACE_URL/api/v1/runtime/status" \
  -H "Authorization: Bearer $ADMIN_TOKEN"

A response with "enabled": false carries a reason explaining which gate closed. See the reference for the full payload.

Step 5 — Read the reconciliation table

Open Inventory → Workloads and scroll to the Gateway bypass card. It lists one row per workload for the selected window (15 minutes to 24 hours; one hour by default), sorted by Bypassed destinations descending. By default it hides workloads with no bypassed destinations. Turn on Show all workloads to see them. For a workload that isn't registered, select Register to open the registration wizard prefilled with its namespace, kind, and name. Select Exclude namespace to stop observing the row's namespace. Select a row to see its kernel observations in Security → Activity.

1. Start with Bypassed destinations. It is the count of distinct provider destinations the workload reached directly, outside governance. It is a set count, not a volume — 0 means the set was empty.

2. Do not subtract M from N. M counts kernel events and N counts gateway requests, over non-overlapping populations. They are on the row as corroborating evidence only; the column tooltips say so. See M and N are evidence, not terms in a formula.

3. Check the Evidence badge. AIControls observes both connections and DNS questions, and every row with evidence says which fired. A destination tagged network means the pod opened a socket to it — traffic happened. A row badged dns only is backed entirely by resolutions: a name was looked up, but nothing establishes that traffic followed.

   The absence of the badge is not by itself the stronger finding. dns only describes a row that has unsuppressed evidence and where all of it is resolutions, so a row with no unsuppressed evidence at all is also unbadged — and since the network observation gives nearly every pod in the cluster a row, most rows are exactly that. Read the badge next to Bypassed destinations: no badge and a non-zero count is the connection-backed finding; no badge with a zero count is just a quiet row.

4. Expand the row. Each destination shows its evidence class, count, identity confidence, and — critically — whether it was suppressed and why. Suppressed destinations stay visible with the reason attached.

5. Read "also governed (n)" as a stronger signal, not an excuse. It means the workload uses the governed path for that host and something in it also went around the path. Split traffic, typically a library ignoring HTTPS_PROXY, is the strongest bypass signal there is. It never clears the destination from the count.

6. Click a row to open the observations drawer: names resolved, destinations reached, and — where the open/exec hooks are available — paths opened and binaries executed.

An empty table is not a clean bill of health

Encrypted DNS, a cached answer, and a hardcoded IP all hide the lookup, but the pod still opens a socket to the destination and that connection is observed — so those three no longer produce a clean-looking workload that is actively reaching providers. What survives is narrower: a destination the sensor never saw resolved in the clear is counted but shown at its address rather than a name, and observations can still be lost to kernel map pressure or the sensor's per-report result cap. Always read the table together with the Coverage card and the ingest lag banner — a lag figure that persists across refreshes means findings were dropped, and a lost observation is indistinguishable from a workload that did nothing.

Troubleshooting

The Runtime nav entry is missing. The integration is not available. Check that AIControls is running in Kubernetes, is not a Nirmata-hosted tenant, and that runtimeControls.enabled: true was actually applied — helm get values aicontrols -n aicontrols will tell you.

"The runtime sensor is not installed on this cluster." The RuntimePolicy CRD is not served. Complete Step 2; the CRD is re-probed every five minutes, so no restart is needed.

"The proxy's ServiceAccount is missing permissions." The startup access review found a gap. The alert lists each missing verb, group, and resource. Confirm the ClusterRoleBinding from Step 3 exists and that nothing in your cluster removed rules from the ClusterRole.

A workload shows "Not applied: enforcement unavailable on the sensor". The sensor can't set up its open and exec hooks on at least one node, so it enforces and observes no open or exec rule there. kyverno-runtime v0.1.3 and later report this on every open and exec policy as EnforcementAvailable=False (Enforce mode) or ObservationAvailable=False (Monitor mode), with Applied=False and a message that names the cause. AIControls reads those conditions, so the workload's kernel plane shows as not applied with that message, and its entries read as monitored rather than enforced. To see the message on the policy, run kubectl get runtimepolicy <name> -o yaml.

The most common cause is a node without bpffs mounted at /sys/fs/bpf. Most node images mount it; kind and some minimal node images don't. Check with mount | grep /sys/fs/bpf, and check the RuntimePolicy condition that matches its mode (EnforcementAvailable for enforce, ObservationAvailable for monitor) and the sensor's startup log, which names the hook type the sensor chose. Mount bpffs on each node, then restart the sensor pods. On kind, run docker exec <node> sh -c 'mount -t bpf bpf /sys/fs/bpf'. You don't need to boot the node with lsm=bpf: the sensor attaches BPF-LSM programs when the kernel has BPF-LSM, and otherwise attaches a program on security_file_open that also enforces.

kyverno-runtime v0.1.2 and earlier don't report these conditions. They log failed to create openexec manager at startup and keep reporting their policies as applied.

Node state is unknown rather than available or unavailable. This is deliberate and means the evidence to decide is not there — most often no status shard has been reported recently, or the capability probe has not completed yet. Do not treat unknown as clean.

"Network-based bypass detection is disabled for this window." The detector stood down. Read the stated reason — there are two causes, and they need different responses. In both, DNS-based findings continue, and because connections are the dominant source of evidence, treat such a window as thinner than usual, not as clean.

• Egress NAT. More than 90% of the cluster's external destinations collapsed to a single IP, the signature of an egress NAT gateway, so addresses no longer distinguish destinations. Nothing to fix — this is the cluster's network topology, and the detector says so rather than emitting identities it knows to be meaningless.
• Cluster facts unavailable or stale. The Kubernetes listing of Services, EndpointSlices, Pods, and Nodes has not completed, has gone stale, or is being refused. A kernel-observed connection is an address and nothing more, so without that listing a connection to cluster DNS or a peer pod cannot be told apart from one to a provider — and reporting the cluster's own plumbing as bypassed destinations would be worse than reporting nothing. If it clears within a few minutes of a restart or upgrade, it was the startup window and needs no action. If it persists, check Health → Runtime for an RBAC gap: a missing list grant on pods, services, nodes, or endpointslices is exactly what produces it, and the startup access review surfaces it as a structured gap listing the missing verb, group, and resource (see the entry above and the role table in Step 3).

The ingest lag figure never clears. Findings the kernel recorded are not becoming stored rows — kernel map pressure, or the sensor's 500-results-per-report-per-node cap. On a busy cluster the usual cause is volume: the network observation deliberately carries no allow list, so ordinary in-cluster service-to-service traffic is reported alongside external traffic and competes for the same cap, and a real provider destination can be crowded out. The cap is the sensor's and cannot be raised from AIControls, so the direct lever is to scope the network observation away from the namespace producing the volume with networkObserveExcludeNamespaces — that returns the namespace's report budget to the dns, open, and exec observations, at the cost of no connection evidence there. Narrowing the observation window or reducing the noise on the affected nodes also helps. A persistent lag figure means the table is incomplete — do not read a low count from that window as a result.

A destination shows as ip:<address> (unattributed). The connection was observed, and it counts — only its name is missing. This happens when the sensor never saw the pod resolve that address in the clear (encrypted DNS, a cached answer, or a hardcoded IP). AIControls resolves in-cluster addresses from the Kubernetes API server; for an external address with no observed answer there is no source to name it from, so it is shown as an address rather than guessed at.

Counts look inflated on a sidecar deployment. The reconciliation currently assumes a central in-cluster gateway. With a gateway deployed as a per-pod sidecar, the gateway's own upstream egress originates from the workload's pod and is not yet suppressed. See Known scope limits.

See also

• Runtime Controls — the model, and what the evidence does and does not prove
• Runtime Controls Reference — endpoints, capability states, and row fields
• Self-Hosted Kubernetes — installing and operating AIControls in your own cluster