Skip to main content

AI Workload Discovery

Self-Hosted Kubernetes

AIControls periodically lists the workload objects in your cluster and records the ones that look like AI workloads, so you get an inventory of what is already running — not only what has been routed through the proxy. The result appears under Inventory > Workloads. This page covers the scan as a piece of cluster infrastructure: what it reads, what it never reads, and how to scope or extend it.

Self-managed installations only

Workload Trust is available only in private, self-managed AIControls installations, on premises or in a cloud cluster that you run. It needs a dedicated cluster that you run, with Kyverno, Kyverno Runtime, and cluster OIDC and RBAC access in place. Nirmata-hosted (SaaS) workspaces don't include it. See Feature availability.

What it reads​

The scan is on by default and read-only. It has its own ClusterRole, separate from Runtime Controls, granting list — and only list — on:

API groupResources
appsdeployments, statefulsets, daemonsets
batchjobs, cronjobs
agents.x-k8s.iosandboxes: Agent Sandbox's Sandbox, which OpenShell creates for each sandbox. The rule grants nothing on a cluster without the CRD.
core ("")namespaces: names and labels only, for the namespace picker when you register a workload

There is no get, no watch, and no write verb of any kind. Secrets are never in this grant, and neither are pods. Pods are deliberately excluded: a workload with no pods running right now — a CronJob between schedules, a scaled-to-zero inference service, a suspended Job — is still a workload, so a pod-derived inventory would lose it and then report it as gone.

Discovery skips itself automatically when AIControls is not running in Kubernetes and no kubeconfig is available, and it never reports the AIControls workload as an AI workload of yours.

What ends up in the inventory​

Each workload gets one row, identified by the tuple of cluster, namespace, kind, and name. The row records the workload's container images, its ServiceAccount, and the names of its environment variables — never their values. Environment variable names are what tell an OPENAI_API_KEY workload from a CLAUDE_API_KEY one, so the inventory does reveal which AI providers a workload is built against.

Each row also records the Pod Security level of the workload's pod template: Privileged, Baseline, or Restricted. The scan computes it from the complete pod template, then keeps only the level and the failing check IDs. It never stores the pod template, and it needs no extra permissions.

Each row also carries two independent facts:

  • Type — AI agent, AI assistant, MCP server, model, Jupyter notebook, coding assistant, OpenShell sandbox, or unknown.
  • Detection confidence — suspected when name, image, or environment-variable heuristics matched, and confirmed when the workload declares itself with a nirmata.io/ai or nirmata.io/ai-type label or annotation, matches the label selector, or is registered. Heuristics alone never confirm.

Choose which workloads are inventoried​

Go to Settings > Workloads > Cluster Discovery to decide which listed workloads appear in the inventory. A workload is included when any of these is true:

  • You registered it. A registered workload is always included, whatever its labels or score, so registering a workload is enough to get it observed. Registering a new workload also scans its namespace right away. Discovery lists only the kinds it scans, so a registered workload of another kind appears in the inventory once you add its kind to discovery.additionalKinds.
  • It matches the label selector. The default selector is nirmata.io/ai=true. You can enter any Kubernetes label selector, such as team in (ml,search). The selector matches the workload's own labels or its pod template's labels, and a match is included whatever its score, even in an excluded namespace.
  • Heuristics score it as AI. With Detect unlabeled AI workloads turned on (the default), discovery still scores every workload the selector does not match, so AI nobody labeled — shadow AI — is still found. Turn it off to include only labeled and registered workloads.

Turn the label selector off to scan every workload with heuristics. Detect unlabeled AI workloads stays on while the selector is off, because otherwise nothing but registrations would be included.

The nirmata.io/discovery-exclude label and the AIControls workload itself are never included, whatever the selector or registration says. See Exclude a workload.

Saved changes apply on every replica within two minutes and appear in the configuration audit trail. An invalid selector is rejected when you save it, and the running scan keeps its previous scope.

Shadow AI alerts​

Shadow AI is any AI workload whose traffic does not go through AIControls. It applies to every workload that Cluster Discovery finds, whether or not it is registered. With Settings > Workloads > Shadow AI > Shadow AI alerts on (the default), AIControls sends one notification per namespace when a scan finds shadow AI workloads it has not seen before. Each workload alerts once, the first time it is found. Notifications go to the channels configured under Settings > Notifications.

How discovery scores a workload​

For a workload that is neither registered nor matched by the label selector, discovery combines the evidence it finds into a score, and lists the workload only when the score reaches minConfidence and at least one signal other than the name matched. A keyword in a name is not evidence by itself: infrastructure agents such as konnectivity-agent and do-node-agent share the word "agent" with AI agents, so a name-only match is never listed, whatever minConfidence is.

SignalWhat matchesWeight
DeclarationThe nirmata.io/ai or nirmata.io/ai-type label or annotation0.95
Environment variableA variable name contains an AI provider or model-endpoint keyword, such as OPENAI_API_KEY, ANTHROPIC_MODEL, LLM_BASE_URL, or MCP_URL0.85
Container imageThe image path contains an AI framework or model server name, such as langchain, vllm, or ollama0.65
Model server kindThe workload kind is a model server by definition, such as a KServe InferenceService0.65
GPU model serverA container requests a GPU and the workload is a model server0.65
Gateway callThe workload called a model or MCP server through the gateway while it had no registration. See Observed AI activity0.9
Provider egressThe runtime sensor saw the workload connect directly to a known AI service host, outside the gateway0.8
NameThe name contains an AI keyword, such as agent, llm, or assistant0.55, only with another signal

To make discovery list a workload with no other evidence, label it so it matches the default selector, or register it:

metadata:
  labels:
    nirmata.io/ai: "true"
    nirmata.io/ai-type: agent

Suspected and confirmed workloads​

A workload that declares itself is confirmed. A workload that matched on environment variables, images, or other signals, without a declaration, is suspected. In the Workloads list, a suspected workload shows a Suspected tag and Not rated in the Risk band column instead of a band, and its primary action is View instead of Register…. The detail drawer and page show the same Not rated state. If the list cannot read the discovery record, it treats the workload the same way until the read succeeds. Select Register anyway… in the row menu to register it. Label the workload to confirm it.

Discovery rates a confirmed workload as usual. Registering an infrastructure component as an AI workload compiles a runtime policy against it, so confirm a workload before you register it.

Excluded namespaces​

Discovery heuristics and the cluster-wide runtime observations skip cluster, cloud-provider, and platform namespaces, so their components never appear as AI workloads. Edit the list under Excluded namespaces in Settings > Workloads > Cluster Discovery. The default list is kube-system, kube-public, kube-node-lease, kyverno, cert-manager, local-path-storage, istio-system, ingress-nginx, gatekeeper-system, gke-managed-system, gke-gmp-system, gmp-system, amazon-cloudwatch, and aws-observability. The AIControls namespace is always excluded, even for a registered or label-selected workload. A workload in one of these namespaces that declares itself with a nirmata.io/ai or nirmata.io/ai-type label or annotation, on the workload or its pod template, matches the label selector, or is registered, is still listed. To seed a different list on first install, set workloadTrust.excludeNamespaces in the Helm values; after that, edit it in Settings.

Runtime Controls honors the same list: ingest drops every runtime finding from an excluded namespace, including findings from a workload's own enforcement policy. AIControls also deletes the runtime events already stored for excluded namespaces within 15 minutes of saving the list. To exclude a namespace and delete its stored runtime events at once, use Exclude namespace on a runtime event or POST /api/v1/runtime/exclusions/namespaces.

Detection confidence is not a protection status or risk score. An observed workload is Discovered until a user creates its registration. Enforcement and Verification remain independent.

Observed AI activity​

Two kinds of observed behavior also count as evidence, because they show what a workload does rather than how it's configured. Either one corroborates a workload on its own, so it's listed even when its specification has no AI environment variable, image, or declaration:

  • A gateway call from an unregistered workload. When the gateway attributes an MCP or LLM call to a workload that has no registration, it applies the unregistered workload mode and records the call. The first such call starts a discovery scan of the workload's namespace, under the same single-replica lease as the periodic scan, so the workload appears in Inventory → Workloads within seconds instead of at the next periodic scan. A workload that shares a registered workload's ServiceAccount never inherits that registration, and a Pod whose workload AIControls can't resolve is denied in Enforce, including when the live Pod lookup is refused by RBAC or runs over its budget. See Unregistered workloads.
  • Direct egress to an AI provider. A Runtime Controls ungoverned-egress finding whose destination host is in the AIControls catalogue of AI services, such as api.openai.com or api.anthropic.com. The gateway can't block this traffic; the network and runtime tiers do. A destination the catalogue doesn't list, or one it marks as a broad domain that mostly carries non-AI traffic, isn't evidence.

Activity from the last 30 days counts. A workload listed this way is Suspected unless it also declares itself, and its detail drawer shows the evidence under Why this workload is listed, for example "Called openai/gpt-4o through the gateway at 2026-10-06T12:00:00Z". Select Register anyway… in the row menu to register it. Discovery still skips excluded namespaces, the workload kinds it doesn't scan, and the AIControls workload itself.

Static discovery and runtime evidence​

Discovery reads workload specifications; Runtime Controls observes what pods do within the sensor's coverage. Apart from the observed AI activity above, these are independent signals. A workload with no recognizable AI name, image, label, or environment-variable name, and no gateway call or AI-provider egress finding, can be absent from Inventory while still producing other runtime findings. An empty inventory does not establish that no AI activity occurred. Conversely, an inventory entry does not prove that its traffic reached the gateway or that enforcement is active.

Agent Sandbox and OpenShell​

Discovery scans the Agent Sandbox Sandbox kind (agents.x-k8s.io/v1beta1) automatically when the cluster serves it. You don't add it to discovery.additionalKinds. On a cluster without the CRD, discovery skips the kind without a list call, a warning, or an incomplete-scan record, and it starts scanning the kind on the next pass after you install the CRD.

A Sandbox that OpenShell manages, which carries the openshell.ai/managed-by: openshell label, gets the OpenShell sandbox type. The Workloads table shows it next to the kind, and the workload drawer shows it in the header. The type wins over a declared nirmata.io/ai-type, so a sandbox that the OpenShell sample script labels nirmata.io/ai-type=agent still shows the OpenShell sandbox type. A nirmata.io/ai-type value of openshell-sandbox is ignored: only the runtime label sets the type. The type is a fact about the runtime and doesn't make the workload AI: the sandbox is confirmed only when you label it nirmata.io/ai or it matches the label selector.

To stop scanning the kind, add a discovery.additionalKinds entry with group: agents.x-k8s.io, version: v1beta1, resource: sandboxes, kind: Sandbox, and enabled: false. An explicit disable wins over the automatic enablement.

If the cluster's API discovery call fails with an error other than "not found", such as a timeout, discovery records the scan as incomplete and doesn't mark existing Sandbox rows as gone.

CronJobs and Jobs​

The scan records the CronJob, not the Jobs it creates. A CronJob produces a new Job on every tick, so recording those would add an inventory row per schedule and mark it gone minutes later. A standalone Job — one no CronJob controls — has nothing else representing it, so it gets its own row.

Configure it​

discovery:
  enabled: true
  # How often to re-scan.
  intervalSeconds: 900
  # Drop workloads scoring below this signal threshold. Range 0.0-1.0.
  minConfidence: 0.50
  # Restrict the scan to one namespace. Empty scans every namespace.
  namespace: ""
# Seeds the excluded namespaces on first install only; edit them later in
# Settings > Workloads > Cluster Discovery.
workloadTrust:
  excludeNamespaces: [kube-system, kube-public, kube-node-lease, datadog]

To turn the scan off entirely, set discovery.enabled: false. That removes the ClusterRole as well as stopping the scan, so no standing permission is left behind for a feature that is not running.

The scan also needs a self-managed installation. The chart sets deployment.mode: selfManaged by default, which is the value you want for a cluster you run. The only other accepted value, hosted, is reserved for Nirmata-hosted tenants and turns discovery off, because a hosted tenant runs in Nirmata's cluster rather than yours. You can also set the mode with the AICONTROLS_DEPLOYMENT_MODE environment variable outside the chart.

Add a custom workload kind​

Custom resources that run AI workloads — a KServe InferenceService, a Ray RayCluster, your own operator's CRD — are not scanned until you add them. Each entry also renders its own list-only RBAC rule, so the grant can never be wider than the coverage you asked for:

discovery:
  additionalKinds:
    - group: serving.kserve.io
      version: v1beta1
      resource: inferenceservices
      kind: InferenceService
    - group: ray.io
      version: v1
      resource: rayclusters
      kind: RayCluster
      templatePath: [spec, headGroupSpec, template, spec]

Set templatePath to the field path holding a pod spec, one field name per segment. AIControls cannot know where a third-party CRD keeps its pod template, so a custom kind needs you to say. Omit it when the kind has no single pod template, as KServe's InferenceService does: the workload is still inventoried from its own labels and annotations, with no container evidence and no ServiceAccount.

You can also correct a built-in kind's templatePath the same way. The rule is that only a path you set explicitly is stored, and only a stored path overrides the one AIControls ships:

KindWhat is stored
Built-in, untouchedNothing. The shipped path applies, so a correction in a later release reaches you automatically.
Built-in, explicitly changedYour path, recorded as an override, and it survives upgrades.
CustomYour path. There is no shipped default to fall back to.

Saved workload-kind overrides appear in the configuration audit trail. If the settings or audit record cannot be persisted, the save fails and the running scanner keeps its previous kind table. Retry after storage is available.

When a path is wrong​

A path is checked for shape when you save it — each segment a single field name, no JSONPath syntax, and a sane depth — so a typo is rejected immediately with a message naming the problem.

A path that saves cleanly but does not actually point at a pod spec fails at scan time, and it fails that kind only: the kind is skipped, the scan records itself incomplete, and every other kind is still recorded. Because the scan is incomplete, nothing is swept, so no workload is marked as gone over a configuration mistake. The Inventory page shows the skipped kind, and the proxy log names the object the path could not be read on.

A path that is simply absent on a particular object is not an error — an optional component that is not configured is a normal thing to find. That workload is inventoried without container evidence, alongside the ones that had a template.

Exclude a workload​

Add the nirmata.io/discovery-exclude: "true" label to a workload or its pod template, and the scan skips it. Use this to silence a known false positive — a workload whose name or environment happens to match the heuristics without being an AI workload.

When the inventory is incomplete​

If the ServiceAccount cannot list one of the configured kinds — a CRD that is not installed, or an RBAC rule you have not applied yet — the scan skips that kind and continues, and records itself as incomplete. An incomplete scan does not sweep, so no workload is marked gone because the scan could not see it. AIControls probes every configured kind at startup and reports any gaps on the Inventory page, so an inventory that is short of a kind says so rather than looking merely empty.

Request-time Pod-to-owner attribution needs cluster-wide get access. The chart grants it by default with workloadIdentity.podRead.enabled: true: a separate ClusterRole with get on Pods and their supported owner kinds, and no list or watch. Pod specs can contain sensitive configuration, so set it to false if the proxy must not read them. This value controls only that RBAC template: enabled runtime controls can grant the same reads independently. If no effective grant exists, the lookup fails closed to an empty tuple. Helm renders custom-kind rules only for an enabled kind with a non-core API group and a templatePath. Adding or enabling a kind in Settings hot-updates the resolver, but it cannot change Kubernetes RBAC. Add the same kind to the chart's discovery.additionalKinds and apply a Helm upgrade before relying on its attribution. Until the grant is present, the Kubernetes lookup fails closed and the request's workload tuple is empty; this never grants access. Template- less inventory kinds do not receive the request-time get grant.

Discovery and runtime controls also need the cluster's identity, which is the kube-system namespace UID. If AIControls can't read it at startup, for example because the RBAC rule is missing or the API server is unreachable, both stay off and say why. AIControls keeps retrying, backing off to once every 5 minutes. When the UID becomes readable, AIControls moves the workload records it kept under the cluster's display name to the UID and starts discovery and runtime controls without a restart.

If discovery or runtime controls still can't start after the UID becomes readable, for example because the Kubernetes client can't be built or the managed runtime policies fail to render, AIControls retries the start with backoff, up to once every 5 minutes, until it succeeds. The Inventory page, the runtime controls endpoints, and the findings collector state the latest failed attempt; the proxy log has the cause. After discovery starts, protection annotations use its live kind table, so a kind you add on the Inventory page can be annotated without a restart.

How new Pods are attributed​

With runtime controls on, each AIControls replica keeps a snapshot of every Pod and the workload that owns it. Runtime findings, gateway identities, and intent recommendations all use it to attribute a Pod to its workload. The snapshot isn't a Pod informer: it holds only Pod metadata and owner references, and it caches no Pod specs.

Three things keep the snapshot current:

  • Relist. Every runtimeControls.snapshotRelistInterval (15 minutes by default), each replica lists every Pod, ReplicaSet, and Job in the cluster and rebuilds its snapshot. This is the backstop.
  • Pod-scheduled event watch. With runtimeControls.podEvents.enabled (the default), each replica watches core/v1 Events, filtered on the API server to involvedObject.kind=Pod,reason=Scheduled. For each new Pod, it reads that Pod and, when the snapshot lacks it, the Pod's ReplicaSet or Job, so the Pod is attributed within seconds. Events from excluded namespaces are skipped. The watch starts from the current state and doesn't replay old events. After a disconnect or a 410 Gone, it resumes with backoff and lists the events it missed.
  • On-demand lookup. When a verified, Pod-bound ServiceAccount token names a Pod the snapshot doesn't hold, AIControls reads that Pod and adds it. The lookup runs in the background, never on the request, and the snapshot attributes the Pod only when the token's Pod UID matches the live Pod. This works with the watch off and needs only the existing get on Pods.

Each replica runs its own watch rather than one replica behind a lease, because each replica keeps its own snapshot and serves requests from it. Lookups share a per-replica bound, runtimeControls.podEvents.maxConcurrentLookups (4 by default), one lookup per Pod at a time, and a short cache of Pods that don't exist.

Once a Pod is attributed, AIControls requests a workload trust pass, so the Pod's gateway activity reaches the next intent recommendation pass, which runs at most every 5 minutes.

Rescan on demand​

An administrator can select Rescan on Inventory > Workloads to relist every replica's Pod snapshot and scan the whole cluster immediately instead of waiting for the next interval. The text under the page title shows what the scan includes, when it last ran, and a Configure discovery link to the settings above. The configuration audit log records each rescan. To scan one namespace, send POST /api/v1/discovery/scan with {"namespace": "<name>"}. A scan writes the inventory, so both are administrator actions rather than reads. See Rescan workloads for the API.

See also​

Last updated on