Skip to main content

Getting Started

Self-Hosted Kubernetes

Install AIControls into an existing Kubernetes cluster using the published Helm chart. No source code required.

Prerequisites​

  • Kubernetes ≥ 1.27
  • Helm ≥ 3.12
  • kubectl with cluster-admin access (for the initial install)
  • A persistent volume provisioner (any StorageClass that supports ReadWriteOnce)

Quickstart on kind​

To try AIControls on your workstation, and you have a checkout of the AIControls source, run this from the checkout instead of the steps below. It needs kind, kubectl, helm, docker, go, jq, and python3.

make kind-quickstart

The command:

  1. Creates a kind cluster with its own kubeconfig file. It never changes your default kubeconfig or its current context, and every command it runs names that file and context.
  2. Builds the AIControls image from the checkout and loads it into the cluster.
  3. Installs the Helm chart with its defaults and llmProxy.enabled: true, and waits until AIControls is ready.
  4. Through the admin API, adds the public DeepWiki MCP server, https://mcp.deepwiki.com/mcp, as deepwiki, and adds the LLM upstreams you pass keys for. With OPENROUTER_API_KEY, it also grants the OpenRouter free model nvidia/nemotron-3.5-lightning:free to the default Workload Blueprint's Access Binding, through the same audited binding edit the setup wizard's Model step uses.
  5. Prints the UI URL, the admin email, and the admin password it set, and starts a port-forward to the UI and the gateway.
VariableDefaultWhat it does
CLUSTERaicontrols-quickstartThe kind cluster name.
QUICKSTART_KUBECONFIG~/.kube/aicontrols-quickstartThe kubeconfig file for this cluster. The KUBECONFIG variable is ignored, because it often names your default kubeconfig.
ANTHROPIC_API_KEYAdds an Anthropic upstream with this key.
OPENROUTER_API_KEYAdds an OpenRouter upstream, as an OpenAI-compatible provider at https://openrouter.ai/api/v1, and grants OPENROUTER_MODEL to the default Workload Blueprint. The setup wizard shows the upstream as configured, with the key stored. See An upstream that's already set up.
OPENROUTER_MODELnvidia/nemotron-3.5-lightning:freeThe OpenRouter model to grant to the default Blueprint: the sample agent's model.
DEPLOY_AGENT0Set to 1 to deploy the sample agent from Protect your first agent.
LICENSE_FILEPath to a license (.lic) file. The command creates the Secret aicontrols-license from it in the release namespace and installs the chart with license.existingSecret, which mounts the file and sets proxy.licenseFile. A license is bound to the cluster's kube-system namespace UID, so request it after the cluster exists (see Step 9). The proxy rejects a license that doesn't match the cluster. When you leave it unset, the command creates no Secret and mounts nothing.
ADMIN_EMAIL, ADMIN_PASSWORDadmin@local, generatedThe admin login.
ADMIN_PORT, GATEWAY_PORT8081, 8080The local ports for the port-forward. If one is in use, the command prints the port-forward command instead.
PORT_FORWARD1Set to 0 to print the port-forward command instead of running it.
DEEPWIKI1Set to 0 to skip adding the DeepWiki MCP server. It needs outbound internet access; if the add fails, the command warns and goes on.

To delete the cluster, run make kind-quickstart-down with the same CLUSTER and QUICKSTART_KUBECONFIG.

To start over with an empty database and keep the cluster, run make kind-quickstart-reset with the same variables you passed to make kind-quickstart:

make kind-quickstart-reset
warning

The reset deletes all AIControls data in the quickstart cluster: users, registered workloads, policies, audit events, and settings. The command prints this warning before it starts.

The command uninstalls the release, deletes its PersistentVolumeClaims, and installs the chart again. It reuses the images already loaded into the cluster and doesn't recreate the cluster, so it's faster than make kind-quickstart-down followed by make kind-quickstart. It uses the same CLUSTER and QUICKSTART_KUBECONFIG defaults, passes --kubeconfig and --context on every command, and never touches other clusters. Because the cluster stays, its kube-system UID doesn't change, so a license you passed with LICENSE_FILE still matches.

Optional demo MCP server​

The Helm chart doesn't install an MCP server for agents. To try tool governance in a cluster with no outbound internet access, a checkout of the AIControls source includes an optional demo MCP server sample: the MCP Everything reference server, with a Dockerfile, a standalone manifest you apply yourself, and a README. On kind, make demo-mcp-kind-load CLUSTER=<name> builds its image and loads it into the cluster. The sample's README shows how to register it as the MCP server demo through the chart's datasources value.

warning

The upstream project describes the Everything server as a reference implementation, not for production. Don't run it in production.

With outbound internet access, a public server is simpler. See Try AIControls with free models and a public MCP server.

Step 1 — Get the Helm chart​

The AIControls chart is published as an OCI artifact. No helm repo add needed — reference it directly with oci://:

helm show chart oci://ghcr.io/nirmata/charts/aicontrols
VERSION=1.12.2
helm show chart oci://ghcr.io/nirmata/charts/aicontrols --version $VERSION

Step 2 — Prepare your values file​

Create a values.yaml. Uncomment and fill in the sections that apply to your cluster.

What belongs in values.yaml vs. the Admin UI

values.yaml only covers deployment settings — things like storage, networking, and TLS. Rate limits, budgets, LLM upstreams, OIDC, and other app-level settings are configured after install, from Settings in the Admin UI, and only ever read values.yaml once, on first boot. See Configuration Model for the full split.

# ── Admin account ─────────────────────────────────────────────────────────────
# Required — the email address for the bootstrap admin account. This becomes the
# first admin user's login identity and is used to issue Personal Access Tokens
# during onboarding. It cannot be changed after the account is created, and the
# install fails with a clear error if left unset.
admin:
  email: ""   # REQUIRED — set to your email address; install fails until this is set

# ── Storage ─────────────────────────────────────────────────────────────────
# The data directory is a ReadWriteOnce PersistentVolumeClaim by default
# (5Gi). Set enabled: false for an emptyDir, which a pod restart wipes.
storage:
  persistentVolume:
    enabled: true
    existingClaim: ""  # optional: mount a claim you manage; no claim is created
    size: 10Gi
    storageClass: ""   # leave empty to use the cluster default

# ── External access ──────────────────────────────────────────────────────────
# Gateway API — single ingress point with path-based routing (recommended).
# Requires Gateway API CRDs in the cluster (e.g. Envoy Gateway, Istio, Cilium).
# Install Envoy Gateway: helm upgrade --install eg oci://docker.io/envoyproxy/gateway-helm \
#   --version v1.3.0 -n envoy-gateway-system --create-namespace --wait
gateway:
  enabled: true
  className: eg      # Envoy Gateway default. Use "istio", "cilium", "nginx", etc.
  hosts:
    - aicontrols.example.com
  # tls:
  #   - secretName: aicontrols-tls
  #     hosts: [aicontrols.example.com]

# Classic Ingress — alternative for clusters without Gateway API CRDs.
# ingress:
#   enabled: true
#   className: nginx
#   hosts:
#     - host: aicontrols.example.com
#       paths:
#         - { path: /mcp/admin, pathType: Prefix, port: admin }
#         - { path: /mcp,       pathType: Prefix, port: mcp   }
#         - { path: /sse,       pathType: Prefix, port: mcp   } # legacy, disabled by default — see #sse-legacy below
#         - { path: /auth,      pathType: Prefix, port: mcp   }
#         - { path: /v1,        pathType: Prefix, port: mcp   }
#         - { path: /.well-known, pathType: Prefix, port: mcp }
#         - { path: /oauth,     pathType: Prefix, port: mcp   }
#         - { path: /oauth2,    pathType: Prefix, port: mcp   }
#         - { path: /servers,   pathType: Prefix, port: mcp   }
#         - { path: /cursor,    pathType: Prefix, port: mcp   }
#         - { path: /bedrock,   pathType: Prefix, port: mcp   }
#         - { path: /vertex,    pathType: Prefix, port: mcp   }
#         - { path: /foundry,   pathType: Prefix, port: mcp   }
#         - { path: /api/v1,    pathType: Prefix, port: admin }
#         - { path: /,          pathType: Prefix, port: admin }
#   tls:
#     - secretName: aicontrols-tls
#       hosts: [aicontrols.example.com]

# ── Action links in notifications ────────────────────────────────────────────
# The externally reachable base URL of this deployment, e.g.
# https://aicontrols.example.com — normally the same hostname as gateway.hosts
# or ingress.hosts above. Required for working action links in Slack, Teams,
# and email notifications ("Review request", budget alerts, etc.); without it,
# or when it names this machine (localhost, 127.0.0.1), those links are left
# out. The VS Code extension also reads it
# (as <publicURL>/ui/) to open the Developer Portal; set it to the address
# where the portal is served, even if that is not the gateway origin.
ui:
  publicURL: ""

# Optional, restart-only inline Judge for unmatched workload access.
# The delegate must return stringResult plus a real confidence in [0,1].
workloadAccess:
  judge: false
llmExt:
  endpoint: ""
  timeoutSeconds: 3
Set ui.publicURL before configuring notification integrations

If you plan to connect Slack, Teams, or email notifications, set ui.publicURL to the hostname from Step 6 first. Every "Review request" / "Manage budgets" link in those notifications is built from this value. Until it is set to an address other machines can reach, notifications go out without these links.

Optional: ML-based prompt-injection detection

The chart can also deploy an ML classifier service that scores LLM prompts, most LLM responses, and MCP tool arguments/responses for prompt-injection/jailbreak intent. It's off by default, provisional (not production-validated), and has known request/response coverage gaps on the cloud proxy — see Enable the injection classifier for the full enablement, streaming-buffer, trust-boundary, and calibration workflow before turning it on.

LLM upstream and policies are configured in the UI

After first login, the setup wizard walks you through connecting an LLM upstream (Anthropic, OpenAI, Bedrock, Ollama, etc.) and creating your first policies. You do not need to put API keys in values.yaml.

For an installation without external policy sources, explicitly set sources to an empty list under policy in your Helm values:

policy:
  sources: []

The empty list disables every external policy source, including the chart's default. Built-in and administrator-created policies still load from the configured database automatically, with either SQLite or PostgreSQL.

Omitting policy.sources from your Helm overrides keeps the chart's default source list, so set it to [] explicitly. Setting it to null removes the key from the rendered configuration and restores the default local file source instead; it does not disable external sources. Do not configure type: db: it is not a supported policy-source type.

Don't have Gateway API installed yet?

Install Envoy Gateway — a CNCF project that implements the Gateway API spec:

helm upgrade --install eg \
  oci://docker.io/envoyproxy/gateway-helm \
  --version v1.3.0 \
  -n envoy-gateway-system \
  --create-namespace \
  --wait

This installs the Gateway API CRDs and the controller. The chart does not create a GatewayClass — create one yourself, pointing at the Envoy Gateway controller:

kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: eg
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
EOF

If your cluster already has Istio, Cilium, or another Gateway API provider, set gateway.className to match and skip this step.

If you don't want to install a gateway provider right now, set gateway.enabled: false and use the classic Ingress or port-forward options in Step 6.

Step 3 — Install via Helm​

helm install aicontrols \
  oci://ghcr.io/nirmata/charts/aicontrols \
  -n aicontrols \
  --create-namespace \
  -f values.yaml

Helm prints next steps when the install completes. To upgrade later:

helm upgrade aicontrols \
  oci://ghcr.io/nirmata/charts/aicontrols \
  --version NEW_VERSION \
  -n aicontrols \
  -f values.yaml

An upgrade can run database migrations at startup that take minutes on a large audit log. See Startup migrations on upgrade.

Chart defaults​

With no identity or storage values, the chart:

  • Discovers the cluster OIDC issuer. At startup, AIControls reads the ServiceAccount token issuer from the Kubernetes API and verifies agents' projected tokens for the aicontrols audience. See Issuer discovery.
  • Reads Pods for workload attribution. workloadIdentity.podRead.enabled: true adds a ClusterRole with get on Pods, on the apps Deployments, StatefulSets, DaemonSets, and ReplicaSets, and on the batch Jobs and CronJobs. The gateway uses it to map each caller's Pod to the workload that owns it. The ClusterRoleBinding is cluster-wide, so AIControls can get any Pod in any namespace, including its full spec. Pod specs can contain sensitive values, such as environment variables; set it to false to drop the grant, and calls are then verified but not attributed to a workload.
  • Keeps its data on a PersistentVolumeClaim. The data directory is a 5 GiB ReadWriteOnce claim, <release>-aicontrols-data (or aicontrols-data for a release named aicontrols), from your default StorageClass, and the Deployment uses the Recreate strategy so only one pod uses the volume. With more than one replica, use PostgreSQL: each replica then gets an emptyDir data directory. More than one replica with SQLite and the claim fails the install, and the error names the fix.
Upgrading a release that used the old defaults
  • A release that had no identity.clusterOIDCIssuer moves from passthrough to verified tokens when discovery succeeds, so agents must present a projected ServiceAccount token for the aicontrols audience, or for the identity.clusterOIDCAudience you set. To keep passthrough, set identity.discoverClusterOIDCIssuer: false.
  • A release that used the emptyDir data directory gets a new, empty claim. Data in the emptyDir isn't migrated; a pod restart already lost it. To keep the emptyDir, set storage.persistentVolume.enabled: false.
  • An existing chart-managed claim keeps its size. To grow it, set storage.persistentVolume.size to a value other than 5Gi; the StorageClass must allow volume expansion.
  • GitOps renders from your values only. Argo CD and helm template can't read the cluster, so the chart can't see an existing claim. The chart decides the data volume from storage.persistentVolume.enabled and storage.persistentVolume.existingClaim, and it renders the claim size you set. To mount a claim you manage, set storage.persistentVolume.existingClaim to its name; the chart then creates no claim and ignores size and storageClass. The setting has no effect when enabled is false, or with PostgreSQL and more than one replica, where each replica gets an emptyDir. To keep the chart's existing claim without tracking its size, set existingClaim to that claim's name; Helm leaves the claim in place because it carries helm.sh/resource-policy: keep. Argo CD with automated pruning can still delete a claim that no longer appears in the rendered manifests, so turn off pruning for the application, or annotate the claim with argocd.argoproj.io/sync-options: Prune=false, before you set existingClaim. Otherwise, set size to the current size of the chart's data claim (the release's full name followed by -data; run kubectl get pvc to find it), because only helm upgrade keeps an existing claim's size when you leave the 5Gi default.

Step 4 — Retrieve the bootstrap admin password​

The Helm chart generates a random 20-character password on first install and stores it in a Kubernetes Secret. Retrieve it now.

kubectl get secret aicontrols-admin-bootstrap \
  -n aicontrols \
  -o jsonpath='{.data.password}' | base64 -d
echo   # print newline after the password
First login

After you log in for the first time, the UI will prompt you to set a permanent password. The bootstrap Secret is kept after helm uninstall (via helm.sh/resource-policy: keep) so it is not accidentally deleted if you reinstall. Delete it manually once you have set a permanent password.

Setting a known password

To set a specific password instead of a random one, add admin.password: YOUR_PASSWORD to values.yaml before installing. This is useful for dev clusters where you want a stable, known password.

admin.email is required

helm install/helm upgrade fails with admin.email is required unless admin.email is set in values.yaml (Step 2). This applies to upgrades of existing installs too — set it before upgrading a release created with an older chart version.

Step 5 — Verify the deployment​

# Wait for the pod to become ready:
kubectl rollout status deployment/aicontrols -n aicontrols

# Check pod and service:
kubectl get pods,svc -n aicontrols
# NAME                               READY   STATUS    RESTARTS   AGE
# pod/aicontrols-7d9f84c5b9-xk2pq   1/1     Running   0          45s
#
# NAME               TYPE        CLUSTER-IP      PORT(S)
# svc/aicontrols     ClusterIP   10.96.142.31    8080/TCP,8081/TCP,9081/TCP,9082/TCP

# Tail logs:
kubectl logs -n aicontrols deploy/aicontrols --follow

Wait for the rollout to succeed and verify that /readyz returns HTTP 200 through your chosen admin access path. A successful Helm submission or a started proxy log message alone does not prove readiness. The health endpoints are:

EndpointPort
GET /healthz8081 (admin)
GET /readyz8081 (admin)
GET /metrics8081 (admin)
GET/HEAD /8080 (proxy), 8081 (admin)

GET/HEAD / returns 200 OK on both ports specifically for cloud load balancers and Gateways whose default health check probes / and expects a 200 (GKE, EKS, and others do this unless you attach a custom health-check policy). This lets those default checks pass with no extra configuration. On port 8081, when the admin UI is enabled, a browser request to / (one sending Accept: text/html) is redirected to /ui/ instead — health-check probes, which don't send that header, still get a plain 200. For explicit liveness and readiness wiring, prefer /healthz and /readyz.

Storage must be reachable at startup and while running

If the pod can't open its database (SQLite on the persistent volume, or your configured PostgreSQL connection) at startup, it exits instead of starting up without persistence, and Kubernetes retries via its normal restart/back-off policy. /readyz also checks the database connection on every poll once the pod is running, so a pod that later loses connectivity (a PostgreSQL failover, for example) is taken out of the Service's rotation until the connection recovers, instead of continuing to serve traffic with a broken audit trail and budget enforcement.

Step 6 — Expose the service​

Forward directly to the aicontrols Service — no gateway needed:

kubectl port-forward -n aicontrols svc/aicontrols 8080:8080 8081:8081
  • Admin UI: http://localhost:8081/ui/
  • MCP proxy: http://localhost:8080

If you expose AIControls through a gateway or ingress, set its request timeout above the longest human-review window (30 minutes for MCP tool calls and 5 minutes for LLM requests by default). A request held for approval keeps its connection open until a reviewer decides, and a shorter timeout cuts it off while it is still pending. For the chart's Gateway API route, set gateway.requestTimeout (for example "1h"); it is empty by default, which leaves the gateway's own default in place. See Gateway and ingress timeouts.

/sse is legacy and disabled by default​

The /sse path above is the legacy 2024-11-05 SSE MCP transport, kept for clients that can't yet use Streamable HTTP (/mcp). It now returns 404 unless proxy.enableLegacySSE: true is set in values.yaml:

proxy:
  enableLegacySSE: true

Only enable it if you have clients that require SSE. The legacy transport pins a session's identity, tenant, and upstream credentials to whichever request opened the session — there is no way to re-derive them per call the way the Streamable HTTP transport does. Since the session ID travels as a URL query parameter on every request (commonly captured by access logs, load balancers, and APM tooling), anyone who obtains a live session's ID can drive tool calls under that session's identity. Prefer Streamable HTTP wherever possible.

The MCP OAuth issuer must match the public hostname​

MCP clients that sign in through AIControls (Claude Desktop, Claude Code, VS Code) start by fetching /.well-known/oauth-authorization-server and then follow the URLs it advertises. Those URLs come from proxy.mcpIssuerURL.

When it is empty, the chart derives it from the first Ingress host or Gateway hostname you configured above — correct for most installs. Set it explicitly whenever clients reach AIControls through a hostname the chart doesn't know about (an external load balancer, a CDN, or custom DNS):

helm upgrade aicontrols oci://ghcr.io/nirmata/charts/aicontrols \
  -n aicontrols --reuse-values \
  --set proxy.mcpIssuerURL=https://aicontrols.example.com

Confirm the advertised issuer is the URL clients actually use:

curl -s https://aicontrols.example.com/.well-known/oauth-authorization-server | jq .issuer

If it returns http://localhost:8080, the value is unset and undeducible — clients complete discovery and then dial their own machine. The post-install notes print a warning in that case.

caution

The issuer is the iss claim of every token AIControls mints, and the token verifier matches on it. Changing it invalidates tokens already issued, so clients sign in again after the change.

Step 7 — TLS (optional)​

Skip this step if you are using port-forward for local access — TLS is not needed there.

The chart can generate a self-signed certificate and store it in a Secret. The cert is created once on first install and reused on every helm upgrade (idempotent via lookup). Valid for 10 years.

warning

Self-signed certificates are not trusted by browsers or most clients without importing the CA. Use only for evaluation or air-gapped clusters. For production, use cert-manager or provide your own certificate.

Enable in values.yaml and upgrade:

tls:
  selfSigned:
    enabled: true

gateway:
  enabled: true
  hosts:
    - aicontrols.example.com   # used as the certificate CN / SAN

The chart creates a Secret named <release>-tls and automatically adds an HTTPS listener (port 443) to the Gateway.

Step 8 — Log in and complete setup​

Open the admin UI in a browser. Use the URL that matches how you exposed the service in Step 6:

Exposure methodAdmin UI URL
Port-forward (direct)http://localhost:8081/ui/
Gateway API port-forwardhttp://localhost:8080/ui/
Ingress / hostnamehttps://aicontrols.example.com/ui/

Log in with username admin and the bootstrap password retrieved in Step 4. You will be prompted to set a permanent password on first login.

After changing your password, the setup wizard launches automatically. It walks you through:

  1. LLM provider — connect Anthropic, OpenAI, AWS Bedrock, Azure OpenAI, GCP Vertex AI, or a self-hosted model (Ollama, vLLM). Your API key is stored as a Kubernetes Secret and never exposed in the UI.
  2. MCP servers — add upstream MCP servers that AIControls will proxy and govern.
  3. First policy — create a starter governance policy (allowlist, HITL trigger, or budget).

Complete the wizard before connecting agent pods — agents need at least one upstream configured to route traffic through.

Step 9 — Activate your license​

AIControls needs a signed license before it will process any AI traffic (MCP tool calls or LLM requests) — the Admin UI stays browsable, but requests are blocked with 402 until a license is loaded. helm install/helm upgrade also prints this reminder in the post-install notes if proxy.licenseFile isn't set.

On-prem licenses are bound to this cluster's kube-system namespace UID, not to a Nirmata account, so a license issued for one cluster won't activate on another. Retrieve the UID:

kubectl get namespace kube-system -o jsonpath='{.metadata.uid}'

Email that UID to support@nirmata.com to request a license. Once you receive the signed .lic file, see License management to provision it.

Usage telemetry

Nothing is sent until you set telemetry.endpoint. Once it is set, the Free plan sends a daily usage document that cannot be turned off; paid plans send it only with telemetry.enabled: true. See Usage telemetry for the fields.

Step 10 — Connect agent pods​

Agent pods must send MCP and LLM traffic through AIControls instead of calling upstream servers directly.

Set MCP_SERVER_URL (and optionally OPENAI_BASE_URL for the LLM proxy) in your agent pod spec:

apiVersion: v1
kind: Pod
metadata:
  name: my-claude-agent
  namespace: agents
spec:
  serviceAccountName: my-agent-sa
  containers:
    - name: agent
      image: my-org/my-ai-agent:latest
      env:
        - name: MCP_SERVER_URL
          value: "http://aicontrols.aicontrols.svc.cluster.local:8080"
        # For OpenAI-compatible clients (Claude Code, Cursor, etc.):
        - name: OPENAI_BASE_URL
          value: "http://aicontrols.aicontrols.svc.cluster.local:8080/v1"

For OIDC identity validation, each agent pod must also present a projected ServiceAccount token:

spec:
  serviceAccountName: my-agent-sa
  containers:
    - name: agent
      env:
        - name: MCP_AUTH_TOKEN_FILE
          value: /var/run/secrets/aicontrols/token
      volumeMounts:
        - name: proxy-token
          mountPath: /var/run/secrets/aicontrols
  volumes:
    - name: proxy-token
      projected:
        sources:
          - serviceAccountToken:
              path: token
              expirationSeconds: 3600
              audience: aicontrols   # must match clusterOIDCAudience in values.yaml

Uninstalling​

helm uninstall aicontrols -n aicontrols

The bootstrap Secret (aicontrols-admin-bootstrap) and the PersistentVolumeClaim are kept after uninstall. Delete them manually if you want a clean slate:

kubectl delete secret aicontrols-admin-bootstrap -n aicontrols
kubectl delete pvc -n aicontrols -l app.kubernetes.io/name=aicontrols
kubectl delete namespace aicontrols

Startup migrations on upgrade​

A new pod applies database migrations before it starts serving, including its /healthz health check. Most migrations take seconds. A release that adds indexes to the audit log builds them at this point, and on an audit_events table with millions of rows each index can take a minute or more.

The chart gives the AIControls container a startup probe so that a slow migration isn't mistaken for a hung pod. Kubernetes doesn't run the liveness or readiness probe until /healthz answers. The pod has periodSeconds (10 seconds) times startupProbe.failureThreshold to start, which is about 5 minutes with the default of 30. If the budget runs out, Kubernetes restarts the pod, and the restarted pod repeats the unfinished migration. For a very large audit log, raise the budget before you upgrade:

startupProbe:
  failureThreshold: 48   # 48 x 10s = 8 minutes

Keep the budget well under 10 minutes. That's the Deployment's default progress deadline, which also counts the time spent scheduling the pod and pulling its image. Past it, kubectl rollout status reports the rollout as failed even though the pod is still migrating. If you run helm upgrade --wait, set --timeout to cover the whole rollout: the startup budget plus scheduling and image pull time. Helm's default wait is 5 minutes.

With the default SQLite backend, every migration runs at pod startup, and audit-event writes wait until it finishes. There's no online pre-build for SQLite, so size startupProbe.failureThreshold for the largest migration you expect, and upgrade at a quiet time. On PostgreSQL, you can build the new session indexes online before the upgrade instead; see Pre-build session indexes before upgrading.

Virtual keys without a cluster ID stop working​

A namespace-scoped virtual key must carry the cluster ID (the kube-system namespace UID) of the cluster its workload runs in. At startup, AIControls expires every active namespace-scoped key that has none, and agents using those keys get 401 until you reissue them. Check for them before you upgrade:

curl -s "$YOUR_WORKSPACE_URL/api/v1/virtual-keys" -H "Authorization: Bearer $ADMIN_TOKEN" \
  | jq '[.[] | select(.status == "active" and (.namespace // "") != "" and (.clusterId // "") == "")] | length'

After the upgrade, the pod log reports how many keys it expired, for example expired 2 namespace-scoped virtual keys without a cluster ID; reissue them. Reissue each one from Virtual Keys with Reissue for cluster…, which fills in this cluster's ID, and hand the new token to the agent. See Keys retired for a missing cluster ID.

Analytics execution deadlines​

Dashboard JSON reads default to a 30-second execution deadline, including database connection waits. To override it, add this setting to your Helm values and apply it with your normal Helm upgrade:

analytics:
  timeout: "20s"

The value must be a positive duration below the 60-second admin response write deadline. Application deployments can also set AICONTROLS_ANALYTICS_TIMEOUT. An earlier caller deadline takes precedence. Configure ingress timeouts above the analytics deadline with time for cancellation and response delivery. An ingress timeout alone does not cancel application database work.

See analytics API response behavior for timeout and unavailable responses. Increasing the deadline allows expensive queries to occupy resources for longer; inspect query and resource metrics first.

Bound dashboard database work​

Dashboard analytics share a bounded execution budget per replica. The defaults limit analytics to two simultaneous database operations. They also have room for 33 concurrent analytics reads, more than the reads of several dashboard pages sent at once. In testing, such a burst completed without 503s while some of its reads took 10 to 20 seconds. Requests beyond the limits receive HTTP 503 with Retry-After rather than growing an unlimited queue.

These Helm values show the defaults:

analytics:
  timeout: "30s"
  execution:
    maxQueries: 2
    maxActive: 8
    maxQueued: 32
    maxWaiters: 40
    # queueTimeout is unset: half of timeout (15s with the 30s default)
    cacheTTL: "2s"
    cacheEntries: 64
    cacheBytes: 4194304
    maxResponseBytes: 4194304

The response limit applies to buffered analytics JSON; exceeding it returns an explicit unavailable response. Cache limits cover retained keys and response bodies. Only stats, cost summary and savings summary retain completed responses; those views may lag writes by up to the cache TTL. Set cacheTTL: "0s" to disable retention while still sharing identical in-flight reads. Authentication and authorization run for every caller. Results are never shared across credentials or authorization scopes.

The identity counts and average cache efficiency on Home and Insights are the exception. They are reused for 15 seconds, independent of cacheTTL, by every caller with the same role, organization and tenant, so they can lag a new or removed identity by up to 15 seconds.

SettingDefaultWhat it limits
maxQueries2Database operations that analytics can run at once. Shrunk to fit the database pool together with the Cost overview reads, which leave two connections free (the SQLite pool has 8 connections; the PostgreSQL pool defaults to 16). It is never below one: with a pool of one to three connections, analytics uses the pool size minus one.
maxActive8Computations that run at once. A running computation still waits for a maxQueries permit before each database operation.
maxQueued32Computations waiting to start. When maxActive and maxQueued are both full, new requests get an immediate 503.
maxWaiters40Concurrent analytics HTTP callers, including cache hits and response writing. Callers beyond this get an immediate 503. Must be at least maxActive; keep it at or above maxActive + maxQueued so every queued computation has a caller slot. Each caller can hold a copy of a response of up to maxResponseBytes. See memory sizing.
queueTimeouthalf of timeoutHow long a computation can wait to start before it gets a 503. The timeout deadline starts when the request arrives and keeps running while the request is queued; if it expires first, the request gets a 504 instead. A value equal to or longer than timeout is replaced with half of timeout.

Tune the limits​

Size the limits to the number of analytics reads that one dashboard load sends at once; the browser sends all of them together. Keep maxActive + maxQueued and maxWaiters at or above that number, so a page's requests are queued instead of rejected on arrival. A queued read still gets a 503 if it can't start within queueTimeout, or a 504 if the request deadline expires first, for example when every active computation is a read that runs longer than the queue wait. Keep queueTimeout long enough for queued reads to start behind the slowest reads, and short enough to leave time for them to run.

Use the aicontrols_analytics_* Prometheus metrics to decide what to change:

SymptomMetricsChange
Fast 503s during a single dashboard loadaicontrols_analytics_responses_total{outcome="unavailable"} rises quickly, with aicontrols_analytics_state{resource="queued"} at maxQueued, or with no executor rejections because maxWaiters callers are still running or writing responsesRaise maxQueued and maxWaiters to cover the page's concurrent reads. The waiters state counts callers waiting for a result and can be lower than the number still writing responses.
503s after about queueTimeoutaicontrols_analytics_events_total{resource="rejected"} rises; aicontrols_analytics_wait_seconds_total{resource="admission"} grows quicklyReads are queued behind slow reads. Raise maxActive or queueTimeout, or fix the slow reads.
504saicontrols_analytics_responses_total{outcome="timeout"} rises; aicontrols_analytics_wait_seconds_total{resource="query_admission"} grows quicklyReads are waiting for database permits or running too long. Check query duration and CPU throttling before raising maxQueries or timeout.
Gateway latency rises with dashboard useaicontrols_analytics_state{resource="db_pool_wait_count"} and db_pool_wait_seconds riseAnalytics is competing for database connections. Lower maxQueries, or raise the PostgreSQL pool size.

Memory sizing​

Analytics buffers each JSON response before sending it. A computation keeps a response buffer and a result copy, and the result can stay in memory while each caller makes its own copy. A conservative upper bound for buffered responses is 2 x (maxWaiters + maxActive) x maxResponseBytes + cacheBytes: about 388 MiB with the defaults, under the chart's default 512 MiB memory limit. That bound needs every response to be near the 4 MiB limit, and it doesn't count temporary allocations while a handler builds and encodes a response. Most dashboard responses are well under 4 MiB, so typical use is a small fraction of it. On replicas with a low memory limit, lower maxResponseBytes or maxWaiters so the worst case fits with room for the gateway.

On SQLite, raising maxQueries above 2 made a seeded burst of dashboard reads slower and caused 504s, because reads compete for the same CPU and file. Raise it only on PostgreSQL, and only after you measure the change. A second dashboard load that arrives while the first is still running can exceed the limits and receive 503s. That is the intended overload protection.

Execution limits exclude LLM/MCP traffic, authoritative budget checks, streaming, exports and backups. They preserve database connection headroom where possible, but do not reserve CPU, memory or disk bandwidth. Background jobs and other replicas still compete for shared resources. Increasing concurrency can worsen latency; compare API errors, database pool waits and gateway latency before and after changing these settings.

The health API exposes connections_in_use, connections_idle, wait_count and wait_duration_ms. Open connections include idle connections. Pool wait counts and duration are cumulative; compare their changes over the same time interval. These health fields share a storage snapshot cached for up to one minute; faster polling does not produce a fresh sample. The analytics Prometheus pool metrics are sampled once per second for shorter spikes, subject to the scrape interval. The separate readiness ping can remain fast while application queries wait.

Prometheus metrics prefixed aicontrols_analytics_ report active/queued work, query admission waits, cache/coalescing events, and response outcomes. A fast 503 shows overload protection working; it does not mean the underlying performance problem is resolved.

Bound the Cost overview reads​

The Cost overview endpoints read through their own small pool of database permits, separate from analytics, so a busy analytics page can't delay them. A section that can't finish in time shows a partial result instead of failing the page. See the Cost overview API for the response fields.

These Helm values show the defaults:

dashboard:
  fastLane:
    maxQueries: 4
    queueWait: "250ms"
    sectionDeadline: "2s"
    requestDeadline: "5s"
    maxWaiters: 16
    maxRawGap: "2h"
  cache:
    ttl: "45s"
    heatmapTTL: "5m"
    partialTTL: "5s"
    entries: 32
    refreshMinInterval: "10s"
SettingDefaultWhat it limits
fastLane.maxQueries4Database operations that Cost overview reads can run at once. Shrunk, together with analytics.execution.maxQueries, so that two connections stay free. With a pool of three connections or fewer, the lane is off and the reads share the analytics permits.
fastLane.queueWait250msHow long a read waits for a permit before its section reports lane_busy.
fastLane.sectionDeadline2sHow long one section can run before it reports timeout.
fastLane.requestDeadline5sThe deadline for a whole request, including waiting for another replica to compute the response.
fastLane.maxWaiters16Requests admitted at once. Requests beyond this get an immediate response with every section lane_busy.
fastLane.maxRawGap2hThe longest span of per-minute counts that AIControls reads from raw events to complete a window.
cache.ttl, cache.heatmapTTL45s, 5mHow long a response with every section ok is reused.
cache.partialTTL5sHow long a response with a partial section is reused.
cache.entries32Responses kept in memory per replica with SQLite. PostgreSQL replicas share a table instead.
cache.refreshMinInterval10sThe shortest time between refresh=1 recomputations of one response.

The aicontrols_dashboard_fastlane_*, aicontrols_dashboard_section_* and aicontrols_dashboard_cache_total Prometheus metrics report permit use, section duration, partial sections by reason, and cache results. aicontrols_dashboard_fastlane_capped is 1 when the pool size forced a smaller setting or turned the lane off.

Find what is holding the SQLite writer​

With the SQLite backend, AIControls lets one write run at a time. When one write runs for a long time, other writes queue behind it. Background jobs then fail with waiting for write lock: context deadline exceeded, and dashboards that depend on them stop updating.

MetricWhat it shows
aicontrols_sqlite_writer_waiters{mode}Writes waiting right now. A value that stays high means the writer is saturated.
aicontrols_sqlite_writer_hold_seconds{mode}Histogram of how long each write held the writer. Query its _bucket, _sum, and _count series.
aicontrols_sqlite_writer_slow_holds_total{mode,caller}Writes that held the writer for at least one second, by the function that issued them.
aicontrols_sqlite_writer_slow_hold_seconds_total{mode,caller}Total time those slow writes held the writer, by function.

To find the operation starving the others, rank callers by held time:

topk(5, sum by (caller) (rate(aicontrols_sqlite_writer_slow_hold_seconds_total[15m])))

Each slow write also logs a WARN sqlite writer slow hold line with the caller, mode, wait, hold, and number of waiting writes. Holds shorter than 15 seconds are sampled to at most one line every 15 seconds; longer holds are always logged. Include these lines and metrics when you report the problem. If audit volume keeps the writer saturated, move to PostgreSQL.

Next steps​

  • Protect your first agent — deploy a sample agent, register it in one step, learn its access, and switch it to Enforce
  • Agent Identity — assign each agent a unique ServiceAccount identity and configure dual identity for user-delegated agents
  • Write a Policy — create CEL rules for your governance requirements
  • Set a Budget — configure per-agent token and cost limits
  • Configuration Model — how deployment settings (Helm) and app settings (Settings UI) stay in sync across upgrades
Last updated on