Skip to main content

Getting Started

Self-Hosted Kubernetes

Install AIControls into an existing Kubernetes cluster using the published Helm chart. No source code required.

Prerequisites​

  • Kubernetes ≥ 1.27
  • Helm ≥ 3.12
  • kubectl with cluster-admin access (for the initial install)
  • A persistent volume provisioner (any StorageClass that supports ReadWriteOnce)

Step 1 — Get the Helm chart​

The AIControls chart is published as an OCI artifact. No helm repo add needed — reference it directly with oci://:

helm show chart oci://ghcr.io/nirmata/charts/aicontrols
VERSION=1.12.2
helm show chart oci://ghcr.io/nirmata/charts/aicontrols --version $VERSION

Step 2 — Prepare your values file​

Create a values.yaml. Uncomment and fill in the sections that apply to your cluster.

What belongs in values.yaml vs. the Admin UI

values.yaml only covers deployment settings — things like storage, networking, and TLS. Rate limits, budgets, LLM upstreams, OIDC, and other app-level settings are configured after install, from Settings in the Admin UI, and only ever read values.yaml once, on first boot. See Configuration Model for the full split.

# ── Admin account ─────────────────────────────────────────────────────────────
# Required — the email address for the bootstrap admin account. This becomes the
# first admin user's login identity and is used to issue Personal Access Tokens
# during onboarding. It cannot be changed after the account is created, and the
# install fails with a clear error if left unset.
admin:
  email: ""   # REQUIRED — set to your email address; install fails until this is set

# ── Storage ─────────────────────────────────────────────────────────────────
storage:
  persistentVolume:
    enabled: true
    size: 10Gi
    storageClass: ""   # leave empty to use the cluster default

# ── External access ──────────────────────────────────────────────────────────
# Gateway API — single ingress point with path-based routing (recommended).
# Requires Gateway API CRDs in the cluster (e.g. Envoy Gateway, Istio, Cilium).
# Install Envoy Gateway: helm upgrade --install eg oci://docker.io/envoyproxy/gateway-helm \
#   --version v1.3.0 -n envoy-gateway-system --create-namespace --wait
gateway:
  enabled: true
  className: eg      # Envoy Gateway default. Use "istio", "cilium", "nginx", etc.
  hosts:
    - aicontrols.example.com
  # tls:
  #   - secretName: aicontrols-tls
  #     hosts: [aicontrols.example.com]

# Classic Ingress — alternative for clusters without Gateway API CRDs.
# ingress:
#   enabled: true
#   className: nginx
#   hosts:
#     - host: aicontrols.example.com
#       paths:
#         - { path: /mcp/admin, pathType: Prefix, port: admin }
#         - { path: /mcp,       pathType: Prefix, port: mcp   }
#         - { path: /sse,       pathType: Prefix, port: mcp   } # legacy, disabled by default — see #sse-legacy below
#         - { path: /auth,      pathType: Prefix, port: mcp   }
#         - { path: /v1,        pathType: Prefix, port: mcp   }
#         - { path: /.well-known, pathType: Prefix, port: mcp }
#         - { path: /oauth,     pathType: Prefix, port: mcp   }
#         - { path: /oauth2,    pathType: Prefix, port: mcp   }
#         - { path: /servers,   pathType: Prefix, port: mcp   }
#         - { path: /cursor,    pathType: Prefix, port: mcp   }
#         - { path: /bedrock,   pathType: Prefix, port: mcp   }
#         - { path: /vertex,    pathType: Prefix, port: mcp   }
#         - { path: /foundry,   pathType: Prefix, port: mcp   }
#         - { path: /api/v1,    pathType: Prefix, port: admin }
#         - { path: /,          pathType: Prefix, port: admin }
#   tls:
#     - secretName: aicontrols-tls
#       hosts: [aicontrols.example.com]

# ── Action links in notifications ────────────────────────────────────────────
# The externally reachable base URL of this deployment, e.g.
# https://aicontrols.example.com — normally the same hostname as gateway.hosts
# or ingress.hosts above. Required for working action links in Slack, Teams,
# and email notifications ("Review request", budget alerts, etc.); without it,
# those links fall back to the pod-local http://localhost:<adminAddr>, which
# does not resolve outside the cluster.
ui:
  publicURL: ""

# Optional, restart-only inline Judge for unmatched workload access.
# The delegate must return stringResult plus a real confidence in [0,1].
workloadAccess:
  judge: false
llmExt:
  endpoint: ""
  timeoutSeconds: 3
Set ui.publicURL before configuring notification integrations

If you plan to connect Slack, Teams, or email notifications, set ui.publicURL to the hostname from Step 6 first. Every "Review request" / "Manage budgets" link in those notifications is built from this value.

Optional: ML-based prompt-injection detection

The chart can also deploy an ML classifier service that scores LLM prompts, most LLM responses, and MCP tool arguments/responses for prompt-injection/jailbreak intent. It's off by default, provisional (not production-validated), and has known request/response coverage gaps on the cloud proxy — see Enable the injection classifier for the full enablement, streaming-buffer, trust-boundary, and calibration workflow before turning it on.

LLM upstream and policies are configured in the UI

After first login, the setup wizard walks you through connecting an LLM upstream (Anthropic, OpenAI, Bedrock, Ollama, etc.) and creating your first policies. You do not need to put API keys in values.yaml.

For an installation without external policy sources, explicitly set sources to an empty list under policy in your Helm values:

policy:
  sources: []

The empty list disables file, Kubernetes, and Nirmata platform policy sources, even if platform.url is set. Built-in and administrator-created policies still load from the configured database automatically, with either SQLite or PostgreSQL. Leave platform.url empty when you do not use a platform connection.

Omitting policy.sources from your Helm overrides retains the chart's default nirmata source. Setting it to null removes the key from the rendered configuration and restores the default local file source instead; it does not disable external sources. Do not configure type: db: it is not a supported policy-source type.

Don't have Gateway API installed yet?

Install Envoy Gateway — a CNCF project that implements the Gateway API spec:

helm upgrade --install eg \
  oci://docker.io/envoyproxy/gateway-helm \
  --version v1.3.0 \
  -n envoy-gateway-system \
  --create-namespace \
  --wait

This installs the Gateway API CRDs and the controller. The chart does not create a GatewayClass — create one yourself, pointing at the Envoy Gateway controller:

kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: eg
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
EOF

If your cluster already has Istio, Cilium, or another Gateway API provider, set gateway.className to match and skip this step.

If you don't want to install a gateway provider right now, set gateway.enabled: false and use the classic Ingress or port-forward options in Step 6.

Step 3 — Install via Helm​

helm install aicontrols \
  oci://ghcr.io/nirmata/charts/aicontrols \
  -n aicontrols \
  --create-namespace \
  -f values.yaml

Helm prints next steps when the install completes. To upgrade later:

helm upgrade aicontrols \
  oci://ghcr.io/nirmata/charts/aicontrols \
  --version NEW_VERSION \
  -n aicontrols \
  -f values.yaml

An upgrade can run database migrations at startup that take minutes on a large audit log. See Startup migrations on upgrade.

Step 4 — Retrieve the bootstrap admin password​

The Helm chart generates a random 20-character password on first install and stores it in a Kubernetes Secret. Retrieve it now.

kubectl get secret aicontrols-admin-bootstrap \
  -n aicontrols \
  -o jsonpath='{.data.password}' | base64 -d
echo   # print newline after the password
First login

After you log in for the first time, the UI will prompt you to set a permanent password. The bootstrap Secret is kept after helm uninstall (via helm.sh/resource-policy: keep) so it is not accidentally deleted if you reinstall. Delete it manually once you have set a permanent password.

Setting a known password

To set a specific password instead of a random one, add admin.password: YOUR_PASSWORD to values.yaml before installing. This is useful for dev clusters where you want a stable, known password.

admin.email is required

helm install/helm upgrade fails with admin.email is required unless admin.email is set in values.yaml (Step 2). This applies to upgrades of existing installs too — set it before upgrading a release created with an older chart version.

Step 5 — Verify the deployment​

# Wait for the pod to become ready:
kubectl rollout status deployment/aicontrols -n aicontrols

# Check pod and service:
kubectl get pods,svc -n aicontrols
# NAME                               READY   STATUS    RESTARTS   AGE
# pod/aicontrols-7d9f84c5b9-xk2pq   1/1     Running   0          45s
#
# NAME               TYPE        CLUSTER-IP      PORT(S)
# svc/aicontrols     ClusterIP   10.96.142.31    8080/TCP,8081/TCP,9081/TCP,9082/TCP

# Tail logs:
kubectl logs -n aicontrols deploy/aicontrols --follow

Wait for the rollout to succeed and verify that /readyz returns HTTP 200 through your chosen admin access path. A successful Helm submission or a started proxy log message alone does not prove readiness. The health endpoints are:

EndpointPort
GET /healthz8081 (admin)
GET /readyz8081 (admin)
GET /metrics8081 (admin)
GET/HEAD /8080 (proxy), 8081 (admin)

GET/HEAD / returns 200 OK on both ports specifically for cloud load balancers and Gateways whose default health check probes / and expects a 200 (GKE, EKS, and others do this unless you attach a custom health-check policy). This lets those default checks pass with no extra configuration. On port 8081, when the admin UI is enabled, a browser request to / (one sending Accept: text/html) is redirected to /ui/ instead — health-check probes, which don't send that header, still get a plain 200. For explicit liveness and readiness wiring, prefer /healthz and /readyz.

Storage must be reachable at startup and while running

If the pod can't open its database (SQLite on the persistent volume, or your configured PostgreSQL connection) at startup, it exits instead of starting up without persistence, and Kubernetes retries via its normal restart/back-off policy. /readyz also checks the database connection on every poll once the pod is running, so a pod that later loses connectivity (a PostgreSQL failover, for example) is taken out of the Service's rotation until the connection recovers, instead of continuing to serve traffic with a broken audit trail and budget enforcement.

Step 6 — Expose the service​

Forward directly to the aicontrols Service — no gateway needed:

kubectl port-forward -n aicontrols svc/aicontrols 8080:8080 8081:8081
  • Admin UI: http://localhost:8081/ui/
  • MCP proxy: http://localhost:8080

/sse is legacy and disabled by default​

The /sse path above is the legacy 2024-11-05 SSE MCP transport, kept for clients that can't yet use Streamable HTTP (/mcp). It now returns 404 unless proxy.enableLegacySSE: true is set in values.yaml:

proxy:
  enableLegacySSE: true

Only enable it if you have clients that require SSE. The legacy transport pins a session's identity, tenant, and upstream credentials to whichever request opened the session — there is no way to re-derive them per call the way the Streamable HTTP transport does. Since the session ID travels as a URL query parameter on every request (commonly captured by access logs, load balancers, and APM tooling), anyone who obtains a live session's ID can drive tool calls under that session's identity. Prefer Streamable HTTP wherever possible.

The MCP OAuth issuer must match the public hostname​

MCP clients that sign in through AIControls (Claude Desktop, Claude Code, VS Code) start by fetching /.well-known/oauth-authorization-server and then follow the URLs it advertises. Those URLs come from proxy.mcpIssuerURL.

When it is empty, the chart derives it from the first Ingress host or Gateway hostname you configured above — correct for most installs. Set it explicitly whenever clients reach AIControls through a hostname the chart doesn't know about (an external load balancer, a CDN, or custom DNS):

helm upgrade aicontrols oci://ghcr.io/nirmata/charts/aicontrols \
  -n aicontrols --reuse-values \
  --set proxy.mcpIssuerURL=https://aicontrols.example.com

Confirm the advertised issuer is the URL clients actually use:

curl -s https://aicontrols.example.com/.well-known/oauth-authorization-server | jq .issuer

If it returns http://localhost:8080, the value is unset and undeducible — clients complete discovery and then dial their own machine. The post-install notes print a warning in that case.

caution

The issuer is the iss claim of every token AIControls mints, and the token verifier matches on it. Changing it invalidates tokens already issued, so clients sign in again after the change.

Step 7 — TLS (optional)​

Skip this step if you are using port-forward for local access — TLS is not needed there.

The chart can generate a self-signed certificate and store it in a Secret. The cert is created once on first install and reused on every helm upgrade (idempotent via lookup). Valid for 10 years.

warning

Self-signed certificates are not trusted by browsers or most clients without importing the CA. Use only for evaluation or air-gapped clusters. For production, use cert-manager or provide your own certificate.

Enable in values.yaml and upgrade:

tls:
  selfSigned:
    enabled: true

gateway:
  enabled: true
  hosts:
    - aicontrols.example.com   # used as the certificate CN / SAN

The chart creates a Secret named <release>-tls and automatically adds an HTTPS listener (port 443) to the Gateway.

Step 8 — Log in and complete setup​

Open the admin UI in a browser. Use the URL that matches how you exposed the service in Step 6:

Exposure methodAdmin UI URL
Port-forward (direct)http://localhost:8081/ui/
Gateway API port-forwardhttp://localhost:8080/ui/
Ingress / hostnamehttps://aicontrols.example.com/ui/

Log in with username admin and the bootstrap password retrieved in Step 4. You will be prompted to set a permanent password on first login.

After changing your password, the setup wizard launches automatically. It walks you through:

  1. LLM provider — connect Anthropic, OpenAI, AWS Bedrock, Azure OpenAI, GCP Vertex AI, or a self-hosted model (Ollama, vLLM). Your API key is stored as a Kubernetes Secret and never exposed in the UI.
  2. MCP servers — add upstream MCP servers that AIControls will proxy and govern.
  3. First policy — create a starter governance policy (allowlist, HITL trigger, or budget).

Complete the wizard before connecting agent pods — agents need at least one upstream configured to route traffic through.

Step 9 — Activate your license​

AIControls needs a signed license before it will process any AI traffic (MCP tool calls or LLM requests) — the Admin UI stays browsable, but requests are blocked with 402 until a license is loaded. helm install/helm upgrade also prints this reminder in the post-install notes if proxy.licenseFile isn't set.

On-prem licenses are bound to this cluster's kube-system namespace UID, not to a Nirmata account, so a license issued for one cluster won't activate on another. Retrieve the UID:

kubectl get namespace kube-system -o jsonpath='{.metadata.uid}'

Email that UID to support@nirmata.com to request a license. Once you receive the signed .lic file, see License management to provision it.

Step 10 — Connect agent pods​

Agent pods must send MCP and LLM traffic through AIControls instead of calling upstream servers directly.

Set MCP_SERVER_URL (and optionally OPENAI_BASE_URL for the LLM proxy) in your agent pod spec:

apiVersion: v1
kind: Pod
metadata:
  name: my-claude-agent
  namespace: agents
spec:
  serviceAccountName: my-agent-sa
  containers:
    - name: agent
      image: my-org/my-ai-agent:latest
      env:
        - name: MCP_SERVER_URL
          value: "http://aicontrols.aicontrols.svc.cluster.local:8080"
        # For OpenAI-compatible clients (Claude Code, Cursor, etc.):
        - name: OPENAI_BASE_URL
          value: "http://aicontrols.aicontrols.svc.cluster.local:8080/v1"

For OIDC identity validation, each agent pod must also present a projected ServiceAccount token:

spec:
  serviceAccountName: my-agent-sa
  containers:
    - name: agent
      env:
        - name: MCP_AUTH_TOKEN_FILE
          value: /var/run/secrets/aicontrols/token
      volumeMounts:
        - name: proxy-token
          mountPath: /var/run/secrets/aicontrols
  volumes:
    - name: proxy-token
      projected:
        sources:
          - serviceAccountToken:
              path: token
              expirationSeconds: 3600
              audience: aicontrols   # must match clusterOIDCAudience in values.yaml

Uninstalling​

helm uninstall aicontrols -n aicontrols

The bootstrap Secret (aicontrols-admin-bootstrap) and the PersistentVolumeClaim are kept after uninstall. Delete them manually if you want a clean slate:

kubectl delete secret aicontrols-admin-bootstrap -n aicontrols
kubectl delete pvc -n aicontrols -l app.kubernetes.io/name=aicontrols
kubectl delete namespace aicontrols

Startup migrations on upgrade​

A new pod applies database migrations before it starts serving, including its /healthz health check. Most migrations take seconds. A release that adds indexes to the audit log builds them at this point, and on an audit_events table with millions of rows each index can take a minute or more.

The chart gives the AIControls container a startup probe so that a slow migration isn't mistaken for a hung pod. Kubernetes doesn't run the liveness or readiness probe until /healthz answers. The pod has periodSeconds (10 seconds) times startupProbe.failureThreshold to start, which is about 5 minutes with the default of 30. If the budget runs out, Kubernetes restarts the pod, and the restarted pod repeats the unfinished migration. For a very large audit log, raise the budget before you upgrade:

startupProbe:
  failureThreshold: 48   # 48 x 10s = 8 minutes

Keep the budget well under 10 minutes. That's the Deployment's default progress deadline, which also counts the time spent scheduling the pod and pulling its image. Past it, kubectl rollout status reports the rollout as failed even though the pod is still migrating. If you run helm upgrade --wait, set --timeout to cover the whole rollout: the startup budget plus scheduling and image pull time. Helm's default wait is 5 minutes.

With the default SQLite backend, every migration runs at pod startup, and audit-event writes wait until it finishes. There's no online pre-build for SQLite, so size startupProbe.failureThreshold for the largest migration you expect, and upgrade at a quiet time. On PostgreSQL, you can build the new session indexes online before the upgrade instead; see Pre-build session indexes before upgrading.

Analytics execution deadlines​

Dashboard JSON reads default to a 30-second execution deadline, including database connection waits. To override it, add this setting to your Helm values and apply it with your normal Helm upgrade:

analytics:
  timeout: "20s"

The value must be a positive duration below the 60-second admin response write deadline. Application deployments can also set AICONTROLS_ANALYTICS_TIMEOUT. An earlier caller deadline takes precedence. Configure ingress timeouts above the analytics deadline with time for cancellation and response delivery. An ingress timeout alone does not cancel application database work.

See analytics API response behavior for timeout and unavailable responses. Increasing the deadline allows expensive queries to occupy resources for longer; inspect query and resource metrics first.

Bound dashboard database work​

Dashboard analytics share a bounded execution budget per replica. The defaults limit analytics to two simultaneous database operations. They also have room for 33 concurrent analytics reads, more than the reads of several dashboard pages sent at once. In testing, such a burst completed without 503s while some of its reads took 10 to 20 seconds. Requests beyond the limits receive HTTP 503 with Retry-After rather than growing an unlimited queue.

These Helm values show the defaults:

analytics:
  timeout: "30s"
  execution:
    maxQueries: 2
    maxActive: 8
    maxQueued: 32
    maxWaiters: 40
    # queueTimeout is unset: half of timeout (15s with the 30s default)
    cacheTTL: "2s"
    cacheEntries: 64
    cacheBytes: 4194304
    maxResponseBytes: 4194304

The response limit applies to buffered analytics JSON; exceeding it returns an explicit unavailable response. Cache limits cover retained keys and response bodies. Only stats, cost summary and savings summary retain completed responses; those views may lag writes by up to the cache TTL. Set cacheTTL: "0s" to disable retention while still sharing identical in-flight reads. Authentication and authorization run for every caller. Results are never shared across credentials or authorization scopes.

SettingDefaultWhat it limits
maxQueries2Database operations that analytics can run at once. Capped at the database pool size minus two, but never below one (the SQLite pool has 8 connections; the PostgreSQL pool defaults to 16). With a pool of one or two connections, analytics can use the last free connection.
maxActive8Computations that run at once. A running computation still waits for a maxQueries permit before each database operation.
maxQueued32Computations waiting to start. When maxActive and maxQueued are both full, new requests get an immediate 503.
maxWaiters40Concurrent analytics HTTP callers, including cache hits and response writing. Callers beyond this get an immediate 503. Must be at least maxActive; keep it at or above maxActive + maxQueued so every queued computation has a caller slot. Each caller can hold a copy of a response of up to maxResponseBytes. See memory sizing.
queueTimeouthalf of timeoutHow long a computation can wait to start before it gets a 503. The timeout deadline starts when the request arrives and keeps running while the request is queued; if it expires first, the request gets a 504 instead. A value equal to or longer than timeout is replaced with half of timeout.

Tune the limits​

Size the limits to the number of analytics reads that one dashboard load sends at once; the browser sends all of them together. Keep maxActive + maxQueued and maxWaiters at or above that number, so a page's requests are queued instead of rejected on arrival. A queued read still gets a 503 if it can't start within queueTimeout, or a 504 if the request deadline expires first, for example when every active computation is a read that runs longer than the queue wait. Keep queueTimeout long enough for queued reads to start behind the slowest reads, and short enough to leave time for them to run.

Use the aicontrols_analytics_* Prometheus metrics to decide what to change:

SymptomMetricsChange
Fast 503s during a single dashboard loadaicontrols_analytics_responses_total{outcome="unavailable"} rises quickly, with aicontrols_analytics_state{resource="queued"} at maxQueued, or with no executor rejections because maxWaiters callers are still running or writing responsesRaise maxQueued and maxWaiters to cover the page's concurrent reads. The waiters state counts callers waiting for a result and can be lower than the number still writing responses.
503s after about queueTimeoutaicontrols_analytics_events_total{resource="rejected"} rises; aicontrols_analytics_wait_seconds_total{resource="admission"} grows quicklyReads are queued behind slow reads. Raise maxActive or queueTimeout, or fix the slow reads.
504saicontrols_analytics_responses_total{outcome="timeout"} rises; aicontrols_analytics_wait_seconds_total{resource="query_admission"} grows quicklyReads are waiting for database permits or running too long. Check query duration and CPU throttling before raising maxQueries or timeout.
Gateway latency rises with dashboard useaicontrols_analytics_state{resource="db_pool_wait_count"} and db_pool_wait_seconds riseAnalytics is competing for database connections. Lower maxQueries, or raise the PostgreSQL pool size.

Memory sizing​

Analytics buffers each JSON response before sending it. A computation keeps a response buffer and a result copy, and the result can stay in memory while each caller makes its own copy. A conservative upper bound for buffered responses is 2 x (maxWaiters + maxActive) x maxResponseBytes + cacheBytes: about 388 MiB with the defaults, under the chart's default 512 MiB memory limit. That bound needs every response to be near the 4 MiB limit, and it doesn't count temporary allocations while a handler builds and encodes a response. Most dashboard responses are well under 4 MiB, so typical use is a small fraction of it. On replicas with a low memory limit, lower maxResponseBytes or maxWaiters so the worst case fits with room for the gateway.

On SQLite, raising maxQueries above 2 made a seeded burst of dashboard reads slower and caused 504s, because reads compete for the same CPU and file. Raise it only on PostgreSQL, and only after you measure the change. A second dashboard load that arrives while the first is still running can exceed the limits and receive 503s. That is the intended overload protection.

Execution limits exclude LLM/MCP traffic, authoritative budget checks, streaming, exports and backups. They preserve database connection headroom where possible, but do not reserve CPU, memory or disk bandwidth. Background jobs and other replicas still compete for shared resources. Increasing concurrency can worsen latency; compare API errors, database pool waits and gateway latency before and after changing these settings.

The health API exposes connections_in_use, connections_idle, wait_count and wait_duration_ms. Open connections include idle connections. Pool wait counts and duration are cumulative; compare their changes over the same time interval. These health fields share a storage snapshot cached for up to one minute; faster polling does not produce a fresh sample. The analytics Prometheus pool metrics are sampled once per second for shorter spikes, subject to the scrape interval. The separate readiness ping can remain fast while application queries wait.

Prometheus metrics prefixed aicontrols_analytics_ report active/queued work, query admission waits, cache/coalescing events, and response outcomes. A fast 503 shows overload protection working; it does not mean the underlying performance problem is resolved.

Next steps​

  • Agent Identity — assign each agent a unique ServiceAccount identity and configure dual identity for user-delegated agents
  • Write a Policy — create CEL rules for your governance requirements
  • Set a Budget — configure per-agent token and cost limits
  • Configuration Model — how deployment settings (Helm) and app settings (Settings UI) stay in sync across upgrades
Last updated on