Monitor system health

Task

Use the Health page to check whether AIControls and everything it depends on is working, and to find out why when it isn't. For the status rules behind it, see Health Status Reference.

Open the Health page

Go to Setup → Health in the admin UI. The same rolled-up status this page shows also appears on the public status page and in the developer portal's status indicator, so anything you fix here clears everywhere else too.

The page refreshes itself every 15 seconds. The line under the title shows how old the current snapshot is.

Read the banner

The banner at the top of the page leads with the status and a count of open issues and warnings, for example Degraded: 1 issue · 2 warnings. When nothing is open it reads All systems operational.

Status	Meaning
Operational	No blocking issue is open. Warnings may still be listed.
Degraded	AIControls is still serving traffic, but at least one component needs attention.
Down	A required component has failed and AIControls cannot serve traffic normally.
Unknown	The admin UI couldn't load the status. An error message with a Retry button appears above the banner.

Below the headline, each open issue is listed with its severity, its message, and the time it started (since 14:02). The right side of the banner shows the proxy version, mode, uptime, and replica readiness: Replicas shows how many of the replicas that recorded metrics in the last 3 minutes, fleet-wide, are ready, for example 2 / 3 ready. A single-instance deployment reports a fleet of one (1 / 1 ready) in the same format.

For which conditions raise which severity, see Status rules.

Work through the tabs

Each tab lists the issues that belong to it, then its own detail below. The tab label shows a count of its blocking issues (orange, or red when one is down) and a separate gray count of its warnings. A tab with nothing open shows a green OK.

Tab	What it shows
Upstreams	Issues for LLM upstreams, then the Upstreams tab: summary tiles, model routes with two or more providers under Failover, every provider with the routes it serves, and a failover-events view.
MCP	Issues for MCP servers, then the MCP tab: summary tiles and a searchable table of every configured MCP server.
Providers	The Providers tab: one card per LLM provider, combining its public status page (if tracked) with the traffic AIControls observed across the upstreams that use it. Raises no issues of its own.
Gateway	Issues from the proxy's request path: the policy engine, policy compilation, the audit pipeline, gateway error rate, and gateway overhead.
Services	Issues from storage, readiness, services, collectors and background jobs, then the Components, Notification delivery and Background jobs tables — see Services tab.
Productivity	Administrator-only assessment job completions, failures, queue history, daily reserved/charged balance, and bounded retries — see Productivity tab.
Runtime	Issues from Kubernetes runtime monitoring. Only shown when Runtime Controls is enabled for your workspace — see Enable Runtime Controls.

Click a column header in the Upstreams, MCP, or Services tab's tables to sort by that column. If a table can't load its data, or the snapshot's own check for it timed out, it shows Could not load upstream status. (or Could not load service status. for the whole Services tab) with a Retry button in place of its content. Use Refresh to try again.

The selected tab is kept in the page URL (?tab=upstreams, ?tab=mcp, ?tab=providers, ?tab=gateway, ?tab=services, ?tab=runtime, or ?tab=productivity), so a link you share opens to the same tab.

Productivity tab

Administrators with PR attribution enabled can open Health → Productivity. The tab polls every 30 seconds; durable queue and budget samples are recorded once per minute and retained for seven days. The Last 1 hour view shows minute buckets; Last 24 hours shows hourly outcomes and the last queue/budget sample in each hour.

Completed and Failed count terminal assessment-job settlements in the selected execution-time window. Existing assessments retained by a job count as completions. A later explicit retry can produce another outcome for the same PR. The failure percentage is failed settlements divided by completed plus failed settlements; it is not a model-request error rate or a count of distinct developers. Historical periods before this version was deployed, and missed sampler intervals, are shown as unavailable rather than invented zeroes. Partial hourly totals contain only recorded outcomes.

The current snapshot shows pending/running jobs, retained terminal totals, the last completed job, and the oldest scheduled-ready wait. Schedule-ready jobs can still be blocked by repository eligibility, configuration, capacity, or budget. Their wait starts at their scheduled availability, not their original PR merge date.

The cost chart is the daily reserved / charged balance, including conservative reservations. It is not actual provider-invoiced spending. Each UTC day has its own balance: a drop at midnight UTC is a daily reset, not a refund. Use Assessment settings to inspect or change the daily budget and automatic assessment configuration.

To retry a bounded group after correcting the cause:

1. Select the failure category and a maximum of 1–100 jobs (default 25).
2. Choose Retry up to … failed jobs and review the confirmation. New model calls can incur additional cost.
3. Choose Queue retries. The operation applies to retained eligible failures in the selected category; the chart's time window does not narrow its scope. Counts update at the next sample.

An interrupted worker may already have dispatched a paid request. Inspect that PR's request history before retrying it. The tab keeps detailed productivity information private to administrators; it does not publish these counters or costs on the public status page.

Services tab

The Services tab has three sub-tabs, all backed by GET /api/v1/health/services: Components, Notification delivery, and Background jobs. Each sub-tab's label carries a badge with its own problem count (configured components that are degraded or down, channels whose latest delivery failed, and jobs that are failing or stale).

• Components lists every service AIControls depends on — storage, policy enforcement, policy sources, the OIDC issuer, Okta sync, PromptGuard, the LLM Judge, the PolicyReport sink, OpenTelemetry export, workload discovery, runtime posture and session intelligence — with its status, whether it's required, and a short detail message. A component configured elsewhere links to that page from its Name cell. The table only shows configured components by default; when some aren't configured, a Show N not configured link reveals them, and the Required column reads "—" for a component that isn't configured, since requiredness doesn't mean anything until it's set up.
• Notification delivery lists each channel and kind that delivered a notification in the last 24 hours, with counts sent and failed.
• Background jobs lists every scheduled job, its status, which replica last ran it, and when it last and next runs. Sort by Last run to find stale jobs, or filter Status to just Failing or Stale.

Click a row in any of the three tables to open its details in a side panel. For Notification delivery and Background jobs, when a row's last attempt failed, the panel shows the last error class (e.g. HTTP 429, timeout) in a red alert — never the raw upstream error text, which the server redacts before it reaches the browser. The table itself never shows this error class either, so a long one doesn't push other columns around.

If notification records or job runs can't be read, the affected table is replaced with the error instead of showing an empty list, and a failed job-runs read also raises a warning issue on the tab.

For the full field reference, including the list of components and every status value, see GET /api/v1/health/services.

Read the Gateway tab

The Gateway tab covers the proxy's own request path — everything AIControls does to a request between accepting it and forwarding it upstream. It uses the time window selector.

Five tiles lead the tab:

Tile	Meaning
Active sessions	Sessions with a request in the last few minutes.
Requests/min	The current request rate, fleet-wide.
Tokens/min	The current token throughput, fleet-wide. Shows — when there's no traffic.
Overhead p95	The 95th-percentile time AIControls itself adds on top of the upstream call (identity, content scanning, policy evaluation, forwarding). Colored red above 250 ms, the same threshold that raises a gateway:overhead_p95 issue — see Status rules.
Gateway error rate	The share of requests in the window that AIControls itself failed with a gateway 5xx or a policy-evaluation error (an upstream 5xx passed through is not counted here — see the Upstreams tab). Every LLM gateway response counts as a request, including requests rejected before they were audited. Colored red above 5%, the same threshold that raises a gateway:error_rate issue. Its subtext breaks the count down into gateway 5xxs, policy errors, scan errors, and fail-open requests.

Click any tile to chart that metric below — the chart card's title is itself a dropdown with the same five choices: Active sessions, Request throughput (requests/min), Token throughput (tokens/min), Latency (overhead — avg, min, max, p95), and Error rate. The latency choice shows a note that its min and max are estimated from latency histogram buckets, not exact. Request and token throughput draw a dashed line marking the window's average; latency and error rate draw a dashed line at their alert threshold (250 ms and 5%).

Below the chart:

• Policy engine shows how many policies are enabled, loaded, and stored, evaluation P95 and error count, when policies were last reloaded, and their sources. Any policy that failed to load is listed by name.
• Audit pipeline shows the write and retry backlog, events dropped in the last 15 minutes and in the window, write-lag P95, how many audit events are retained, and the most recent write failure, if any.
• Approvals (HITL) shows the oldest pending human-in-the-loop approval, the configured timeout, and how many expired in the last 24 hours.
• Storage shows the audit database's driver, ping time, size, and open/max connections.
• Process (per replica) lists every reporting replica with its readiness, database ping, memory, goroutine count, GC pause P99, decision-cache hit rate, whether it holds the sampler lease, and when it last reported in.

Read the Runtime tab

The Runtime tab adds three tiles above its issues — Sensors alive (nodes reporting a live sensor out of the fleet), Ingest lag (the p95 shortfall across nodes once at least two report one, otherwise the fleet total), and RBAC gaps (permissions the proxy's ServiceAccount is missing) — followed by a summary of per-node coverage, an install prompt when the runtime sensor is missing, and a table of every node's sensor and open/exec hook state. This tab is the single home of runtime coverage. For workloads that reach providers outside the gateway, see the Gateway bypass card on Inventory → Workloads. It only renders once the runtime controller has completed its own capability probe; until then, or if runtime observation isn't active on the cluster, the tab shows the reason instead. For the full field reference behind these tiles, see Runtime Controls Reference.

The Upstreams tab

The Upstreams tab reads GET /api/v1/health/upstreams for whichever time window is selected, and refreshes itself every 15 seconds. It covers LLM upstreams only — MCP servers have their own MCP tab.

Five tiles across the top summarize the window: how many critical upstreams are healthy (with how many LLM upstreams and routes are configured), LLM requests, the overall error rate, provider P95/TTFT latency, and failovers in the last 24 hours (with the most recent one's time and route, or none).

Below the tiles, a Routes / Failover events switch picks the view:

• Routes shows, under Failover (Routes with two or more providers), a card for each model route with two or more members: its name, rolled-up status, and strategy (for example Priority failover (sticky sessions) or Lowest latency (sticky sessions)), with route-level stats — active member (Any member (…) for a strategy other than failover), healthy/total members, requests, error rate, P95 latency, and failovers in the last 24 hours — and an expandable table of its members. Below them, the Providers card lists every LLM provider, with a Routes column naming the routes it serves. A search box filters the rows by name. The Providers table and each route's member table use the same row shape:
  • Name (or Member) and reachability (a colored status icon next to the name), plus whether a route member is active, primary, standby, or fallback-only.
  • Type — LLM · proxy or LLM · passthrough. A passthrough upstream isn't actively probed; its status is inherited from its provider and shown as "via <provider> provider".
  • Criticality — a dropdown you can change inline; see Set an upstream's criticality below.
  • Probe — P50/P99 probe latency and probe error rate; Passthrough — uses provider health for a passthrough upstream; Local — not probed for a local (stdio) upstream; otherwise Not probed.
  • Traffic — request count, error rate, and P95 latency observed from real traffic in the selected window.
  • Availability (24h) — a bar of 48 half-hour segments covering the last 24 hours, colored green (ok), yellow (degraded), red (down), or gray (no probes ran in that slot). Hover a segment for its exact time and state.
  • Issue — the open issue for that upstream, if any.
• Failover events lists failovers from the last 24 hours, aggregated per minute across all replicas (up to 200 rows, newest first): time, Route, from → to, a count badge when the same failover with the same reason repeated in a minute, and the reason when known.

Click a column header to sort a table by that column. If the tab can't load upstream data, it shows Could not load upstream status. instead of an empty table. If the tab already had data loaded and a later refresh (the 15-second auto-refresh, a criticality change, or switching the time window) fails, it keeps showing that last snapshot with a Data may be stale. warning and a Retry button, rather than silently going quiet about the failure.

Click any row to open a drawer with that upstream's full detail, including its probe and traffic figures, its criticality control, and the same 24-hour probe history as a larger bar and a chart.

The MCP tab

The MCP tab covers MCP servers — the same GET /api/v1/health/upstreams response, filtered to type: "mcp" — for whichever time window is selected.

Four tiles summarize the window: how many critical MCP servers are healthy (with how many servers are configured), tool calls, the error rate (with a count of failed calls), and the slowest server's P95 latency. Below the tiles, a search box filters the server table by name; the table uses the same row shape as the Upstreams tab (Name, Type, Criticality, Probe, Traffic, Availability (24h), Issue). Every configured MCP server is listed, including local (stdio) servers and servers with no traffic yet. Click a row to open the same detail drawer. If the tab can't load server data, it shows Could not load MCP server status. with a Retry button.

The Providers tab

The Providers tab is per-provider health: each LLM provider's public status page, where one is tracked, combined with the traffic AIControls observed across every upstream that uses it. It raises no issues of its own.

Three tiles summarize the window: how many tracked status pages are operational, how many have an open incident, and the slowest provider's P95 latency. Below the tiles, one card per provider shows its status page (a tag with the page's status, or Not tracked), requests, error rate, P95 latency, and which upstreams use it. A tooltip on the status-page detail shows when it last changed.

Set an upstream's criticality

Every upstream is critical by default, so an outage on a fallback or experimental upstream can make AIControls look degraded. Lower its criticality so it reports a warning instead, or nothing at all. Setting criticality requires the admin role — the control is disabled for viewers.

1. Find the upstream's row in the Upstreams tab (in the Providers table or inside a route) or the MCP tab's table, or open it from the row drawer.

2. Use the Criticality dropdown to choose Critical, Non-critical, or Ignored. The change is saved immediately (PUT /api/v1/health/upstreams/{name}/criticality); the upstream's issue now shows as a warning (non-critical) or is gone (ignored), and the banner status is recalculated. Other replicas pick up the change within a minute.

   If the name isn't a configured or probed upstream — for example, one seen only in traffic (type: unknown) — the request is rejected and an error toast explains why.

You can also set criticality directly through the API, which is useful for scripting:

curl -X PUT "$YOUR_WORKSPACE_URL/api/v1/health/upstreams/openai-fallback/criticality" \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"criticality": "non_critical"}'

To make the upstream count again, send {"criticality": "critical"}. For what each level does, see Upstream criticality.

Choose a time window

The Last 1 hour / Last 24 hours selector next to the Refresh button is kept in the page URL (?window=1h or ?window=24h), so a link you share opens to the same window.

Refresh on demand

Click Refresh to re-fetch the latest snapshot instead of waiting for the next automatic refresh.

Enable the public status page

The public status page at /status shows the rolled-up status, uptime and, optionally, upstream reachability to anyone who can reach your workspace, without signing in. It never shows issues, the version, or mode. See GET /api/v1/status for exactly which fields public callers, developers, and admins or viewers each receive.

It's off by default. Turn it on in either of two ways:

• In your AIControls configuration file:

  ui:
    publicViews:
      status:
        enabled: true

• Through the admin API:

  curl -X PUT "$YOUR_WORKSPACE_URL/api/v1/ui/settings" \
    -H "Authorization: Bearer $ADMIN_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{"publicStatusEnabled": true}'

Then open $YOUR_WORKSPACE_URL/status. To hide the upstream table from public callers, set ui.publicViews.status.showUpstreams: false in the configuration file, or send {"publicStatusShowUpstreams": false} to the same endpoint. A value saved through the API overrides the file.

While the page is disabled, /api/v1/status returns 404 Not Found to callers who aren't signed in.

Inspect PR assessment failures

Administrators can use Health → Productivity to compare terminal failure categories over the selected hour or day and review the affected PRs.

1. Review Automatic job failures by category to find the error type and interval.
2. Select a category count to review the whole window, or choose a Failure interval and category, then select Review failed PRs.
3. Optionally filter by an exact repository name, such as owner/repo. Select a PR to inspect its evidence and request history.

If the selected interval exceeds the failure list’s paging limit, a warning shows how many retained events are accessible. Narrow the repository filter or choose a shorter chart interval to inspect older events. The paging limit does not reduce the chart totals.

The list records terminal automatic-job attempts. The same PR can appear more than once after retries, and a later successful assessment does not remove its earlier events. It does not include every provider error or manual analysis request. Category totals and charts cover all repositories; the repository filter narrows only the event list.

Individual events are retained for seven days and start when event capture becomes available. The view shows this boundary; older aggregate counters cannot identify individual PRs. Deleting a source PR also removes its retained failure events. Missing sampler intervals remain gaps, and partial hourly totals contain only recorded outcomes. The current minute is marked ongoing: its counters and PR list can change until the displayed interval ends. Closed intervals use an inclusive start and exclusive end. Safe diagnostic codes and available model, prompt, and source-selector versions help compare attempts without exposing private prompts or model output. The queue version identifies scheduling history and is separate from the classification prompt version.

Reviewing events does not retry work. The separate retry action applies to eligible current failures in the selected category, independently of the chart interval, and can incur additional provider cost.

Inspect manual operation failures

1. Open Health → Productivity → Manual assessment operation failures.
2. Choose an operation stage, safe cause, or exact repository name. These filters apply to the hourly chart, recorded-failure and distinct-PR totals, and associated PRs.
3. Select the whole Health window or an hourly interval, then choose Review associated PRs.
4. Open an event to inspect its correlation ID, analysis request ID, model dispatch state, suggested retry time, and available GitHub HTTP status/request ID. Inspect the PR evidence and request history before deciding to try another operation.

This panel records manual source retrieval and analysis saving/accounting failures. Model/provider and output-validation failures remain in request history. A source failure marked Model request not dispatched made no model call. Other dispatch states alone do not establish provider billing or a saved result. These are recorded failures without a successful-attempt denominator; zero recorded failures does not prove healthy coverage. Events retain seven days of history, and the first observed failure does not establish complete instrumentation coverage. Reviewing this panel does not queue a retry.

When an assessment alert says its failure could not be saved to Health history, retain its correlation ID and available GitHub request metadata for troubleshooting in server logs.