Health Status Reference

Reference

Lookup material for the health status that one evaluator produces and every health surface shares: the status values and the rules behind them, the issue fields, the source-prefix vocabulary, and the endpoints that return them. For how to read this on the Health page, see Monitor system health.

Status values

One evaluator computes a single rolled-up status. It's the same value shown on the Health page banner, the public status page, and the developer portal's status indicator.

Status	Banner label	Meaning
ok	Operational	No down or degraded issue is open. Warnings may still be listed.
degraded	Degraded	AIControls is still serving traffic, but at least one degraded issue is open.
down	Down	At least one down issue is open. AIControls cannot serve traffic normally.

When the admin UI can't load a status at all, it shows Unknown instead of guessing. That value is never returned by the API.

Status rules

Each rule below raises one issue with the listed severity. The rolled-up status is the worst severity among open issues; warning issues never change it.

Severity	Raised when	Source
down	The replica isn't ready to serve traffic.	readiness
down	The storage backend returns an error.	storage
down	The policy engine failed to load policies.	policy_engine
down	Every critical upstream is unreachable.	upstream:<name>
degraded	A critical upstream is unreachable or degraded (while at least one other critical upstream still responds).	upstream:<name>
degraded	A required service is unhealthy.	service:<key>
degraded	The audit pipeline dropped events.	audit:dropped
degraded	The gateway error rate is above the threshold.	gateway:error_rate
degraded	The gateway's P95 overhead is above the threshold.	gateway:overhead_p95
degraded	A background job is failing.	job:<name>
degraded	Gateway requests were refused in the last 15 minutes because of a setup problem: an upstream with no credential, a model the routed upstream can't serve, or an ambiguous or unsupported upstream.	gateway:config:<reason>
warning	Callers sent a credential the gateway refused: passthrough mode is disabled, or the upstream doesn't accept it.	gateway:config:<reason>
warning	A non-critical upstream is unreachable or degraded.	upstream:<name>
warning	An optional service is unhealthy.	service:<key>
warning	One or more policies failed to compile.	policy:compile
warning	A background job is stale.	job:<name>

Upstreams marked as ignored are never evaluated.

Upstream criticality

Each LLM or MCP upstream has a criticality that sets how its problems count toward the rolled-up status. Every upstream is critical until an admin changes it — see Mark an upstream non-critical or ignored.

Criticality	When the upstream is unreachable or degraded	Counted in critical_total
critical	Raises a degraded issue, or a down issue when every critical upstream is unreachable.	Yes
non_critical	Raises a warning issue. The rolled-up status doesn't change.	No
ignored	Raises no issue. The upstream is still listed with its probe and traffic data.	No

Provider status pages are informational and have no criticality.

The Issue shape

Each problem the evaluator finds is one issue:

Field	Type	Meaning
severity	"down" | "degraded" | "warning"	How this issue affects the rolled-up status — see Status rules.
source	string	Which component raised the issue — see Source prefixes.
message	string	Human-readable description of what's wrong.
since	timestamp	When the replica that answered the request first saw the issue. Each replica tracks this on its own, so two replicas can report different since values for the same issue, and a replica restart resets it.

Source prefixes

The source field determines which Health page tab surfaces an issue:

Source	Tab	Examples
upstream:<name>	MCP, when <name> is a configured MCP server; Upstreams otherwise	upstream:openai-prod (Upstreams), upstream:github-mcp (MCP)
storage, readiness, service:<key>, collector:<what>, job:<name>	Services	storage, service:okta, job:discovery-sync
runtime:*	Runtime	runtime:sensor
everything else	Gateway	policy_engine, policy:compile, audit:dropped, gateway:error_rate, gateway:overhead_p95

The Providers tab shows no issues of its own — provider status pages are informational, and a provider's upstream-facing problems surface as upstream:<name> issues on the Upstreams or MCP tab instead.

Thresholds

Metric	Threshold	Summary field
Gateway error rate	Above 5%	thresholds.error_rate_pct
Gateway P95 proxy overhead	Above 250 ms	thresholds.overhead_p95_ms

Both thresholds, and the audit:dropped rule, are evaluated over the last 15 minutes of traffic across every replica. With no traffic in that window, neither threshold raises an issue. See Overhead and end-to-end latency and Gateway errors for what each metric counts.

Endpoints

Endpoint	Auth	Backs
GET /api/v1/health/summary	Any admin UI role, including viewer	The Health page.
GET /api/v1/health/services	Any admin UI role, including viewer	The Health page's Services tab. See below.
GET /api/v1/health/productivity	Admin	Productivity processing history.
GET /api/v1/health/productivity/failures	Admin	PR failure history for a selected interval.
GET /api/v1/health/productivity/operations	Admin	Manual source and finalization failures, with hourly counts and PR drilldown.
GET /api/v1/health/gateway	Any admin UI role, including viewer	The Gateway tab of the Health page.
GET /api/v1/health/upstreams	Any admin UI role, including viewer	The Health page's Upstreams, MCP and Providers tabs. The MCP tab filters upstreams[] to type: "mcp"; the Providers tab reads providers[]. See below.
PUT /api/v1/health/upstreams/{name}/criticality	Admin	Setting an upstream's criticality.
GET /api/v1/status	Signed in, or public when the public status page is enabled	The public status page (/status) and the developer portal's status indicator. Fields vary by caller — see below.

GET /api/v1/health/summary

curl "$YOUR_WORKSPACE_URL/api/v1/health/summary" \
  -H "Authorization: Bearer $ADMIN_TOKEN"

Field	Meaning
status	The rolled-up status.
generated_at	When the replica built this snapshot.
snapshot_age_seconds	How old the snapshot was when it was served.
issues	Open issues, sorted by severity then source. Always an array; empty when nothing is wrong.
components[]	One row per checked component: name, healthy, status (healthy, degraded, down, or unknown), required, and message. Storage, Readiness and Policy cache are required. A failing component's message is a fixed description; the underlying error, which can include hostnames or URLs, is written to the proxy logs instead.
proxy	version, mode, uptime_seconds, started_at.
replicas	reporting is the number of replicas that recorded metrics in the last 3 minutes; ready is how many of those were ready. Before any replica has recorded metrics (for example, right after the first start), both counts describe the replica that answered the request.
thresholds	overhead_p95_ms and error_rate_pct — see Thresholds.

GET /api/v1/health/gateway

curl "$YOUR_WORKSPACE_URL/api/v1/health/gateway?window=24h" \
  -H "Authorization: Bearer $ADMIN_TOKEN"

window is 1h (the default) or 24h; any other value returns 400 Bad Request. Counts and latencies cover every replica that recorded metrics in the window. Latency percentiles are computed from the combined distribution of all replicas, not averaged per replica. The endpoint returns 503 Service Unavailable when storage is disabled.

Field	Meaning
window	The window the response covers: 1h or 24h.
generated_at	When the response was built.
bucket_seconds	Width of each throughput.series point: 60 for 1h, 300 for 24h.
sessions.active_now	Developer sessions that are currently active.
sessions.unique_1h, sessions.unique_24h	Developer sessions with activity in the last hour and the last 24 hours.
throughput.rpm_current	Requests per minute across all replicas in the most recent recorded minute. 0 when no replica recorded a minute in the last 3 minutes.
throughput.avg, min, max, p95, p99	Requests per minute across all replicas, over the per-minute values in the window.
throughput.tokens_per_min_current	Tokens per minute across all replicas in the most recent recorded minute. 0 when no replica recorded a minute in the last 3 minutes.
throughput.tokens_per_min_avg	Tokens per minute across all replicas, averaged over the per-minute values in the window.
throughput.series[]	One point per bucket: ts, total_rpm, allowed_rpm, denied_rpm, escalated_rpm (averaged over the bucket's minutes), error_count (summed over the bucket), active_sessions, tokens_per_min, error_rate_pct, and overhead_avg_ms/overhead_min_ms/overhead_max_ms/overhead_p95_ms (overhead_min_ms and overhead_max_ms are estimated from latency histogram buckets, not exact). These back the Gateway tab's chart, whose title dropdown switches between active sessions, request throughput, token throughput, overhead latency, and error rate.
latency.overhead	avg_ms, p95_ms, p99_ms of the time AIControls adds to an LLM request — see Overhead and end-to-end latency.
latency.end_to_end	avg_ms, p95_ms, p99_ms of the full request time, including the upstream provider.
latency.mcp	avg_ms, p95_ms, p99_ms of MCP tool calls.
latency.stages	P95 time spent in each request stage: identity_ms, content_scan_ms, policy_eval_ms, forward_ms.
errors.gateway_5xx	Gateway errors in the window.
errors.policy_eval_errors	Policy evaluations that returned an error.
errors.content_scan_errors	Content scans that returned an error.
errors.fail_open	Requests allowed because a check failed and was configured to fail open.
errors.error_rate_pct	Gateway errors plus policy evaluation errors, as a percentage of all requests in the window. See Gateway errors for which requests count. 0 when there were no requests. The gateway:error_rate rule uses the same calculation over the last 15 minutes.
policy_engine.enabled	Enabled policies.
policy_engine.loaded	Policies compiled and loaded into the policy engine.
policy_engine.failed[]	Names of policies that failed to compile.
policy_engine.stored	Policies in storage, enabled or not.
policy_engine.last_reload_at	When the policy engine last reloaded policies. null before the first reload.
policy_engine.eval_p95_ms	P95 policy evaluation time. Same value as latency.stages.policy_eval_ms.
policy_engine.eval_errors	Same value as errors.policy_eval_errors.
policy_engine.sources[]	Where enabled policies come from, for example local or kubernetes.
audit.write_backlog	Audit events waiting to be written, summed over each replica's latest value.
audit.retry_backlog	Audit events waiting to be retried after a storage error, summed over each replica's latest value.
audit.dropped_window	Audit events dropped in the window.
audit.dropped_15m	Audit events dropped in the last 15 minutes. The audit:dropped rule uses this count.
audit.write_lag_p95_ms	P95 time between an event happening and its audit record being stored, over the last 5 to 10 minutes, on the replica that answered.
audit.last_error, audit.last_error_at	Whether the replica that answered has had an audit write error, and when the most recent one happened. last_error is a fixed message; the full error is in that replica's proxy logs. Empty and null when there has been none since that replica started.
audit.events_retained	Audit events in storage. On PostgreSQL this is the database's row estimate. Refreshed at most once a minute.
audit.retention_days	How many days audit events are kept. 0 means events are kept indefinitely.
hitl.oldest_pending_seconds	Age of the oldest approval request still waiting for a decision. 0 when none are pending.
hitl.timeout_minutes	The default approval timeout.
hitl.expired_24h	Approval requests that expired without a decision in the last 24 hours.
storage.driver	sqlite or postgres.
storage.ping_ms	Round-trip time of a storage ping from the replica that answered. -1 when the ping failed.
storage.size_bytes	Database size. -1 when it could not be read. Refreshed at most once a minute.
storage.connections_open, storage.connections_max	Open connections in the answering replica's pool, and the pool limit (0 means no limit). Refreshed at most once a minute.
storage.connections_in_use, storage.connections_idle	Connections currently in use or idle in the answering replica's pool at the cached sample time. Refreshed at most once a minute.
storage.wait_count	Cumulative number of waits for a pool connection since the answering replica's pool was created. Refreshed at most once a minute.
storage.wait_duration_ms	Cumulative time spent waiting for pool connections, in milliseconds, since the answering replica's pool was created. Refreshed at most once a minute.
replicas[]	One row per replica that recorded metrics in the last 3 minutes: id, ready, db_ping_ms, memory_bytes, goroutines, gc_pause_p99_ms, decision_cache_hit_rate (null before any lookup), sampler_lease (true on the replica that runs fleet-wide sampling), and last_seen.

Pool statistics share the storage snapshot cache and can lag by up to one minute; frequent health polling does not refresh that cache sooner. Compare cumulative counters across distinct samples from the same replica. For shorter contention spikes, use the analytics Prometheus pool metrics, sampled once per second per replica (subject to your monitoring system's scrape interval).

Overhead and end-to-end latency

• Overhead is the time AIControls itself adds to a request: identity, content scanning, policy evaluation and request handling. It excludes the time spent waiting for the upstream provider, so a slow model does not raise it. The gateway:overhead_p95 rule uses overhead.
• End-to-end is the full time from receiving the request to finishing the response, including the upstream provider. It shows what developers experience, but it moves with provider latency, so no health rule is based on it.

Gateway errors

A gateway error is a 5xx response that AIControls produced itself, such as an internal error while handling the request. A 5xx that the upstream provider returned and AIControls passed through is not a gateway error, so a provider outage does not raise gateway:error_rate; it shows up on the Upstreams tab instead.

The error rate divides these errors by every response the LLM gateway served plus every MCP request. Each LLM gateway response counts once, whether or not it produced an audit event, so a request that AIControls rejects with a 5xx before it is audited still counts toward the rate. When every request fails this way, the rate is 100% and gateway:error_rate is raised. In a window with no LLM gateway responses, the rate uses the audited request count instead.

GET /api/v1/health/upstreams

curl "$YOUR_WORKSPACE_URL/api/v1/health/upstreams?window=1h" \
  -H "Authorization: Bearer $ADMIN_TOKEN"

window is 1h (the default) or 24h and sets the period for the summary and traffic fields. Any other value returns 400 Bad Request. Traffic, request counts, failovers and availability_24h are totals across every replica. probe, status and the route members' live status come from the replica that answers the request.

Field	Meaning
window	The window applied: 1h or 24h.
generated_at	When this response was built.
summary.critical_total	Number of probed upstreams whose criticality is critical. Upstreams of type unknown have no live status, so they aren't counted.
summary.critical_healthy	How many of those are reachable.
summary.requests	Requests AIControls handled in the window.
summary.llm_requests, summary.mcp_requests	The LLM and MCP share of requests.
summary.error_rate_pct	Upstream errors as a percentage of upstream requests in the window.
summary.provider_latency_p95_ms	P95 upstream response time across all upstreams, in milliseconds.
summary.ttft_p95_ms	P95 time to first token for streamed LLM responses, in milliseconds.
upstreams[]	One row per upstream, described below. The Upstreams tab shows type: "llm" rows (in its Providers table and under each route); the MCP tab shows type: "mcp" rows.
routes[]	One row per route the router uses, including implicit routes, described below. The Upstreams tab shows routes with two or more members under Failover.
status_pages[]	One row per provider status page, described below.
providers[]	One row per LLM provider, described below. Backs the Providers tab.
failover_events[]	Failovers in the last 24 hours, aggregated per minute across all replicas, newest first, at most 200 rows. Described below. Backs the Upstreams tab's Failover events view.

Each upstreams[] row:

Field	Meaning
name	The upstream name.
type	llm, mcp, or unknown for a name seen only in traffic that isn't a configured upstream, MCP server, or known provider. Every configured LLM upstream and MCP server is listed, even before it is probed or receives traffic; LLM upstreams come from the same source the gateway routes with (stored upstreams, or the configuration file when none is enabled). Unconfigured (shadow) MCP servers aren't listed. The admin UI shows unknown rows on neither the Upstreams nor the MCP tab.
status	The upstream's live status.
criticality	critical, non_critical or ignored — see Upstream criticality.
provider	The LLM provider this upstream uses, when applicable.
transport	The MCP transport (for example stdio or http), for MCP upstreams.
passthrough	true for a passthrough LLM upstream. Passthrough upstreams aren't actively probed — their status is inherited from via_provider.
via_provider	Set on a passthrough upstream: the provider name its status is read from. The admin UI renders this as "via <via_provider> provider".
probe	Live probe results: p50_ms, p99_ms, error_rate_pct and last_checked_at. null for an upstream that isn't probed, including every passthrough upstream.
traffic	Real traffic in the window: requests, errors, error_rate_pct, latency_p95_ms, and last_used_at. LLM traffic is keyed by the upstream name, else the provider; MCP tool calls are keyed by the configured MCP server name. Each attempt of a failed-over request counts toward the upstream it was sent to.
availability_24h[]	48 half-hour slots covering the last 24 hours, oldest first, each with start and state. state is down when every probe in the slot failed, degraded when more than 5% failed, ok otherwise, and none when the slot has no probes.
issue	The open issue for this upstream (severity, message, since), or null. Always null for an ignored upstream.
routes[]	Names of the model routes this provider is a member of. Omitted when none.

Each routes[] row:

Field	Meaning
name	The route name.
strategy	The route's strategy: priority_failover for failover (shown in the admin UI as "Priority failover (sticky sessions)"), round_robin for round-robin ("Round-robin (sticky sessions)"), lowest_cost ("Lowest cost (sticky sessions)"), lowest_latency ("Lowest latency (sticky sessions)"), or weighted ("Weighted (sticky sessions)").
members[]	Each member's name and live status, in member order.
active_member	The member a new, non-sticky request is sent to first: the first reachable member in member order that isn't fallback-only, else the first reachable fallback-only member. Set only for priority_failover: other strategies start new sessions on any member, and the admin UI shows "Any member (round-robin)", "Any member (lowest cost)", "Any member (lowest latency)", or "Any member (weighted)". Failover happens per request, so a recent failover doesn't change it. Empty when no member is reachable.
healthy_members	How many members are currently reachable.
traffic	Route-level traffic in the window, same shape as an upstreams[] row's traffic.
failovers_24h	How many times a request failed over from one member to another in the last 24 hours.
last_failover	The most recent failover's at, from and to, or null when there was none in the last 24 hours.

Each status_pages[] row:

Field	Meaning
name	The status page's display name.
status	operational, degraded, outage or unknown.
detail	The component status as the provider reports it, for example partial_outage.
since	When status last changed.

Each providers[] row:

Field	Meaning
name	The provider name.
status_page	The matching status_pages[] row, or null when this provider has no status page configured — shown in the admin UI as Not tracked.
status	The provider's rolled-up status, based on the LLM upstreams that use it.
traffic	Combined traffic in the window across every upstream that uses this provider: requests, errors, error_rate_pct, latency_p95_ms, last_used_at.
used_by[]	Names of the upstreams that use this provider.

Each failover_events[] row:

Field	Meaning
at	The time of the latest failover in this row's minute.
route	The route the failover happened in.
from, to	The member failed away from, and the member failed over to.
reason	Why the failover happened, when known. Omitted otherwise.
count	How many failovers this row represents. Greater than 1 when the same from → to transition with the same reason repeated within the same minute, on any replica.

Failover history is kept for 7 days.

PUT /api/v1/health/upstreams/{name}/criticality

curl -X PUT "$YOUR_WORKSPACE_URL/api/v1/health/upstreams/openai-prod/criticality" \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"criticality": "non_critical"}'

The body's criticality is critical, non_critical or ignored. The response is the updated upstreams[] row, and the rolled-up status is re-evaluated before it returns.

Response	When
200 OK	The criticality was saved.
400 Bad Request	The body isn't a single JSON object or criticality isn't one of the three values.
403 Forbidden	The caller isn't an admin.
404 Not Found	No configured LLM upstream, MCP server or probed upstream has that name. Upstreams of type unknown (seen only in traffic) can't be given a criticality.

Each change is recorded in the configuration audit log as upstream.configuration_updated.

GET /api/v1/status

What a caller receives depends on who it is:

• Public caller — not signed in.
• Developer — a developer portal session or a developer token (sess_, pat_ or aic_), or any caller with the developer role.
• Admin or viewer — an admin UI session with the admin or viewer role.

Field	Public caller	Developer	Admin or viewer
status	Yes	Yes	Yes
generated_at	Yes	Yes	Yes
proxy.uptime_seconds	Yes	Yes	Yes
upstreams	When upstreams are shown	Yes	Yes
proxy.version, proxy.mode, proxy.policy_count	No	No	Yes
issues	No	No	Yes
load.*	No	No	Yes. Deprecated: the counts are kept for API compatibility only, and load.budget_warnings is always 0. Read business activity from the Dashboard instead.

The public_status_show_upstreams setting controls upstreams for public callers only. It defaults to ui.publicViews.status.showUpstreams in the configuration file, which defaults to true; a value saved through the admin API overrides the file. Signed-in callers always receive upstreams. upstreams is omitted when there are none to list. When the public status page is disabled, public callers receive 404 Not Found. A change to either setting applies at once on the replica that saved it and within about 15 seconds on other replicas.

Upstream status values

Each entry in upstreams carries its own status, independent of the rolled-up status:

Status	Meaning
reachable	Responding normally.
degraded	Responding, but with elevated latency or error rate.
unreachable	Not responding.
unknown	Not yet checked.

GET /api/v1/health/services

curl "$YOUR_WORKSPACE_URL/api/v1/health/services" \
  -H "Authorization: Bearer $ADMIN_TOKEN"

Field	Meaning
generated_at	When the replica built this response.
services[]	One row per service, in the fixed order below: key, name, status (see Service status values), required, configured (false for a service not set up on this deployment; the admin UI hides these rows by default behind a Show N not configured toggle, and shows required as "—" for them), alwaysVisible (present and true only for a not_configured row the admin UI shows regardless, such as the Workload Trust row), detail (a short summary with no raw upstream errors or URL credentials; admins find the raw error in the proxy logs), last_checked_at (omitted when never checked), and link (the admin UI page that configures the service; empty when there is none).
notifications[]	One row per notification channel and kind that delivered in the last 24 hours: channel, kind, sent_24h, failed_24h, last_success_at, last_failure_at, and last_error (the error class of the most recent failure; empty unless the caller is an admin).
notifications_error	Set, and notifications empty, when delivery records could not be read or the read timed out. Omitted otherwise.
jobs[]	One row per background job: job, status (see Job status values), holder (the replica that last ran it), last_run_at, last_duration_ms, last_error (the error class of the last failed run; empty unless the caller is an admin), next_run_at, and interval_seconds (0 when the interval is unknown).
jobs_error	Set, and jobs empty, when job runs could not be read or the read timed out. The same failure raises a warning issue from source service:job_runs. Omitted otherwise.

services, notifications and jobs are always arrays. Job and notification rows are read from shared storage, so every replica returns the same rows, except that a job's running status is known only to the replica running it: that replica reports running, and the others report the job's last completed run. Service rows reflect the replica that answered the request.

Services

A required service that is degraded or down raises a degraded issue; an optional one raises a warning. Services that are not_configured or unknown raise nothing.

Key	Name	Required
storage	Storage	Yes
policy_enforcement	Policy enforcement	Yes
policy_source_kubernetes	Kubernetes policy source. With several sources, one row per source, keyed policy_source_kubernetes_<n>.	Yes
oidc	OIDC issuer	No
okta_sync	Okta sync	No
promptguard	PromptGuard. Required when it is fail-closed (failOpen: false), because an outage then blocks content-scanned traffic.	When fail-closed
llm_judge	LLM Judge. Reports degraded when most of its recent calls fail, and unknown when it made no calls recently.	Yes
report_sink	PolicyReport sink. The detail names the state and, when the cluster serves no policy-report API, which CRDs to install; the full reason is on GET /api/v1/audit/report-sink/status.	Yes
otel_export	OpenTelemetry export	No
discovery	Workload discovery	No
posture	Runtime posture	No
workload_trust	Workload Trust. Appears only when Workload Trust is unavailable, in place of discovery and posture.	No
intelligence	Session intelligence	No
runtime_sensor	Runtime sensor continuity. One row per cluster, keyed runtime_sensor_<cluster> when there are several. The row is healthy with the time the current unbroken run began, down with the time of the outage and its reason, or unknown with never recorded when no healthy sensor probe exists yet. An outage degrades the overall status, because workload readiness rule R6 fails closed while it lasts. The row is not_configured when runtime controls aren't running. Demo mode simulates a healthy sensor.	Yes

The discovery and posture rows are part of Workload Trust, which is available only in self-managed installations. On a Nirmata-hosted deployment, services omits both rows. In their place, a single workload_trust row reports not_configured and states why in its detail. The row has configured: false and alwaysVisible: true. The Services tab shows alwaysVisible rows even when it hides other not_configured rows, so this row explains the missing rows instead of leaving a gap. The row raises no issue. GET /api/v1/discovery/status returns enabled: false with reasonCode: "hosted_deployment".

If a deployment that held workload registrations or Workload Blueprints switches to hosted, AIControls keeps that data, hides it, and logs the warning workload trust: hidden on this deployment at startup with the number of each. The data reappears if you return the deployment to self-managed.

Service status values

Status	Meaning
healthy	Working normally.
degraded	Working, but with errors — for example a policy source serving its last-known-good policies, or a Judge whose recent calls mostly fail.
down	Not working.
unknown	Configured, but not yet checked, with no recent activity to judge by, or its status check did not answer in time — for example runtime posture before it has observed any workload. A check that runs past its time limit reports unknown with the detail status check timed out.
not_configured	Not set up on this deployment.

Error classes

last_error on job and notification rows never carries the raw error, which can include hosts, addresses, usernames or response bodies. It is one of these fixed classes:

Class	Meaning
HTTP <code>	The remote endpoint answered with that HTTP status code.
timed out	The operation hit a deadline.
canceled	The operation was canceled.
connection refused	Nothing accepted the connection.
connection reset	The remote end dropped the connection.
host not found	The host name did not resolve.
TLS error	The TLS handshake or certificate check failed.
panicked	The job or delivery panicked.
failed (details in server logs)	Any other failure.

The full error is in the proxy logs.

Job status values

Status	Meaning	Issue
ok	The last run succeeded and the job is on schedule.	None
running	The replica that answered the request is running the job now.	None
failing	The last run failed. Takes precedence over running and stale. The issue message is last run failed; admins read the error class from last_error and the full error in the proxy logs.	degraded
stale	The last run started more than three intervals ago. A job whose interval is unknown (interval_seconds is 0) is never stale.	warning
never	The job is scheduled but has not run yet.	None

Productivity processing history

GET /api/v1/health/productivity?window=1h|24h is administrator-only. It reads retained operational samples and never starts collection, enqueues jobs, or calls a model. The endpoint enforces a bounded query window. See Monitor productivity processing.

Field	Meaning
generatedAt, from, to, bucketSeconds	UTC response time, requested execution-time range, and chart bucket width (60 or 3600 seconds).
historyStartedAt	Earliest retained queue/budget sample, or null before the first sample.
failureHistoryStartedAt	Earliest retained immutable failure-event boundary, or null before recording starts. This boundary is separate from queue/budget sampling history.
telemetryState	current, stale (latest sample older than 150 seconds), or unavailable. This describes telemetry, not classification quality.
missingMinutes	Minutes in the window without a queue/cost sample, including the current minute if it has not yet been sampled.
current	Latest retained queue and daily-budget snapshot in the window, or null. Includes sampledAt, pending/running/completed/failed counts, schedule-ready count, wait age, last completed-job timestamp, UTC budgetDay, and accountedUSD.
completed, failed, failureRatePct	Recorded terminal job settlements in the window; rate is failed / (completed + failed) × 100, or null with no outcomes.
series	Bucketed outcome counts plus last sampled pending/running/budget gauges. Missing values are null; sampledMinutes exposes partial buckets.
series[].at, from, to	Bucket timestamp and its bounded execution-time interval [from, to), suitable for the failure-history query. The active minute's to can be later than generatedAt; its counts are still changing.
series[].failureCategories	Counts for all eight supported failure categories when a bucket has a recorded minute row, including explicit zero counts. null when no minute row was recorded; missing telemetry is not zero activity. Hourly buckets sum only retained recorded minutes.
failures	Recorded failed settlements by fixed safe category; no source text or provider response bodies.
enabled, automatic, dailyBudgetUSD	Effective assessment switches and configured daily cap.

The budget gauge combines reserved and charged cost using the current assessment accounting rules. It does not claim invoice precision or reconstruct earlier periods. Counters are written atomically with fenced job settlement, and queue gauges represent the shared store once rather than being summed per replica.

GET /api/v1/health/productivity/failures

Returns retained terminal failures from automatic PR assessments. Administrator access is required. Reading this endpoint does not retry a PR, fetch source, or call a model.

Query parameter	Meaning
from, to	Required RFC3339 timestamps, with optional fractional seconds. Selects execution time in [from, to), with to after from and an interval of at most 24 hours plus one minute.
category	Optional: source_unavailable, invalid_citation, invalid_json, provider, interrupted, other, incomplete_source, or invalid_classification. Omit for all categories.
repository	Optional exact owner/repository filter, case-insensitive, at most 250 bytes. Filters this list only; processing charts remain workspace-wide.
limit	Page size from 1 to 100. Omitted or 0 uses 25.
offset	Zero-based row offset, from 0 to 100000. Defaults to 0. Narrow the interval, repository, or category when results exceed this paging limit.

Invalid parameters return 400 Bad Request; callers without administrator access receive 403 Forbidden. Unavailable failure-history storage returns 503 Service Unavailable. Responses disable caching.

Response field	Meaning
from, to, category, repository, limit, offset	Applied query parameters, including the default page size.
maxOffset	Largest accepted offset, currently 100000.
generatedAt	UTC response timestamp.
historyStartedAt	Earliest retained failure-history boundary, or null before recording starts. History is retained for seven days.
completeWindow	Whether the entire requested interval lies within retained history and ends no later than the response time.
total	Number of retained events matching the filters, including events beyond the paging limit. This counts failed attempts, not distinct PRs.
items	Array of matching events, newest first; ties use descending event ID. Empty when no retained events match.
items[].id, prID, repository, prNumber, title	Event ID and associated PR identity. Repository and title reflect the current stored PR metadata.
items[].at, headSHA, queueVersion	Failure timestamp and the attempt's revision and queue version.
items[].category, diagnosticCode	Safe failure category and diagnostic code. The code can be empty when unavailable.
items[].model, promptVersion, sourceVersion	Bounded processing metadata; empty when unavailable. Raw provider responses and source patches are not returned.

A retry or later success does not rewrite an earlier failure event. Events begin when failure recording is enabled; historical failures are not reconstructed. The chart's active minute can include later events than an earlier drill-down response, so counts at that boundary can differ.

GET /api/v1/health/productivity/operations

Returns recorded manual source-retrieval and analysis-finalization failures. Administrator access is required. GET reads shared storage without retrieving GitHub source, queuing work, or calling a model. See Inspect manual operation failures.

Query parameter	Meaning
from, to	Required RFC3339 timestamps with optional fractional seconds. Selects [from, to), with to after from and a maximum interval of 24 hours plus one minute.
stage	Optional: source_preview, analysis_source, manual_save_source, review_source, or analysis_finalize. Omit for all stages.
diagnosticCode	Optional safe cause from the list below. Omit for all causes.
repository	Optional exact case-insensitive owner/repository, at most 250 bytes. Applies to totals, series, distinct PRs, and event list.
limit, offset	Page size 1–100 (omitted or 0 defaults to 25); row offset 0–100000 (defaults to 0). Pagination affects only items.

Source codes are source_deferred, upstream_rate_limited, upstream_unauthorized, upstream_forbidden, upstream_not_found_or_inaccessible, upstream_error, source_timeout, source_canceled, source_network_error, source_response_too_large, source_malformed_response, source_unavailable, source_changed, incomplete_source, and source_ineligible. Finalization codes are publication_conflict, publication_rejected, settlement_unavailable, and storage_unavailable; unknown is also accepted.

Invalid parameters return 400; non-administrators receive 403; unavailable operation-history storage returns 503. Responses disable caching.

Response field	Meaning
from, to, stage, diagnosticCode, repository, limit, offset	Applied filters and pagination.
generatedAt, maxOffset, bucketSeconds	Response time, maximum accepted offset (100000), and hourly bucket width (3600 seconds).
total, distinctPRs	Matching recorded failures and distinct associated PRs across the full filtered interval. Repeated operations can fail for the same PR.
coverage, completeWindow	Always recorded_failures_only and false. Successful attempts and every provider/validation failure are not collected here, so this stream has no failure-rate denominator.
firstObservedAt, retainedSince	First successfully recorded manual failure (or null) and the current seven-day retention cutoff. First observation survives pruning and does not prove uninterrupted instrumentation coverage.
series[]	Hourly at, clipped [from, to) interval, total, and a diagnosticCodes count map, under the same filters as the event list. Empty maps mean no recorded failures; they do not establish healthy coverage.
items[]	Matching events, newest timestamp first with descending ID as tiebreaker.
items[].id, prID, repository, prNumber, title, headSHA, at	Server correlation ID, PR identity/current metadata, imported attempt revision, and event time.
items[].stage, diagnosticCode, dispatchState, elapsedMS	Failure stage/cause, not_dispatched, dispatched, or unknown, and elapsed milliseconds. Source stages are always not_dispatched. Dispatch status alone does not establish billing or successful publication.
items[].requestID, retryAt, upstreamStatus, upstreamRequestID	Optional analysis request association, suggested retry timestamp, HTTP status, and GitHub request ID. Unknown or unsafe metadata is omitted.

Events are retained for seven days, removed with deleted source PRs, and never reconstructed for older releases. Provider/output-validation errors remain in PR request history. An operation response reports eventRecorded: false if writing its event failed; its correlation ID remains available in safe server logs. No raw upstream body, credential, provider output, or source patch is stored in this stream.

Analytics API execution deadlines

Dashboard and analytics JSON reads have a default 30-second execution deadline. This includes waiting for a database connection. When the deadline expires, query cancellation propagates through the request and the API returns HTTP 504 with an error message. A required analytics read that fails before the deadline returns HTTP 503, rather than presenting incomplete totals as a successful response. Successful response formats are unchanged.

The deadline applies to the selected cost, budget overview/effective/preview, stats, metrics, adoption, insights, activity, security, intelligence, identity-list, session-list, audit-list, and baseline-alert GET endpoints, including developer-scoped insights. It does not apply to LLM or MCP forwarding, streaming responses, exports, backups, or mutations. Gateway error and latency metrics describe forwarded traffic; they do not measure these analytics API responses.

Self-hosted administrators can configure the analytics deadline. An earlier caller deadline takes precedence.

The web UI treats a 504, and a request the browser gives up on after two minutes, as a final answer: it does not retry. Pages that handle load errors, such as the Insights overview and the Gateway activity list, show an error state with a Retry button instead of a zero. Most analytics views that read 7 to 30 day windows (attribution, adoption, contributions, intelligence and Insights) refresh every five minutes, and every fifteen after a deadline error; a few views, such as the security dashboard, keep their own refresh interval. The Insights overview loads its figures in three stages and marks any one that fails with Could not load and its own Retry button.

A 504 identifies an execution deadline, not a diagnosis of insufficient hardware. Inspect database query duration, connection waits, and CPU throttling before increasing the deadline. Increasing it allows expensive work to occupy resources for longer.