Upstreams

Concepts

An upstream is an LLM provider that AIControls routes traffic to on behalf of your developers. You configure API keys once — developers never touch them directly.

How upstreams work

When a developer's AI tool makes a request, AIControls:

1. Authenticates the developer using their personal access token
2. Evaluates policies against the request
3. Forwards the request to the matching upstream using the upstream's API key
4. Returns the response and logs the full audit event

Developers connect with workspace credentials — they don't need Anthropic or OpenAI accounts.

Supported providers

AIControls can route to five classes of upstream provider, each with its own identity and authentication model:

• Anthropic — authenticated with a single sk-ant-... API key when using Anthropic's own API (the default, or an explicit api.anthropic.com URL). AIControls supports all Claude models, including claude-opus-4-8, claude-sonnet-4-6, and claude-haiku-4-5. New models appear automatically as Anthropic releases them, so you don't need to update an upstream definition when a new model ships. Pointing this provider at a custom URL — a third-party gateway that speaks the same Anthropic Messages wire format (e.g. z.ai's GLM models) — is also supported, and the API key can be in that gateway's own format rather than sk-ant-...; the sk-ant-... requirement is only enforced for the canonical Anthropic host.
• OpenAI — authenticated with a single sk-... API key. An OpenAI account can have multiple organizations, so an upstream optionally carries an organization ID to disambiguate which org's models and billing a request should use.
• Azure OpenAI — identity is resource-based rather than account-based: a request targets a specific Azure resource endpoint and a named deployment within it (rather than a model ID directly), and every request is versioned against a specific Azure API version.
• Google Vertex — authenticated via Google Application Default Credentials (a service account key file, or GKE/GCE Workload Identity) rather than a vendor-issued API key. Vertex traffic is only supported through the native /vertex/* route today (Gemini's own generateContent/streamGenerateContent protocol) — the OpenAI-compatible /v1/chat/completions route doesn't yet translate to it (#1578). Model discovery (GET /v1/models) for this provider is served from AIControls' cost table rather than a live probe, since Vertex has no OpenAI-shaped discovery endpoint to call — any explicit Static Model List is included alongside those entries.
• Amazon Bedrock — authenticated with AWS credentials (an access key/secret pair, or an assumed role) rather than a vendor-issued API key, and scoped by AWS region. Because Bedrock authentication is just AWS IAM, you can scope a credential tightly — for example, a dedicated IAM user with bedrock:InvokeModel permission limited to specific model ARNs — instead of using a single all-or-nothing key.

For the exact fields, console steps, and config values needed to add each provider, see Configure an Upstream.

Routing between upstreams

When you configure multiple upstreams, AIControls uses static routing — each upstream declares which requests it should receive, and the first match wins. Routing decisions are computed from this static configuration, but a policy can override the result for an individual request — see Overriding routing with a policy below.

Pool size and licensing

Up to two upstreams can be configured on any tier — enough for the common pairing of one upstream holding an organization-managed API key alongside a second that forwards each caller's own credential. Pools of three or more upstreams require an Enterprise license.

This limit is checked at startup. If a configuration file declares more upstreams than the active tier permits, the process refuses to start and logs an error naming the configured count and the active tier, rather than starting with some upstreams silently dropped. Reduce the pool or load an Enterprise license to proceed.

Each upstream can match on any combination of three signals (all are AND-ed):

Signal	Field	How it works
OIDC groups	groups	Routes requests from members of specific IdP groups. Use ["*"] as a catch-all that matches any caller.
Model globs	match.models	Routes by the model name in the request. Supports prefix, suffix, and substring wildcards — e.g. claude-*, *gpt-4*.
Identity labels	match.labels	Routes by identity source set server-side during token resolution. Common key: source with values pat, github, session, or passthrough. session matches governed session-token (sess_) callers such as CI jobs and headless agents. Cannot be forged by clients.

Selection order

Upstreams are evaluated in order of specificity — more specific predicates always beat catch-alls, regardless of the order you define them in settings:

1. Identity labels present — highest priority
2. Model globs present
3. Non-wildcard groups present
4. Catch-all (groups: ["*"] or no predicates) — lowest priority

Among upstreams with equal specificity, use the priority field as a numeric tiebreaker — lower value is evaluated first.

If specificity and priority are still tied, AIControls uses the requested model to break the tie — the tied upstream whose provider (Anthropic, OpenAI, or Google) matches the model wins. If the model's family can't be determined, or every tied upstream is equally unopinionated (e.g. Azure, LiteLLM, custom deployments), routing falls back to a fixed, deterministic order. If the model's family is known and no single tied upstream confidently matches it, AIControls refuses to guess — the request fails immediately instead of silently going to the wrong provider.

note

Routing rules are configured per upstream in Settings → Upstreams → Edit upstream. Use the Priority field as a numeric tiebreaker when two upstreams have the same specificity — lower value is evaluated first.

Overriding routing with a policy

The upstream this static routing selects is exposed to every policy as object.defaults.routing.upstreamName (with object.defaults.routing.matchedOn explaining why it was picked). A MutatingPolicy with patchType: Defaults can override this per request — for example, routing a call to a data-retention-compliant upstream once content scanning flags PII in the prompt. See Defaults mutations for the worked example and the override's fail-open behavior (an unknown upstream name is ignored, never a denial).

This override is evaluated on every request path, so it also applies to policies that only intend to affect LLM traffic — routing has no meaning for a plain MCP tool call and the override namespace is simply absent there.

note

The routing dry-run API (POST /api/v1/routing/resolve) previews only the static selection described above — it does not simulate a patchType: Defaults override a policy might apply at request time.

Upstream pools — automatic failover

Preview

Upstream pools are a preview feature. The shape described here may change in a future release.

A pool groups several upstreams into a named, ordered failover chain. If the first member returns a retryable error — a rate limit (429), a provider-side error (5xx), or a timeout — AIControls tries the next member before failing the request. Without a pool, that same error is returned straight to the developer, even when an equivalent upstream is configured and healthy.

A 400, 401, or 403 is not retried — those indicate a problem with the request or its credentials, which reproduces identically on every member, so retrying just adds latency without changing the outcome.

Membership rules

Every member of a pool must speak the same wire format:

• Extra API keys for the same provider (e.g. two Anthropic keys on separate billing accounts) always work.
• Anthropic, AWS Bedrock (Claude models), and Google Vertex (Claude-on-Vertex) can mix freely in one pool — they speak the same Anthropic Messages format under the hood, just over different transports and credentials. AIControls translates the model ID for each member automatically (a Bedrock member needs a different wire ID than a native Anthropic member for the same request).
• Azure OpenAI / Azure AI Foundry upstreams can join a pool, but only with an explicit wire format set on that member — Azure Foundry can host either OpenAI- or Anthropic-shaped deployments, so AIControls can't infer which one a given deployment speaks. A member left without this is excluded from the pool (with a warning), not silently misrouted.
• Mixing genuinely incompatible families — Anthropic and OpenAI in the same pool, for example — isn't supported yet; that member is excluded from the chain with a warning rather than being retried against a provider that can't parse the request.

A member can be marked fallback only, meaning it's never treated as the pool's primary — useful for a slower or more expensive region you only want used as a last resort.

Once an upstream is referenced by a pool, the pool's own routing rules (model patterns, groups, priority) take over — the member upstream's own routing settings are no longer consulted. An upstream not referenced by any pool keeps routing exactly as it did before pools existed.

What shows up in the audit log

A request served by a pool records which pool it went through and which attempt ultimately served it (e.g. "attempt 2 of 3") in Security Events, alongside a row for every failed attempt along the way — so a provider incident is visible, not just its absence of impact. A policy can also read the pool a request was routed through via object.llm.upstream.pool.

See Configure an Upstream for setup steps.

See also

For step-by-step setup instructions for each provider, see Configure an Upstream.