Configure an LLM Provider
Create an LLM provider to give your applications a managed proxy URL: Redpanda handles the upstream API keys, forwards requests to the provider, and records usage for you. Create a provider for each upstream you use, whether that’s OpenAI, Anthropic, Google AI, AWS Bedrock, or an OpenAI-compatible endpoint.
After reading this page, you will be able to:
-
Create an LLM provider for OpenAI, Anthropic, Google AI, AWS Bedrock, or an OpenAI-compatible endpoint
-
Select the models you want to expose through the provider
-
Verify the provider is reachable using the built-in Test connection control
Prerequisites
-
An API key (or AWS credentials for Bedrock) for the upstream provider you want to configure.
-
Optional: Secrets already created in the Agentic Data Plane secret store for the provider’s credentials. The create form can also store a new API key inline, so you don’t have to prepare a secret first. Secret identifiers use
UPPER_SNAKE_CASE. For example:OPENAI_API_KEY,ANTHROPIC_API_KEY,AWS_ACCESS_KEY_ID.
Fill in the identity fields
The Identity section collects the provider’s name. Enter a Display name: a human-readable label shown in dashboards and model selectors, up to 253 characters.
Agentic Data Plane derives the provider’s resource ID from the display name in the background; the create form doesn’t show it. The resource ID is the machine identifier used in API calls and CLI commands: it starts with a lowercase letter, contains only lowercase letters, numbers, and hyphens, ends with a letter or number (^[a-z]([a-z0-9-]{0,61}[a-z0-9])?$), is at most 63 characters, and is immutable after creation. It appears in the proxy URL (/llm/v1/providers/<resource-id>/…).
Choose a provider type
In the Provider type section, pick the type that matches your upstream.
| Type | Use when |
|---|---|
OpenAI |
Proxy GPT, o-series, and embeddings through the OpenAI API. Best when you already hold an OpenAI API key or want the broadest GPT model catalog. With passthrough on, forwards each client’s own |
Anthropic |
Call Claude Opus, Sonnet, and Haiku directly. Strong at coding, long-context reasoning, and tool use. With passthrough on, forwards each client’s own |
Google AI |
Reach Gemini Pro, Flash, and multimodal models through Google AI Studio. Ideal for long-context workloads and image/video inputs. |
AWS Bedrock |
Invoke foundation models (Claude, Llama, Titan, Nova, Mistral, AI21 Jamba, Gemma) hosted inside your AWS account. Requires an AWS region and credentials (static, STS-assumed role, or the default credential chain). Supports the native Bedrock APIs ( |
OpenAI-compatible |
Point at any OpenAI-compatible endpoint that ships |
Selecting a type reveals the type-specific configuration fields.
Fill in the type-specific configuration
Each API key reference and credential field points at a secret-store entry, not the secret value itself. On the create form, the Credential section lists the keys already in the Agentic Data Plane secret store: pick one, or select Bring a new API key reference to store a new key inline. Enter a Key name and the API key value, then click Save key. Agentic Data Plane stores the key in the secret store under an identifier derived from the key name, shown under the field; the value is stored encrypted and can’t be viewed again after saving.
To change the key later, open the provider’s Settings tab. Its Credential section uses the same picker as the create form: pick another stored key, or bring a new one.
-
OpenAI
-
Anthropic
-
Google AI
-
AWS Bedrock
-
OpenAI-compatible
| Field | Notes |
|---|---|
|
Optional. Prefilled with the standard OpenAI API ( |
|
Required unless |
|
Optional toggle. When on, AI Gateway forwards each client’s own |
| Field | Notes |
|---|---|
|
Optional. Prefilled with the standard Anthropic API ( |
|
Required unless |
|
Optional toggle. When on, AI Gateway forwards each client’s own |
| Field | Notes |
|---|---|
|
Optional. Prefilled with the standard Google AI API ( |
|
Required. Secret-store reference for the Google AI API key. |
|
Google’s SDKs send their API key as |
| Field | Notes |
|---|---|
|
Required. AWS region where the Bedrock endpoint is deployed, for example |
|
Optional. Override the default regional Bedrock endpoint. With a custom base URL, AI Gateway sends every request to that URL, so models that Bedrock serves only on its separate |
|
How AI Gateway authenticates to Bedrock: Bedrock API key, Default chain, Static keys, or Assume IAM role. Rows that name a mode apply only to that mode. |
|
Bedrock API key only. Required. Secret-store reference for an Amazon Bedrock API key, |
|
Static keys only. Secret-store reference for the AWS access key ID, |
|
Static keys only. Secret-store reference for the AWS secret access key, |
|
Assume IAM role only. Required. ARN of the IAM role AI Gateway assumes through AWS STS, for example |
|
Assume IAM role only. Optional. External ID for cross-account role assumption. Set it only when the role’s trust policy mandates an external ID. |
|
Assume IAM role only. Optional. Session name that appears in AWS CloudTrail audit logs, for example |
|
Optional. Name of a guardrail to attach to this provider, or empty for none. The UI shows this field on Bedrock providers only. For other provider types, set it with the |
Pick a Credential type to control how AI Gateway authenticates to Bedrock:
-
Bedrock API key: An Amazon Bedrock API key stored in the secret store, which AI Gateway sends as a bearer token instead of signing requests with SigV4. Use this when you want a single credential and no IAM key pair. Generate the key in the Amazon Bedrock console, and store the key value on its own. The identity behind the key needs the
bedrock:CallWithBearerTokenpermission, plusbedrock:ListFoundationModelsfor Test connection and the invoke permissions for the models you call. AI Gateway never generates or renews a Bedrock API key, so an expired or revoked key fails every call until you update the secret. A short-term key expires within 12 hours and works only in the region it was generated for, so a provider that uses one stops working when the key expires unless you update the secret first. If you set aBase URLfor this provider, it must start withhttps://. See Generate Amazon Bedrock API keys for how to create a key and which permissions it carries. -
Default chain (default): Leave the credentials unset to use the AWS SDK’s default provider chain (environment variables, shared config, EKS Pod Identity, IRSA, or instance profile). Use this when the gateway already runs with an AWS identity. An Agentic Data Plane environment hosted on a cloud other than AWS has no such identity, so this mode can’t obtain credentials there.
-
Static keys: An IAM access key pair stored in the secret store. Use this when no ambient AWS identity is available, including on an environment hosted on a cloud other than AWS. This is the path the Bedrock setup guide walks through. This mode takes an IAM access key pair only. A Bedrock API key belongs in Bedrock API key, and a temporary access key that needs a session token belongs in Assume IAM role or Default chain.
-
Assume IAM role: AI Gateway assumes an IAM role through AWS STS. Use this for cross-account access or when your security policy requires short-lived credentials. The role ARN names the role to assume. AI Gateway still authenticates the AssumeRole call through the default chain, so this mode also needs an ambient AWS identity. A provider carries one credential type, so you can’t pair a role ARN with an access key pair. See Bedrock troubleshooting for the error a provider reports when it has no reachable AWS identity.
For how Bedrock model identifiers and IAM ARNs map to these fields, see Inference profiles and IAM.
| Field | Notes |
|---|---|
|
Required. URL of your OpenAI-compatible endpoint, for example |
|
Optional. Leave empty for endpoints with no authentication (common for local runtimes), or when |
|
Optional toggle. Behaves as it does on an OpenAI provider: AI Gateway forwards each client’s own |
OpenAI-compatible endpoints can serve any model. Enter the exact model identifiers your upstream server exposes (for example, meta-llama/Llama-3.3-70B-Instruct or qwen3:8b).
|
|
Save-time validation checks the credential reference itself, not the secret behind it. The Google AI type rejects a create or update with a missing or empty API key reference. The OpenAI and Anthropic types require exactly one of an API key reference and |
Select models
Models you select on this form become the catalog the provider exposes.
For OpenAI, Anthropic, Google AI, and AWS Bedrock, the form shows a picker backed by the provider’s catalog, with the full catalog selected by default. Keep the default, or deselect models to narrow what the provider exposes. Each model in the picker shows its input and output price per million tokens. Pick from the list, or type a model identifier the catalog doesn’t show. For OpenAI-compatible, the form takes a freeform list: type the exact identifiers your upstream serves.
Redpanda maintains the catalog of available models in the picker. When an upstream provider publishes a new model, it usually appears in the picker within a day or two. Admins don’t have to wait for a Redpanda release. New models aren’t enabled automatically: an admin still selects the model in the catalog to make it callable through this provider. A model the upstream provider has retired drops out of the picker list, though you can still type its identifier. A provider that already has it selected keeps serving it, its recorded spend keeps pricing at the catalog rates, and agents using it keep working.
For Bedrock, the picker lists inference-profile IDs for models that Bedrock serves only through a profile, such as Claude, and the bare model ID for models invoked in-region, such as Mistral Large 3. See Inference profiles and IAM.
The Bedrock picker groups models under the vendor that publishes them, such as Anthropic, Amazon, Google, Meta, Mistral, and OpenAI.
|
Redpanda stores models as structured |
Beyond pricing, the catalog carries each model’s capabilities and context-window limits. The model discovery API (the ListModels and GetModel methods on ModelService) reports max_input_tokens (the largest context the model accepts) and max_output_tokens (the most it can generate in a single response) for each model. Both are read-only catalog metadata: a limit the catalog doesn’t declare stays unset rather than reported as zero. Clients read max_input_tokens to show how full an agent’s context window is, such as the context-window indicator on the agent’s Playground tab.
Override per-model pricing
Cost reporting prices each call at the catalog rates for the model. If your organization negotiates non-standard rates, or you track spend against an internal chargeback rate, override the rates per model on this provider.
Each model you select in the model picker, and each enabled model on a saved provider’s Models tab, carries a pencil icon (Override pricing). Click it to open the pricing dialog for that model. A model the catalog doesn’t price has the same dialog, with empty fields and no default rate to fall back on, so you can set every rate yourself.
The dialog lists one field per billing bucket, in the same order as the provider’s published rate card:
| Bucket | What it bills |
|---|---|
Input |
Per 1M input tokens. Tool-use input also bills at this rate. |
Output |
Per 1M output tokens. Reasoning tokens also bill at this rate. |
Cached input |
Per 1M tokens read from prompt cache. |
Cache write (5-minute TTL) |
Per 1M tokens written to a 5-minute prompt cache. |
Cache write (1-hour TTL) |
Per 1M tokens written to a 1-hour prompt cache. |
Enter rates in dollars per million tokens. Each field is independent:
-
Leave a field blank to keep the catalog rate for that bucket. The catalog rate shows as the field’s placeholder.
-
Enter a positive value to replace the catalog rate for that bucket only.
-
Enter
0to make that bucket explicitly free, which is different from leaving it blank.
Cache writes with an unknown TTL always bill at the catalog rate; they have no override field.
Click Reset on a field to clear a single override, or click Reset all to clear every field at once. Saving with every field empty drops all overrides for the model and returns it to the catalog rates. Overrides are scoped to this provider and model, and they change what Agentic Data Plane’s cost reporting computes, not what the upstream provider actually charges you.
If saving fails, the dialog stays open with your values and reports the reason, so you can fix the problem and retry without retyping the rates.
To see at a glance which models carry an override, look for the dollar-sign badge. On the Models tab it sits beside the model name, and in the model picker it ends the price line. It appears on every enabled model that has a pricing override, and its tooltip confirms that the model uses custom pricing instead of the catalog default. In the model picker’s price line, an overridden bucket’s rate is highlighted too, so you can tell an overridden input rate from an overridden output rate. Enabled models are listed first on the Models tab, which keeps the models you price where you can find them.
You can set the same overrides from the CLI instead of the pricing dialog. The repeatable --pricing flag on rpk ai llm-provider create and rpk ai llm-provider update takes one entry per model, written as comma-separated key=value pairs with rates in dollars per million tokens:
rpk ai llm-provider update openai \
--pricing "model=gpt-4o,input=2.50,output=10.00,cached=1.25" \
--pricing "model=gpt-4o-mini,input=0.15,output=0.60"Each entry names the model with model, which is required, then sets one rate key per billing bucket: input, output, cached, cache_write_5m, and cache_write_1h. As in the dialog, an omitted rate keeps the catalog rate and 0 sets an explicit free rate; set at least one rate per model. On update, --pricing sets the provider’s full model list, so include every model you want the provider to keep, either as its own --pricing entry or through --provider-models.
Explore the provider detail page
After you create the provider, its detail page organizes everything about it into tabs: Overview, Models, Connect, Playground (when enabled for your organization), and Settings. The page header carries the provider’s logo, its display name with an Enabled or Disabled badge beside it, its resource identifier, and its description if it has one.
Overview
The Overview tab reports over a single time range, which you pick with the range selector at the top of the tab. The range covers everything on the tab: the metric cards, the Gateway traffic chart, Who calls what, and the Callers table. It opens on the last seven days, and you can pick a preset or an exact window of your own.
Three metric cards give total spend, requests, and tokens over that range. Each card gives the total, how it changed against the range of the same length just before it, a trend line, and a breakdown of the figure by cost type, request outcome, or token type. The token card also reports what share of prompt tokens came from cache. Click View more on a card to open a detail panel for that metric, where you can group and filter the figure.
Below the cards, a Gateway traffic chart plots token volume as bars and spend as a line, with the totals for the range in its header, next to the Connection card. Drag across the chart, or pinch it, to zoom into a narrower window, which then becomes the range for the whole tab. Click View all to open the Cost and usage page with this provider charted against the gateway total over the same range. The Connection card lists the provider type, the authentication passthrough state, and any attached guardrail, then the endpoint values with a copy control each: the proxy URL, the upstream base URL, the API key secret reference, and, for an AWS Bedrock provider, the region and its credential references.
A Who calls what card then draws the caller-to-model traffic for that range. Hover a node in that graph to isolate its connections, which a hint below the graph reminds you of on a machine with a mouse.
A Callers table lists, over the same range, each agent or user that called the provider, with its request count, the models it used, its error count, spend, and when it was last seen. Sort the table by requests, errors, spend, or last seen. It opens on the busiest caller first. Search the table by caller name, slug, or email. Select an agent’s row to open the Activity tab of the Cost and usage page for that agent, or a user’s row to open that page with this provider charted against the gateway total. For analysis across providers, see View cost and usage.
Investigate failed calls
When a caller has failed requests in the selected range, Who calls what shows a warning above the graph counting the callers that need attention, and says so when some callers aren’t shown. Expand it to see each failing caller with its count of failed requests, then click Inspect to open the details. An affected caller’s node in the graph and a caller’s error count in the Callers table open the same details.
The Failed provider calls panel lists that caller’s failed calls over the range, newest first. Each entry carries a label for what went wrong, such as Rate limited, Authentication failed, Permission denied, Resource not found, Timed out, Provider unavailable, Safety policy blocked, Request too large, or Invalid request, along with when the call started, a summary of the error, the model, the latency, and the trace, span, and response identifiers with a copy control each. Click Load more to page through a longer list. The panel reports the call’s own diagnostics, not the prompt or the model’s output.
Each entry offers the next steps that fit its failure:
-
Open transcript opens the conversation for a call an agent made.
-
Review usage breaks down what hit the rate limit.
-
Review models opens the model list for a model the provider no longer serves.
-
Review connection opens the credential and endpoint settings behind an authentication or permission failure.
-
A link to the upstream provider’s own status page appears when that provider timed out or was unavailable.
-
Review guardrail appears for a call the safety policy blocked, on a provider with an attached guardrail.
-
Manage budgets and Test in Playground appear when those features are enabled for your organization.
Models
The Models tab lists the chat models available on this provider. Each row shows the model’s name and identifier, icons for the capabilities it supports (such as vision, tools, JSON, or reasoning), its context-window limit, and its input and output prices per million tokens. Hover over a capability icon to read what that capability means. Use the toggle on a row to enable or disable that model on the provider. Select a model to open its detail page (see View a model’s detail page).
Enabled models sort to the top. Use the search box to find a model by display name or identifier, and the Status and Capability filters to narrow the list.
AI Gateway rejects a request for a model that isn’t enabled on the provider with HTTP 403 and the error type model_not_allowed. This applies to every provider type.
On an AWS Bedrock provider, disabling a model also affects agents: Agentic Data Plane rejects an agent that references a disabled model, at agent creation or when the model reference changes. Agents that already reference a model you disable stay saveable until someone edits their model. See Create an Agent.
Connect
The Connect tab generates ready-made client configuration for this provider: a gateway-token step, ready-to-run rpk ai setup steps, setup instructions for popular clients such as Claude Code, and code examples in several languages, all with the provider’s proxy URL prefilled. For a Bedrock provider, the Claude Code snippets run against the gateway without local AWS credentials, because the gateway signs the upstream AWS requests for you. See Connect your app to AI Gateway for the underlying flow.
On an OpenAI or OpenAI-compatible provider with authorization passthrough on, the tab shows a passthrough setup guide in place of the client instructions, and omits the direct code examples. The guide explains that a client sends its upstream credential in the Authorization header and its gateway token in the X-Redpanda-Cloud-Token header. Passthrough therefore needs a client that can send an upstream credential and a separate custom header. A client that can’t doesn’t support this mode, and the tab’s standard rpk ai launch steps and SDK examples use gateway-only authentication, so they don’t apply to a passthrough provider. When the provider’s base URL is the Codex endpoint, the guide also prints a ready-to-paste Codex client configuration and the shell commands that set the gateway token from rpk ai auth token. Use that manual setup rather than rpk ai run codex, which configures API-key authentication. Refresh the gateway token and restart the client when it expires.
The tab also carries a Verify provider panel. Click Test connection to check the saved provider against the upstream service. The result appears as a message under the panel. See Save and verify for what the check covers.
Playground
When the Playground tab is available, use it to test one of the provider’s models before you connect application code. Choose a model, send a prompt, and optionally adjust response controls such as reasoning effort and maximum output tokens. The response streams into the conversation while the Playground shows the request, response events, status, and timing. Playground conversations are ephemeral.
Settings
The Settings tab is where you edit the provider after you create it. It is organized into sections:
-
General: Edit the display name. The provider type and resource ID are fixed after creation and shown read-only.
-
Credential: Change how the gateway authenticates to the upstream provider. To rotate a stored key’s value for every provider that uses it, follow the link to Provider keys in the Secret Store.
-
Connection: The read-only proxy URL and the editable upstream base URL. To confirm the provider is reachable, use Test connection on the Connect tab after you save.
-
Models: Choose which models this provider exposes to agents and developers.
-
Transcripts: Turn message-body recording on or off (see Configure message recording).
When you change a field, a save bar appears with Save changes and Discard. Below the form, an enable/disable control stops or resumes routing without deleting anything, and a Delete this provider danger zone permanently removes the provider after you confirm.
View a model’s detail page
From the Models tab, select a model to open its detail page. The page gathers what the catalog knows about that model on this provider in one place:
-
An overview strip with the model’s context window (largest input it accepts), maximum output (most it can generate in a single response), and price (input and output rates per million tokens). Each value appears only when the catalog declares it.
-
Model details: The model ID and the provider’s proxy URL, each with a copy control.
-
Pricing: A read-only rate card of the model’s effective per-bucket rates, that is, the catalog rates with any per-model overrides applied (see Override per-model pricing). This section appears only when rates are known, and links to the upstream provider’s pricing reference.
-
Capabilities: The model’s capabilities from the catalog, with a link to the upstream provider’s capabilities reference. This section appears only when the catalog declares capabilities.
-
Usage: Spend, Requests, and Tokens cards for traffic routed to this model through this provider over the last 7 days. Click View more on a card to open a metric detail drawer.
Configure message recording
The Transcripts section controls whether AI Gateway records the message bodies this provider proxies. It appears on the create form and on the provider’s Settings tab, and it has two independent toggles:
| Toggle | What it captures |
|---|---|
|
Captures the full request body (prompt content and tool-call arguments) on observability traces. |
|
Captures the full response body (completion content and tool-call results) on observability traces. |
When you create a provider, recording starts on, so a new provider records message bodies from its first request. This holds on every create path: the Create provider form starts with both toggles on, and a create with rpk ai llm-provider create that doesn’t mention transcripts gets the same default. The bodies land on the AI Gateway’s own trace data for the calls it proxies. They don’t feed agent transcripts, which a managed agent records under its own transcript recording mode. Turn either toggle off, on the create form or later on the Settings tab, for workloads where the message body must not be retained, such as regulated PII or customer secrets. To keep that content out of agent transcripts too, set each agent’s recording mode to Metadata only or Off.
Two cases don’t get that default:
-
An existing provider keeps whatever it was saved with. The default applies at create time and doesn’t change any provider you already had.
-
A create that explicitly disables recording stays off. From the CLI, pass
--transcripts.record-input-messages=false --transcripts.record-output-messages=false. Set both flags together, or neither: a create that sets exactly one of them leaves the other off.
These are per-provider settings, not per-request: applications cannot opt in or out at call time. To split sensitive from non-sensitive traffic, create one provider with recording on and another with it off, and route each application to whichever proxy URL matches its data class.
| These toggles govern only the message bodies the AI Gateway records for the calls it proxies. An agent’s Transcripts tab is assembled from spans the agent exports itself. A managed agent’s transcript recording mode, or a self-managed agent’s own instrumentation capture settings, controls the text those transcripts show, not these toggles. See Transcript recording and Message content capture. |
Recording settings do not affect cost and usage telemetry. Token counts, latency, and provider/model attribution are always recorded, so the Cost and usage page reports spend for traffic on the provider regardless of these toggles; only the message bodies are withheld when the toggles are off.
| Changing a toggle takes effect for new requests. Message bodies already recorded under the previous setting are not retroactively redacted; delete or rotate the provider if you need to purge historical content. |
Save and verify
Before you click Create provider, check the Transcripts section. Both toggles arrive on, so the provider starts recording prompts and responses with its first request, and what it captures is not retroactively redacted. See Configure message recording.
-
Optional: In the Connection settings section, click Test connection under Verify connection to check the credential and endpoint you entered before you commit to them. The check creates nothing, so a failure costs you only the retry. It needs a provider key (for the types that require one) and a base URL that starts with
http://orhttps://. For an OpenAI-compatible provider, the base URL is required. For other provider types, it’s checked only if you set one. -
Click Create provider. Until the form is complete, the button stays disabled and the hint next to it tells you what’s still needed (for example, saving the new key).
-
After create, you land on the provider’s Connect tab, with client setup instructions and code examples that have the proxy URL prefilled. To copy the proxy URL on its own, open the Overview tab and use the copy control on the Connection card. The proxy URL is where your applications point.
-
To confirm the saved provider is reachable, click Test connection under Verify provider on the Connect tab. The result appears as a message under the panel.
-
To wire up an application, use the Connect tab or follow Connect your app to AI Gateway.
What a successful result proves depends on the provider type, and the panel says so before you run it:
-
For most types, the check lists the models the upstream service offers, using the credential and endpoint you configured. It sends no model request, so it confirms authentication and the network path rather than access to any one model.
-
For AWS Bedrock, the check confirms the AWS credentials in the configured region. It doesn’t verify access to a model, which IAM grants separately (see Inference profiles and IAM).
-
For a Bedrock provider that authenticates with a Bedrock API key, the check lists the foundation models in the region, so the key’s identity needs
bedrock:ListFoundationModelsas well asbedrock:CallWithBearerToken. AWS answers a rejected key and a key that lacks a permission the same way, so the check reports both as an authentication failure, and it names the cause when AWS’s response does, such as an expired key. As with the other Bedrock credential types, a check that passes doesn’t confirm that the key can invoke a model or use a guardrail. -
For any provider with authorization passthrough on, there’s no server-side credential to test. The first real request is what verifies upstream access, because each client authenticates with its own key.
If the check fails, see Troubleshooting.
Browse providers in the list view
The LLM providers list page is the at-a-glance home for every provider in your Agentic Data Plane environment. Open it from the sidebar’s LLM providers entry.
| Column | What it shows |
|---|---|
|
The provider-type icon (OpenAI, Anthropic, Google AI, AWS Bedrock, or OpenAI-compatible), the display name, and the resource identifier beneath it. Select the provider name to open its detail page. |
|
|
|
How many models are configured on the provider. Select the count to read the full list. A provider with no models configured shows a dash. |
|
Request count over the last 24 hours. |
|
Spend over the last 30 days. For longer-range or cross-provider analysis, use the Cost and usage page (see View cost and usage). |
Click the 24h requests or 30d spend column header to order the list by that figure. A metric sort replaces any other column’s sort. Use the search box to find a provider by name, and the Filter button to narrow the list by provider type, model, or status. The Add provider button opens the create flow described in Open the Create LLM provider page. Each row’s actions menu can enable or disable the provider, copy its proxy URL, or delete it, and offers to copy the upstream base URL for an OpenAI or OpenAI-compatible provider. Disabling asks you to confirm, and deleting asks you to type the provider’s display name. The list paginates, with a rows-per-page selector in the footer.
View cost and usage
The Cost and usage page reports spend, requests, and tokens for a time window against the period before it, breaks spend down by agent, model, provider, user, or tag, charts any one of them against the gateway total, and exports CSV. See Analyze Cost and Usage.
Edit, disable, or delete a provider
-
Edit: Open the Settings tab and change any field except the provider type and resource ID, which are immutable after creation. The display name, credential reference, base URL, and model list can all change. When you edit a field, a save bar appears; click Save changes to apply your edits.
-
Disable: On the Settings tab, click Disable provider and confirm. The provider remains in the list, but requests to its proxy URL are rejected until you enable it again. Use this when you want to pause traffic without losing configuration.
-
Delete: On the Settings tab, use the Delete this provider danger zone, click Delete provider, and type
deleteto confirm. The action is permanent. In-flight requests fail and downstream clients receive errors until reconfigured.
Troubleshooting
| Symptom | What to check |
|---|---|
|
Confirm the secret exists in the Agentic Data Plane secret store and the reference in the provider configuration is spelled identically ( |
Bedrock returns |
These come from the provider’s AWS region, its IAM policy, or the model identifier you called. See Bedrock troubleshooting. |
The upstream provider returns 401 when passthrough is enabled |
Confirm the client is sending its own |
A passthrough request fails with HTTP 400 and |
With passthrough on, the |
A passthrough request fails with HTTP 502 and |
The upstream endpoint answered with a redirect, which AI Gateway refuses to follow when passthrough is on so that a client’s credentials never travel to another host. Point the provider’s |
Gemini returns 401 |
The client didn’t send its gateway token in a header AI Gateway reads. Send it in |
Requests fail with HTTP 403 and |
The requested model isn’t enabled on the provider. Enable it on the provider’s Models tab, or request a model that is enabled. |
Provider list empty or 403 |
Confirm your account can act on LLM providers. Listing them needs |
Limitations
AI Gateway does not provide these capabilities. For current status, see the Agentic Data Plane release notes.
-
Multi-provider routing, failover, and retries across providers. A synthetic provider that fans requests to multiple upstreams is not part of AI Gateway.
-
Rate limits. Requests-per-second, per-minute, or per-day limits are not available. To cap spend rather than request rate, use budgets, which enforce a per-agent hard cap.
