Agentic Data Plane

Configure an LLM Provider

Create an LLM provider to give your applications a managed proxy URL: Redpanda handles the upstream API keys, forwards requests to the provider, and records usage for you. Create a provider for each upstream you use, whether that’s OpenAI, Anthropic, Google AI, AWS Bedrock, or an OpenAI-compatible endpoint.

After reading this page, you will be able to:

  • Create an LLM provider for OpenAI, Anthropic, Google AI, AWS Bedrock, or an OpenAI-compatible endpoint

  • Select the models you want to expose through the provider

  • Verify the provider is reachable using the built-in Test connection control

Prerequisites

  • An API key (or AWS credentials for Bedrock) for the upstream provider you want to configure.

  • Optional: Secrets already created in your dataplane’s secret store for the provider’s credentials. The create form can also store a new API key inline, so you don’t have to prepare a secret first. Secret identifiers use UPPER_SNAKE_CASE. For example: OPENAI_API_KEY, ANTHROPIC_API_KEY, AWS_ACCESS_KEY_ID.

Open the Create LLM provider page

  1. Open LLM Providers in the sidebar.

  2. Click Add provider.

Fill in the identity fields

The Identity section collects the provider’s name. Enter a Display name: a human-readable label shown in dashboards and model selectors, up to 253 characters.

Agentic Data Plane derives the provider’s resource ID from the display name in the background; the create form doesn’t show it. The resource ID is the machine identifier used in API calls and CLI commands: lowercase letters, numbers, and hyphens only (^[a-z][a-z0-9-]*$), up to 63 characters, and immutable after creation. It appears in the proxy URL (/llm/v1/providers/<resource-id>/…​).

Choose a provider type

In the Provider type section, pick the type that matches your upstream.

Type Use when

OpenAI

Proxy GPT, o-series, and embeddings through the OpenAI API. Best when you already hold an OpenAI API key or want the broadest GPT model catalog.

Anthropic

Call Claude Opus, Sonnet, and Haiku directly. Strong at coding, long-context reasoning, and tool use. Supports forwarding client Authorization headers to Anthropic for enterprise and Max-plan subscription passthrough (see Anthropic: Authorization passthrough).

Google AI

Reach Gemini Pro, Flash, and multimodal models through Google AI Studio. Ideal for long-context workloads and image/video inputs.

AWS Bedrock

Invoke foundation models (Claude, Llama, Titan, Nova, Mistral, AI21 Jamba, Gemma) hosted inside your AWS account. Requires an AWS region and credentials (static, STS-assumed role, or the default credential chain). Supports the native Bedrock APIs (InvokeModel, Converse) and an OpenAI-compatible Chat Completions endpoint for gpt-oss and Gemma models. See AWS Bedrock: Inference profiles and IAM for picking the right model identifier, and Set up AWS Bedrock as an LLM provider for a step-by-step IAM and access-key walkthrough.

OpenAI-compatible

Point at any OpenAI-compatible endpoint that ships /v1/chat/completions (vLLM, Ollama, LM Studio, LocalAI, Together, Groq, OpenRouter). Useful for self-hosted models and aggregator gateways. Requires a Base URL. Authentication is optional.

Selecting a type reveals the type-specific configuration fields.

Fill in the type-specific configuration

Each API key reference and credential field points at a secret-store entry, not the secret value itself. On the create form, the Credential section lists the keys already in your dataplane’s secret store: pick one, or select Bring a new API key reference to store a new key inline. Enter a Key name and the API key value, then click Save key. Agentic Data Plane stores the key in the secret store under an identifier derived from the key name, shown under the field; the value is stored encrypted and can’t be viewed again after saving.

When you edit a provider later, the API key reference field offers an Existing tab to pick a stored secret and a New tab to create one inline.

  • OpenAI

  • Anthropic

  • Google AI

  • AWS Bedrock

  • OpenAI-compatible

Field Notes

Base URL

Optional. Leave empty for the standard OpenAI API (https://api.openai.com/v1). Override for Azure OpenAI or other OpenAI-hosted endpoints.

API key reference

Required. Secret-store reference for the OpenAI API key. Must be UPPER_SNAKE_CASE, for example OPENAI_API_KEY.
Field Notes

Base URL

Optional. Leave empty for the standard Anthropic API (https://api.anthropic.com).

API key reference

Required unless Authorization passthrough is on. UPPER_SNAKE_CASE, for example ANTHROPIC_API_KEY.

Authorization passthrough

Optional toggle. When on, AI Gateway forwards the client’s Authorization header to Anthropic instead of using a server-side API key. Used for enterprise and Max-plan OAuth passthrough: each client authenticates with its own Anthropic subscription. Leave the API key reference empty when using passthrough.
Field Notes

Base URL

Optional. Leave empty for the standard Google AI API (https://generativelanguage.googleapis.com).

API key reference

Required. Secret-store reference for the Google AI API key. UPPER_SNAKE_CASE, for example GOOGLE_AI_API_KEY.

Gemini uses the x-goog-api-key header for authentication, not Authorization: Bearer. This matters when you wire up clients. See Connect your app to AI Gateway.
Field Notes

Region

Required. AWS region where the Bedrock endpoint is deployed, for example us-east-1.

Base URL

Optional. Override the default regional Bedrock endpoint.

Credential type

How AI Gateway authenticates to Bedrock: Default chain, Static keys, or Assume IAM role. The fields below depend on the mode you pick.

Access key ID reference

Static keys only. Secret-store reference for the AWS access key ID, UPPER_SNAKE_CASE (typically AWS_ACCESS_KEY_ID).

Secret access key reference

Static keys only. Secret-store reference for the AWS secret access key, UPPER_SNAKE_CASE (typically AWS_SECRET_ACCESS_KEY).

Role ARN

Assume IAM role only. Required. ARN of the IAM role AI Gateway assumes through AWS STS, for example arn:aws:iam::123456789012:role/BedrockRole.

External ID

Assume IAM role only. Optional. External ID for cross-account role assumption. Set it only when the role’s trust policy mandates an external ID.

Session name

Assume IAM role only. Optional. Session name that appears in AWS CloudTrail audit logs, for example redpanda-adp.

Guardrail

Optional. Name of a guardrail to attach to this provider, or empty for none. Only the Bedrock provider type exposes this setting. AI Gateway validates the name when you save: it rejects a guardrail that doesn’t exist or is being deleted, so set the field to an existing guardrail or leave it empty. See Create a guardrail.

Pick a Credential type to control how AI Gateway authenticates to Bedrock:

  • Default chain (default): Leave the credentials unset to use the AWS SDK’s default provider chain (environment variables, shared config, EKS Pod Identity, IRSA, or instance profile). Use this when the gateway already runs with an AWS identity.

  • Static keys: An access key pair stored in the secret store. Use this when no ambient AWS identity is available. This is the path the Bedrock setup guide walks through.

  • Assume IAM role: AI Gateway assumes an IAM role through AWS STS. Use this for cross-account access or when your security policy requires short-lived credentials.

Field Notes

Base URL

Required. URL of your OpenAI-compatible endpoint, for example http://vllm.internal:8000/v1, http://ollama.local:11434/v1, or an aggregator like Together / Groq / OpenRouter.

API key reference

Optional. Leave empty for endpoints with no authentication (common for local runtimes). UPPER_SNAKE_CASE if set.
OpenAI-compatible endpoints can serve any model. Enter the exact model identifiers your upstream server exposes (for example, meta-llama/Llama-3.3-70B-Instruct or qwen3:8b).

For the OpenAI, Google AI, and AWS Bedrock provider types, AI Gateway validates that the credential references resolve before it accepts the create or update. AI Gateway rejects a missing or empty secret reference at save time instead of failing at first call. The OpenAI-compatible type does not require a credential reference, so it can be created with no authentication for local runtimes such as Ollama or vLLM.

Select models

Models you select on this form become the catalog the provider exposes.

For OpenAI, Anthropic, Google AI, and AWS Bedrock, the form shows a picker backed by the provider’s catalog, with the full catalog selected by default. Keep the default, or deselect models to narrow what the provider exposes. Each model in the picker shows its input and output price per million tokens. Pick from the list, or type a model identifier the catalog doesn’t show. For OpenAI-compatible, the form takes a freeform list: type the exact identifiers your upstream serves.

Redpanda maintains the catalog of available models in the picker. When an upstream provider publishes a new model, it usually appears in the picker within a day or two; admins don’t have to wait for a Redpanda release. New models aren’t enabled automatically: an admin still selects the model in the catalog to make it callable through this provider.

For Bedrock, the picker exposes inference profiles, not raw foundation-model IDs. See AWS Bedrock: Inference profiles and IAM.

Redpanda stores models as structured ProviderModel entries (one entry per model, with the model name as the only required field). Each model can carry custom pricing overrides that replace the catalog rates for that model in cost reporting; see Override per-model pricing. The legacy flat models field still works on writes for backward compatibility.

Beyond pricing, the catalog carries each model’s capabilities and context-window limits. The model discovery API (the ListModels and GetModel methods on ModelService) reports max_input_tokens (the largest context the model accepts) and max_output_tokens (the most it can generate in a single response) for each model. Both are read-only catalog metadata: a limit the catalog doesn’t declare stays unset rather than reported as zero. Clients read max_input_tokens to show how full an agent’s context window is, such as the context-window indicator on the agent’s Playground tab.

Override per-model pricing

Cost reporting prices each call at the catalog rates for the model. If your organization negotiates non-standard rates, or you track spend against an internal chargeback rate, override the rates per model on this provider.

In the model picker, each selected model carries a pencil icon (Override pricing). Click it to open the pricing dialog for that model. The dialog lists one field per billing bucket, in the same order as the provider’s published rate card:

Bucket What it bills

Input

Per 1M input tokens. Tool-use input also bills at this rate.

Output

Per 1M output tokens. Reasoning tokens also bill at this rate.

Cached input

Per 1M tokens read from prompt cache.

Cache write (5-minute TTL)

Per 1M tokens written to a 5-minute prompt cache.

Cache write (1-hour TTL)

Per 1M tokens written to a 1-hour prompt cache.

Enter rates in dollars per million tokens. Each field is independent:

  • Leave a field blank to keep the catalog rate for that bucket. The catalog rate shows as the field’s placeholder.

  • Enter a positive value to replace the catalog rate for that bucket only.

  • Enter 0 to make that bucket explicitly free, which is different from leaving it blank.

Cache writes with an unknown TTL always bill at the catalog rate; they have no override field.

Use the reset control on a field to clear a single override, or clear every field to drop all overrides for the model. Overrides are scoped to this provider and model, and they change what Agentic Data Plane’s cost reporting computes, not what the upstream provider actually charges you.

Explore the provider detail page

After you create the provider, its detail page organizes everything about it into tabs: Overview, Models, Connect, Playground (when enabled for your organization), and Settings.

Overview

The Overview tab opens on a row of stat tiles: Models enabled (the models configured on this provider), Requests over the last 24 hours, Spend over the last 30 days, and Health over the last 24 hours, shown with a status indicator that reads healthy, degraded, failing, or idle based on the provider’s recent request and error volume.

Below the tiles, a Gateway traffic chart plots request and token volume over time next to the Connection card. The Connection card shows the provider type, an Active or Disabled status badge, the authentication passthrough state, the proxy URL, the upstream base URL, the API key secret reference, and any attached guardrail, each with a copy control where relevant.

A Who calls what card then draws the caller-to-model traffic for a time range you pick with the range selector, and a Callers table lists, over the same range, each agent or user that called the provider, with its request count, the models it used, its error count, spend, and when it was last seen. Search the table by caller name, slug, or email, and select a row to open that caller’s details on the Cost & Usage page. For analysis across providers, use the Cost & Usage page under Governance (see View cost and usage).

Models

The Models tab lists the chat models available on this provider. Each row shows the model’s name and identifier, icons for the capabilities it supports (such as vision, tools, JSON, or reasoning), its context-window limit, and its input and output prices per million tokens. Hover over a capability icon to read what that capability means. Use the toggle on a row to enable or disable that model on the provider. Select a model to open its detail page (see View a model’s detail page).

Enabled models sort to the top. Use the search box to find a model by display name or identifier, and the Status and Capability filters to narrow the list.

Connect

The Connect tab generates ready-made client configuration for this provider: a gateway-token step, ready-to-run rpk ai setup steps, setup instructions for popular clients such as Claude Code, and code examples in several languages, all with the provider’s proxy URL prefilled. For a Bedrock provider, the Claude Code snippets run against the gateway without local AWS credentials, because the gateway signs the upstream AWS requests for you. See Connect your app to AI Gateway for the underlying flow.

Playground

When the Playground tab is available, use it to test one of the provider’s models before you connect application code. Choose a model, send a prompt, and optionally adjust response controls such as reasoning effort and maximum output tokens. The response streams into the conversation while the Playground shows the request, response events, status, and timing. Playground conversations are ephemeral.

LLM Playground test loop. A user selects a model, writes a prompt, and chooses response controls. AI Gateway translates and sends the request to the configured provider, streams the response back, and exposes request details, response events, status, and timing in the Playground.
Figure 1. Playground turns a model test into an inspectable gateway request

Settings

The Settings tab is where you edit the provider after you create it. It is organized into sections:

  • General: Edit the display name. The provider type and resource ID are fixed after creation and shown read-only.

  • Credential: Change how the gateway authenticates to the upstream provider. To rotate a stored key’s value for every provider that uses it, follow the link to Provider keys in the Secret Store.

  • Connection: The read-only proxy URL and the editable upstream base URL. Use Test connection to confirm the provider is reachable before you save.

  • Models: Choose which models this provider exposes to agents and developers.

  • Transcripts: Turn message-body recording on or off (see Configure transcript logging).

When you change a field, a save bar appears with Save changes and Discard. Below the form, an enable/disable control stops or resumes routing without deleting anything, and a Delete this provider danger zone permanently removes the provider after you confirm.

View a model’s detail page

From the Models tab, select a model to open its detail page. The page gathers what the catalog knows about that model on this provider in one place:

  • An overview strip with the model’s context window (largest input it accepts), maximum output (most it can generate in a single response), and price (input and output rates per million tokens). Each value appears only when the catalog declares it.

  • Model details: The model ID and the provider’s proxy URL, each with a copy control.

  • Pricing: A read-only rate card of the model’s effective per-bucket rates, that is, the catalog rates with any per-model overrides applied (see Override per-model pricing). This section appears only when rates are known, and links to the upstream provider’s pricing reference.

  • Capabilities: The model’s capabilities from the catalog, with a link to the upstream provider’s capabilities reference. This section appears only when the catalog declares capabilities.

  • Usage: Spend, Requests, and Tokens cards for traffic routed to this model through this provider over the last 7 days. Select View more on a card to open a metric detail drawer.

Configure transcript logging

The Transcripts section on the provider’s Settings tab controls whether AI Gateway records the message bodies this provider proxies. It has two independent toggles, both off by default:

Toggle What it captures

Record inputs

Captures the full request body (prompt content and tool-call arguments) on observability traces.

Record outputs

Captures the full response body (completion content and tool-call results) on observability traces.

Because both toggles default to off, AI Gateway does not retain message bodies for a new provider until you turn them on. Enable them to power turn-by-turn investigation and per-conversation drill-down in the Transcripts view. Leave them off for workloads where the message body must not be retained, such as regulated PII or customer secrets.

These are per-provider settings, not per-request: applications cannot opt in or out at call time. To split sensitive from non-sensitive traffic, create one provider with recording on and another with it off, and route each application to whichever proxy URL matches its data class.

Recording settings do not affect cost and usage telemetry. Token counts, latency, and provider/model attribution are always recorded, so the Cost & Usage page reports spend for traffic on the provider regardless of these toggles; only the message bodies are withheld when the toggles are off.

Changing a toggle takes effect for new requests. Transcripts already captured under the previous setting are not retroactively redacted; delete or rotate the provider if you need to purge historical content.

Save and verify

  1. Click Create provider. Until the form is complete, the button stays disabled and the hint next to it tells you what’s still needed (for example, saving the new key).

  2. After create, you land on the provider’s Connect tab, with client setup instructions and code examples that have the proxy URL prefilled. To copy the proxy URL on its own, open the Overview tab and use the copy control on the Connection card. The proxy URL is where your applications point.

  3. To confirm the provider is reachable, open the Settings tab and click Test connection in the Connection section. The result appears as a message below the button.

  4. To wire up an application, use the Connect tab or follow Connect your app to AI Gateway.

A successful Test connection result confirms that the provider’s credentials, region (Bedrock), and network path are all correct. If the call fails, see Troubleshooting.

AWS Bedrock: Inference profiles and IAM

Bedrock has three concepts that affect how you configure a provider: foundation models, cross-region inference profiles, and IAM. Get these right and the Test connection check passes. Get them wrong and you see AccessDenied or ValidationException errors.

Foundation models versus inference profiles

A foundation model is the base model AWS exposes (for example, anthropic.claude-sonnet-4-6). It runs in the AWS region you call.

A cross-region inference profile wraps a foundation model with a geography prefix that routes requests across multiple regions for higher availability and throughput. The prefix tells AWS which geography the request should run in:

Prefix Geography

us.

US regions

eu.

EU regions

apac.

Asia-Pacific regions

au.

Australia regions

jp.

Japan regions

global.

Any region; routes for lowest cost

Examples: us.anthropic.claude-sonnet-4-6 (Claude Sonnet 4.6 routed across US regions), eu.anthropic.claude-haiku-4-5 (Haiku 4.5 routed across EU regions).

Anthropic Claude 4.6+ models (Sonnet 4.6, Opus 4.6, Opus 4.7) cannot be invoked with the bare foundation-model ID; they require an inference profile. If you try the bare ID, Bedrock returns:

"Invocation of model ID … with on-demand throughput isn’t supported. Retry your request with the ID or ARN of an inference profile that contains this model."

Older 4.5 and earlier Claude models still accept bare IDs.

Pricing varies by profile. The bare foundation-model ID and the global. profile share AWS’s headline rate; geo profiles (us., eu., apac., au., jp.) carry approximately a 10% cross-region inference premium. Use global. when you want the headline rate and don’t need a specific geography. Use us. / eu. / apac. when data residency matters.

AI Gateway preserves the regional prefix end to end when it records spend, so the Cost & Usage page attributes usage to the correct regional rate. A call to eu.anthropic.claude-haiku-4-5 is billed at the EU Haiku rate, not the headline foundation-model rate.

IAM policy patterns

Bedrock IAM resources have different ARN structures depending on whether you reference a foundation model, a system-defined inference profile, or an account-scoped application inference profile. The provider’s IAM principal needs bedrock:InvokeModel and bedrock:InvokeModelWithResponseStream on every resource it calls.

Resource type ARN shape

Foundation model

arn:aws:bedrock:{region}::foundation-model/{model-id} (no account ID; AWS-owned)

System-defined inference profile

arn:aws:bedrock:{region}:*:inference-profile/{profile-id} (wildcard account; system-defined)

Application inference profile (account-scoped)

arn:aws:bedrock:{region}:{account-id}:application-inference-profile/{profile-id}

A minimal policy granting access to all foundation models plus all cross-region profiles:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": ["bedrock:InvokeModel", "bedrock:InvokeModelWithResponseStream"],
    "Resource": [
      "arn:aws:bedrock:*::foundation-model/*",
      "arn:aws:bedrock:*:*:inference-profile/*"
    ]
  }]
}

For production, scope to specific models and regions instead of using wildcards.

Anthropic: Authorization passthrough

If you want each client to authenticate against Anthropic with its own subscription (Claude Pro, Max, Team, or enterprise), enable Authorization passthrough instead of configuring a server-side API key. In this mode:

  • Leave the API key field empty.

  • Clients must send their own Anthropic Authorization header with every request. AI Gateway forwards it unchanged.

  • Use this when you want to aggregate individual client subscriptions rather than share a single API account.

The provider detail page shows whether Authorization passthrough is enabled in the Connection card.

Browse providers in the list view

The LLM Providers list page is the at-a-glance home for every provider in your dataplane. Open it from the sidebar’s LLM Providers entry.

Column What it shows

Provider

The provider-type icon (OpenAI, Anthropic, Google, AWS Bedrock, or OpenAI-compatible), the display name, and the resource identifier beneath it. Select the provider name to open its detail page.

Status

Active for an enabled provider. A disabled provider rejects requests to its proxy URL until you enable it again.

Models

The model identifiers configured on the provider, shown as chips. A provider with no models configured shows a dash.

24h requests

Request count over the last 24 hours.

30d spend

Spend over the last 30 days. For longer-range or cross-provider analysis, use the Cost & Usage page under Governance (see View cost and usage).

Use the search box to find a provider by name, and the Filter button to narrow the list by provider type, model, or status. The Add provider button opens the create flow described in Open the Create LLM provider page. Each row’s actions menu can activate or deactivate the provider, copy its proxy or base URL, or delete it. The list paginates, with a rows-per-page selector in the footer.

View cost and usage

The Cost & Usage page under Governance tracks spend, request volume, and token volume over time across providers, models, agents, and users, with period comparisons and CSV export. This report moved to its own page: see Analyze Cost and Usage.

Edit, disable, or delete a provider

  • Edit: Open the Settings tab and change any field except the provider type and resource ID, which are immutable after creation. The display name, credential reference, base URL, and model list can all change. When you edit a field, a save bar appears; click Save changes to apply your edits.

  • Disable: On the Settings tab, click Disable provider and confirm. The provider remains in the list, but requests to its proxy URL are rejected until you enable it again. Use this when you want to pause traffic without losing configuration.

  • Delete: On the Settings tab, use the Delete this provider danger zone, click Delete provider, and type delete to confirm. The action is permanent. In-flight requests fail and downstream clients receive errors until reconfigured.

Troubleshooting

Symptom What to check

secret "<NAME>" not found

Confirm the secret exists in your dataplane’s secret store and the reference in the provider configuration is spelled identically (UPPER_SNAKE_CASE, no typos).

Bedrock returns AccessDenied or region errors

Verify the AWS region field matches the region where your Bedrock models are enabled. Bedrock model availability varies by region. Confirm the IAM principal has bedrock:InvokeModel on the foundation-model and inference-profile ARNs you use. See AWS Bedrock: Inference profiles and IAM.

Bedrock returns "Invocation of model ID … with on-demand throughput isn’t supported"

You called a Claude 4.6+ model with a bare foundation-model ID. Switch to an inference profile (for example, us.anthropic.claude-sonnet-4-6 instead of anthropic.claude-sonnet-4-6). See AWS Bedrock: Inference profiles and IAM.

Anthropic returns 401 when passthrough is enabled

Confirm the client is sending its own Authorization header and the API key field on the provider is empty.

Gemini returns 401

Gemini uses the x-goog-api-key header, not Authorization. If you’re seeing 401s on Gemini, check that the client is sending the correct header. See Connect your app to AI Gateway.

Provider list empty or 403

Confirm your account can act on LLM providers. Listing them needs Action::"LLMProvider.list" and creating one needs Action::"LLMProvider.create". Outside the Admin role, both come from an access policy. See LLM provider permissions.

Limitations

AI Gateway does not provide these capabilities. For current status, see the Agentic Data Plane release notes.

  • Multi-provider routing, failover, and retries across providers. A synthetic provider that fans requests to multiple upstreams is not part of AI Gateway.

  • Rate limits. Requests-per-second, per-minute, or per-day limits are not available. To cap spend rather than request rate, use budgets, which enforce a per-agent hard cap.

  • Managed MCP aggregation at the gateway. Register MCP tool servers separately under MCP Servers in Agentic Data Plane.

Next steps