Build Remote MCP Servers in Redpanda Cloud

beta

This guide teaches you how to build Remote MCP servers that are managed in Redpanda Cloud. Remote MCP servers run inside your Redpanda Cloud cluster and expose tools that AI clients can call using MCP.

Prerequisites

For a quickstart, see Remote MCP Server Quickstart.

You should also be familiar with YAML, HTTP APIs, and basic event-stream processing concepts.

Concepts and architecture

Remote MCP server

A managed, hosted MCP service deployed to your cluster. You write each tool’s logic using Redpanda Connect configuration and annotate them with MCP metadata so clients can discover and invoke them.

Redpanda Connect

A framework for building event-driven data pipelines. Tools inside your Remote MCP servers are implemented as Redpanda Connect configurations that run inside your Redpanda Cloud cluster.

Tool

A single request/response operation exposed to MCP.

Secrets

Credentials and tokens that you want to use in your tools. These values must be stored in the Secrets Store and referenced as ${secrets.NAME}. Use uppercase snakecase for secret names (such as DATAPLANE_TOKEN).

Development workflow

  1. Access your Redpanda Cloud cluster: Select a Redpanda Cloud cluster to host your MCP server.

  2. Design the tool contract: Annotate MCP metadata with meta.mcp.enabled: true, provide a concise description, and declare parameters.

  3. Provision secrets (optional): Create secrets in the Secrets Store for any credentials. Never inline secrets in YAML.

  4. Implement the logic: Build the logic for your MCP tool using Redpanda Connect configurations.

  5. Lint and deploy: Validate your configuration, catch schema errors, then deploy and test with the MCP Inspector.

  6. Authenticate and connect your MCP client: Install a proxy entry with rpk cloud mcp proxy in your AI assistant MCP configuration, or connect with your own MCP client.

  7. Observe and debug: Add temporary log processors, adjust timeouts/retries, and right-size resources.

Access MCP Servers in Redpanda Cloud

  1. Log into your Redpanda Cloud Console.

  2. Select your cluster from the cluster list.

  3. In the left sidebar, navigate to Remote MCP.

  4. If this is your first MCP server, you’ll see an empty list with a Create MCP Server button.

  5. Fill in the required fields:

    • Name: A unique identifier for your server (for example weather-tool, customer-lookup).

    • Description: Brief description of what your server does.

    • Configuration: This is where to put your YAML configuration to define an MCP tool.

Design the tool contract and MCP metadata

Each MCP tool must declare its interface using meta.mcp metadata. This metadata allows AI clients to discover and invoke the tool correctly.

Define a clear, stable interface for each tool. Keep the description task-oriented and keep properties to a minimum.

meta:
  mcp:
    enabled: true (1)
    description: "Publishes a message to a specified Redpanda topic" (2)
    properties: (3)
      - name: message
        type: string
        description: "The message content to publish"
        required: true
      - name: topic_name
        type: string
        description: "The Redpanda topic to publish to"
        required: true
1 Set meta.mcp.enabled: true to expose the tool using MCP.
2 A concise description that explains what the tool does. This should be understandable by an AI model.
3 Define the properties that AI clients can provide when calling this tool.

Properties define the inputs that AI clients can provide when calling your tool. Each property has:

  • name: The parameter name AI clients will use

  • type: Data type (string, number, or boolean)

  • description: Human-readable explanation of what this property does

  • required: Whether the property is mandatory (true) or optional (false)

Property guidance:

  • Use descriptive names that clearly indicate the property’s purpose.

  • Provide clear descriptions that explain both the purpose and any constraints.

  • Mark only mandatory fields as required. Prefer optional properties with sensible defaults.

  • Document default values in the description and implement them in your tool logic.

  • Consider component type restrictions: input components only support a count property, cache components have no custom properties, and output components receive properties as arrays for batch operations.

After defining your tool contract, implement the logic to handle input validation, defaults, and the main processing steps.

Provision secrets

All credentials and sensitive values must be stored in the Redpanda Cloud Secrets Store. Follow these best practices:

  • Reference secrets as ${secrets.NAME} in your YAML configuration.

  • Never commit secrets to Git or reference them directly inline in configuration files.

  • Use uppercase snakecase for secret names (such as DATAPLANE_TOKEN).

  • Rotate secrets in the Secrets Store as needed.

  • Only request the scopes/roles required by your tool (principle of least privilege).

See an example of using secrets in MCP patterns.

Implement the logic of your MCP tool

Use Redpanda Connect components to implement the logic of your MCP tool. Here are some best practices:

  • Single responsibility: Each tool should do one thing well.

  • Descriptive naming: Use clear, specific labels like fetch-user-profile instead of generic names like get-data. The top-level label becomes the tool’s name in the MCP server.

  • Input validation: Always validate and sanitize user inputs using Bloblang.

  • Error handling: Provide meaningful error messages.

  • Documentation: Write clear descriptions that explain what the tool does and what it returns.

For common patterns and examples, see Patterns for Remote MCP Servers.

Here’s a complete example that demonstrates best practices:

label: weather-service
processors:
  - label: validate_inputs
    mutation: |
      # Validate and sanitize city input
      meta city = this.city.string().
        re_replace_all("[^a-zA-Z\\s\\-]", "").
        trim()

      # Check for empty input
      root = if @city == "" {
        throw("City name cannot be empty")
      } else { "" }

  - label: fetch_weather_data
    try:
      - http:
          url: "https://wttr.in/${! @city }?format=j1"
          verb: GET
          headers:
            User-Agent: "redpanda-connect-mcp/1.0"
          timeout: "10s"
      - mutation: |
          root = {
            "city": @city,
            "temperature_c": this.current_condition.0.temp_C.number(),
            "temperature_f": this.current_condition.0.temp_F.number(),
            "feels_like_c": this.current_condition.0.FeelsLikeC.number(),
            "humidity": this.current_condition.0.humidity.number(),
            "description": this.current_condition.0.weatherDesc.0.value,
            "wind_speed_kmh": this.current_condition.0.windspeedKmph.number(),
            "timestamp": now().format_timestamp("2006-01-02T15:04:05Z07:00")
          }
      - log:
          message: "Weather data fetched for city: ${! @city }"
          level: "INFO"

  - label: handle_weather_errors
    catch:
      - mutation: |
          root = {
            "error": "Failed to fetch weather data",
            "city": @city,
            "details": error(),
            "timestamp": now().format_timestamp("2006-01-02T15:04:05Z07:00")
          }
      - log:
          message: "Weather API error for city ${! @city }: ${! error() }"
          level: "ERROR"

meta:
  tags: [ weather, example ]
  mcp:
    enabled: true
    description: "Get current weather conditions for any city worldwide"
    properties:
      - name: city
        type: string
        description: "Name of the city (such as 'San Francisco', 'London')"
        required: true

YAML configuration rules

Each YAML configuration (tool) should contain exactly one component type. The component type is inferred from the type you choose in the dropdown when creating or editing the MCP server. Valid component types are:

  • input (for data sources)

  • output (for data sinks)

  • processor (for data transformations and data access)

  • cache (for caching intermediate results)

Property restrictions by component type

Different component types have different property capabilities when exposed as MCP tools:

Component Type Property Support Details

input

Only supports count property

AI clients can specify how many messages to read, but you cannot define custom properties.

cache

No custom properties

Properties are hardcoded to key and value for cache operations.

output

Custom properties supported

AI sees properties as an array for batch operations: [{prop1, prop2}, {prop1, prop2}].

processor

Custom properties supported

You can define any properties needed for data processing operations.

Configuration examples

Correct example
label: event-reader
redpanda:
  seed_brokers: [ "${REDPANDA_BROKERS}" ]
  topics: [ "events" ]
  consumer_group: "mcp-reader"

meta:
  mcp:
    enabled: true
    description: "Consume events from Redpanda"
Correct example
label: fetch-example-data
processors:
  - label: safe_operation
    try:
      - http:
          url: "https://api.example.com/data"
          timeout: "10s"
      - mutation: |
          root = this.merge({"processed": true})

  - label: handle_errors
    catch:
      - mutation: |
          root = {
            "error": "Operation failed",
            "details": error()
          }
Incorrect (do not include the input wrapper)
label: incorrect-example
input:
  redpanda:
    seed_brokers: [ "${REDPANDA_BROKERS}" ]
    topics: [ "events" ]
Incorrect (multiple component types in one file)
label: incorrect-example
input:
  redpanda: { ... }
processors:
  - mutation: { ... }
output:
  redpanda: { ... }

Lint and deploy your MCP server

Lint your configuration

Before deploying, validate your YAML configuration to catch syntax errors and schema violations:

  1. In the Redpanda Cloud Console, after entering your YAML configuration, click Lint.

  2. The linter checks for:

    • Valid YAML syntax

    • Correct Redpanda Connect component schemas

    • Required MCP metadata fields

    • Secret reference formatting

    • Component configuration completeness

  3. Fix any errors shown in the linting output before proceeding to deployment.

  4. A successful lint shows no errors and confirms your configuration is ready to deploy.

Always lint your configuration after making changes. This catches issues early and prevents deployment failures.

Deploy your MCP server

  1. Click Create MCP Server to deploy the server.

  2. Monitor the Status column. It will show one of:

    • Starting: Server is being deployed.

    • Running: Server is active and ready to receive requests.

    • Error: Check logs for deployment issues.

Test with MCP Inspector

  1. When your server status shows Running, click on the server name.

  2. Navigate to the MCP Inspector tab.

  3. This built-in tool lets you:

    • View all available tools your server exposes.

    • Test tool calls with sample parameters.

    • See request/response data.

    • Debug issues without setting up external clients.

Manage your MCP server

From the MCP Servers list, you can:

  • View logs: Click the server name, then go to the Logs tab to see execution logs and errors.

  • Update configuration: Edit the YAML and redeploy.

  • Start/Stop: Control server lifecycle.

  • Delete: Remove the server and clean up resources.

See Manage Remote MCP Servers.

Authenticate and connect your MCP client

To connect your local MCP client to your Remote MCP server:

  1. First, authenticate to Redpanda Cloud:

    rpk cloud login

    This opens a browser window for sign-in and stores your authentication token locally.

  2. Install the MCP proxy for your client (Claude/Claude Code).

    For BYOC and Dedicated clusters, use:

    rpk cloud mcp proxy \
--cluster-id <cluster-id> \ (1)
--mcp-server-id <server-id> \ (2)
--install --client claude-code

    For Serverless clusters, use:

    rpk cloud mcp proxy \
--serverless-cluster-id <cluster-id> \ (1)
--mcp-server-id <server-id> \ (2)
--install --client claude-code
    1 The target Redpanda Cloud cluster ID where your Remote MCP server is hosted.
    2 The unique ID of your deployed Remote MCP server
    You can find this command and the IDs in the Connection tab of your MCP server in Redpanda Cloud.

The proxy acts as a bridge between your local client and the Remote MCP server running in your cluster. It:

  • Connects to your Remote MCP server using your authentication token

  • Discovers and registers all tools from the Remote server locally

  • Proxies MCP requests from your client to the Remote server

  • Handles authentication by injecting your token into each request

    1. Restart your client and invoke your tool.

You can implement the auth workflow directly against Redpanda Cloud APIs and skip the use of rpk. The proxy is a convenience, not a requirement of Remote MCP. For code examples, see the Connection tab in your MCP server in Redpanda Cloud.

Observe and debug your MCP tools

You can view execution logs in the Redpanda Cloud Console under your MCP server’s Logs tab.

Suggested reading