Build an MCP Server in Redpanda Connect

This guide teaches you how to build MCP servers with self-managed Redpanda Connect. MCP servers run as part of your Redpanda Connect deployment and expose tools that AI clients can call using the Model Context Protocol.

Prerequisites

  • The Redpanda CLI (rpk)

  • At least version 4.56.0 of Redpanda Connect

    If you need to upgrade, see Install or Upgrade with rpk.

  • Basic understanding of YAML, HTTP APIs, and event-stream processing concepts

  • (Optional) Claude Desktop or other MCP-compatible AI client for testing

For a quickstart, see Redpanda Connect MCP Server Quickstart.

Concepts and architecture

MCP server

A service that exposes tools through the Model Context Protocol. In Redpanda Connect, this is enabled using the rpk connect mcp-server command to start a dedicated MCP server process.

Tool

A single request/response operation exposed to MCP clients. Each tool is implemented as a Redpanda Connect configuration and described to MCP with meta.mcp metadata.

Contract

The formal interface a tool exposes to AI clients, including its description, input parameters and types. The contract ensures that clients know how to call the tool and what results to expect. In Redpanda Connect MCP, the contract is declared in the tool’s meta.mcp metadata.

Redpanda Connect configuration

A YAML file that defines how data flows through inputs, processors, and outputs. When used as an MCP tool, Redpanda Connect configurations typically receive JSON input and return structured results.

Secrets

Credentials and tokens should be stored as environment variables and referenced as ${ENV_VAR}. Never hardcode secrets in YAML files.

Development workflow

  1. Initialize the project: Use the Redpanda CLI to scaffold your project.

  2. Design the tool contract: Annotate MCP metadata with meta.mcp.enabled: true, provide a concise description, and declare parameters.

  3. Implement the configuration: Build the logic for your MCP tool using Redpanda Connect configurations.

  4. Lint and start the MCP server: Launch your server using rpk connect mcp-server to expose your tools.

  5. Connect your AI assistants: Set up Claude Code or other MCP clients to use your tools.

  6. Publish your MCP server (optional): Make your MCP server discoverable by AI clients and agents by publishing it to an MCP registry. This enables authentication and access control for your tools.

Initialize a new MCP server project

  1. Initialize a new MCP server project using the Redpanda CLI. This sets up the recommended folder structure and example configuration files.

    rpk connect mcp-server init

    This command scaffolds your project with the necessary directories and template YAML files for defining tools and observability resources.

    ├── o11y
│   ├── metrics.yaml
│   └── tracer.yaml
└── resources
    ├── caches
    │   └── example-cache.yaml
    ├── inputs
    │   └── example-input.yaml
    ├── outputs
    │   └── example-output.yaml
    └── processors
        └── example-processor.yaml

    The resources directory is where you define your tools such as inputs, outputs, and processors. Each tool is defined in its own YAML file. By default, the example- files are provided as templates. The o11y directory contains configuration for observability, including metrics and tracing.

  2. Remove the example- files and add your own tools:

    rm resources/inputs/example-input.yaml
rm resources/outputs/example-output.yaml
rm resources/processors/example-processor.yaml
rm resources/caches/example-cache.yaml

    Then, you can create new YAML files for each tool you want to expose. Make sure each file contains only one component type (input, output, processor, or cache). Put inputs in the resources/inputs/ directory, outputs in resources/outputs/, processors in resources/processors/, and caches in resources/caches/.

See the next sections for details on customizing these files for your use case.

Design the tool contract and MCP metadata

Each MCP tool must declare its interface using meta.mcp metadata. This metadata allows AI clients to discover and invoke the tool correctly.

Define a clear, stable interface for each tool. Keep the description task-oriented and keep parameters to a minimum.

meta:
  mcp:
    enabled: true (1)
    description: "Fetches a compact summary from an external API using two optional parameters." (2)
    properties: (3)
      - name: parameter1
        type: string
        description: "Primary filter; defaults to provider standard when omitted."
        required: false
      - name: parameter2
        type: number
        description: "Limit of results (1-100)."
        required: false
1 Set meta.mcp.enabled: true to expose the tool using MCP.
2 Add a concise description that explains what the tool does. The description is passed as a tool option, making it available to clients and documentation. This should be understandable by an AI model.
3 List the input parameters (properties) for the tool.

Property guidance:

  • Use string, number, or boolean types.

  • Validate ranges and enums using Bloblang.

  • Mark only mandatory fields as required.

  • Document defaults in the description and enforce them in the configuration.

After defining your tool contract, implement the configuration to handle input validation, defaults, and the main processing steps.

Implement the configuration

Use Redpanda Connect components to implement the logic of your MCP tool. Here are some best practices:

  • Single responsibility: Each tool should do one thing well.

  • Descriptive naming: Use clear, specific labels like fetch-user-profile instead of generic names like get-data. The top-level label becomes the tool’s name in the MCP server.

  • Input validation: Always validate and sanitize user inputs using Bloblang.

  • Error handling: Provide meaningful error messages.

  • Documentation: Write clear descriptions that explain what the tool does and what it returns.

For common patterns and examples, see Redpanda Connect Patterns for MCP Servers.

Here’s a complete example that demonstrates best practices:

label: weather-service
processors:
  - label: validate_inputs
    mutation: |
      # Validate and sanitize city input
      meta city = this.city.string().
        re_replace_all("[^a-zA-Z\\s\\-]", "").
        trim()

      # Check for empty input
      root = if @city == "" {
        throw("City name cannot be empty")
      } else { "" }

  - label: fetch_weather_data
    try:
      - http:
          url: "https://wttr.in/${! @city }?format=j1"
          verb: GET
          headers:
            User-Agent: "redpanda-connect-mcp/1.0"
          timeout: "10s"
      - mutation: |
          root = {
            "city": @city,
            "temperature_c": this.current_condition.0.temp_C.number(),
            "temperature_f": this.current_condition.0.temp_F.number(),
            "feels_like_c": this.current_condition.0.FeelsLikeC.number(),
            "humidity": this.current_condition.0.humidity.number(),
            "description": this.current_condition.0.weatherDesc.0.value,
            "wind_speed_kmh": this.current_condition.0.windspeedKmph.number(),
            "timestamp": now().format_timestamp("2006-01-02T15:04:05Z07:00")
          }
      - log:
          message: "Weather data fetched for city: ${! @city }"
          level: "INFO"

  - label: handle_weather_errors
    catch:
      - mutation: |
          root = {
            "error": "Failed to fetch weather data",
            "city": @city,
            "details": error(),
            "timestamp": now().format_timestamp("2006-01-02T15:04:05Z07:00")
          }
      - log:
          message: "Weather API error for city ${! @city }: ${! error() }"
          level: "ERROR"

meta:
  tags: [ weather, example ]
  mcp:
    enabled: true
    description: "Get current weather conditions for any city worldwide"
    properties:
      - name: city
        type: string
        description: "Name of the city (such as 'San Francisco', 'London')"
        required: true

YAML configuration rules

Each YAML file should contain exactly one component type. The component type is inferred from the directory structure:

Directory Component Type

resources/inputs/

Input component

resources/outputs/

Output component

resources/processors/

Processor component

resources/caches/

Cache component

Property restrictions by component type

Different component types have different property capabilities when exposed as MCP tools:

Component Type Property Support Details

input

Only supports count property

AI clients can specify how many messages to read, but you cannot define custom properties.

cache

No custom properties

Properties are hardcoded to key and value for cache operations.

output

Custom properties supported

AI sees properties as an array for batch operations: [{prop1, prop2}, {prop1, prop2}].

processor

Custom properties supported

You can define any properties needed for data processing operations.
Correct example when inside resources/inputs/
label: event-reader
redpanda:
  seed_brokers: [ "${REDPANDA_BROKERS}" ]
  topics: [ "events" ]
  consumer_group: "mcp-reader"

meta:
  mcp:
    enabled: true
    description: "Consume events from Redpanda"
Correct example when inside resources/processors/
label: fetch-example-data
processors:
  - label: safe_operation
    try:
      - http:
          url: "https://api.example.com/data"
          timeout: "10s"
      - mutation: |
          root = this.merge({"processed": true})

  - label: handle_errors
    catch:
      - mutation: |
          root = {
            "error": "Operation failed",
            "details": error()
          }
Incorrect (do not include the input wrapper)
label: incorrect-example
input:
  redpanda:
    seed_brokers: [ "${REDPANDA_BROKERS}" ]
    topics: [ "events" ]
Incorrect (multiple component types in one file)
label: incorrect-example
input:
  redpanda: { ... }
processors:
  - mutation: { ... }
output:
  redpanda: { ... }
Incorrect (try/catch as single processor)
label: incorrect-example
processors:
  - label: operation
    try:
      - http: { ... }
    catch:
      - mutation: { ... }

Start the MCP server

  1. Navigate to the directory where you initialized the MCP server project.

  2. Lint your configuration files before starting the server:

    rpk connect mcp-server lint

    This command checks all YAML files in your resources directory for errors or misconfigurations. Fix any issues reported before proceeding.

  3. Start the MCP server to expose all your tools over HTTP:

    rpk connect mcp-server --address localhost:4195

    This command creates an MCP server listening on localhost:4195.

    You should see output like this:

    time=2025-06-27T15:20:27.976+01:00 level=INFO msg="Registering processor tool" label=...
time=2025-06-27T15:20:27.978+01:00 level=INFO msg="Successfully loaded Redpanda license" expires_at=2035-06-25T15:20:27+01:00 license_org="" license_type="open source"

Expose a subset of tools

To expose only a subset of tools, use the --tag flag. This helps you:

  • Keep experiments isolated.

  • Avoid exposing sensitive functionality accidentally.

  • Create sets of tools that are relevant to specific agents or workflows.

For example, to expose only tools tagged with example, start the server with:

rpk connect mcp-server --address localhost:4195 --tag example

Connect an MCP client

You can connect any MCP-compatible client to your MCP server using the server’s HTTP endpoint. Most clients require the server address and may support authentication or custom headers.

To connect a generic MCP client:

  1. Ensure your MCP server is running and accessible at the desired address (for example, http://localhost:4195).

  2. Configure your client with the MCP server’s endpoint. For example: http://localhost:4195/sse.

Refer to your client’s documentation for details on supported options and advanced configuration. For a list of example clients, see the MCP documentation.

Connect Claude Code to your MCP server

To connect Claude Code to your MCP server, you need to expose a live event stream that Claude can consume. This is done using the mcp-remote utility, which bridges your local service to Claude’s MCP interface. mcp-remote is a lightweight bridge that turns any streaming HTTP endpoint into a source of MCP-compatible messages.

  1. To install mcp-remote, run:

    claude mcp add local -- npx mcp-remote http://localhost:4195/sse

    You should see output like this:

    Added stdio MCP server local with command: npx mcp-remote http://localhost:4195/sse to local config

    • claude mcp add local --

      This tells Claude to set up a new local input channel, which is a subprocess or pipe that streams messages. The -- delimiter ensures that everything after it is treated as a shell command to execute.

    • npx mcp-remote http://localhost:4195/sse

      This runs the mcp-remote utility, which:

      • Connects to the provided SSE endpoint

      • Converts events into MCP message format

      • Writes them to stdout

      Claude reads these messages from the subprocess, treating them as if they were emitted by a native MCP agent.

  2. Verify that the local input channel is set up correctly by running:

    claude /mcp

    You should see an entry for local with the command you just added.

    Press Enter until you see the tools list.

    Tools for local (1 tools)
│ ❯ 1. search-bluesky-posts

In summary, the flow is:

  • Your MCP server exposes a Server-Sent Events (SSE) stream at http://localhost:4195/sse.

  • mcp-remote connects to that stream and reads the events.

  • mcp-remote converts each event into a structured MCP message.

  • Those messages are forwarded to Claude Code through the local input channel created by claude mcp add local.

Publish your MCP server

To make your MCP server discoverable by AI clients and agents, you can publish it to an MCP registry. This also enables you to provide authentication and access control for your tools.

See the official guide for publishing and securing your server: MCP Registry Publishing Guide.

Troubleshooting

This section covers common issues and debugging techniques when developing MCP tools with Redpanda Connect.

Tool not appearing in MCP client

  • Check that meta.mcp.enabled: true is set.

  • Verify the MCP server is running on the expected port.

  • Ensure the tool has the correct tag specified in the server startup command.

Unable to infer component type

If you see errors like:

resources/inputs/redpanda-consume.yaml(1,1) unable to infer component type: [input processors cache_resources meta]
resources/outputs/redpanda-publish.yaml(1,1) unable to infer component type: [processors output meta]

This means your YAML file contains more than one component type, or uses a wrapper (such as input: or output:) that is not allowed. Each YAML file must contain only a single component type, and should not be wrapped in an input: or output: block.

To fix this:

  • Split out each component type into its own file (for example, one file for the input, one for the processors, one for the output).

  • See YAML configuration best practices for correct examples.

Example of incorrect YAML
input:
  redpanda: { ... }
processors:
  - mutation: { ... }
output:
  redpanda: { ... }
Example of correct YAML (for an input)
label: my_input
redpanda:
  seed_brokers: [ "${REDPANDA_BROKERS}" ]
  topics: [ "events" ]
  consumer_group: "mcp-reader"
meta:
  mcp:
    enabled: true
    description: "Consume events from Redpanda"

Suggested reading