Redpanda Connect MCP Server Quickstart

This quickstart builds a local MCP server that exposes Redpanda Connect configurations as tools that AI assistants can use. It creates a tool that searches Bluesky posts, then asks Claude Code to perform the search using natural language:

Find recent posts on Bluesky mentioning Redpanda.

What you’ll learn

By completing this quickstart, you’ll understand:

  • How MCP enables AI assistants to discover and use your tools.

  • How to structure Redpanda Connect configurations as MCP tools.

  • How to run a local MCP server and connect it to Claude Code.

  • How AI agents orchestrate your tools through natural language.

How MCP works

The Model Context Protocol (MCP) makes your tools AI-consumable by exposing them with structured metadata:

  • MCP Server: A local HTTP service that hosts your Redpanda Connect configurations as discoverable tools

  • Tools: Individual Redpanda Connect configurations (processors, inputs, outputs, caches) exposed through the server

  • Metadata: Structured descriptions of what each tool does, what inputs it needs, and what outputs it returns

  • Client: An AI assistant (like Claude Code) that discovers your tools and calls them based on user requests

When you ask Claude to "search Bluesky for Redpanda news," Claude:

  1. Discovers your search-bluesky-posts tool through the MCP server

  2. Understands it needs a query parameter

  3. Calls your tool with the appropriate query

  4. Returns the results in natural language

This enables natural-language automation for internal workflows using your own business logic.

Prerequisites

You’ll need the following to complete this quickstart:

Initialize an MCP server project

To get started, create a new MCP server project using the Redpanda CLI.

  1. Create a directory for your MCP server project:

    mkdir redpanda-connect-mcp
cd redpanda-connect-mcp

  2. Initialize the MCP server project:

    rpk connect mcp-server init

    This command scaffolds your project with the necessary directories and template YAML files for defining tools and observability resources.

    ├── o11y
│   ├── metrics.yaml
│   └── tracer.yaml
└── resources
    ├── caches
    │   └── example-cache.yaml
    ├── inputs
    │   └── example-input.yaml
    ├── outputs
    │   └── example-output.yaml
    └── processors
        └── example-processor.yaml

  3. Make sure your Redpanda Connect version is at least 4.66.1:

    rpk connect --version

    If you need to upgrade, see Upgrade Redpanda Connect.

Create a tool definition

In this step, you’ll create a Redpanda Connect configuration that searches public Bluesky posts using a customizable query. You’ll define this configuration as a tool that Claude can discover and use.

Save the following YAML to a file named bluesky.yaml and save it in the resources/processors/ directory:

label: search-bluesky-posts (1)
try: (2)
  - mutation: |
      root.limit = root.limit.number(10)
      root.limit = [ root.limit, 100 ].min()
      root.limit = [ root.limit, 1 ].max()
      meta query_string = "q=" + root.query.escape_url_query() + "&limit=%v".format(root.limit)
      root = ""
  - http:
      url: "https://api.bsky.app/xrpc/app.bsky.feed.searchposts?${! @query_string }"
      verb: GET
  - mapping: 'root = this.posts'
  - unarchive:
      format: json_array

meta:
  tags: [ bluesky ] (3)
  mcp:
    enabled: true
    description: Search public Bluesky posts based on a query string. (4)
    properties: (5)
      - name: query
        type: string
        required: true
        description: A Lucene-style query string to search posts.
      - name: limit
        type: number
        description: Number of posts to return (1-100). Defaults to 10.

This example defines a tool that searches public Bluesky posts based on a query string. Important parts of the configuration include:

1 A unique label for identification. Claude uses this label to discover and call the tool.
2 A series of processors to build the HTTP request, handle the response, and format the output.
3 A unique tag (bluesky) to control exposure. You can use tags to group related tools or isolate experiments.
4 A clear description for LLMs helps Claude understand what the tool does.
5 Structured properties for inputs like query and limit.

Start the MCP server

  1. Navigate to your MCP server project directory if you’re not already there:

    cd redpanda-connect-mcp

  2. Start the MCP server to expose the tool over HTTP:

    rpk connect mcp-server --address localhost:4195 --tag bluesky

    You should see output like this:

    time=2025-06-27T15:20:27.976+01:00 level=INFO msg="Registering processor tool" label=search-bluesky-posts
time=2025-06-27T15:20:27.978+01:00 level=INFO msg="Successfully loaded Redpanda license" expires_at=2035-06-25T15:20:27+01:00 license_org="" license_type="open source"

    This command creates an MCP server listening on localhost:4195, and makes your tool discoverable to AI agents. Leave this terminal open while you interact with Claude Code.

Only tools with the specified --tag are exposed. This helps you:

  • Keep experiments isolated

  • Avoid exposing sensitive functionality accidentally

  • Create sets of tools that are relevant to specific agents or workflows

Connect Claude Code to your MCP server

Now that your MCP server is running and exposing tools, you can connect Claude Code so it can discover and use them.

Claude Code communicates with MCP servers using the standard input/output (stdio) transport protocol. Because your Redpanda Connect MCP server exposes tools over HTTP with Server-Sent Events (SSE), you need a bridge to convert between these protocols. The mcp-remote utility provides this bridge.

When you connect Claude Code:

  • mcp-remote subscribes to your server’s SSE endpoint (http://localhost:4195/sse).

  • It translates HTTP/SSE messages into stdio format that Claude understands.

  • Claude automatically discovers your tools and their metadata.

  • You can ask Claude in natural language to use your tools.

  • Claude calls the appropriate tool based on your request.

To set up this connection:

  1. Open a new terminal window.

  2. To install mcp-remote, run:

    claude mcp add local -- npx mcp-remote http://localhost:4195/sse

    You should see output like this:

    Added stdio MCP server local with command: npx mcp-remote http://localhost:4195/sse to local config

  3. Verify that the local input channel is set up correctly by running:

    claude /mcp

    You should see an entry for local.

  4. Press Enter until you see the tools list.

    Tools for local (1 tools)
│ ❯ 1. search-bluesky-posts

  5. Press Esc until you return to the main prompt.

Use the tool with Claude

Now you can see MCP in action. When you ask Claude to search Bluesky, watch what happens behind the scenes.

  1. Enter the following prompt to start a conversation with Claude Code:

    Search Bluesky for the latest news about Redpanda Data

    If you exited the prompt, you can re-enter it by running:

    claude Search Bluesky for the latest news about Redpanda Data

  2. When Claude prompts you for permissions, select Yes:

    > Search Bluesky for the latest news about Redpanda Data

⏺ local:search-bluesky-posts (MCP)(value: "Redpanda")

╭─────────────────────────────────────────────
│ Tool use
│   local:search-bluesky-posts(value: "Redpanda") (MCP)
│   Search public Bluesky posts based on a query string.
│ Do you want to proceed?
│ ❯ 1. Yes
│   2. Yes, and don't ask again for local:search-bluesky-posts commands in /Users/jakecahill/Documents/my-agent  │
│   3. No, and tell Claude what to do differently (esc)

    This is what happens:

    • Claude analyzes your natural language request.

    • Claude identifies the search-bluesky-posts tool as the right one to use.

    • Claude extracts "Redpanda Data" as the query parameter.

    • Claude calls your local MCP server via mcp-remote.

    • Your Redpanda Connect configuration executes the Bluesky API request.

    • Results flow back through the MCP server to Claude.

    • Claude synthesizes the results into a conversational response.

Here’s an example of what the result might look like:

⏺ Based on the search results from Bluesky, here are the latest posts about Redpanda Data:

  Recent Company Activity:
  - Product Development: Expanding Redpanda Connect beyond data infrastructure to support AI agent tools like
   Slack, GitHub, and Google Drive
  - New Snowflake Connector: Launched with 2x faster throughput than Kafka Connect for data streaming

  Events & Marketing:
  - Participated in Data+AI Summit (Booth E500)
  - Hosting disaster recovery streamcast on June 12th
  - Fireside chat with Entain's Chief Data Officer about replacing Confluent with Redpanda in 2 weeks

  Success Stories:
  - Gaming company Entain replaced Confluent with Redpanda in just two weeks
  - Focus on real-time financial data streaming to Snowflake for analytics

  The posts show Redpanda positioning itself as a faster, simpler alternative to Kafka/Confluent with strong
  momentum in hiring and customer wins.

If you change the YAML configuration of your tools, restart the MCP server to pick up the changes.

Stop or disconnect the MCP server

To disconnect or stop the MCP server, press Ctrl+C in the terminal where the server is running. This will gracefully shut down the MCP server process and disconnect any connected clients.

You can also close the terminal window or kill the process using standard OS commands (such as kill <pid> on Linux/macOS).

Summary

You extended Claude’s capabilities with a custom tool built from Redpanda Connect components. Claude can now search Bluesky through natural language commands without manual API calls or scripting.

This same pattern works for any workflow you can build with Redpanda Connect: consuming from Kafka topics, transforming data, calling APIs, querying databases, or orchestrating complex data pipelines. Each configuration becomes an AI-consumable tool.

Troubleshoot

This section covers issues you might encounter when setting up and using the MCP server.

JSON schema errors

This error indicates you’re using an outdated version of Redpanda Connect with an incompatible JSON schema format:

{
  "type": "error",
  "error": {
    "type": "invalid_request_error",
    "message": "tools.17.custom.input_schema: JSON schema is invalid. It must match JSON Schema draft 2020-12 (https://json-schema.org/draft/2020-12). Learn more about tool use at https://docs.claude.com/en/docs/tool-use."
  },
  "request_id": "req_011CTWPsqnC26DJVUhxnQnn6"
}

Solution: Upgrade to at least version 4.66.1 of Redpanda Connect.

rpk connect --version

If you need to upgrade, see Upgrade Redpanda Connect.

Tool not found or not exposed

If Claude Code doesn’t show your tool in the tools list, or shows "No tools available":

  1. Ensure your YAML files are in the correct directory structure. The expected structure is:

    • Processors: resources/processors/

    • Inputs: resources/inputs/

    • Outputs: resources/outputs/

    • Caches: resources/caches/

  2. Verify MCP metadata:

    • Confirm your tool has the meta.mcp.enabled: true property

    • Check that the meta.tags array includes the tag you specified when starting the server

      Example correct structure
      label: my-tool
# ... processor configuration ...
meta:
  tags: [ bluesky ]  # Must match --tag argument
  mcp:
    enabled: true    # Required for exposure
    description: Tool description

Connection issues

If Claude Code can’t connect to your MCP server:

  • Server not running:

    • Verify the MCP server is still running in your terminal.

    • Check that you see the "Registering processor tool" log messages.

    • Ensure the server is listening on the expected address (localhost:4195 by default).

  • If port 4195 is already in use, specify a different port:

    rpk connect mcp-server --address localhost:4196 --tag bluesky

    Then update your mcp-remote connection:

    claude mcp add local -- npx mcp-remote http://localhost:4196/sse

Next steps

Now that you understand how MCP works with Redpanda Connect, you can create custom tools for your specific workflows:

  • Data pipeline orchestration: Build tools that consume from Kafka topics, transform data, and write to destinations.

  • API integrations: Create tools that call internal APIs, third-party services, or webhooks.

  • Database operations: Add tools for querying, inserting, or updating data using natural language.

  • Monitoring and observability: Expose tools that check system health, query metrics, or analyze logs.

  • Multi-step workflows: Combine multiple tools with different tags to handle complex operations.

  • Advanced patterns: See Build an MCP Server in Redpanda Connect to learn advanced patterns for building MCP tools.

  • Component reference: View the full catalog of Redpanda Connect components you can use to build tools.