Create and Manage Connectors in Kubernetes

When you have Redpanda Connectors deployed, you can create and manage connectors using Redpanda Console or the Kafka Connect REST API.

The Redpanda Connectors Docker image is a community-supported artifact. For support, reach out to the Redpanda team in the community Slack.

This topic describes how to use the Redpanda Connectors Docker image, which includes a pre-configured instance of Kafka Connect that works with Redpanda.

The Redpanda Connectors Docker image contains only the MirrorMaker2 connector but you can build a custom image to install additional connectors. For a smoother experience, consider using the Managed Connectors available in Redpanda Cloud.

Prerequisites

Manage connectors in Redpanda Console

By default, Redpanda Console is deployed with a ClusterIP Service. To access Redpanda Console, you can use the kubectl port-forward command to forward one of your local ports to the Pod.

The kubectl port-forward command is a development tool. To expose services to external traffic in a more permanent and controlled manner, use Kubernetes Services such as LoadBalancer or NodePort.

  1. Expose Redpanda Console to your localhost:

    kubectl --namespace <namespace> port-forward svc/redpanda-console 8080:8080

    This command actively runs in the command-line window. To execute other commands while the command is running, open another command-line window.

  2. Open Redpanda Console on http://localhost:8080.

You can create and manage connectors by clicking Connectors in the navigation menu.

Manage connectors with the REST API

This section provides examples of requesting data from the REST API using cURL. Execute all cURL commands in the Pod that’s running Redpanda Connectors.

To view the name of the Pod that’s running Redpanda Connectors:

kubectl get pod -l app.kubernetes.io/name=connectors --namespace <namespace>

View version of Kafka Connect

To view the version of Kafka Connect, run:

curl localhost:8083 | jq

View a list of connectors

To view all available connectors, run:

curl localhost:8083/connector-plugins | jq

View active connectors

To view all active connectors, run:

curl 'http://localhost:8083/connectors?expand=status&expand=info' | jq

Create a connector

To create a connector, run:

curl "localhost:8083/connectors" -H 'Content-Type: application/json' --data-raw '<connector-config>'

For example:

curl "localhost:8083/connectors" \
  -H 'Content-Type: application/json' \
  --data-raw '{ "name": "heartbeat-connector", "config": { "connector.class": "org.apache.kafka.connect.mirror.MirrorHeartbeatConnector", "heartbeats.topic.replication.factor": "1", "replication.factor": "1", "source.cluster.alias": "source", "source.cluster.bootstrap.servers": "redpanda:29092", "target.cluster.bootstrap.servers": "redpanda:29092"}}'

View connector status

To view the status of a connector, run:

curl localhost:8083/connectors/<connector-name>/status

For example:

curl localhost:8083/connectors/heartbeat-connector/status

Delete a connector

To delete a connector, run:

curl "localhost:8083/connectors/<connector-name>" -X 'DELETE'

For example:

curl "localhost:8083/connectors/heartbeat-connector" -X 'DELETE'

Next steps