Create and Manage Kafka Connect Connectors in Kubernetes

When you have Kafka Connect deployed, you can create and manage connectors using Redpanda Console or the Kafka Connect REST API.

The Redpanda Connectors Docker image is a community-supported artifact. Redpanda Data does not provide enterprise support for this image. For support, reach out to the Redpanda team in Redpanda Community Slack.

Prerequisites

Manage connectors in Redpanda Console

By default, Redpanda Console is deployed with a ClusterIP Service. To access Redpanda Console, you can use the kubectl port-forward command to forward one of your local ports to the Pod.

The kubectl port-forward command is a development tool. To expose services to external traffic in a more permanent and controlled manner, use Kubernetes Services such as LoadBalancer or NodePort.

  1. Expose Redpanda Console to your localhost:

    kubectl --namespace <namespace> port-forward svc/redpanda-console 8080:8080

    This command actively runs in the command-line window. To execute other commands while the command is running, open another command-line window.

  2. Open Redpanda Console on http://localhost:8080.

You can create and manage connectors by clicking Connectors in the navigation menu.

Manage connectors with the REST API

This section provides examples of requesting data from the REST API using cURL. Execute all cURL commands in the Pod that’s running Kafka Connect.

To view the name of the Pod that’s running Kafka Connect:

kubectl get pod -l app.kubernetes.io/name=connectors --namespace <namespace>

View version of Kafka Connect

To view the version of Kafka Connect, run:

curl localhost:8083 | jq

View a list of connectors

To view all available connectors, run:

curl localhost:8083/connector-plugins | jq

View active connectors

To view all active connectors, run:

curl 'http://localhost:8083/connectors?expand=status&expand=info' | jq

Create a connector

To create a connector, run:

curl "localhost:8083/connectors" -H 'Content-Type: application/json' --data-raw '<connector-config>'

For example:

curl "localhost:8083/connectors" \
  -H 'Content-Type: application/json' \
  --data-raw '{ "name": "heartbeat-connector", "config": { "connector.class": "org.apache.kafka.connect.mirror.MirrorHeartbeatConnector", "heartbeats.topic.replication.factor": "1", "replication.factor": "1", "source.cluster.alias": "source", "source.cluster.bootstrap.servers": "redpanda:29092", "target.cluster.bootstrap.servers": "redpanda:29092"}}'

View connector status

To view the status of a connector, run:

curl localhost:8083/connectors/<connector-name>/status

For example:

curl localhost:8083/connectors/heartbeat-connector/status

Delete a connector

To delete a connector, run:

curl "localhost:8083/connectors/<connector-name>" -X 'DELETE'

For example:

curl "localhost:8083/connectors/heartbeat-connector" -X 'DELETE'

Next steps