Create and Manage Kafka Connect Connectors in Kubernetes
When you have Kafka Connect deployed, you can create and manage connectors using Redpanda Console or the Kafka Connect REST API.
The Redpanda Connectors Docker image is a community-supported artifact. Redpanda Data does not provide enterprise support for this image. For support, reach out to the Redpanda team in Redpanda Community Slack. |
Manage connectors in Redpanda Console
By default, Redpanda Console is deployed with a ClusterIP Service. To access Redpanda Console, you can use the kubectl port-forward
command to forward one of your local ports to the Pod.
The kubectl port-forward command is a development tool. To expose services to external traffic in a more permanent and controlled manner, use Kubernetes Services such as LoadBalancer or NodePort.
|
-
Expose Redpanda Console to your localhost:
kubectl --namespace <namespace> port-forward svc/redpanda-console 8080:8080
bashThis command actively runs in the command-line window. To execute other commands while the command is running, open another command-line window.
-
Open Redpanda Console on http://localhost:8080.
You can create and manage connectors by clicking Connectors in the navigation menu.
Manage connectors with the REST API
This section provides examples of requesting data from the REST API using cURL. Execute all cURL commands in the Pod that’s running Kafka Connect.
To view the name of the Pod that’s running Kafka Connect:
kubectl get pod -l app.kubernetes.io/name=connectors --namespace <namespace>
View version of Kafka Connect
To view the version of Kafka Connect, run:
curl localhost:8083 | jq
View a list of connectors
To view all available connectors, run:
curl localhost:8083/connector-plugins | jq
View active connectors
To view all active connectors, run:
curl 'http://localhost:8083/connectors?expand=status&expand=info' | jq
Create a connector
To create a connector, run:
curl "localhost:8083/connectors" -H 'Content-Type: application/json' --data-raw '<connector-config>'
For example:
curl "localhost:8083/connectors" \
-H 'Content-Type: application/json' \
--data-raw '{ "name": "heartbeat-connector", "config": { "connector.class": "org.apache.kafka.connect.mirror.MirrorHeartbeatConnector", "heartbeats.topic.replication.factor": "1", "replication.factor": "1", "source.cluster.alias": "source", "source.cluster.bootstrap.servers": "redpanda:29092", "target.cluster.bootstrap.servers": "redpanda:29092"}}'