Use the Data Plane APIs

Beta

The Redpanda Cloud API is a collection of REST APIs that allow you to interact with different parts of Redpanda Cloud. The Data Plane APIs enable you to programmatically manage the resources within your clusters, including topics, users, access control lists (ACLs), and connectors. You can call the API endpoints directly, or use tools like Terraform or Python scripts to automate resource management.

For the full Cloud API reference documentation, see Redpanda Cloud API Reference.

The data plane contains the actual Redpanda clusters. Every cluster is its own data plane, and so it has its own distinct Data Plane API URL.

Get Data Plane API URL

  • BYOC or Dedicated

  • Serverless

To retrieve the Data Plane API URL of a cluster, make a request to GET /v1beta2/clusters/{id}.

To retrieve the Data Plane API URL of a cluster, make a request to GET /v1beta2/serverless/clusters/{id}.

The response includes a dataplane_api.url value:

  "id": "....",
  "name": "my-cluster",
....
  "dataplane_api": {
    "url": "https://api-xyz.abc.fmc.ppd.cloud.redpanda.com"
  },
...

Data Plane APIs

Create a user

To create a new user in your Redpanda cluster, make a POST request to the /v1alpha2/users endpoint, including the SASL mechanism, username, and password in the request body:

curl -X POST "https://<dataplane-api-url>/v1alpha2/users" \
 -H "Authorization: Bearer <token>" \
 -H "accept: application/json" \
 -H "content-type: application/json" \
 -d '{"mechanism":"SASL_MECHANISM_SCRAM_SHA_256","name":"payment-service","password":"secure-password"}'
When using a shell substitution variable for the token, use double quotes to wrap the header value.

The success response returns the newly-created username and SASL mechanism:

{
  "user": {
    "name": "payment-service",
    "mechanism": "SASL_MECHANISM_SCRAM_SHA_256"
  }
}

Create an ACL

To create a new ACL in your Redpanda cluster, make a POST /v1alpha2/acls request. The following example ACL allows all operations on any Redpanda topic for a user with the name payment-service.

curl -X POST "https://<dataplane-api-url>/v1alpha2/acls" \
 -H "Authorization: Bearer <token>" \
 -H "accept: application/json" \
 -H "content-type: application/json" \
 -d '{"host":"*","operation":"OPERATION_ALL","permission_type":"PERMISSION_TYPE_ALLOW","principal":"User:payment-service","resource_name":"*","resource_pattern_type":"RESOURCE_PATTERN_TYPE_LITERAL","resource_type":"RESOURCE_TYPE_TOPIC"}'

The success response is empty, with a 201 status code.

{}

Create a topic

To create a new Redpanda topic without specifying any further parameters, such as the desired topic-level configuration or partition count, make a POST request to /v1alpha2/topics endpoint:

curl -X POST "<dataplane-api-url>/v1alpha2/topics" \
 -H "Authorization: Bearer <token>" \
 -H "accept: application/json" \
 -H "content-type: application/json" \
 -d '{"name":"<topic-name>"}'

Use Redpanda Connect

Use the API to manage Redpanda Connect pipelines in Redpanda Cloud.

The Pipeline APIs for Redpanda Connect are supported in BYOC and Serverless clusters only.

Get Redpanda Connect pipeline

To get details of a specific pipeline, make a GET /v1alpha2/redpanda-connect/pipelines/{id} request.

curl "https://<dataplane-url>/v1alpha2/redpanda-connect/pipelines/<pipeline-id>"

Stop a Redpanda Connect pipeline

To stop a running pipeline, make a PUT /v1alpha2/redpanda-connect/pipelines/{id}/stop request.

curl -X PUT "https://<dataplane-url>/v1alpha2/redpanda-connect/pipelines/<pipeline-id>/stop"

Start a Redpanda Connect pipeline

To start a previously stopped pipeline, make a PUT /v1alpha2/redpanda-connect/pipelines/{id}/start request.

curl -X PUT "https://<dataplane-url>/v1alpha2/redpanda-connect/pipelines/<pipeline-id>/start"

Update a Redpanda Connect pipeline

To update a pipeline, make a PUT /v1alpha2/redpanda-connect/pipelines/{id} request. You update a pipeline configuration to scale resources, for example the number of CPU cores and amount of memory allocated.

curl -X PUT "https://api.redpanda.com/v1alpha2/redpanda-connect/pipelines/" \
 -H 'accept: application/json'\
 -H 'content-type: application/json' \
 -d '{"resources":{"cpu_shares":"8","memory_shares":"8G"}}'

Manage Kafka Connect

Use the API to configure your Kafka Connect clusters.

Kafka Connect is supported in BYOC and Dedicated clusters only.

Create a Kafka Connect cluster secret

Kafka Connect cluster secret data must first be in JSON format, and then Base64-encoded.

  1. Prepare the secret data in JSON format:

    {"secret.access.key": "<secret-access-key-value>"}

  2. Encode the secret data in Base64:

    echo '{"secret.access.key": "<secret-access-key-value>"}' | base64

  3. Use the Secrets API to create a secret that stores the Base64-encoded secret data:

    curl -X POST "https://<dataplane-api-url>/v1alpha2/kafka-connect/clusters/redpanda/secrets" \
 -H 'accept: application/json'\
 -H 'content-type: application/json' \
 -d '{"name":"<connector-name>","secret_data":"<secret-data-base64-encoded>"}'

The response returns an id that you can use to create the Kafka Connect connector.

Create a Kafka Connect connector

To create a connector, make a POST request to /v1alpha2/kafka-connect/clusters/{cluster_name}/connectors.

The following example shows how to create an S3 sink connector with the name my-connector:

curl -X POST "<dataplane-api-url>/v1alpha2/kafka-connect/clusters/redpanda/connectors" \
 -H "Authorization: Bearer <token>" \
 -H "accept: application/json" \
 -H "content-type: application/json" \
 -d '{"config":{"connector.class":"com.redpanda.kafka.connect.s3.S3SinkConnector","topics":"test-topic","aws.secret.access.key":"${secretsManager:<secret-id>:secret.access.key}","aws.s3.bucket.name":"bucket-name","aws.access.key.id":"access-key","aws.s3.bucket.check":"false","region":"us-east-1"},"name":"my-connector"}'

The field aws.secret.access.key in this example contains sensitive information that usually shouldn’t be added to a configuration directly. Redpanda recommends that you first create a secret and then use the secret ID to inject the secret in your Create Connector request.

If you had created a secret following the example from the previous section Create a Kafka Connect cluster secret, use the id returned in the Create Secret response to replace the placeholder <secret-id> in this Create Connector example. The syntax ${secretsManager:<secret-id>:secret.access.key} tells the Kafka Connect cluster to load <secret-id>, specifying the key secret.access.key from the secret JSON.

Example success response:

{
  "name": "my-connector",
  "config": {
    "aws.access.key.id": "access-key",
    "aws.s3.bucket.check": "false",
    "aws.s3.bucket.name": "bucket-name",
    "aws.secret.access.key": "secret-key",
    "connector.class": "com.redpanda.kafka.connect.s3.S3SinkConnector",
    "name": "my-connector",
    "region": "us-east-1",
    "topics": "test-topic"
  },
  "tasks": [],
  "type": "sink"
}

Restart a Kafka Connect connector

To restart a connector, make a POST request to the /v1alpha2/kafka-connect/clusters/{cluster_name}/connectors/{name}/restart endpoint:

curl -X POST "<dataplane-api-url>/v1alpha2/kafka-connect/clusters/redpanda/connectors/my-connector/restart" \
 -H "Authorization: Bearer <token>" \
 -H "accept: application/json"\
 -H "content-type: application/json" \
 -d '{"include_tasks":false,"only_failed":false}'

Limitations

  • Client SDKs are not available.