Docs Self-Managed Get Started Quickstarts Redpanda Self-Managed Redpanda Self-Managed Quickstart Learn how to quickly start working with a local Redpanda cluster that comes with a free 30-day license for Enterprise Edition. You can also extend the trial license for a further 30 days. Redpanda Self-Managed is a modern streaming platform, compatible with Kafka APIs, designed for speed, simplicity, and efficiency. In this quickstart, you: Deploy a three-broker Redpanda cluster. Explore streaming data in Redpanda Console. Learn the basics of streaming with the Redpanda CLI (rpk). Deploy a streaming pipeline with Redpanda Connect. Learn how to extend your trial license for an additional 30 days. This quickstart uses Docker to run Redpanda, which is only for development and testing purposes. For production deployments, refer to the Linux deployment guides or the Kubernetes deployment guides. To download the Redpanda binary, see GitHub. Looking for a managed solution? You can also get started quickly with a hosted Redpanda cluster by signing up for Redpanda Cloud. Enterprise features The Enterprise Edition of Redpanda Self-Managed adds advanced capabilities to help scale, manage, optimize, and secure your cluster. All new Redpanda clusters include a built-in 30-day Enterprise Edition license, so you can evaluate enterprise features. Some features highlighted in this quickstart require an enterprise license: Redpanda Console Authentication: Securely control login access to Redpanda Console. Audit Logging: Track and monitor all user actions in a Redpanda topic. Continuous Data Balancing: Automatically balance data across brokers to optimize performance. Tiered Storage: Lower storage costs by offloading log segments to object storage. Your trial license also includes additional enterprise features. For more information, see Redpanda Licenses and Enterprise Features. Sign up for an extended trial license. Prerequisites You need Docker Compose. For installation instructions, see the Docker Compose documentation. To check if you have Docker Compose installed, run: docker compose version You should see the version of Docker Compose that’s installed on your local machine. If you’re on Windows, use a compatible terminal that supports Unix-like commands. Or, consider using Windows Subsystem for Linux (WSL). Deploy Redpanda Self-Managed To download, extract, and start Redpanda Self-Managed in Docker, run: mkdir redpanda-quickstart && cd redpanda-quickstart && \ (1) curl -sSL https://docs.redpanda.com/redpanda-quickstart.tar.gz | tar xzf - && \ (2) cd docker-compose && docker compose up -d (3) 1 Create and navigate to the redpanda-quickstart directory. 2 Download and extract the archive. Explore the downloaded files docker-compose.yml name: redpanda-quickstart-multi-broker networks: redpanda_network: driver: bridge volumes: redpanda-0: null redpanda-1: null redpanda-2: null minio: null services: ################## # Redpanda Brokers # ################## redpanda-0: command: - redpanda - start - --kafka-addr internal://0.0.0.0:9092,external://0.0.0.0:19092 # Address the broker advertises to clients that connect to the Kafka API. # Use the internal addresses to connect to the Redpanda brokers # from inside the same Docker network. # Use the external addresses to connect to the Redpanda brokers # from outside the Docker network. - --advertise-kafka-addr internal://redpanda-0:9092,external://localhost:19092 - --pandaproxy-addr internal://0.0.0.0:8082,external://0.0.0.0:18082 # Address the broker advertises to clients that connect to the HTTP Proxy. - --advertise-pandaproxy-addr internal://redpanda-0:8082,external://localhost:18082 - --schema-registry-addr internal://0.0.0.0:8081,external://0.0.0.0:18081 # Redpanda brokers use the RPC API to communicate with each other internally. - --rpc-addr redpanda-0:33145 - --advertise-rpc-addr redpanda-0:33145 # Mode dev-container uses well-known configuration properties for development in containers. - --mode dev-container # Tells Seastar (the framework Redpanda uses under the hood) to use 1 core on the system. - --smp 1 - --default-log-level=info image: docker.redpanda.com/redpandadata/redpanda:v24.3.5 container_name: redpanda-0 # Sets the username and password of the bootstrap SCRAM superuser # See https://docs.redpanda.com/current/deploy/deployment-option/self-hosted/manual/production/production-deployment/#bootstrap-a-user-account environment: RP_BOOTSTRAP_USER: "superuser:secretpassword" volumes: - redpanda-0:/var/lib/redpanda/data - ./bootstrap.yml:/etc/redpanda/.bootstrap.yaml networks: - redpanda_network ports: - 18081:18081 - 18082:18082 - 19092:19092 - 19644:9644 healthcheck: test: ["CMD", "rpk", "cluster", "info", "-X", "user=superuser", "-X", "pass=secretpassword"] interval: 10s timeout: 5s retries: 10 depends_on: - minio redpanda-1: command: - redpanda - start - --kafka-addr internal://0.0.0.0:9092,external://0.0.0.0:29092 - --advertise-kafka-addr internal://redpanda-1:9092,external://localhost:29092 - --pandaproxy-addr internal://0.0.0.0:8082,external://0.0.0.0:28082 - --advertise-pandaproxy-addr internal://redpanda-1:8082,external://localhost:28082 - --schema-registry-addr internal://0.0.0.0:8081,external://0.0.0.0:28081 - --rpc-addr redpanda-1:33145 - --advertise-rpc-addr redpanda-1:33145 - --mode dev-container - --smp 1 - --default-log-level=info - --seeds redpanda-0:33145 image: docker.redpanda.com/redpandadata/redpanda:v24.3.5 container_name: redpanda-1 environment: RP_BOOTSTRAP_USER: "superuser:secretpassword" volumes: - redpanda-1:/var/lib/redpanda/data - ./bootstrap.yml:/etc/redpanda/.bootstrap.yaml networks: - redpanda_network ports: - 28081:28081 - 28082:28082 - 29092:29092 - 29644:9644 depends_on: - redpanda-0 - minio redpanda-2: command: - redpanda - start - --kafka-addr internal://0.0.0.0:9092,external://0.0.0.0:39092 - --advertise-kafka-addr internal://redpanda-2:9092,external://localhost:39092 - --pandaproxy-addr internal://0.0.0.0:8082,external://0.0.0.0:38082 - --advertise-pandaproxy-addr internal://redpanda-2:8082,external://localhost:38082 - --schema-registry-addr internal://0.0.0.0:8081,external://0.0.0.0:38081 - --rpc-addr redpanda-2:33145 - --advertise-rpc-addr redpanda-2:33145 - --mode dev-container - --smp 1 - --default-log-level=info - --seeds redpanda-0:33145 image: docker.redpanda.com/redpandadata/redpanda:v24.3.5 container_name: redpanda-2 environment: RP_BOOTSTRAP_USER: "superuser:secretpassword" volumes: - redpanda-2:/var/lib/redpanda/data - ./bootstrap.yml:/etc/redpanda/.bootstrap.yaml networks: - redpanda_network ports: - 38081:38081 - 38082:38082 - 39092:39092 - 39644:9644 depends_on: - redpanda-0 - minio #################### # Redpanda Console # #################### console: container_name: redpanda-console image: docker.redpanda.com/redpandadata/console:v2.8.2 networks: - redpanda_network entrypoint: /bin/sh command: -c 'echo "$$CONSOLE_CONFIG_FILE" > /tmp/config.yml && echo "$$CONSOLE_ROLEBINDINGS_CONFIG_FILE" > /tmp/role-bindings.yml && /app/console' volumes: - ./config:/tmp/config/ environment: CONFIG_FILEPATH: ${CONFIG_FILEPATH:-/tmp/config.yml} CONSOLE_CONFIG_FILE: | # Configure a connection to the Redpanda cluster # See https://docs.redpanda.com/current/console/config/connect-to-redpanda/ kafka: brokers: ["redpanda-0:9092"] schemaRegistry: enabled: true urls: ["http://redpanda-0:8081","http://redpanda-1:8081","http://redpanda-2:8081"] sasl: enabled: true username: superuser password: secretpassword mechanism: SCRAM-SHA-256 redpanda: adminApi: enabled: true urls: ["http://redpanda-0:9644","http://redpanda-1:9644","http://redpanda-2:9644"] username: superuser password: secretpassword console: # Configures Redpanda Console to fetch topic documentation from GitHub and display it in the UI. # See https://docs.redpanda.com/current/console/config/topic-documentation/ topicDocumentation: enabled: true git: enabled: true repository: url: https://github.com/redpanda-data/docs branch: main baseDirectory: tests/docker-compose login: enabled: true jwtSecret: change-this-to-something-secret useSecureCookies: false # Enable plain login authentication (enterprise feature) # See https://docs.redpanda.com/current/console/config/security/plain/ plain: enabled: true credentials: - username: "jane" password: "some-other-secret-password" - username: "john" password: "some-secret-password" enterprise: # Configures role-based access control to data and features in Redpanda Console. # See https://docs.redpanda.com/current/console/config/security/authorization/ rbac: enabled: true roleBindingsFilepath: /tmp/role-bindings.yml CONSOLE_ROLEBINDINGS_CONFIG_FILE: | roleBindings: - metadata: name: Platform Ops subjects: - kind: user provider: Plain name: jane roleName: admin - metadata: name: Software Engineers subjects: - kind: user provider: Plain name: john roleName: editor ports: - 8080:8080 depends_on: redpanda-0: condition: service_healthy #################### # Redpanda Connect # #################### connect: container_name: redpanda-connect image: docker.redpanda.com/redpandadata/connect networks: - redpanda_network entrypoint: /bin/sh depends_on: redpanda-0: condition: service_healthy command: -c 'echo "$$CONNECT_CFG_FILE" > /tmp/connect.yml; /redpanda-connect -c /tmp/connect.yml' environment: # This Redpanda Connect configuration creates fake data, # processes it, and writes the output to a set of topics. # # Input: # - Uses Redpanda Connect's generate input to generate fake data. # See https://docs.redpanda.com/redpanda-connect/components/inputs/generate/ # Pipeline: # - Bloblang mapping to batch each input and map 1 message to 'logins' # topic, and a random number (1-3) of messages to 'transaction' topic # - Unarchive processor to parse the JSON array and extract each # element into its own message. # See https://docs.redpanda.com/redpanda-connect/guides/bloblang/about/ # Output: # - kafka_franz output to write the messages to the Redpanda brokers. # See https://docs.redpanda.com/redpanda-connect/components/outputs/kafka_franz/ CONNECT_CFG_FILE: | input: generate: interval: 1s mapping: | let first_name = fake("first_name") let last_name = fake("last_name") root.user_id = counter() root.name = $$first_name + " " + $$last_name root.email = ($$first_name.slice(0,1) + $$last_name + "@" + fake("domain_name")).lowercase() root.ip = fake("ipv4") root.login_time = now() pipeline: processors: - mapping: | root = range(0, random_int(min:2, max:4)).map_each(cust -> this) - unarchive: format: "json_array" - mapping: | if batch_index() == 0 { meta topic = "logins" root = this } else { meta topic = "transactions" root.user_id = this.user_id root.email = this.email root.index = batch_index() - 1 root.product_url = fake("url") root.price = fake("amount_with_currency") root.timestamp = now() } output: kafka_franz: seed_brokers: [ "redpanda-0:9092" ] topic: $${! metadata("topic") } sasl: - mechanism: SCRAM-SHA-256 password: secretpassword username: superuser #################### # rpk container to create the edu-filtered-domains topic # # See https://docs.redpanda.com/current/reference/rpk/rpk-topic/rpk-topic-create/ #################### createtopic: command: - topic - create - edu-filtered-domains - -X user=superuser - -X pass=secretpassword - -X brokers=redpanda-0:9092 image: docker.redpanda.com/redpandadata/redpanda:v24.3.5 networks: - redpanda_network depends_on: redpanda-0: condition: service_healthy #################### # rpk container to register the schema # # See https://docs.redpanda.com/current/manage/schema-reg/schema-reg-api/ #################### registerschema: command: - registry - schema - create - transactions - --schema - /etc/redpanda/transactions-schema.json - -X user=superuser - -X pass=secretpassword - -X registry.hosts=redpanda-0:8081 image: docker.redpanda.com/redpandadata/redpanda:v24.3.5 # Mount the local directory that contains your schema to the container. volumes: - ./transactions-schema.json:/etc/redpanda/transactions-schema.json networks: - redpanda_network depends_on: redpanda-0: condition: service_healthy #################### # rpk container to deploy the pre-built data transform # # See https://docs.redpanda.com/current/develop/data-transforms/deploy/ #################### deploytransform: command: - transform - deploy - --file=/etc/redpanda/regex.wasm - --name=regex - --input-topic=logins - --output-topic=edu-filtered-domains - --var=PATTERN="[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.edu" - --var=MATCH_VALUE=true - -X user=superuser - -X pass=secretpassword - -X admin.hosts=redpanda-0:9644 image: docker.redpanda.com/redpandadata/redpanda:v24.3.5 volumes: - ./transform/regex.wasm:/etc/redpanda/regex.wasm networks: - redpanda_network depends_on: redpanda-0: condition: service_healthy createtopic: condition: service_started #################### # MinIO for Tiered Storage # # See https://min.io/ # # NOTE: MinIO is included in this quickstart for development and evaluation purposes only. # It is not supported for production deployments of Redpanda. # # For production environments, use one of the supported object storage providers: # https://docs.redpanda.com/current/deploy/deployment-option/self-hosted/manual/production/requirements/#object-storage-providers-for-tiered-storage #################### minio: container_name: minio image: minio/minio command: server --console-address ":9001" /data ports: - 9000:9000 - 9001:9001 environment: MINIO_ROOT_USER: minio MINIO_ROOT_PASSWORD: redpandaTieredStorage7 MINIO_SERVER_URL: "http://minio:9000" MINIO_REGION_NAME: local MINIO_DOMAIN: minio volumes: - minio:/data networks: redpanda_network: aliases: - redpanda.minio mc: depends_on: - minio image: minio/mc container_name: mc networks: - redpanda_network environment: - AWS_ACCESS_KEY_ID=minio - AWS_SECRET_ACCESS_KEY=redpandaTieredStorage7 - AWS_REGION=local entrypoint: > /bin/sh -c " until (/usr/bin/mc config host add minio http://minio:9000 minio redpandaTieredStorage7) do echo '...waiting...' && sleep 1; done; /usr/bin/mc mb minio/redpanda; /usr/bin/mc policy set public minio/redpanda; tail -f /dev/null " rpk-profile.yaml # This file configures `rpk` to connect to a remote Redpanda cluster running in the same local network as `rpk`. # Configuration for connecting to the Kafka API of the Redpanda cluster. kafka_api: # SASL (Simple Authentication and Security Layer) settings for authentication. sasl: user: superuser # The username used for authentication password: secretpassword # The password associated with the username mechanism: scram-sha-256 # Authentication mechanism; SCRAM-SHA-256 provides secure password-based authentication # List of Kafka brokers in the Redpanda cluster. # These brokers ensure high availability and fault tolerance for Kafka-based communication. brokers: - 127.0.0.1:19092 # Broker 1: Accessible on localhost, port 19092 - 127.0.0.1:29092 # Broker 2: Accessible on localhost, port 29092 - 127.0.0.1:39092 # Broker 3: Accessible on localhost, port 39092 # Configuration for connecting to the Redpanda Admin API. # The Admin API allows you to perform administrative tasks such as managing configurations, monitoring, and scaling. admin_api: # List of Admin API endpoints for managing the cluster. addresses: - 127.0.0.1:19644 # Admin API for Broker 1: Accessible on localhost, port 19644 - 127.0.0.1:29644 # Admin API for Broker 2: Accessible on localhost, port 29644 - 127.0.0.1:39644 # Admin API for Broker 3: Accessible on localhost, port 39644 bootstrap.yml # ================================================================= # This file defines initial cluster properties for a Redpanda cluster. # Some of these settings are intended for quickstart development and evaluation # and are not suitable for production environments. # # For more information on bootstrap files, see: # https://docs.redpanda.com/current/deploy/deployment-option/self-hosted/manual/production/production-deployment/#configure-a-bootstrap-file # ================================================================= # # Enable SASL authentication for the Kafka and Admin APIs. # https://docs.redpanda.com/current/reference/properties/cluster-properties/#admin_api_require_auth admin_api_require_auth: true # At least one superuser is required to be able to create other SASL users # https://docs.redpanda.com/current/reference/properties/cluster-properties/#superusers superusers: - superuser # https://docs.redpanda.com/current/reference/properties/cluster-properties/#enable_sasl enable_sasl: true # Allow topics to be created on first access. # https://docs.redpanda.com/current/reference/properties/cluster-properties/#auto_create_topics_enabled auto_create_topics_enabled: true # Enable data transforms. # https://docs.redpanda.com/current/develop/data-transforms/how-transforms-work/ data_transforms_enabled: true # Enable audit logging (enterprise feature). # https://docs.redpanda.com/current/manage/audit-logging/ audit_enabled: true # Enable Tiered Storage (enterprise feature). # https://docs.redpanda.com/current/manage/tiered-storage/ cloud_storage_enabled: true cloud_storage_region: local cloud_storage_access_key: minio cloud_storage_secret_key: redpandaTieredStorage7 cloud_storage_api_endpoint: minio cloud_storage_api_endpoint_port: 9000 cloud_storage_disable_tls: true cloud_storage_bucket: redpanda # Forces segments to be uploaded to Tiered Storage faster for the purposes of the quickstart # https://docs.redpanda.com/current/reference/properties/object-storage-properties/#cloud_storage_segment_max_upload_interval_sec cloud_storage_segment_max_upload_interval_sec: 60 # Continuous Data Balancing (enterprise feature) continuously monitors your node and rack availability and disk usage. This enables self-healing clusters that dynamically balance partitions, ensuring smooth operations and optimal cluster performance. # https://docs.redpanda.com/current/manage/cluster-maintenance/continuous-data-balancing/ partition_autobalancing_mode: continuous 3 Start Docker Compose. When the containers are running, you have: A Redpanda cluster with a 30-day free trial license. A running data streaming pipeline generating data into the logins and transactions topics. A Wasm data transform processing data from logins into edu-filtered-domains. A secured instance of Redpanda Console with login authentication. Explore Redpanda Console Redpanda Console is a developer-friendly web UI for managing and debugging your Redpanda cluster and your applications. This section provides practical examples and scenarios to help you understand how to leverage Redpanda Console for different use cases, including data observability, Redpanda management, access control, and connectivity. Redpanda Console was deployed as part of the docker-compose.yml file and is running locally on port 8080. Log in to Redpanda Console Redpanda Console is configured with plain login credentials (enterprise feature) for an admin user called jane. To start using Redpanda Console, you first need to log in with the credentials defined in the docker-compose.yml file. Open your web browser and go to http://localhost:8080/login. Enter the following credentials: Username: jane Password: some-other-secret-password This user is an admin user that has access to all features, including the Admin panel. You should now see an overview of your cluster’s status, health, and brokers. To view details about a specific broker, click View at the end of the row in the Broker Details table. See also: Plain Login Setup. View topics and filter messages This quickstart deployment comes with two pre-configured topics, logins and transactions, actively receiving data from a Redpanda Connect pipeline. These topics simulate real-world data flows, enabling you to explore and interact with Redpanda Console features. You can filter messages in topics using several methods: Text search: Search for specific text within the message. Partition offset filters: Set the start and end offsets to narrow down the message range. JavaScript filters: Use JavaScript function bodies to create complex filters that run server-side, allowing you to filter through topics with millions of records. Suppose you’re asked to find all transactions related to the .edu domain. You can use a JavaScript filter to display only messages that include email addresses in that domain. In the left menu, click Topics. Click the transactions topic. When the topic’s page opens, the Messages tab is selected by default. Click Add filter > JavaScript Filter. Give your filter a name, such as "Find .edu domains". Replace the default JavaScript code with the following: return value.email.includes(".edu"); Click Save to apply the filter. You should see only messages that include the specific domain. By default, the filter runs on the newest 50 messages. You can run the filter on older messages by changing the start offset. For example, to start from the oldest messages, select Beginning. See also: Filter Messages with JavaScript in Redpanda Console. Explore Schema Registry On the Schema Registry page, you see an overview of your schemas. You can create, manage, and inspect your schemas without leaving Redpanda Console. See also: Use Schema Registry in Redpanda Console. Configure access control for Redpanda Managing access control lists (ACLs) can be complex, but Redpanda Console simplifies this with an intuitive interface. This quickstart deployment includes a bootstrapped superuser with full administrative privileges. The superuser has unrestricted access to all topics, consumer groups, transactions, and cluster-level operations. By default, the credentials for the superuser are: Username: superuser Password: secretpassword For improved security and role-based access control (RBAC), you should create additional users with restricted permissions as needed. On the Security page, you can: Create, view, and edit ACLs for topics, consumer groups, transactions, and the entire cluster. Quickly understand what each principal (user) is authorized to do. Suppose you’re onboarding a new team member who needs access to specific topics. You can use Redpanda Console to ensure users have the right permissions to perform their tasks. In the left menu, click Security. On the Users tab, click Create user. Enter "Sasha" in the username field. Enter "sashapassword" in the password field. Click Create. Click Done, and you see a new user called Sasha. This user has no permissions yet. To set permissions on the transactions topic: On the Access control page, open to the Roles tab. Click Create role. Enter "transaction-managers" as the role name. In the topic selector dropdown, select Literal and enter "transactions" in the input field. Under Operations, click the All dropdown and select Allow. Scroll down to the bottom of the page and under Principals select Sasha from the dropdown. Click Create. Now Sasha has full access only to the topic called transactions. To test these permissions, use rpk to connect to Redpanda as the user Sasha. Try to access the logins topic as Sasha: docker exec -it redpanda-0 rpk topic describe logins \ -X user=Sasha \ -X pass=sashapassword \ -X sasl.mechanism=SCRAM-SHA-256 \ -X brokers=redpanda-0:9092 You are not authorized to view the topic. SUMMARY ======= NAME logins PARTITIONS 0 ERROR TOPIC_AUTHORIZATION_FAILED: Not authorized to access topics: [Topic authorization failed.] CONFIGS ======= config response contained error: TOPIC_AUTHORIZATION_FAILED: Not authorized to access topics: [Topic authorization failed.] Now try to access the transactions topic as Sasha: docker exec -it redpanda-0 rpk topic describe transactions \ -X user=Sasha \ -X pass=sashapassword \ -X sasl.mechanism=SCRAM-SHA-256 \ -X brokers=redpanda-0:9092 You have access to this topic. Example output SUMMARY ======= NAME transactions PARTITIONS 1 REPLICAS 1 CONFIGS ======= KEY VALUE SOURCE cleanup.policy delete DEFAULT_CONFIG compression.type producer DEFAULT_CONFIG delete.retention.ms -1 DEFAULT_CONFIG flush.bytes 262144 DEFAULT_CONFIG flush.ms 100 DEFAULT_CONFIG initial.retention.local.target.bytes -1 DEFAULT_CONFIG initial.retention.local.target.ms -1 DEFAULT_CONFIG max.message.bytes 1048576 DEFAULT_CONFIG message.timestamp.type CreateTime DEFAULT_CONFIG redpanda.iceberg.delete true DEFAULT_CONFIG redpanda.iceberg.mode disabled DEFAULT_CONFIG redpanda.leaders.preference none DEFAULT_CONFIG redpanda.remote.delete true DEFAULT_CONFIG redpanda.remote.read true DEFAULT_CONFIG redpanda.remote.write true DEFAULT_CONFIG retention.bytes -1 DEFAULT_CONFIG retention.local.target.bytes -1 DEFAULT_CONFIG retention.local.target.ms 86400000 DEFAULT_CONFIG retention.ms 604800000 DEFAULT_CONFIG segment.bytes 134217728 DEFAULT_CONFIG segment.ms 1209600000 DEFAULT_CONFIG write.caching true DEFAULT_CONFIG Explore Data transforms Data transforms let you run common data streaming tasks on the Redpanda broker, like filtering, scrubbing, and transcoding. For example, you may have consumers that require you to redact credit card numbers or convert JSON to Avro. This quickstart deployment comes with one transform function called regex running in your cluster. Its job is to find records in the logins topic that contain email addresses with the .edu domain and add those to a new topic called edu-filtered-domains. In the left menu, click Transforms. On the Transforms page, you see your transform. You can use Redpanda Console to manage and monitor your transforms. The source code for the data transform function is in the docker-compose/transform/transform.go file that you downloaded. This file is commented to explain how it works. See also: Manage Data Transforms in Redpanda Console Data Transforms View audit logs Audit logs provide a record of all user actions. You can use these logs to track changes, troubleshoot issues, and maintain compliance with your organization’s security policies. In the left menu, click Topics. Click the Show internal topics checkbox. Click the _redpanda.audit_log topic in the table. See also: Audit Logging. Start streaming To start building a basic streaming application, you can use the Redpanda CLI (rpk) to create a topic, produce messages to it, and consume messages from it. Each Redpanda broker comes preinstalled with rpk, so you can use it inside one of the Redpanda broker’s Docker containers. For simplicity, this quickstart uses the bootstrapped superuser (superuser) to perform all operations. The superuser has full administrative privileges, making it convenient for setup and exploration. However, to enhance security in a production environment, Redpanda Data recommends creating users with restricted permissions for each task. To use rpk inside the Redpanda broker’s Docker container: Get information about the cluster: docker exec -it redpanda-0 rpk cluster info -X user=superuser -X pass=secretpassword Create a topic called chat-room: docker exec -it redpanda-0 rpk topic create chat-room \ --replicas 3 \ (1) --topic-config redpanda.remote.read=true \ (2) --topic-config redpanda.remote.write=true \ (3) -X user=superuser \ -X pass=secretpassword 1 Set a replication factor of 3 to replicate the topic across all 3 brokers. This replication factor provides high availability and data durability. For more details, see Choose the replication factor. 2 Enable remote reads for this topic to read offloaded records from object storage. 3 Enable remote writes for this topic to offload older records to object storage. For more details, see Use Tiered Storage. Output: TOPIC STATUS chat-room OK Produce a message to the topic: docker exec -it redpanda-0 rpk topic produce chat-room -X user=superuser -X pass=secretpassword Enter a message, then press Enter: Pandas are fabulous! Example output: Produced to partition 0 at offset 0 with timestamp 1663282629789. Press Ctrl+C to finish producing messages to the topic. Consume one message from the topic: docker exec -it redpanda-0 rpk topic consume chat-room --num 1 -X user=superuser -X pass=secretpassword Your message is displayed along with its metadata: { "topic": "chat-room", "value": "Pandas are fabulous!", "timestamp": 1663282629789, "partition": 0, "offset": 0 } Connect to the cluster externally It’s often more practical to connect to a remote cluster from your local machine with rpk. The docker-compose.yml file configured the containers to expose ports on your localhost, so you can communicate with the cluster outside the Docker network. To do so, download rpk on your local machine and configure it to connect to your cluster on the exposed ports. Install rpk on your local machine: Linux macOS amd64 arm64 curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-amd64.zip && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-linux-amd64.zip -d ~/.local/bin/ curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-arm64.zip && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-linux-arm64.zip -d ~/.local/bin/ You can use rpk on Windows only with WSL. However, commands that require Redpanda to be installed on your machine are not supported, such as rpk container commands, rpk iotune, and rpk redpanda commands. If you don’t have Homebrew installed, install it. To install or update rpk, run: brew install redpanda-data/tap/redpanda Create a profile to configure rpk to connect to your cluster: rpk profile create quickstart --from-profile rpk-profile.yaml The profile is configured in the docker-compose/rpk-profile.yaml file that you downloaded. This file is commented to explain how it works. Get information about the cluster to test the connection: rpk cluster info The Redpanda broker returns a list of all broker addresses, so rpk can communicate with all brokers directly. Each broker returns its configured advertise-* address that matches the port to which the initial connection has been made. You can see examples of these addresses in the Docker Compose file that you downloaded. View offloaded data in Tiered Storage Tiered Storage keeps local disk usage stable and performance consistent by offloading older data to object storage. You can also use Tiered Storage to recover clusters or topics, mount and unmount topics from a cluster, and more. Redpanda supports Amazon S3, Azure ADLS, and Google GCS as object storage providers. For this quickstart, Redpanda uses MinIO as the object storage provider. MinIO is not supported for production deployments. Check the size of remote and local disks for the chat-room topic you created earlier: rpk cluster logdirs describe --topics chat-room BROKER DIR TOPIC PARTITION SIZE ERROR 0 /var/lib/redpanda/data chat-room 0 611 0 remote://redpanda chat-room 0 319 1 /var/lib/redpanda/data chat-room 0 611 1 remote://redpanda chat-room 0 319 2 /var/lib/redpanda/data chat-room 0 611 2 remote://redpanda chat-room 0 319 Local storage: /var/lib/redpanda/data holds recent data. Remote storage: remote://redpanda holds offloaded data. Open MinIO at http://localhost:9001/browser to view your data stored in the S3-compatible object store. Login credentials: Username: minio Password: redpandaTieredStorage7 See also: Use Tiered Storage. Deploy a pipeline with Redpanda Connect To deploy a data streaming pipeline in Redpanda, you can configure and run connectors that ingest, transform, and route data between various sources and sinks. Redpanda Connect simplifies this process by allowing you to manage these pipelines using declarative YAML configuration files and the rpk CLI. Deploy the example pipeline: rpk connect run ../generate-profiles.yaml This pipeline is configured to generate fake user profiles and produce them to the profiles topic in your Redpanda cluster. It’ll run until you stop it. The pipeline configuration is in the docker-compose/generate-profiles.yaml file that you downloaded. This file is commented to explain how it works. After a few seconds, stop the pipeline by pressing Ctrl+C. Consume 10 messages from the profiles topic to confirm that the pipeline produced data to that topic. rpk topic consume profiles --num 10 Extend your 30-day trial Your initial trial license is valid for 30 days. To extend it by another 30 days: Sign up for a new trial license. Upload the new license to your running cluster using rpk cluster license set: rpk cluster license set <license> Replace the <license> placeholder with your license. This command applies the new license to your Redpanda cluster, extending your access to enterprise features for another 30 days. Restart Redpanda Console so that it can fetch the new license from the Redpanda brokers: docker restart redpanda-console You can extend the trial license only once. Upgrade to Redpanda Enterprise. See also: Redpanda Licensing. Customize the Docker quickstart To customize this quickstart, you can configure Redpanda, Redpanda Console, or Redpanda Connect using the provided Docker Compose file. This allows you to tailor the services to fit your specific requirements. Redpanda broker properties To configure the Redpanda services with broker configuration properties, pass properties with the --set option in the redpanda start command in your Docker Compose file. Example: docker-compose.yml redpanda-0: (1) command: (2) - redpanda - start - --set pandaproxy_client.retries=6 (3) 1 The service name in Docker Compose. 2 Overrides the default command for the Redpanda container. 3 Sets the pandaproxy_client.retries broker configuration property to 6. Redpanda cluster properties Cluster-level configurations are defined in the bootstrap.yml file. This file is included in your Docker Compose setup. It contains essential settings that apply to the entire Redpanda cluster. You can find all cluster properties in Cluster Configuration Properties. Example: docker-compose.yml redpanda: volumes: - ./bootstrap.yml:/etc/redpanda/.bootstrap.yaml (1) 1 Mounts the bootstrap.yml file from your local parent directory to the container’s /etc/redpanda/ directory. Redpanda Console configuration Redpanda Console configuration is managed with the environment variable CONSOLE_CONFIG_FILE. You can specify the path to your configuration file in the Docker Compose file. Example: docker-compose.yml console: environment: CONSOLE_CONFIG_FILE: | # Configure a connection to the Redpanda cluster # See https://docs.redpanda.com/current/console/config/connect-to-redpanda/ kafka: brokers: ["redpanda-0:9092"] schemaRegistry: enabled: true urls: ["http://redpanda-0:8081","http://redpanda-1:8081","http://redpanda-2:8081"] sasl: enabled: true username: superuser password: secretpassword mechanism: SCRAM-SHA-256 Redpanda Connect configuration Existing data pipelines are configured using the CONNECT_CFG_FILE environment variable in the connect service. This environment variable contains YAML configuration settings for Redpanda Connect. Example: docker-compose.yml connect: environment: CONNECT_CFG_FILE: | input: generate: interval: 1s mapping: | let first_name = fake("first_name") let last_name = fake("last_name") root.user_id = counter() root.name = $$first_name + " " + $$last_name root.email = ($$first_name.slice(0,1) + $$last_name + "@" + fake("domain_name")).lowercase() root.ip = fake("ipv4") root.login_time = now() pipeline: processors: - mapping: | root = range(0, random_int(min:2, max:4)).map_each(cust -> this) - unarchive: format: "json_array" - mapping: | if batch_index() == 0 { meta topic = "logins" root = this } else { meta topic = "transactions" root.user_id = this.user_id root.email = this.email root.index = batch_index() - 1 root.product_url = fake("url") root.price = fake("amount_with_currency") root.timestamp = now() } output: kafka_franz: seed_brokers: [ "redpanda-0:9092" ] topic: $${! metadata("topic") } sasl: - mechanism: SCRAM-SHA-256 password: secretpassword username: superuser To create new data pipelines, run rpk connect run, specifying the path to your own YAML configuration file: rpk connect run <path-to-config> Clean up To reset your environment, shut down the running processes and delete the containers: docker compose down To delete the volumes along with all your cluster data: docker compose down -v Next steps Contact Redpanda to discuss using Redpanda Enterprise Edition in production. Try more examples in Redpanda Labs Learn more about enterprise features for Redpanda Self-Managed Deploy for development or production Suggested reading Redpanda Connect Documentation rpk Commands Introduction to Redpanda Console Docker images Docker images for Redpanda Console Docker images for Redpanda Back to top × Simple online edits For simple changes, such as fixing a typo, you can edit the content directly on GitHub. Edit on GitHub Or, open an issue to let us know about something that you want us to change. Open an issue Contribution guide For extensive content updates, or if you prefer to work locally, read our contribution guide . Was this helpful? thumb_up thumb_down group Ask in the community mail Share your feedback group_add Make a contribution Quickstarts Redpanda CLI Quickstart