# Redpanda Streaming Quickstart > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Redpanda Streaming Quickstart latest-redpanda-tag: v24.3.9 latest-console-tag: v3.7.3 latest-operator-version: v26.1.4 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "true" page-eol-date: December 3, 2025 latest-connect-version: 4.93.0 docname: quick-start page-component-name: streaming page-version: "24.3" page-component-version: "24.3" page-component-title: Streaming page-relative-src-path: quick-start.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/v/24.3/modules/get-started/pages/quick-start.adoc description: Learn how to quickly start working with a local Redpanda cluster that comes with a free 30-day license for Enterprise Edition. You can also extend the trial license for a further 30 days. page-git-created-date: "2023-05-30" page-git-modified-date: "2026-05-26" support-status: past end-of-life --- Learn how to quickly start working with a local Redpanda cluster that comes with a free 30-day license for Enterprise Edition. You can also extend the trial license for a further 30 days. Redpanda Streaming is a modern streaming platform, compatible with Kafka APIs, designed for speed, simplicity, and efficiency. In this quickstart, you: - Deploy a three-broker Redpanda cluster. - Explore streaming data in Redpanda Console. - Learn the basics of streaming with the Redpanda CLI (`rpk`). - Deploy a streaming pipeline with Redpanda Connect. - Learn how to extend your trial license for an additional 30 days. > 📝 **NOTE** > > This quickstart uses Docker to run Redpanda, which is only for development and testing purposes. For production deployments, refer to the [Linux deployment guides](https://docs.redpanda.com/streaming/24.3/deploy/deployment-option/self-hosted/manual/) or the [Kubernetes deployment guides](https://docs.redpanda.com/streaming/24.3/deploy/deployment-option/self-hosted/kubernetes/). To download the Redpanda binary, see [GitHub](https://github.com/redpanda-data/redpanda/releases/latest). > > **Looking for a managed solution?** You can also get started quickly with a hosted Redpanda cluster by signing up for [Redpanda Cloud](https://cloud.redpanda.com). ## [](#enterprise-features)Enterprise features The Enterprise Edition of Redpanda Streaming adds advanced capabilities to help scale, manage, optimize, and secure your cluster. All new Redpanda clusters include a built-in 30-day Enterprise Edition license, so you can evaluate enterprise features. Some features highlighted in this quickstart require an enterprise license: - **Redpanda Console Authentication**: Securely control login access to Redpanda Console. - **Audit Logging**: Track and monitor all user actions in a Redpanda topic. - **Continuous Data Balancing**: Automatically balance data across brokers to optimize performance. - **Tiered Storage**: Lower storage costs by offloading log segments to object storage. Your trial license also includes additional enterprise features. For more information, see [Redpanda Licenses and Enterprise Features](https://docs.redpanda.com/streaming/24.3/get-started/licensing/overview/). > 💡 **TIP: Want a longer trial?** > > [Sign up for an extended trial license](https://cloud.redpanda.com/try-enterprise). ## [](#prerequisites)Prerequisites You need Docker Compose. For installation instructions, see the [Docker Compose documentation](https://docs.docker.com/compose/install/). To check if you have Docker Compose installed, run: ```bash docker compose version ``` You should see the version of Docker Compose that’s installed on your local machine. > 📝 **NOTE: Windows users** > > If you’re on Windows, use a compatible terminal that supports Unix-like commands. Or, consider using [Windows Subsystem for Linux (WSL)](https://docs.microsoft.com/en-us/windows/wsl/install). ## [](#deploy-redpanda-streaming)Deploy Redpanda Streaming To download, extract, and start Redpanda Streaming in Docker, run: ```bash mkdir redpanda-quickstart && cd redpanda-quickstart && \ (1) curl -sSL https://docs.redpanda.com/24.3-redpanda-quickstart.tar.gz | tar xzf - && \ (2) cd docker-compose && docker compose up -d (3) ``` | 1 | Create and navigate to the redpanda-quickstart directory. | | --- | --- | | 2 | Download and extract the archive. Explore the downloaded filesdocker-compose.ymlname: redpanda-quickstart-multi-broker networks: redpanda_network: driver: bridge volumes: redpanda-0: null redpanda-1: null redpanda-2: null minio: null services: ################## # Redpanda Brokers # ################## redpanda-0: command: - redpanda - start - --kafka-addr internal://0.0.0.0:9092,external://0.0.0.0:19092 # Address the broker advertises to clients that connect to the Kafka API. # Use the internal addresses to connect to the Redpanda brokers # from inside the same Docker network. # Use the external addresses to connect to the Redpanda brokers # from outside the Docker network. - --advertise-kafka-addr internal://redpanda-0:9092,external://localhost:19092 - --pandaproxy-addr internal://0.0.0.0:8082,external://0.0.0.0:18082 # Address the broker advertises to clients that connect to the HTTP Proxy. - --advertise-pandaproxy-addr internal://redpanda-0:8082,external://localhost:18082 - --schema-registry-addr internal://0.0.0.0:8081,external://0.0.0.0:18081 # Redpanda brokers use the RPC API to communicate with each other internally. - --rpc-addr redpanda-0:33145 - --advertise-rpc-addr redpanda-0:33145 # Mode dev-container uses well-known configuration properties for development in containers. - --mode dev-container # Tells Seastar (the framework Redpanda uses under the hood) to use 1 core on the system. - --smp 1 - --default-log-level=info image: docker.redpanda.com/redpandadata/redpanda:24.3.9 container_name: redpanda-0 # Sets the username and password of the bootstrap SCRAM superuser # See https://docs.redpanda.com/current/deploy/deployment-option/self-hosted/manual/production/production-deployment/#bootstrap-a-user-account environment: RP_BOOTSTRAP_USER: "superuser:secretpassword" volumes: - redpanda-0:/var/lib/redpanda/data - ./bootstrap.yml:/etc/redpanda/.bootstrap.yaml networks: - redpanda_network ports: - 18081:18081 - 18082:18082 - 19092:19092 - 19644:9644 healthcheck: test: ["CMD", "rpk", "cluster", "info", "-X", "user=superuser", "-X", "pass=secretpassword"] interval: 10s timeout: 5s retries: 10 depends_on: - minio redpanda-1: command: - redpanda - start - --kafka-addr internal://0.0.0.0:9092,external://0.0.0.0:29092 - --advertise-kafka-addr internal://redpanda-1:9092,external://localhost:29092 - --pandaproxy-addr internal://0.0.0.0:8082,external://0.0.0.0:28082 - --advertise-pandaproxy-addr internal://redpanda-1:8082,external://localhost:28082 - --schema-registry-addr internal://0.0.0.0:8081,external://0.0.0.0:28081 - --rpc-addr redpanda-1:33145 - --advertise-rpc-addr redpanda-1:33145 - --mode dev-container - --smp 1 - --default-log-level=info - --seeds redpanda-0:33145 image: docker.redpanda.com/redpandadata/redpanda:24.3.9 container_name: redpanda-1 environment: RP_BOOTSTRAP_USER: "superuser:secretpassword" volumes: - redpanda-1:/var/lib/redpanda/data - ./bootstrap.yml:/etc/redpanda/.bootstrap.yaml networks: - redpanda_network ports: - 28081:28081 - 28082:28082 - 29092:29092 - 29644:9644 depends_on: - redpanda-0 - minio redpanda-2: command: - redpanda - start - --kafka-addr internal://0.0.0.0:9092,external://0.0.0.0:39092 - --advertise-kafka-addr internal://redpanda-2:9092,external://localhost:39092 - --pandaproxy-addr internal://0.0.0.0:8082,external://0.0.0.0:38082 - --advertise-pandaproxy-addr internal://redpanda-2:8082,external://localhost:38082 - --schema-registry-addr internal://0.0.0.0:8081,external://0.0.0.0:38081 - --rpc-addr redpanda-2:33145 - --advertise-rpc-addr redpanda-2:33145 - --mode dev-container - --smp 1 - --default-log-level=info - --seeds redpanda-0:33145 image: docker.redpanda.com/redpandadata/redpanda:24.3.9 container_name: redpanda-2 environment: RP_BOOTSTRAP_USER: "superuser:secretpassword" volumes: - redpanda-2:/var/lib/redpanda/data - ./bootstrap.yml:/etc/redpanda/.bootstrap.yaml networks: - redpanda_network ports: - 38081:38081 - 38082:38082 - 39092:39092 - 39644:9644 depends_on: - redpanda-0 - minio #################### # Redpanda Console # #################### console: container_name: redpanda-console image: docker.redpanda.com/redpandadata/console:3.7.3 networks: - redpanda_network entrypoint: /bin/sh command: -c 'echo "$$CONSOLE_CONFIG_FILE" > /tmp/config.yml && echo "$$CONSOLE_ROLEBINDINGS_CONFIG_FILE" > /tmp/role-bindings.yml && /app/console' volumes: - ./config:/tmp/config/ environment: CONFIG_FILEPATH: ${CONFIG_FILEPATH:-/tmp/config.yml} CONSOLE_CONFIG_FILE: | # Configure a connection to the Redpanda cluster # See https://docs.redpanda.com/current/console/config/connect-to-redpanda/ kafka: brokers: ["redpanda-0:9092"] schemaRegistry: enabled: true urls: ["http://redpanda-0:8081","http://redpanda-1:8081","http://redpanda-2:8081"] sasl: enabled: true username: superuser password: secretpassword mechanism: SCRAM-SHA-256 redpanda: adminApi: enabled: true urls: ["http://redpanda-0:9644","http://redpanda-1:9644","http://redpanda-2:9644"] username: superuser password: secretpassword console: # Configures Redpanda Console to fetch topic documentation from GitHub and display it in the UI. # See https://docs.redpanda.com/current/console/config/topic-documentation/ topicDocumentation: enabled: true git: enabled: true repository: url: https://github.com/redpanda-data/docs branch: main baseDirectory: tests/docker-compose login: enabled: true jwtSecret: change-this-to-something-secret useSecureCookies: false # Enable plain login authentication (enterprise feature) # See https://docs.redpanda.com/current/console/config/security/plain/ plain: enabled: true credentials: - username: "jane" password: "some-other-secret-password" - username: "john" password: "some-secret-password" enterprise: # Configures role-based access control to data and features in Redpanda Console. # See https://docs.redpanda.com/current/console/config/security/authorization/ rbac: enabled: true roleBindingsFilepath: /tmp/role-bindings.yml CONSOLE_ROLEBINDINGS_CONFIG_FILE: | roleBindings: - metadata: name: Platform Ops subjects: - kind: user provider: Plain name: jane roleName: admin - metadata: name: Software Engineers subjects: - kind: user provider: Plain name: john roleName: editor ports: - 8080:8080 depends_on: redpanda-0: condition: service_healthy #################### # Redpanda Connect # #################### connect: container_name: redpanda-connect image: docker.redpanda.com/redpandadata/connect networks: - redpanda_network entrypoint: /bin/sh depends_on: redpanda-0: condition: service_healthy command: -c 'echo "$$CONNECT_CFG_FILE" > /tmp/connect.yml; /redpanda-connect -c /tmp/connect.yml' environment: # This Redpanda Connect configuration creates fake data, # processes it, and writes the output to a set of topics. # # Input: # - Uses Redpanda Connect's generate input to generate fake data. # See https://docs.redpanda.com/redpanda-connect/components/inputs/generate/ # Pipeline: # - Bloblang mapping to batch each input and map 1 message to 'logins' # topic, and a random number (1-3) of messages to 'transaction' topic # - Unarchive processor to parse the JSON array and extract each # element into its own message. # See https://docs.redpanda.com/redpanda-connect/guides/bloblang/about/ # Output: # - kafka_franz output to write the messages to the Redpanda brokers. # See https://docs.redpanda.com/redpanda-connect/components/outputs/kafka_franz/ CONNECT_CFG_FILE: | input: generate: interval: 1s mapping: | let first_name = fake("first_name") let last_name = fake("last_name") root.user_id = counter() root.name = $$first_name + " " + $$last_name root.email = ($$first_name.slice(0,1) + $$last_name + "@" + fake("domain_name")).lowercase() root.ip = fake("ipv4") root.login_time = now() pipeline: processors: - mapping: | root = range(0, random_int(min:2, max:4)).map_each(cust -> this) - unarchive: format: "json_array" - mapping: | if batch_index() == 0 { meta topic = "logins" root = this } else { meta topic = "transactions" root.user_id = this.user_id root.email = this.email root.index = batch_index() - 1 root.product_url = fake("url") root.price = fake("amount_with_currency") root.timestamp = now() } output: kafka_franz: seed_brokers: [ "redpanda-0:9092" ] topic: $${! metadata("topic") } sasl: - mechanism: SCRAM-SHA-256 password: secretpassword username: superuser #################### # rpk container to create the edu-filtered-domains topic # # See https://docs.redpanda.com/current/reference/rpk/rpk-topic/rpk-topic-create/ #################### createtopic: command: - topic - create - edu-filtered-domains - -X user=superuser - -X pass=secretpassword - -X brokers=redpanda-0:9092 image: docker.redpanda.com/redpandadata/redpanda:24.3.9 networks: - redpanda_network depends_on: redpanda-0: condition: service_healthy #################### # rpk container to register the schema # # See https://docs.redpanda.com/current/manage/schema-reg/schema-reg-api/ #################### registerschema: command: - registry - schema - create - transactions - --schema - /etc/redpanda/transactions-schema.json - -X user=superuser - -X pass=secretpassword - -X registry.hosts=redpanda-0:8081 image: docker.redpanda.com/redpandadata/redpanda:24.3.9 # Mount the local directory that contains your schema to the container. volumes: - ./transactions-schema.json:/etc/redpanda/transactions-schema.json networks: - redpanda_network depends_on: redpanda-0: condition: service_healthy #################### # rpk container to deploy the pre-built data transform # # See https://docs.redpanda.com/current/develop/data-transforms/deploy/ #################### deploytransform: command: - transform - deploy - --file=/etc/redpanda/regex.wasm - --name=regex - --input-topic=logins - --output-topic=edu-filtered-domains - --var=PATTERN="[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.edu" - --var=MATCH_VALUE=true - -X user=superuser - -X pass=secretpassword - -X admin.hosts=redpanda-0:9644 image: docker.redpanda.com/redpandadata/redpanda:24.3.9 volumes: - ./transform/regex.wasm:/etc/redpanda/regex.wasm networks: - redpanda_network depends_on: redpanda-0: condition: service_healthy createtopic: condition: service_started #################### # MinIO for Tiered Storage # # See https://min.io/ # # NOTE: MinIO is included in this quickstart for development and evaluation purposes only. # It is not supported for production deployments of Redpanda. # # For production environments, use one of the supported object storage providers: # https://docs.redpanda.com/current/deploy/deployment-option/self-hosted/manual/production/requirements/#object-storage-providers-for-tiered-storage #################### minio: container_name: minio image: minio/minio command: server --console-address ":9001" /data ports: - 9000:9000 - 9001:9001 environment: MINIO_ROOT_USER: minio MINIO_ROOT_PASSWORD: redpandaTieredStorage7 MINIO_SERVER_URL: "http://minio:9000" MINIO_REGION_NAME: local MINIO_DOMAIN: minio volumes: - minio:/data networks: redpanda_network: aliases: - redpanda.minio mc: depends_on: - minio image: minio/mc container_name: mc networks: - redpanda_network environment: - AWS_ACCESS_KEY_ID=minio - AWS_SECRET_ACCESS_KEY=redpandaTieredStorage7 - AWS_REGION=local entrypoint: > /bin/sh -c " until (/usr/bin/mc config host add minio http://minio:9000 minio redpandaTieredStorage7) do echo '...waiting...' && sleep 1; done; /usr/bin/mc mb minio/redpanda; /usr/bin/mc policy set public minio/redpanda; tail -f /dev/null "rpk-profile.yaml# This file configures `rpk` to connect to a remote Redpanda cluster running in the same local network as `rpk`. # Configuration for connecting to the Kafka API of the Redpanda cluster. kafka_api: # SASL (Simple Authentication and Security Layer) settings for authentication. sasl: user: superuser # The username used for authentication password: secretpassword # The password associated with the username mechanism: scram-sha-256 # Authentication mechanism; SCRAM-SHA-256 provides secure password-based authentication # List of Kafka brokers in the Redpanda cluster. # These brokers ensure high availability and fault tolerance for Kafka-based communication. brokers: - 127.0.0.1:19092 # Broker 1: Accessible on localhost, port 19092 - 127.0.0.1:29092 # Broker 2: Accessible on localhost, port 29092 - 127.0.0.1:39092 # Broker 3: Accessible on localhost, port 39092 # Configuration for connecting to the Redpanda Admin API. # The Admin API allows you to perform administrative tasks such as managing configurations, monitoring, and scaling. admin_api: # List of Admin API endpoints for managing the cluster. addresses: - 127.0.0.1:19644 # Admin API for Broker 1: Accessible on localhost, port 19644 - 127.0.0.1:29644 # Admin API for Broker 2: Accessible on localhost, port 29644 - 127.0.0.1:39644 # Admin API for Broker 3: Accessible on localhost, port 39644bootstrap.yml# ================================================================= # This file defines initial cluster properties for a Redpanda cluster. # Some of these settings are intended for quickstart development and evaluation # and are not suitable for production environments. # # For more information on bootstrap files, see: # https://docs.redpanda.com/current/deploy/deployment-option/self-hosted/manual/production/production-deployment/#configure-a-bootstrap-file # ================================================================= # # Enable SASL authentication for the Kafka and Admin APIs. # https://docs.redpanda.com/current/reference/properties/cluster-properties/#admin_api_require_auth admin_api_require_auth: true # At least one superuser is required to be able to create other SASL users # https://docs.redpanda.com/current/reference/properties/cluster-properties/#superusers superusers: - superuser # https://docs.redpanda.com/current/reference/properties/cluster-properties/#enable_sasl enable_sasl: true # Allow topics to be created on first access. # https://docs.redpanda.com/current/reference/properties/cluster-properties/#auto_create_topics_enabled auto_create_topics_enabled: true # Enable data transforms. # https://docs.redpanda.com/current/develop/data-transforms/how-transforms-work/ data_transforms_enabled: true # Enable audit logging (enterprise feature). # https://docs.redpanda.com/current/manage/audit-logging/ audit_enabled: true # Enable Tiered Storage (enterprise feature). # https://docs.redpanda.com/current/manage/tiered-storage/ cloud_storage_enabled: true cloud_storage_region: local cloud_storage_access_key: minio cloud_storage_secret_key: redpandaTieredStorage7 cloud_storage_api_endpoint: minio cloud_storage_api_endpoint_port: 9000 cloud_storage_disable_tls: true cloud_storage_bucket: redpanda # Forces segments to be uploaded to Tiered Storage faster for the purposes of the quickstart # https://docs.redpanda.com/current/reference/properties/object-storage-properties/#cloud_storage_segment_max_upload_interval_sec cloud_storage_segment_max_upload_interval_sec: 60 # Continuous Data Balancing (enterprise feature) continuously monitors your node and rack availability and disk usage. This enables self-healing clusters that dynamically balance partitions, ensuring smooth operations and optimal cluster performance. # https://docs.redpanda.com/current/manage/cluster-maintenance/continuous-data-balancing/ partition_autobalancing_mode: continuous | | 3 | Start Docker Compose. | When the containers are running, you have: - A Redpanda cluster with a 30-day free trial license. - A running data streaming pipeline generating data into the `logins` and `transactions` topics. - A Wasm data transform processing data from `logins` into `edu-filtered-domains`. - A secured instance of Redpanda Console with login authentication. ## [](#explore-redpanda-console)Explore Redpanda Console Redpanda Console is a developer-friendly web UI for managing and debugging your Redpanda cluster and your applications. This section provides practical examples and scenarios to help you understand how to leverage Redpanda Console for different use cases, including data observability, Redpanda management, access control, and connectivity. Redpanda Console was deployed as part of the `docker-compose.yml` file and is running locally on port 8080. ### [](#log-in-to-redpanda-console)Log in to Redpanda Console Redpanda Console is configured with plain login credentials (enterprise feature) for an admin user called `jane`. To start using Redpanda Console, you first need to log in with the credentials defined in the `docker-compose.yml` file. 1. Open your web browser and go to [http://localhost:8080/login](http://localhost:8080/login). 2. Enter the following credentials: - Username: `jane` - Password: `some-other-secret-password` This user is an admin user that has access to all features, including the Admin panel. You should now see an overview of your cluster’s status, health, and brokers. ![overview](https://docs.redpanda.com/streaming/24.3/console/_images/overview.png) To view details about a specific broker, click **View** at the end of the row in the **Broker Details** table. ![broker overview](https://docs.redpanda.com/streaming/24.3/console/_images/broker-overview.png) See also: [Plain Login Setup](https://docs.redpanda.com/streaming/24.3/console/config/security/plain/). ### [](#view-topics-and-filter-messages)View topics and filter messages This quickstart deployment comes with two pre-configured topics, `logins` and `transactions`, actively receiving data from a Redpanda Connect pipeline. These topics simulate real-world data flows, enabling you to explore and interact with Redpanda Console features. You can filter messages in topics using several methods: - Text search: Search for specific text within the message. - Partition offset filters: Set the start and end offsets to narrow down the message range. - JavaScript filters: Use JavaScript function bodies to create complex filters that run server-side, allowing you to filter through topics with millions of records. Suppose you’re asked to find all transactions related to the `.edu` domain. You can use a JavaScript filter to display only messages that include email addresses in that domain. 1. In the left menu, click **Topics**. 2. Click the **transactions** topic. When the topic’s page opens, the **Messages** tab is selected by default. 3. Click **Add filter** > **JavaScript Filter**. 4. Give your filter a name, such as "Find .edu domains". ![js filter](https://docs.redpanda.com/streaming/24.3/console/_images/js-filter.png) 5. Replace the default JavaScript code with the following: ```js return value.email.includes(".edu"); ``` 6. Click **Save** to apply the filter. You should see only messages that include the specific domain. By default, the filter runs on the newest 50 messages. You can run the filter on older messages by changing the start offset. For example, to start from the oldest messages, select **Beginning**. See also: [Filter Messages with JavaScript in Redpanda Console](https://docs.redpanda.com/streaming/24.3/console/ui/programmable-push-filters/). ### [](#explore-schema-registry)Explore Schema Registry On the **Schema Registry** page, you see an overview of your schemas. You can create, manage, and inspect your schemas without leaving Redpanda Console. ![schema reg](https://docs.redpanda.com/streaming/24.3/console/_images/schema-reg.png) See also: [Use Schema Registry in Redpanda Console](https://docs.redpanda.com/streaming/24.3/console/ui/schema-reg/). ### [](#configure-access-control-for-redpanda)Configure access control for Redpanda Managing access control lists (ACLs) can be complex, but Redpanda Console simplifies this with an intuitive interface. This quickstart deployment includes a bootstrapped superuser with full administrative privileges. The superuser has unrestricted access to all topics, consumer groups, transactions, and cluster-level operations. By default, the credentials for the superuser are: - Username: `superuser` - Password: `secretpassword` For improved security and role-based access control (RBAC), you should create additional users with restricted permissions as needed. On the **Security** page, you can: - Create, view, and edit ACLs for topics, consumer groups, transactions, and the entire cluster. - Quickly understand what each principal (user) is authorized to do. Suppose you’re onboarding a new team member who needs access to specific topics. You can use Redpanda Console to ensure users have the right permissions to perform their tasks. 1. In the left menu, click **Security**. 2. On the **Users** tab, click **Create user**. 3. Enter "Sasha" in the username field. 4. Enter "sashapassword" in the password field. 5. Click **Create**. ![user](https://docs.redpanda.com/streaming/24.3/console/_images/user.png) Click **Done**, and you see a new user called Sasha. This user has no permissions yet. To set permissions on the `transactions` topic: 1. On the Access control page, open to the **Roles** tab. 2. Click **Create role**. 3. Enter "transaction-managers" as the role name. 4. In the topic selector dropdown, select **Literal** and enter "transactions" in the input field. 5. Under **Operations**, click the **All** dropdown and select **Allow**. 6. Scroll down to the bottom of the page and under **Principals** select **Sasha** from the dropdown. 7. Click **Create**. Now Sasha has full access only to the topic called transactions. To test these permissions, use `rpk` to connect to Redpanda as the user Sasha. 1. Try to access the `logins` topic as Sasha: ```bash docker exec -it redpanda-0 rpk topic describe logins \ -X user=Sasha \ -X pass=sashapassword \ -X sasl.mechanism=SCRAM-SHA-256 \ -X brokers=redpanda-0:9092 ``` You are not authorized to view the topic. SUMMARY ======= NAME logins PARTITIONS 0 ERROR TOPIC\_AUTHORIZATION\_FAILED: Not authorized to access topics: \[Topic authorization failed.\] CONFIGS ======= config response contained error: TOPIC\_AUTHORIZATION\_FAILED: Not authorized to access topics: \[Topic authorization failed.\] 2. Now try to access the `transactions` topic as Sasha: ```bash docker exec -it redpanda-0 rpk topic describe transactions \ -X user=Sasha \ -X pass=sashapassword \ -X sasl.mechanism=SCRAM-SHA-256 \ -X brokers=redpanda-0:9092 ``` You have access to this topic. Example output SUMMARY ======= NAME transactions PARTITIONS 1 REPLICAS 1 CONFIGS ======= KEY VALUE SOURCE cleanup.policy delete DEFAULT\_CONFIG compression.type producer DEFAULT\_CONFIG delete.retention.ms -1 DEFAULT\_CONFIG flush.bytes 262144 DEFAULT\_CONFIG flush.ms 100 DEFAULT\_CONFIG initial.retention.local.target.bytes -1 DEFAULT\_CONFIG initial.retention.local.target.ms -1 DEFAULT\_CONFIG max.message.bytes 1048576 DEFAULT\_CONFIG message.timestamp.type CreateTime DEFAULT\_CONFIG redpanda.iceberg.delete true DEFAULT\_CONFIG redpanda.iceberg.mode disabled DEFAULT\_CONFIG redpanda.leaders.preference none DEFAULT\_CONFIG redpanda.remote.delete true DEFAULT\_CONFIG redpanda.remote.read true DEFAULT\_CONFIG redpanda.remote.write true DEFAULT\_CONFIG retention.bytes -1 DEFAULT\_CONFIG retention.local.target.bytes -1 DEFAULT\_CONFIG retention.local.target.ms 86400000 DEFAULT\_CONFIG retention.ms 604800000 DEFAULT\_CONFIG segment.bytes 134217728 DEFAULT\_CONFIG segment.ms 1209600000 DEFAULT\_CONFIG write.caching true DEFAULT\_CONFIG ### [](#explore-data-transforms)Explore Data transforms Data transforms let you run common data streaming tasks on the Redpanda broker, like filtering, scrubbing, and transcoding. For example, you may have consumers that require you to redact credit card numbers or convert JSON to Avro. This quickstart deployment comes with one transform function called `regex` running in your cluster. Its job is to find records in the `logins` topic that contain email addresses with the `.edu` domain and add those to a new topic called `edu-filtered-domains`. In the left menu, click **Transforms**. On the **Transforms** page, you see your transform. You can use Redpanda Console to manage and monitor your transforms. > 📝 **NOTE** > > The source code for the data transform function is in the [`docker-compose/transform/transform.go` file](https://github.com/redpanda-data/docs/tree/v/24.3/tests/docker-compose/transform/transform.go) that you downloaded. This file is commented to explain how it works. ![transforms](https://docs.redpanda.com/streaming/24.3/console/_images/transforms.png) See also: - [Manage Data Transforms in Redpanda Console](https://docs.redpanda.com/streaming/24.3/console/ui/data-transforms/) - [Data Transforms](https://docs.redpanda.com/streaming/24.3/develop/data-transforms/) ### [](#view-audit-logs)View audit logs Audit logs provide a record of all user actions. You can use these logs to track changes, troubleshoot issues, and maintain compliance with your organization’s security policies. 1. In the left menu, click **Topics**. 2. Click the **Show internal topics** checkbox. 3. Click the **\_redpanda.audit\_log** topic in the table. ![audit logs](https://docs.redpanda.com/streaming/24.3/console/_images/audit-logs.png) See also: [Audit Logging](https://docs.redpanda.com/streaming/24.3/manage/audit-logging/). ## [](#start-streaming)Start streaming To start building a basic streaming application, you can use the Redpanda CLI (`rpk`) to create a topic, produce messages to it, and consume messages from it. Each Redpanda broker comes preinstalled with `rpk`, so you can use it inside one of the Redpanda broker’s Docker containers. For simplicity, this quickstart uses the bootstrapped superuser (`superuser`) to perform all operations. The superuser has full administrative privileges, making it convenient for setup and exploration. However, to enhance security in a production environment, Redpanda Data recommends creating users with restricted permissions for each task. To use `rpk` inside the Redpanda broker’s Docker container: 1. Get information about the cluster: ```bash docker exec -it redpanda-0 rpk cluster info -X user=superuser -X pass=secretpassword ``` 2. Create a topic called **chat-room**: ```bash docker exec -it redpanda-0 rpk topic create chat-room \ --replicas 3 \ (1) --topic-config redpanda.remote.read=true \ (2) --topic-config redpanda.remote.write=true \ (3) -X user=superuser \ -X pass=secretpassword ``` | 1 | Set a replication factor of 3 to replicate the topic across all 3 brokers. This replication factor provides high availability and data durability. For more details, see Choose the replication factor. | | --- | --- | | 2 | Enable remote reads for this topic to read offloaded records from object storage. | | 3 | Enable remote writes for this topic to offload older records to object storage. For more details, see Use Tiered Storage.Output:TOPIC STATUS chat-room OK | 3. Produce a message to the topic: ```bash docker exec -it redpanda-0 rpk topic produce chat-room -X user=superuser -X pass=secretpassword ``` 4. Enter a message, then press Enter: ```text Pandas are fabulous! ``` Example output: Produced to partition 0 at offset 0 with timestamp 1663282629789. 5. Press Ctrl+C to finish producing messages to the topic. 6. Consume one message from the topic: ```bash docker exec -it redpanda-0 rpk topic consume chat-room --num 1 -X user=superuser -X pass=secretpassword ``` Your message is displayed along with its metadata: ```json { "topic": "chat-room", "value": "Pandas are fabulous!", "timestamp": 1663282629789, "partition": 0, "offset": 0 } ``` ### [](#connect-to-the-cluster-externally)Connect to the cluster externally It’s often more practical to connect to a remote cluster from your local machine with `rpk`. The `docker-compose.yml` file configured the containers to expose ports on your localhost, so you can communicate with the cluster outside the Docker network. To do so, download `rpk` on your local machine and configure it to connect to your cluster on the exposed ports. 1. Install `rpk` on your local machine: #### Linux > 💡 **TIP** > > You can use `rpk` on Windows only with [WSL](https://learn.microsoft.com/windows/wsl/install). However, commands that require Redpanda to be installed on your machine are not supported, such as [`rpk container`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-container/rpk-container/) commands, [`rpk iotune`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-iotune/), and [`rpk redpanda`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-redpanda/rpk-redpanda/) commands. ##### amd64 ```bash curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-amd64.zip && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-linux-amd64.zip -d ~/.local/bin/ ``` ##### arm64 ```bash curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-arm64.zip && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-linux-arm64.zip -d ~/.local/bin/ ``` #### macOS 1. If you don’t have Homebrew installed, [install it](https://brew.sh/). 2. To install or update `rpk`, run: ```bash brew install redpanda-data/tap/redpanda ``` 2. Create a profile to configure `rpk` to connect to your cluster: ```bash rpk profile create quickstart --from-profile rpk-profile.yaml ``` > 📝 **NOTE** > > The profile is configured in the [`docker-compose/rpk-profile.yaml` file](https://github.com/redpanda-data/docs/tree/v/24.3/tests/docker-compose/rpk-profile.yaml) that you downloaded. This file is commented to explain how it works. 3. Get information about the cluster to test the connection: ```bash rpk cluster info ``` > 📝 **NOTE** > > The Redpanda broker returns a list of all broker addresses, so `rpk` can communicate with all brokers directly. Each broker returns its configured `advertise-*` address that matches the port to which the initial connection has been made. You can see examples of these addresses in the Docker Compose file that you downloaded. ### [](#view-offloaded-data-in-tiered-storage)View offloaded data in Tiered Storage Tiered Storage keeps local disk usage stable and performance consistent by offloading older data to object storage. You can also use Tiered Storage to recover clusters or topics, mount and unmount topics from a cluster, and more. > 📝 **NOTE** > > Redpanda supports Amazon S3, Azure ADLS, and Google GCS as object storage providers. For this quickstart, Redpanda uses MinIO as the object storage provider. **MinIO is not supported for production deployments**. 1. Check the size of remote and local disks for the `chat-room` topic you created earlier: ```bash rpk cluster logdirs describe --topics chat-room ``` BROKER DIR TOPIC PARTITION SIZE ERROR 0 /var/lib/redpanda/data chat-room 0 611 0 remote://redpanda chat-room 0 319 1 /var/lib/redpanda/data chat-room 0 611 1 remote://redpanda chat-room 0 319 2 /var/lib/redpanda/data chat-room 0 611 2 remote://redpanda chat-room 0 319 - Local storage: `/var/lib/redpanda/data` holds recent data. - Remote storage: `remote://redpanda` holds offloaded data. 2. Open MinIO at [http://localhost:9001/browser](http://localhost:9001/browser) to view your data stored in the S3-compatible object store. Login credentials: - Username: ```none minio ``` - Password: ```none redpandaTieredStorage7 ``` See also: [Use Tiered Storage](https://docs.redpanda.com/streaming/24.3/manage/tiered-storage/). ## [](#deploy-a-pipeline-with-redpanda-connect)Deploy a pipeline with Redpanda Connect Redpanda Connect is a powerful tool that allows you to build and manage data streaming pipelines. It supports hundreds of connectors for various systems, enabling seamless data ingestion, transformation, and routing between different sources and sinks. Using declarative YAML configuration files and the `rpk` CLI, you can quickly set up and deploy complex data pipelines in Redpanda. 1. Deploy the example pipeline: ```bash rpk connect run ../generate-profiles.yaml ``` This pipeline is configured to generate fake user profiles and produce them to the `profiles` topic in your Redpanda cluster. It’ll run until you stop it. > 📝 **NOTE** > > The pipeline configuration is in the [`docker-compose/generate-profiles.yaml` file](https://github.com/redpanda-data/docs/tree/v/24.3/tests/docker-compose/generate-profiles.yaml) that you downloaded. This file is commented to explain how it works. 2. After a few seconds, stop the pipeline by pressing Ctrl+C. 3. Consume 10 messages from the `profiles` topic to confirm that the pipeline produced data to that topic. ```bash rpk topic consume profiles --num 10 ``` ## [](#extend-your-30-day-trial)Extend your 30-day trial Your initial trial license is valid for 30 days. To extend it by another 30 days: 1. [Sign up for a new trial license](https://cloud.redpanda.com/try-enterprise). 2. Upload the new license to your running cluster using [`rpk cluster license set`](https://docs.redpanda.com/streaming/24.3/reference/rpk/rpk-cluster/rpk-cluster-license-set/): ```bash rpk cluster license set ``` Replace the `` placeholder with your license. This command applies the new license to your Redpanda cluster, extending your access to enterprise features for another 30 days. 3. Restart Redpanda Console so that it can fetch the new license from the Redpanda brokers: ```bash docker restart redpanda-console ``` > 📝 **NOTE** > > You can extend the trial license only once. > 💡 **TIP: Ready to go into production?** > > [Upgrade to Redpanda Enterprise](https://www.redpanda.com/upgrade). See also: [Redpanda Licensing](https://docs.redpanda.com/streaming/24.3/get-started/licensing/). ## [](#clean-up)Clean up To reset your environment, shut down the running processes and delete the containers: ```bash docker compose down ``` To delete the volumes along with all your cluster data: ```bash docker compose down -v ``` ## [](#customize-the-docker-quickstart)Customize the Docker quickstart To customize this quickstart, you can configure Redpanda, Redpanda Console, or Redpanda Connect using the provided Docker Compose file. This allows you to tailor the services to fit your specific requirements. ### [](#redpanda-broker-properties)Redpanda broker properties To configure the Redpanda services with [broker configuration properties](https://docs.redpanda.com/streaming/24.3/reference/properties/broker-properties/), pass properties with the `--set` option in the `redpanda start` command in your Docker Compose file. **Example:** `docker-compose.yml` ```yaml redpanda-0: (1) command: (2) - redpanda - start - --set pandaproxy_client.retries=6 (3) ``` | 1 | The service name in Docker Compose. | | --- | --- | | 2 | Overrides the default command for the Redpanda container. | | 3 | Sets the pandaproxy_client.retries broker configuration property to 6. | ### [](#redpanda-cluster-properties)Redpanda cluster properties Cluster-level configurations are defined in the `bootstrap.yml` file. This file is included in your Docker Compose setup. It contains essential settings that apply to the entire Redpanda cluster. You can find all cluster properties in [Cluster Configuration Properties](https://docs.redpanda.com/streaming/24.3/reference/properties/cluster-properties/). **Example:** `docker-compose.yml` ```yaml redpanda: volumes: - ./bootstrap.yml:/etc/redpanda/.bootstrap.yaml (1) ``` | 1 | Mounts the bootstrap.yml file from your local parent directory to the container’s /etc/redpanda/ directory. | | --- | --- | ### [](#redpanda-console-configuration)Redpanda Console configuration Redpanda Console configuration is managed with the environment variable `CONSOLE_CONFIG_FILE`. You can specify the path to your configuration file in the Docker Compose file. **Example:** `docker-compose.yml` ```yaml console: environment: CONSOLE_CONFIG_FILE: | # Configure a connection to the Redpanda cluster # See https://docs.redpanda.com/current/console/config/connect-to-redpanda/ kafka: brokers: ["redpanda-0:9092"] schemaRegistry: enabled: true urls: ["http://redpanda-0:8081","http://redpanda-1:8081","http://redpanda-2:8081"] sasl: enabled: true username: superuser password: secretpassword mechanism: SCRAM-SHA-256 ``` ### [](#redpanda-connect-configuration)Redpanda Connect configuration Existing data pipelines are configured using the `CONNECT_CFG_FILE` environment variable in the `connect` service. This environment variable contains YAML configuration settings for Redpanda Connect. **Example:** `docker-compose.yml` ```yaml connect: environment: CONNECT_CFG_FILE: | input: generate: interval: 1s mapping: | let first_name = fake("first_name") let last_name = fake("last_name") root.user_id = counter() root.name = $$first_name + " " + $$last_name root.email = ($$first_name.slice(0,1) + $$last_name + "@" + fake("domain_name")).lowercase() root.ip = fake("ipv4") root.login_time = now() pipeline: processors: - mapping: | root = range(0, random_int(min:2, max:4)).map_each(cust -> this) - unarchive: format: "json_array" - mapping: | if batch_index() == 0 { meta topic = "logins" root = this } else { meta topic = "transactions" root.user_id = this.user_id root.email = this.email root.index = batch_index() - 1 root.product_url = fake("url") root.price = fake("amount_with_currency") root.timestamp = now() } output: kafka_franz: seed_brokers: [ "redpanda-0:9092" ] topic: $${! metadata("topic") } sasl: - mechanism: SCRAM-SHA-256 password: secretpassword username: superuser ``` To create new data pipelines, run `rpk connect run`, specifying the path to your own YAML configuration file: ```bash rpk connect run ``` ## [](#next-steps)Next steps [Contact Redpanda](https://www.redpanda.com/contact) to discuss using Redpanda Enterprise Edition in production. - [Try more examples in Redpanda Labs](https://docs.redpanda.com/labs/) - [Learn more about enterprise features for Redpanda Streaming](https://docs.redpanda.com/streaming/24.3/get-started/licensing/overview/) - [Deploy for development or production](https://docs.redpanda.com/streaming/24.3/deploy/deployment-option/self-hosted/manual/production/) ## [](#suggested-reading)Suggested reading - [Redpanda Connect Documentation](https://docs.redpanda.com/connect/home/) - [rpk Commands](https://docs.redpanda.com/streaming/24.3/reference/rpk/) - [Introduction to Redpanda Console](https://docs.redpanda.com/streaming/24.3/console/) - Docker images - [Docker images for Redpanda Console](https://hub.docker.com/r/redpandadata/console/tags) - [Docker images for Redpanda](https://hub.docker.com/r/redpandadata/redpanda)