Streaming

Deploy for Development

You can deploy Redpanda using well-known configuration properties optimized for a development or test environment. This configuration uses less system resources and simplifies setup, but it’s not suitable for a production deployment. For example, in development mode, the default group_topic_partitions is 3, and the default storage_min_free_bytes is 1 GiB. In production mode, the default group_topic_partitions is 16, and the default storage_min_free_bytes is 5 GiB.

  • Development mode enables write caching by default. This is a relaxed mode of acks=all that acknowledges a message as soon as it is received and acknowledged on a majority of brokers, without waiting for it to fsync to disk. Write caching provides lower latency while still ensuring that a majority of brokers acknowledge the write. For more information, or to disable this, see write caching.

  • Development mode also bypasses fsync, acknowledging messages before they’re stored to disk. This reduces the durability of messages, could cause potential data loss, and could give unrealistic performance characteristics for a production environment.

To deploy for a production environment, see Deploy for Production. Or to try out Redpanda in Docker, see Redpanda Quickstart.

Prerequisites

Make sure you meet the hardware and software requirements.

TCP/IP ports

Redpanda uses the following default ports:

Port Purpose

9092

Kafka API

8082

HTTP Proxy

8081

Schema Registry

9644

Admin API and Prometheus

33145

internal RPC

Install Redpanda

Install Redpanda on each system you want to be part of your cluster. There are binaries available for Fedora/RedHat or Debian systems.

Unless you intend to run Redpanda in FIPS-compliance mode, the following packages should accommodate your needs (for both Debian and Redhat based systems):

redpanda

  • Contains the Redpanda application and all supporting libraries

  • Depends on redpanda-tuners and either redpanda-rpk or redpanda-rpk-fips

redpanda-rpk

  • Contains the pure GoLang compiled rpk application

  • If you wish to use rpk only, then this is the only required install package

redpanda-tuners

  • Contains the files used to run Redpanda tuners

  • Depends on redpanda-rpk or redpanda-rpk-fips

  • Fedora/RedHat

  • Debian/Ubuntu

curl -1sLf 'https://dl.redpanda.com/nzc4ZYQK3WRGd9sy/redpanda/cfg/setup/bash.rpm.sh' | \
sudo -E bash && sudo yum install redpanda -y
curl -1sLf 'https://dl.redpanda.com/nzc4ZYQK3WRGd9sy/redpanda/cfg/setup/bash.deb.sh' | \
sudo -E bash && sudo apt install redpanda -y

Install Redpanda for FIPS compliance

This feature requires an enterprise license. To get a trial license key or extend your trial period, generate a new trial license key. To purchase a license, contact Redpanda Sales.

If Redpanda has enterprise features enabled and it cannot find a valid license, restrictions apply.

To run Redpanda in FIPS compliance mode, you must install the redpanda-fips package, which is separate from and dependent upon the redpanda install package already being installed. Install the following packages after installing redpanda:

redpanda-fips

  • Contains the OpenSSL FIPS-approved module and scripts required to set up and run Redpanda in FIPS-compliance mode.

  • Depends upon the successful installation of the redpanda package.

  • Includes the fips.so cryptographic provider (built from OpenSSL v3.0.9, which is the latest FIPS 140-2 approved module) and a copy of the OpenSSL application.

  • Executes openssl fipsinstall against the fips.so module, which generates a fipsmodule.cnf file that is used during the module’s POST (power-on-self-test) to validate the integrity of the module.

redpanda-rpk-fips

  • RHEL

  • Debian/Ubuntu

To install Redpanda for FIPS compliance, run:

sudo apt install -y redpanda-rpk-fips redpanda-fips
Alternatively, you could run sudo apt install -y redpanda-fips, which also picks up and includes the redpanda install package.

If you wish to only use rpk on a FIPS host, run:

sudo yum install -y redpanda-fips redpanda-rpk-fips

To install Redpanda for FIPS compliance, run:

sudo apt install redpanda-fips redpanda-rpk-fips
Alternatively, you could run sudo apt install redpanda-fips, which also picks up and includes the redpanda install package.

If you wish to only use rpk on a FIPS host, run:

sudo apt install -y redpanda-rpk-fips

See also: Configure Redpanda for FIPS

Install Redpanda Console

Redpanda Console is a developer-friendly web UI for managing and debugging your Redpanda cluster and your applications.

For each new release, Redpanda compiles the Redpanda Console to a single binary for Linux, macOS, and Windows. You can find the binaries in the attachments of each release on GitHub.

  • Fedora/RedHat

  • Debian/Ubuntu

curl -1sLf 'https://dl.redpanda.com/nzc4ZYQK3WRGd9sy/redpanda/cfg/setup/bash.rpm.sh' | \
sudo -E bash && sudo yum install redpanda-console -y
curl -1sLf 'https://dl.redpanda.com/nzc4ZYQK3WRGd9sy/redpanda/cfg/setup/bash.deb.sh' | \
sudo -E bash && sudo apt-get install redpanda-console -y

Bootstrap broker configurations

Each broker requires a set of broker configurations that determine how all brokers communicate with each other and with clients. Bootstrapping a cluster configures the listener, seed servers, and advertised listeners, which ensure proper network connectivity and accessibility.

Starting in version 23.3.8, rpk enhances the bootstrapping process with additional flags for configuring advertised listener addresses directly.

Use the rpk redpanda config bootstrap command to bootstrap Redpanda:

sudo rpk redpanda config bootstrap --self <listener-address> --advertised-kafka <advertised-kafka-address> --ips <seed-server1-ip>,<seed-server2-ip>,<seed-server3-ip> && \
sudo rpk redpanda config set redpanda.empty_seed_starts_cluster false

Replace the following placeholders:

  • <listener-address>: The --self flag tells Redpanda the interfaces to bind to for the Kafka API, the RPC API, and the Admin API. These addresses determine on which network interface and port Redpanda listens for incoming connections.

    • Set the listener address to 0.0.0.0 to listen on all network interfaces available on the machine.

    • Set the listener address to a specific IP address to bind the listener to that address, restricting connections to that interface.

  • <advertised-kafka-address>: The --advertised-kafka flag sets a different advertised Kafka address, which is useful for scenarios where the accessible address differs from the bind address.

    Redpanda does not allow advertised addresses set to 0.0.0.0. If you set any advertised addresses to 0.0.0.0, Redpanda will output startup validation errors.

  • <seed-server-ips>: The --ips flag lists all the seed servers in the cluster, including the one being started.

    The --ips flag must be set identically (with nodes listed in identical order) on each node.

Bootstrapping Redpanda updates your /etc/redpanda/redpanda.yaml configuration file:

/etc/redpanda/redpanda.yaml
redpanda:
    data_directory: /var/lib/redpanda/data
    empty_seed_starts_cluster: false
    seed_servers:
        - host:
            address: <seed-server1-ip>
            port: 33145
        - host:
            address: <seed-server2-ip>
            port: 33145
        - host:
            address: <seed-server3-ip>
            port: 33145
    rpc_server:
        address: <listener-address>
        port: 33145
    kafka_api:
        - address: <listener-address>
          port: 9092
    admin:
        - address: <listener-address>
          port: 9644
    advertised_rpc_api:
        address: <listener-address>
        port: 33145
    advertised_kafka_api:
        - address: <advertised-kafka-address>
          port: 9092

Recommendations

  • Redpanda Data strongly recommends at least three seed servers when forming a cluster. A larger number of seed servers increases the robustness of consensus and minimizes any chance that new clusters get spuriously formed after brokers are lost or restarted without any data.

  • It’s important to have one or more seed servers in each fault domain (for example, in each rack or cloud AZ). A higher number provides a stronger guarantee that clusters don’t fracture unintentionally.

  • It’s possible to change the seed servers for a short period of time after a cluster has been created. For example, you may want to designate one additional broker as a seed server to increase availability. To do this without cluster downtime, add the new broker to the seed_servers property and restart Redpanda to apply the change on a broker-by-broker basis.

Listeners for mixed environments

For clusters serving both internal and external clients, configure multiple listeners for the Kafka API to separate internal from external traffic.

For more details, see Configure Listeners.

Start Redpanda

To start Redpanda:

sudo systemctl start redpanda-tuner redpanda

When a Redpanda cluster starts, it instantiates a controller Raft group with all the seed servers specified in the --ips flag. After all seed servers complete their startup procedure and become accessible, the cluster is then available. After that, non-seed servers start up and are added to the cluster.

Start Redpanda Console

  1. Start Redpanda Console:

    sudo systemctl start redpanda-console

  2. Make sure that Redpanda Console is active and running:

    sudo systemctl status redpanda-console

Verify the installation

To verify that the Redpanda cluster is up and running, use rpk to get information about the cluster:

rpk cluster info

You should see a list of advertised addresses.

To create a topic:

rpk topic create <topic-name>

If topics were initially created in a test environment with a replication factor of 1, use rpk topic alter-config to change the topic replication factor:

rpk topic alter-config <topic-names> --set replication.factor=3

Perform a self test

To understand the performance capabilities of your Redpanda cluster, Redpanda offers built-in self-test features that evaluate the performance of both disk and network operations.

For more information, see Disk and network self-test benchmarks.

Next steps

Suggested reading

Suggested labs

See more
Search all labs