Connecting Remotely to Kubernetes

Deprecated red

The Cluster and Console resources are deprecated. For details, see the deprecation notice. To migrate to the Redpanda resource, see Migrate from Cluster and Console Custom Resources

This section shows how to set up Kubernetes with the Redpanda operator in Google GKE, Amazon EKS, or Digital Ocean, so you can work with Redpanda from outside of the Kubernetes network.

Create a Kubernetes cluster

Create a three-node cluster for your Redpanda deployment on any of the following platforms:

  • AWS EKS

  • Google GKE

  • Digital Ocean

Use the EKS Getting Started guide to set up EKS. When you finish, you have eksctl installed, so that you can create and delete clusters in EKS.

To create a cluster:

eksctl create cluster \
  --name redpanda \
  --nodegroup-name standard-workers \
  --node-type m5.xlarge \
  --nodes 3 \
  --nodes-min 1 \
  --nodes-max 4

The process takes about 10-15 minutes to finish.

Complete the "Before You Begin" steps described in Google Kubernetes Engine Quickstart.

To create a cluster:

gcloud container clusters create redpanda --machine-type e2-standard-4 --cluster-version 1.21 && \
gcloud container clusters get-credentials redpanda
You may need to specify a --region, --zone, or --project in this command.

Set up your Digital Ocean account, and install doctl.

Remember to set up your personal access token. For information, see the Digital Ocean setup docs.

To create a cluster:

doctl kubernetes cluster create redpanda --wait --size s-4vcpu-8gb

kubectl context

Most cloud utility tools automatically change your kubectl config file.

To check if you’re in the correct context:

kubectl config current-context

For Digital Ocean, for example, the output looks similar to this:

do-nyc1-redpanda

If you’re running multiple clusters, or if the config file wasn’t set up automatically, see the Kubernetes documentation.

Prepare TLS certificate infrastructure

The Redpanda cluster uses cert-manager to create TLS certificates for communication between the cluster nodes.

To use Helm to install cert-manager:

helm repo add jetstack https://charts.jetstack.io && \
helm repo update && \
helm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.12.2 \
  --set installCRDs=true

Install the Redpanda operator and cluster

  1. To simplify the commands, create a variable to hold the latest version number:

    export VERSION=$(curl -s https://api.github.com/repos/redpanda-data/redpanda/releases/latest | jq -r .tag_name)

    This section uses jq to help. If you don’t have jq installed:

    • apt

    • brew

    sudo apt-get update && \
sudo apt-get install jq
    brew install jq
    You can also get operator versions from the list of operator releases.

  2. To install the latest Redpanda operator:

    • bash

    • zsh

    kubectl apply -k https://github.com/redpanda-data/redpanda/src/go/k8s/config/crd?ref=$VERSION && \
helm repo add vectorized https://charts.vectorized.io/ && \
helm repo update && \
helm install \
  --namespace redpanda-system \
  --create-namespace redpanda-operator \
  --version $VERSION \
  vectorized/redpanda-operator
    noglob kubectl apply -k https://github.com/redpanda-data/redpanda/src/go/k8s/config/crd?ref=$VERSION && \
helm repo add vectorized https://charts.vectorized.io/ && \
helm repo update && \
helm install \
  --namespace redpanda-system \
  --create-namespace redpanda-operator \
  --version $VERSION \
  vectorized/redpanda-operator

  3. To install a cluster with external connectivity:

    kubectl apply -f https://raw.githubusercontent.com/redpanda-data/redpanda/$VERSION/src/go/k8s/config/samples/external_connectivity.yaml

  4. To get the addresses of the brokers:

    kubectl get clusters external-connectivity -o=jsonpath='{.status.nodes.external}'

    The broker addresses are shown in the command output. For example:

    ["34.121.167.159:30249","34.71.125.54:30249","35.184.221.5:30249"]

    If you don’t get any response for this command, check if the pods are healthy and are running with no errors.

    The following commands help you better understand what’s happening:

    kubectl describe statefulset external-connectivity
kubectl describe pods external-connectivity-0

  5. To configure security access:

    • AWS EKS

    • Google GKE

    • Digital Ocean

    When you run eksctl, it automatically creates a lot of resources for you (dedicated VPC, new Security Group, and others). Because of that, you have to enter your security configurations and open the ports that external-connectivity uses in order to follow the next steps. The easiest way to do that is to:

    1. Get the ports that you need to open with the command you ran in the previous step.

    2. Go to your Security Group configurations and check the newly created rule for your cluster.

    3. Open TCP traffic to the ports.

    For more information, see the AWS guide for configuring VPCs and Security Groups.

    For GKE, open the firewall for access to the cluster:

    1. To get the port number on which Redpanda is listening:

      kubectl get service external-connectivity-external -o=jsonpath='{.spec.ports[0].nodePort}'

      The port is shown in the command output.

    2. To create a firewall rule that allows traffic to Redpanda on that port:

      gcloud compute firewall-rules create redpanda-nodeport --allow tcp:<port_number>

      The port that Redpanda is listening on is shown in the command output; for example: 30249.

    For Digital Ocean, there’s no need for additional configurations.

Verify the connection

  1. From a remote machine that has rpk installed, to get information about the cluster:

    rpk -X brokers=34.121.167.159:30249,34.71.125.54:30249,35.184.221.5:30249 \
cluster info

    Check if you’re using the correct address and ports. Otherwise you may run into errors like the following:

    unable to create topics [chat-rooms]: invalid large response size 1213486160 > limit 104857600

  2. To create a topic in your Redpanda cluster:

    rpk -X brokers=34.121.167.159:30249,34.71.125.54:30249,35.184.221.5:30249 \
topic create chat-rooms -p 5

  3. To show the list of topics:

    rpk -X brokers=34.121.167.159:30249,34.71.125.54:30249,35.184.221.5:30249 \
topic list

Next steps