Connecting Remotely to Kubernetes

This section shows how to set up Kubernetes with the Redpanda operator in Google GKE, Amazon EKS, or Digital Ocean, so you can work with Redpanda from outside of the Kubernetes network.

Create a Kubernetes cluster

Create a three-node cluster for your Redpanda deployment on any of the following platforms:

AWS EKS

Use the EKS Getting Started guide to set up EKS. When you finish, you have eksctl installed, so that you can create and delete clusters in EKS.

To create a cluster:

eksctl create cluster \
--name redpanda \
--nodegroup-name standard-workers \
--node-type m5.xlarge \
--nodes 3 \
--nodes-min 1 \
--nodes-max 4

The process takes about 10-15 minutes to finish.

Google GKE

Complete the "Before You Begin" steps described in Google Kubernetes Engine Quickstart.

To create a cluster:

gcloud container clusters create redpanda --machine-type e2-standard-4 --cluster-version 1.21 && \
gcloud container clusters get-credentials redpanda

note

You may need to specify a --region, --zone, or --project in this command.

Digital Ocean

Set up your Digital Ocean account, and install doctl.

Remember to set up your personal access token. For information, see the Digital Ocean setup docs.

To create a cluster:

doctl kubernetes cluster create redpanda --wait --size s-4vcpu-8gb

kubectl context

Most cloud utility tools automatically change your kubectl config file.

To check if you're in the correct context:

kubectl config current-context

For Digital Ocean, for example, the output looks similar to this:

do-nyc1-redpanda

If you're running multiple clusters, or if the config file wasn't set up automatically, see the Kubernetes documentation.

Prepare TLS certificate infrastructure

The Redpanda cluster uses cert-manager to create TLS certificates for communication between the cluster nodes.

To use Helm to install cert-manager:

helm repo add jetstack https://charts.jetstack.io && \
helm repo update && \
helm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.4.4 \
  --set installCRDs=true

Install the Redpanda operator and cluster

1. To simplify the commands, create a variable to hold the latest version number:

   export VERSION=$(curl -s https://api.github.com/repos/redpanda-data/redpanda/releases/latest | jq -r .tag_name)

   This section uses jq to help. If you don't have jq installed:

   apt

   sudo apt-get update && \
   sudo apt-get install jq

   brew

   brew install jq
   note

   You can also get operator versions from the list of operator releases.

2. To install the latest Redpanda operator:

   bash

   kubectl apply -k https://github.com/redpanda-data/redpanda/src/go/k8s/config/crd?ref=$VERSION && \
   helm repo add redpanda https://charts.vectorized.io/ && \
   helm repo update && \
   helm install \
     --namespace redpanda-system \
     --create-namespace redpanda-operator \
     --version $VERSION \
     redpanda/redpanda-operator

   zsh

   noglob kubectl apply -k https://github.com/redpanda-data/redpanda/src/go/k8s/config/crd?ref=$VERSION && \
   helm repo add redpanda https://charts.vectorized.io/ && \
   helm repo update && \
   helm install \
     --namespace redpanda-system \
     --create-namespace redpanda-operator \
     --version $VERSION \
     redpanda/redpanda-operator

3. To install a cluster with external connectivity:

   kubectl apply -f https://raw.githubusercontent.com/redpanda-data/redpanda/$VERSION/src/go/k8s/config/samples/external_connectivity.yaml

4. To get the addresses of the brokers:

   kubectl get clusters external-connectivity -o=jsonpath='{.status.nodes.external}'

   The broker addresses are shown in the command output. For example:

   ["34.121.167.159:30249","34.71.125.54:30249","35.184.221.5:30249"]

   If you don't get any response for this command, check if the pods are healthy and are running with no errors.

   The following commands help you better understand what's happening:

   kubectl describe statefulset external-connectivity
   kubectl describe pods external-connectivity-0

5. To configure security access:

   AWS EKS

   When you run eksctl, it automatically creates a lot of resources for you (dedicated VPC, new Security Group, and others). Because of that, you have to enter your security configurations and open the ports that external-connectivity uses in order to follow the next steps. The easiest way to do that is to:

   a. Get the ports that you need to open with the command you ran in the previous step.

   b. Go to your Security Group configurations and check the newly created rule for your cluster.

   c. Open TCP traffic to the ports.

   For more information, see the AWS guide for configuring VPCs and Security Groups.

   Google GKE

   For GKE, open the firewall for access to the cluster:

   a. To get the port number on which Redpanda is listening:

   kubectl get service external-connectivity-external -o=jsonpath='{.spec.ports[0].nodePort}'

   The port is shown in the command output.

   b. To create a firewall rule that allows traffic to Redpanda on that port:

   gcloud compute firewall-rules create redpanda-nodeport --allow tcp:<port_number>

   The port that Redpanda is listening on is shown in the command output; for example: 30249.

   Digital Ocean

   For Digital Ocean, there's no need for additional configurations.

Verify the connection

1. From a remote machine that has rpk installed, to get information about the cluster:

   rpk --brokers 34.121.167.159:30249,34.71.125.54:30249,35.184.221.5:30249 \
   cluster info
   note

   Check if you're using the correct address and ports. Otherwise you may run into errors like the following:

   unable to create topics [chat-rooms]: invalid large response size 1213486160 > limit 104857600

2. To create a topic in your Redpanda cluster:

   rpk --brokers 34.121.167.159:30249,34.71.125.54:30249,35.184.221.5:30249 \
   topic create chat-rooms -p 5

3. To show the list of topics:

   rpk --brokers 34.121.167.159:30249,34.71.125.54:30249,35.184.221.5:30249 \
   topic list

Next steps

• For a detailed explanation, see Configuring the Kubernetes Operator for Connectivity.
• Contact us in our Slack community so we can work together to implement your Kubernetes use cases.