Connecting remotely to Kubernetes

The Kubernetes Quick Start guide describes how to quickly get up and running with a Kubernetes cluster. Those instructions only provide access to the cluster from within the Kuberenetes network.

Here we'll show you an example of how to set up Kubernetes in Google GKE, Amazon EKS, or Digital Ocean so you can work with Redpanda from outside of the Kubernetes network.

Let's get started...

Create a Kubernetes cluster

Create a 3-node cluster on the platform of your choice:

AWS EKS

Use the EKS Getting Started guide to set up EKS. When you finish, you'll have eksctl installed so that you can create and delete clusters in EKS. Then, create an EKS cluster with:

eksctl create cluster \
--name redpanda \
--nodegroup-name standard-workers \
--node-type m5.xlarge \
--nodes 3 \
--nodes-min 1 \
--nodes-max 4

It will take about 10-15 minutes for the process to finish.

Google GKE

First complete the "Before You Begin" steps described in Google Kubernetes Engine Quickstart. Then, create a cluster with:

gcloud container clusters create redpanda --machine-type e2-standard-4 --cluster-version 1.21 && \
gcloud container clusters get-credentials redpanda

note

• You may need to add a --region, --zone, or --project to this command.

Digital Ocean

First, set up your Digital Ocean account and install doctl.

Remember to setup your personal access token.

For additional information, check out the Digital Ocean setup docs.

Then you can create a cluster for your Redpanda deployment:

doctl kubernetes cluster create redpanda --wait --size s-4vcpu-8gb

Kubectl context

Most cloud utility tools will automatically change your kubectl config file.
To check if you're in the correct context, run the command:

kubectl config current-context

For Digital Ocean for example, the output will look similar to this:

do-nyc1-redpanda

If you're running multiple clusters or if the config file wasn't set up automatically, look for more information in the Kubernetes documentation.

Prepare TLS certificate infrastructure

The Redpanda cluster uses cert-manager to create TLS certificates for communication between the cluster nodes.

We'll use Helm to install cert-manager:

helm repo add jetstack https://charts.jetstack.io && \
helm repo update && \
helm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.4.4 \
  --set installCRDs=true

Install the Redpanda operator and cluster

1. Just to simplify the commands, create a variable to hold the latest version number:

   export VERSION=$(curl -s https://api.github.com/repos/redpanda-data/redpanda/releases/latest | jq -r .tag_name)

   Note - You can find information about the versions of the operator in the list of operator releases.
   We're using jq to help us. If you don't have it installed run this command:

   apt

   sudo apt-get update && \
   sudo apt-get install jq

   brew

   brew install jq

2. Install the latest redpanda operator:

   bash

   kubectl apply -k https://github.com/redpanda-data/redpanda/src/go/k8s/config/crd?ref=$VERSION && \
   helm repo add redpanda https://charts.vectorized.io/ && \
   helm repo update && \
   helm install \
     --namespace redpanda-system \
     --create-namespace redpanda-operator \
     --version $VERSION \
     redpanda/redpanda-operator

   zsh

   noglob kubectl apply -k https://github.com/redpanda-data/redpanda/src/go/k8s/config/crd?ref=$VERSION && \
   helm repo add redpanda https://charts.vectorized.io/ && \
   helm repo update && \
   helm install \
     --namespace redpanda-system \
     --create-namespace redpanda-operator \
     --version $VERSION \
     redpanda/redpanda-operator

3. Install a cluster with external connectivity:

   kubectl apply -f https://raw.githubusercontent.com/redpanda-data/redpanda/$VERSION/src/go/k8s/config/samples/external_connectivity.yaml

4. Get the addresses of the brokers:

   kubectl get clusters external-connectivity -o=jsonpath='{.status.nodes.external}'

   The broker addresses are shown in the command output, for example:

   ["34.121.167.159:30249","34.71.125.54:30249","35.184.221.5:30249"]

   If you don't get any response for this command, please check if the pods are healthy and are running with no errors.

   Commands like these will help you better understand what's happening:

   kubectl describe statefulset external-connectivity
   kubectl describe pods external-connectivity-0

5. Configure security access

AWS EKS

When you run eksctl it automatically creates a lot of resources for you (dedicated VPC, new Security Group and etc). Because of that, you have to enter your security configurations and open the ports that external-connectivity uses in order to follow the next steps.

The easiest way to do that is to:

a. Get the ports that you need to open with the command that you ran in step 4.

b. Go to your Security Group configurations and check the newly created rule for your cluster.

c. Open TCP traffic to the ports.

If you don't know how to do it, refer to the AWS guide for configuring VPCs and Security Groups.

Google GKE

For GKE, open the firewall for access to the cluster: a. Get the port number that Redpanda is listening on:

kubectl get service external-connectivity-external -o=jsonpath='{.spec.ports[0].nodePort}'

The port is shown in the command output.

b. Create a firewall rule that allows traffic to Redpanda on that port:

gcloud compute firewall-rules create redpanda-nodeport --allow tcp:<port_number>

The port that Redpanda is listening on is shown in the command output, for example:

30249

Digital Ocean

For Digital Ocean, there's no need for additional configurations.

Verify the connection

1. From a remote machine that has rpk installed, get information about the cluster:

   rpk --brokers 34.121.167.159:30249,34.71.125.54:30249,35.184.221.5:30249 \
   cluster info

   Note - Check if you're using the correct address and ports. Otherwise you may run into errors like:

   unable to create topics [chat-rooms]: invalid large response size 1213486160 > limit 104857600

2. Create a topic in your Redpanda cluster:

   rpk --brokers 34.121.167.159:30249,34.71.125.54:30249,35.184.221.5:30249 \
   topic create chat-rooms -p 5

3. Show the list of topics:

   rpk --brokers 34.121.167.159:30249,34.71.125.54:30249,35.184.221.5:30249 \
   topic list

Now you know how to set up a Kubernetes cluster in a cloud and access it from a remote machine.

Next steps

• Check out our in-depth explanation of Kubernetes connectivity.
• Contact us in our Slack community so we can work together to implement your Kubernetes use cases.