Connecting Remotely to Kubernetes
This section shows how to set up Kubernetes with the Redpanda operator in Google GKE, Amazon EKS, or Digital Ocean, so you can work with Redpanda from outside of the Kubernetes network.
Create a Kubernetes cluster
Create a three-node cluster for your Redpanda deployment on any of the following platforms:
- AWS EKS
- Google GKE
- Digital Ocean
Use the EKS Getting Started guide to set up EKS.
When you finish, you have eksctl
installed, so that you can create and delete clusters in EKS.
To create a cluster:
eksctl create cluster \
--name redpanda \
--nodegroup-name standard-workers \
--node-type m5.xlarge \
--nodes 3 \
--nodes-min 1 \
--nodes-max 4
The process takes about 10-15 minutes to finish.
Complete the "Before You Begin" steps described in Google Kubernetes Engine Quickstart.
To create a cluster:
gcloud container clusters create redpanda --machine-type e2-standard-4 --cluster-version 1.21 && \
gcloud container clusters get-credentials redpanda
You may need to specify a --region
, --zone
, or --project
in this command.
Set up your Digital Ocean account, and install doctl
.
Remember to set up your personal access token. For information, see the Digital Ocean setup docs.
To create a cluster:
doctl kubernetes cluster create redpanda --wait --size s-4vcpu-8gb
kubectl context
Most cloud utility tools automatically change your kubectl
config file.
To check if you're in the correct context:
kubectl config current-context
For Digital Ocean, for example, the output looks similar to this:
do-nyc1-redpanda
If you're running multiple clusters, or if the config file wasn't set up automatically, see the Kubernetes documentation.
Prepare TLS certificate infrastructure
The Redpanda cluster uses cert-manager to create TLS certificates for communication between the cluster nodes.
To use Helm to install cert-manager:
helm repo add jetstack https://charts.jetstack.io && \
helm repo update && \
helm install \
cert-manager jetstack/cert-manager \
--namespace cert-manager \
--create-namespace \
--version v1.4.4 \
--set installCRDs=true
Install the Redpanda operator and cluster
To simplify the commands, create a variable to hold the latest version number:
export VERSION=$(curl -s https://api.github.com/repos/redpanda-data/redpanda/releases/latest | jq -r .tag_name)
This section uses
jq
to help. If you don't havejq
installed:- apt
- brew
sudo apt-get update && \
sudo apt-get install jqbrew install jq
noteYou can also get operator versions from the list of operator releases.
To install the latest Redpanda operator:
- bash
- zsh
kubectl apply -k https://github.com/redpanda-data/redpanda/src/go/k8s/config/crd?ref=$VERSION && \
helm repo add redpanda https://charts.vectorized.io/ && \
helm repo update && \
helm install \
--namespace redpanda-system \
--create-namespace redpanda-operator \
--version $VERSION \
redpanda/redpanda-operatornoglob kubectl apply -k https://github.com/redpanda-data/redpanda/src/go/k8s/config/crd?ref=$VERSION && \
helm repo add redpanda https://charts.vectorized.io/ && \
helm repo update && \
helm install \
--namespace redpanda-system \
--create-namespace redpanda-operator \
--version $VERSION \
redpanda/redpanda-operatorTo install a cluster with external connectivity:
kubectl apply -f https://raw.githubusercontent.com/redpanda-data/redpanda/$VERSION/src/go/k8s/config/samples/external_connectivity.yaml
To get the addresses of the brokers:
kubectl get clusters external-connectivity -o=jsonpath='{.status.nodes.external}'
The broker addresses are shown in the command output. For example:
["34.121.167.159:30249","34.71.125.54:30249","35.184.221.5:30249"]
If you don't get any response for this command, check if the pods are healthy and are running with no errors.
The following commands help you better understand what's happening:
kubectl describe statefulset external-connectivity
kubectl describe pods external-connectivity-0To configure security access:
- AWS EKS
- Google GKE
- Digital Ocean
When you run
eksctl
, it automatically creates a lot of resources for you (dedicated VPC, new Security Group, and others). Because of that, you have to enter your security configurations and open the ports that external-connectivity uses in order to follow the next steps. The easiest way to do that is to:a. Get the ports that you need to open with the command you ran in the previous step.
b. Go to your Security Group configurations and check the newly created rule for your cluster.
c. Open TCP traffic to the ports.
For more information, see the AWS guide for configuring VPCs and Security Groups.
For GKE, open the firewall for access to the cluster:
a. To get the port number on which Redpanda is listening:
kubectl get service external-connectivity-external -o=jsonpath='{.spec.ports[0].nodePort}'
The port is shown in the command output.
b. To create a firewall rule that allows traffic to Redpanda on that port:
gcloud compute firewall-rules create redpanda-nodeport --allow tcp:<port_number>
The port that Redpanda is listening on is shown in the command output; for example:
30249
.For Digital Ocean, there's no need for additional configurations.
Verify the connection
From a remote machine that has
rpk
installed, to get information about the cluster:rpk --brokers 34.121.167.159:30249,34.71.125.54:30249,35.184.221.5:30249 \
cluster infonoteCheck if you're using the correct address and ports. Otherwise you may run into errors like the following:
unable to create topics [chat-rooms]: invalid large response size 1213486160 > limit 104857600
To create a topic in your Redpanda cluster:
rpk --brokers 34.121.167.159:30249,34.71.125.54:30249,35.184.221.5:30249 \
topic create chat-rooms -p 5To show the list of topics:
rpk --brokers 34.121.167.159:30249,34.71.125.54:30249,35.184.221.5:30249 \
topic list
Next steps
- For a detailed explanation, see Configuring the Kubernetes Operator for Connectivity.
- Contact us in our Slack community so we can work together to implement your Kubernetes use cases.