Streaming

Deploy Redpanda with Argo CD

Deploy Redpanda with Argo CD, so that Git holds the desired state of your clusters. This guide installs Argo CD, then builds one repository that deploys cert-manager and a Redpanda cluster through a single root Application.

After reading this page, you will be able to:

Install Argo CD on a Kubernetes cluster
Manage cert-manager and a Redpanda cluster from one Git repository
Keep every Application synced with sync waves and server-side diff

Prerequisites

A Kubernetes cluster with one worker node for each broker you plan to deploy. Every step on this page is the same on any conformant cluster. To create one, follow the platform guide you use, and stop before it installs Redpanda:
- Amazon EKS
- Google GKE
- Azure AKS
- Local (kind or minikube)
  
  The three cloud guides also create a StorageClass named csi-driver-lvm-striped-xfs for the worker nodes' local NVMe disks, which the cluster manifest later on this page references. A kind or minikube cluster uses its default StorageClass instead.
kubectl with a kubeconfig that points to your cluster. To check:
```
kubectl get nodes
```
Permission to install custom resource definitions (CRDs):
```
kubectl auth can-i create CustomResourceDefinition --all-namespaces
```
You should see yes in the output.
A Git repository that Argo CD can read, and push access to it. This repository holds every Application manifest and your cluster configuration.

Keep credentials out of Git. This guide commits only the cluster’s shape, and points to Secrets that you create in the cluster.

What you’ll build

One Git repository holds everything, and one root Application deploys it. The root Application points at argocd/, so Argo CD manages the child Applications the same way it manages any other resource. This is the app-of-apps pattern.

Choose how Redpanda itself is deployed. The Redpanda Operator is the recommended option, and the default throughout this page.

Operator
Helm

Argo CD deploys the operator, and the cluster is a Redpanda custom resource that the operator reconciles:

argocd/
  cert-manager.yaml        # Application, sync wave 0
  redpanda-operator.yaml   # Application, sync wave 1
  redpanda-cluster.yaml    # Application, sync wave 2
clusters/
  redpanda.yaml            # the Redpanda custom resource

cert-manager deploys cert-manager, which issues the brokers' TLS certificates.
redpanda-operator deploys the Redpanda Operator from its Helm chart, including the Redpanda Operator CRDs.
redpanda-cluster deploys the Redpanda custom resource from clusters/. The operator reads that resource and renders the StatefulSet, Services, ConfigMaps, and Secrets for the cluster.

This keeps the broker workload out of Argo CD’s diff. The operator owns the StatefulSet, so Argo CD compares only the custom resource, which is a small manifest that you control in full.

Argo CD renders the Redpanda Helm chart itself, and owns every object it produces:

argocd/
  cert-manager.yaml        # Application, sync wave 0
  redpanda-cluster.yaml    # Application, sync wave 1

cert-manager deploys cert-manager, which issues the brokers' TLS certificates.
redpanda-cluster deploys the Redpanda Helm chart, with the cluster’s values in the Application manifest.

There is no operator and no custom resource. This trades the operator’s reconciliation for direct control of the rendered manifests, and it needs one extra setting to sync cleanly.

Only Application manifests live in your repository. Each one pins an upstream chart by version, at https://charts.jetstack.io and https://charts.redpanda.com, so you never vendor a chart into Git. Changing a pinned version, or any cluster setting, is a commit.

Argo CD documents app-of-apps as an admin-only tool, because a child Application can name any Argo CD project. Restrict push access to this repository, and review the project field in every pull request that touches argocd/.

Install Argo CD

Argo CD 2.10 or later is required for the server-side diff option used later on this page. The stable manifests install a 3.x release.

Create the argocd namespace and install Argo CD:

kubectl create namespace argocd
kubectl apply --server-side --force-conflicts -n argocd \
  -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

The --server-side flag avoids the annotation size limit that the Argo CD CRDs exceed.

Wait for the repo server and the application controller to roll out:

kubectl -n argocd rollout status deploy/argocd-repo-server --timeout=5m
kubectl -n argocd rollout status statefulset/argocd-application-controller --timeout=5m

Get the initial admin password:

kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath='{.data.password}' | base64 -d; echo

Open the Argo CD UI on https://localhost:8080 and log in as admin:
```
kubectl port-forward svc/argocd-server -n argocd 8080:443
```
This guide needs no LoadBalancer Service. Leave the port forward running in a separate terminal.
If your repository is private, give Argo CD a credential for it. Without one, every Application that reads from git fails to compare with Failed to load target state:
```
kubectl -n argocd create secret generic repo-redpanda \
  --from-literal=type=git \
  --from-literal=url=https://github.com/<organization>/<repository>.git \
  --from-literal=username=<username> \
  --from-literal=password=<token>
kubectl -n argocd label secret repo-redpanda \
  argocd.argoproj.io/secret-type=repository
```
Replace <username> and <token> with a user and a personal access token that can read the repository. You can also add the credential from the UI, under Settings > Repositories.

Deploy cert-manager

Redpanda clusters have TLS enabled by default, and the Redpanda Helm chart renders cert-manager Certificate and Issuer objects for the broker certificates. Deploy cert-manager as its own Application, so that Git holds its version too, and sync it before the cluster. Both options need it.

Commit the following manifest to argocd/cert-manager.yaml:

argocd/cert-manager.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: cert-manager
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "0" (1)
spec:
  project: default
  source:
    repoURL: https://charts.jetstack.io
    chart: cert-manager
    targetRevision: v1.21.2 (2)
    helm:
      releaseName: cert-manager
      valuesObject:
        crds:
          enabled: true (3)
  destination:
    server: https://kubernetes.default.svc
    namespace: cert-manager
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - ServerSideApply=true (4)
      - CreateNamespace=true

1	Syncs first, before Redpanda.
2	Pins the cert-manager chart version.
3	Installs the cert-manager CRDs. cert-manager documents `crds.enabled` as the value to set when a deployment tool installs the chart, because it renders the CRDs as ordinary templates that Argo CD can then own and upgrade. Don’t use the older `installCRDs` flag, which the chart deprecates.
4	Applies manifests server-side. Three of the six cert-manager CRDs are larger than the 262144-byte annotation that client-side apply writes, the largest at around 325 KB, so a client-side sync fails.

Two behaviors of this chart are worth knowing when you watch the sync. The cert-manager-startupapicheck Job carries the Helm post-install hook annotation, which Argo CD runs as a PostSync hook, so the Application reports Synced only after that Job succeeds. The CRDs carry helm.sh/resource-policy: keep, which Argo CD honors as Delete=false, so deleting this Application leaves the CRDs, and the certificates they hold, in place.

To manage cert-manager outside Argo CD instead, install it with Helm:

helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install cert-manager jetstack/cert-manager \
  --set crds.enabled=true \
  --namespace cert-manager \
  --create-namespace

Deploy Redpanda

Commit the manifests for your chosen option, then deploy all of them in Deploy the root Application.

In both options, replace <namespace> with the namespace to deploy into, such as redpanda, and replace <organization> and <repository> with your Git repository.

Operator
Helm

Commit the operator’s Application to argocd/redpanda-operator.yaml:

argocd/redpanda-operator.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: redpanda-operator
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "1" (1)
spec:
  project: default
  source:
    repoURL: https://charts.redpanda.com
    chart: operator
    targetRevision: "26.2.4" (2)
    helm:
      releaseName: redpanda-operator
      valuesObject:
        crds:
          enabled: true (3)
  destination:
    server: https://kubernetes.default.svc
    namespace: <namespace>
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - ServerSideApply=true (4)
      - CreateNamespace=true

1	Syncs after cert-manager and before the cluster.
2	Pins the Redpanda Operator chart version. Pin it explicitly so that a sync never upgrades the operator without a commit.
3	Installs the Redpanda Operator CRDs, including the `Redpanda` resource used in the next step.
4	Applies manifests server-side. The Redpanda Operator CRDs are too large for the last-applied-configuration annotation that client-side apply writes. This Application syncs clean with no extra configuration. Argo CD ignores the fields that the API server defaults on the operator’s own objects, such as `protocol` and `imagePullPolicy`, so a live object that carries more defaulted fields than the manifest is not a diff.

Commit the cluster resource to clusters/redpanda.yaml:

clusters/redpanda.yaml

apiVersion: cluster.redpanda.com/v1alpha2
kind: Redpanda
metadata:
  name: redpanda
spec:
  clusterSpec:
    image:
      tag: v26.2.3 (1)
    statefulset:
      replicas: 3
    storage:
      persistentVolume:
        enabled: true
        storageClass: csi-driver-lvm-striped-xfs (2)

1 Pins the Redpanda version. Upgrade the cluster by changing this tag in Git.

Points each broker’s PersistentVolumeClaim at the StorageClass that your platform guide created. Leave the field out to use the cluster’s default provisioner, which is what a kind or minikube cluster needs.

To enable SASL authentication, add the credentials to a Secret in the cluster and reference it from auth.sasl.secretRef, as described in Configure Authentication. The operator renders the Redpanda Helm chart inside the cluster, so the generated bootstrap user password stays stable across syncs.

Commit the cluster’s Application to argocd/redpanda-cluster.yaml:

argocd/redpanda-cluster.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: redpanda-cluster
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "2" (1)
spec:
  project: default
  source:
    repoURL: https://github.com/<organization>/<repository>.git (2)
    targetRevision: main
    path: clusters
  destination:
    server: https://kubernetes.default.svc
    namespace: <namespace>
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - ServerSideApply=true

1	Syncs last, once the operator is running and cert-manager can issue the brokers' certificates.
2	Your own repository, the one that holds `argocd/` and `clusters/`.

The Redpanda Helm chart sets a few values that the API server rewrites on write, such as resources.limits.cpu from 1 to "1" and resources.limits.memory from 2.5Gi to 2560Mi. Argo CD compares the manifest against the live object, sees the rewritten values, and reports the StatefulSet as OutOfSync even though the cluster is healthy. Server-side diff fixes this by running the manifest through a server-side apply dry run, so that both sides of the comparison carry the same defaulting.

Commit the following Application to argocd/redpanda-cluster.yaml:

argocd/redpanda-cluster.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: redpanda-cluster
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "1"
    argocd.argoproj.io/compare-options: ServerSideDiff=true (1)
spec:
  project: default
  source:
    repoURL: https://charts.redpanda.com
    chart: redpanda
    targetRevision: "26.2.4"
    helm:
      releaseName: redpanda
      valuesObject:
        image:
          tag: v26.2.3
        statefulset:
          replicas: 3
        storage:
          persistentVolume:
            enabled: true
            storageClass: csi-driver-lvm-striped-xfs
        commonAnnotations: (2)
          argocd.argoproj.io/sync-wave: "1"
        console:
          annotations:
            argocd.argoproj.io/sync-wave: "1"
  destination:
    server: https://kubernetes.default.svc
    namespace: <namespace>
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - ServerSideApply=true (3)
      - CreateNamespace=true

1	Compares a server-side apply dry run against the live object. Add `IncludeMutationWebhook=true` if a mutating webhook still causes drift.
2	Puts every object the Redpanda Helm chart renders into a wave. Requires chart version 26.2.4 or later. Pair it with the same wave on `console.annotations`, or the sync deadlocks. See Order resources with sync waves.
3	Pair server-side diff with server-side apply.

Optionally, enable server-side diff for every Application in the instance, instead of annotating each one. Restart the application controller so the change takes effect:

kubectl -n argocd patch configmap argocd-cmd-params-cm \
  --type merge -p '{"data":{"controller.diff.server.side":"true"}}'
kubectl -n argocd rollout restart statefulset/argocd-application-controller

Prefer server-side diff over ignoreDifferences. A list of ignored paths has to name every rewritten field for every container, and it hides real changes to any path it lists. Reserve ignoreDifferences for a single field that another controller owns, such as spec.replicas when a HorizontalPodAutoscaler writes it.

Deploy the root Application

One root Application deploys the children. Argo CD assesses each child’s health, so the sync waves hold the order.

Restore the health check for Application resources, which Argo CD does not assess by default. Without it, every child reports healthy the moment it exists, and the waves no longer hold:

argocd-cm-application-health.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
  labels:
    app.kubernetes.io/name: argocd-cm
    app.kubernetes.io/part-of: argocd
data:
  resource.customizations.health.argoproj.io_Application: |
    hs = {}
    hs.status = "Progressing"
    hs.message = ""
    if obj.status ~= nil then
      if obj.status.health ~= nil then
        hs.status = obj.status.health.status
        if obj.status.health.message ~= nil then
          hs.message = obj.status.health.message
        end
      end
    end
    return hs

kubectl -n argocd patch configmap argocd-cm --patch-file argocd-cm-application-health.yaml

Push your repository, then apply the root Application. This is the only manifest you apply by hand:

app-root.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: redpanda-root
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/<organization>/<repository>.git
    targetRevision: main
    path: argocd (1)
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd (2)
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

1	The directory of child Application manifests.
2	Application resources live in the Argo CD namespace, not in the cluster’s namespace. `kubectl apply -f app-root.yaml`

Watch the children sync in wave order:

kubectl -n argocd get applications --watch

Confirm each component as its wave completes:

kubectl -n cert-manager rollout status deploy/cert-manager --timeout=5m
kubectl -n cert-manager rollout status deploy/cert-manager-webhook --timeout=5m
kubectl -n cert-manager rollout status deploy/cert-manager-cainjector --timeout=5m
kubectl --namespace <namespace> rollout status --watch statefulset/redpanda

With the operator, also watch the custom resource. The cluster is ready when the READY column shows True:

kubectl get redpanda --namespace <namespace> --watch

From here on, every change is a commit to this repository: a new chart version, a broker count, a cluster property. Argo CD picks it up and syncs it.

Verify the deployment

Check that the root Application and its children are synced and healthy:

kubectl -n argocd get applications \
  -o custom-columns='NAME:.metadata.name,SYNC:.status.sync.status,HEALTH:.status.health.status'

NAME                SYNC     HEALTH
cert-manager        Synced   Healthy
redpanda-cluster    Synced   Healthy
redpanda-operator   Synced   Healthy
redpanda-root       Synced   Healthy

The redpanda-operator row is absent when Argo CD renders the Redpanda Helm chart.

List any resources that Argo CD considers out of sync. The output is empty when the Application is clean:

kubectl -n argocd get application redpanda-cluster \
  -o jsonpath='{range .status.resources[?(@.status=="OutOfSync")]}{.kind}/{.name}{"\n"}{end}'

Confirm that the brokers formed a cluster:

kubectl --namespace <namespace> exec -it redpanda-0 -c redpanda -- \
  rpk cluster info

Order resources with sync waves

Argo CD applies resources in ascending sync wave order, and waits for each wave to become healthy before it starts the next. To put the objects that a Helm chart renders into a wave, annotate them with argocd.argoproj.io/sync-wave.

A wave orders resources within one Application, which is why the child Applications carry waves of their own: the root Application syncs them in that order, the same way any Application orders its resources. That ordering depends on the Application health check restored in Deploy the root Application, because Argo CD moves to the next wave once the current one is healthy.

commonAnnotations requires Redpanda Helm chart 26.2.4 or later. It sets annotations on the objects that the Redpanda Helm chart renders, so it applies when Argo CD renders that chart. It has no effect on a cluster deployed from a Redpanda custom resource, because the Redpanda CRD has no matching field and the API server drops unknown fields. To order a custom resource, annotate the resource itself in Git.

Set commonAnnotations in the Helm values of the Application that renders the Redpanda Helm chart:

commonAnnotations:
  argocd.argoproj.io/sync-wave: "1"

The Redpanda Helm chart adds these annotations to every object it renders, including the StatefulSet object. Two places are deliberately left out:

The StatefulSet’s Pod template. Annotating the Pod template changes the Pod spec, which restarts every broker.
The StatefulSet’s volume claim templates, which are immutable on a live StatefulSet.

Annotations set on an individual resource take precedence over commonAnnotations, as do the Helm hook annotations that the Redpanda Helm chart needs for its jobs.

Console objects come from the Console subchart and don’t inherit commonAnnotations. When you use commonAnnotations to set a sync wave, set the same wave on the Console Deployment, or the sync deadlocks:

commonAnnotations:
  argocd.argoproj.io/sync-wave: "1"
console:
  annotations:
    argocd.argoproj.io/sync-wave: "1"

Without the second annotation the Console Deployment stays in wave 0 while the Certificate whose Secret it mounts moves to wave 1. Argo CD waits for wave 0 to become healthy before it applies wave 1, the Deployment never starts because its certificate doesn’t exist yet, and no brokers are created. The Application sits at waiting for healthy state of apps/Deployment/redpanda-console.

Annotate the Console Service and ServiceAccount through console.service.annotations and console.serviceAccount.annotations if you need the annotation on those too.

Use the same value for any annotation you need on every object, such as the owner or cost center that a policy engine requires.

To annotate the objects that the Redpanda Operator chart renders, use its annotations value, which behaves the same way:

annotations:
  argocd.argoproj.io/sync-wave: "0"

Troubleshoot sync failures

Every Redpanda Operator chart object stays OutOfSync

Redpanda Operator chart versions before 26.2.4 render annotations: null on objects that have no annotations configured. Argo CD compares that null against the absent annotations map on the live object and keeps the object, such as the ServiceMonitor, permanently out of sync.

Upgrade the Redpanda Operator chart to 26.2.4 or later, which renders an empty map instead.

The StatefulSet stays OutOfSync

This affects clusters that Argo CD renders from the Redpanda Helm chart. It sets values that the API server rewrites. Turn on server-side diff, as described in the Helm option of Deploy Redpanda.

Clusters deployed from a Redpanda custom resource don’t hit this, because the operator owns the StatefulSet and Argo CD never diffs it.

The bootstrap user Secret rejects the sync

This affects clusters that Argo CD renders from the Redpanda Helm chart, with SASL enabled. A sync can fail with the following error, which was reported on Argo CD 2.13 with chart 25.3.2:

Secret "redpanda-bootstrap-user" is invalid:
data: Forbidden: field is immutable when `immutable` is set.

When you don’t configure a password for the bootstrap user, the Redpanda Helm chart generates one and marks the Secret immutable. On each render it looks up the existing Secret to reuse that password, but Argo CD renders charts with helm template, which has no cluster access. The lookup returns nothing and the chart generates a new password, so every render produces a different bootstrap user password than the one the cluster is running with. Whether the API server rejects the change outright depends on your Argo CD version, but the churn is a problem either way: configure the password so that it stops.

Configure the password so that every render produces the same Secret. Either point auth.sasl.bootstrapUser.secretKeyRef at a Secret that you create outside Argo CD, in which case the Redpanda Helm chart renders no bootstrap user Secret at all:

auth:
  sasl:
    enabled: true
    secretRef: redpanda-superusers
    bootstrapUser:
      secretKeyRef:
        name: redpanda-bootstrap-user
        key: password

Or set auth.sasl.bootstrapUser.password from your secret manager at render time.

On a cluster that’s already running, reuse the password that the cluster was bootstrapped with. Redpanda creates the bootstrap user when the cluster first starts, and a values change doesn’t rewrite that credential:

kubectl --namespace <namespace> get secret redpanda-bootstrap-user \
  -o jsonpath='{.data.password}' | base64 -d; echo

Clusters deployed from a Redpanda custom resource don’t hit this. The operator renders the Redpanda Helm chart inside the cluster, where the lookup finds the existing Secret.

A sync fails on the commonAnnotations value

Redpanda Operator chart 26.2.4 renames commonAnnotations to connectAnnotations, and the Redpanda Operator’s --common-annotations flag to --connect-annotations. The value only ever annotated the resources that the Redpanda Connect Pipeline controller creates, so the old name read as a chart-wide setting that it never was.

The Redpanda Operator chart rejects unknown values, so the old name fails the sync rather than being ignored. Rename the value in Git before you upgrade the targetRevision:

connectAnnotations:
  owner: platform-team@example.com

Next steps

Customize the Helm chart to change cluster configuration through Git.
Manage Redpanda Connect pipelines, which are also plain manifests that Argo CD can deploy.
Deployment workflow for a production checklist.

Suggested labs

Set Up GitOps for the Redpanda Helm Chart

Search all labs

Was this helpful?

group Ask in the community

mail Share your feedback