# Resilience Testing in Kubernetes

> For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt)

---
title: Resilience Testing in Kubernetes
latest-redpanda-tag: v24.3.9
latest-console-tag: v3.7.3
latest-operator-version: v26.1.4
# EOL = End-of-Life (support lifecycle status)
page-is-nearing-eol: "false"
page-is-past-eol: "true"
page-eol-date: December 3, 2025
latest-connect-version: 4.93.0
docname: kubernetes/k-resilience-testing
page-component-name: streaming
page-version: "24.3"
page-component-version: "24.3"
page-component-title: Streaming
page-relative-src-path: kubernetes/k-resilience-testing.adoc
page-edit-url: https://github.com/redpanda-data/docs/edit/v/24.3/modules/manage/pages/kubernetes/k-resilience-testing.adoc
description: With resilience testing, you can introduce failures and observe how the system behaves under each failure scenario.
page-git-created-date: "2024-01-04"
page-git-modified-date: "2024-02-26"
support-status: past end-of-life
---

<!-- Source: https://docs.redpanda.com/streaming/24.3/manage/kubernetes/k-resilience-testing.md -->

Resilience testing is an important part of ensuring that a system is reliable and can recover from failures. To perform resilience testing for Redpanda in Kubernetes, you can introduce failure scenarios and observe how the system behaves under each scenario.

## [](#prerequisites)Prerequisites

-   Create a test environment that mimics your production environment as closely as possible. The test environment should include a Redpanda cluster with at least three replicas, and any services that your application depends on. You can find guides for deploying Redpanda in [Get Started with Redpanda in Kubernetes](https://docs.redpanda.com/streaming/24.3/deploy/deployment-option/self-hosted/kubernetes/get-started-dev/).

-   [Set up monitoring](https://docs.redpanda.com/streaming/24.3/manage/kubernetes/monitoring/) so that you can observe changes in the system behavior.


## [](#simulate-failure-scenarios)Simulate failure scenarios

This section provides the steps to simulate failure scenarios in Kubernetes. After each simulation, it’s important to monitor the behavior of the Redpanda cluster and any clients that are connected to it.

### [](#broker-going-down)Broker going down

You can simulate a broker going down for an extended period of time by manually terminating one of them.

1.  Find out on which node each of your brokers is running:

    ```bash
    kubectl get pod --namespace <namespace>  \
      -o=custom-columns=NODE:.spec.nodeName,NAME:.metadata.name -l \
      app.kubernetes.io/component=redpanda-statefulset
    ```

2.  Taint the node that’s running the broker that you want to terminate:

    ```bash
    kubectl taint nodes <node-name> isolate-broker=true:NoExecute
    ```

    Replace `<node-name>` with the name of the node you want to taint.

    Any Pods that do not tolerate this taint are terminated and evicted from the node.

3.  Monitor the logs and metrics of the remaining brokers to observe how they behave when a broker is unexpectedly terminated.

4.  Remove the taint when you’re ready for the broker to come back online:

    ```bash
    kubectl taint nodes <node-name> isolate-broker=true:NoExecute-
    ```

5.  Check whether the terminated broker can rejoin the cluster when it is rescheduled on the node and comes back online.


## [](#suggested-reading)Suggested reading

It’s best practice to automate failure scenarios as part of your regular testing to identify any weaknesses in your deployment. You can use tools, such as [Chaos Monkey](https://netflix.github.io/chaosmonkey/) and [LitmusChaos](https://docs.litmuschaos.io/docs/getting-started/installation/).

## Suggested labs

-   [Redpanda Iceberg Docker Compose Example](https://docs.redpanda.com/labs/docker-compose/iceberg/)
-   [Iceberg Streaming on Kubernetes with Redpanda, MinIO, and Spark](https://docs.redpanda.com/labs/kubernetes/iceberg/)

[Search all labs](https://docs.redpanda.com/labs)