# Audit Logging in Kubernetes

> For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt)

---
title: Audit Logging in Kubernetes
latest-operator-version: v26.1.4
# EOL = End-of-Life (support lifecycle status)
page-is-nearing-eol: "false"
page-is-past-eol: "true"
page-eol-date: December 22, 2024
latest-console-tag: v3.7.3
latest-connect-version: 4.93.0
docname: kubernetes/security/k-audit-logging
page-component-name: streaming
page-version: "23.3"
page-component-version: "23.3"
page-component-title: Streaming
page-relative-src-path: kubernetes/security/k-audit-logging.adoc
page-edit-url: https://github.com/redpanda-data/docs/edit/v/23.3/modules/manage/pages/kubernetes/security/k-audit-logging.adoc
description: Learn how to use Redpanda's audit logging capabilities.
page-git-created-date: "2024-01-05"
page-git-modified-date: "2024-02-26"
support-status: past end-of-life
---

<!-- Source: https://docs.redpanda.com/streaming/23.3/manage/kubernetes/security/k-audit-logging.md -->

Many scenarios for streaming data include the need for fine-grained auditing of user activity related to the system. This is especially true for regulated industries such as finance, healthcare, and the public sector. Complying with [PCI DSS v4](https://pcidssguide.com/whats-new-in-pci-dss-v4-0/) standards, for example, requires verbose and detailed activity auditing, alerting, and analysis capabilities.

Redpanda’s auditing capabilities support recording both administrative and operational interactions with topics and with users. Redpanda complies with the Open Cybersecurity Schema Framework (OCSF), providing a predictable and extensible solution that works seamlessly with industry standard tools.

With audit logging enabled, there should be no noticeable changes in performance other than slightly elevated CPU usage.

> 📝 **NOTE**
>
> Audit logging is configured at the cluster level. Redpanda supports excluding specific topics or principals from auditing to help reduce noise in the log. Audit logging is disabled by default.

## [](#audit-log-flow)Audit log flow

The Redpanda audit log mechanism functions similar to the Kafka flow you may be familiar with. When a user interacts with another user or with a topics, Redpanda writes an event to a specialized audit topic. The audit topic is immutable. Only Redpanda can write to it. Users are prevented from writing to the audit topic directly and the Kafka API cannot create or delete it.

![Audit log flow](https://docs.redpanda.com/streaming/23.3/shared/_images/audit-logging-flow.png)

By default, any management and authentication actions performed on the cluster yield messages written to the audit log topic that are retained for seven days. Interactions with all topics by all principals are audited. Actions performed using the Kafka API and Admin API are all audited, as are actions performed directly through `rpk`.

Messages recorded to the audit log topic comply with the [open cybersecurity schema framework](https://schema.ocsf.io/). Any number of analytics frameworks, such as Splunk or Sumo Logic, can receive and process these messages. Using an open standard ensures Redpanda’s audit logs coexist with those produced by other IT assets, powering holistic monitoring and analysis of your assets.

## [](#audit-log-configuration-options)Audit log configuration options

Redpanda’s audit logging mechanism supports several options to control the volume and availability of audit records. Configuration is applied at the cluster level using the standard [cluster configuration mechanism](https://docs.redpanda.com/current/manage/cluster-maintenance/cluster-property-configuration/).

You can configure these options directly in either the Helm values or the Redpanda resource.

-   `auditLogging.enabled`: Sets the value of the [`audit_enabled`](https://docs.redpanda.com/streaming/23.3/reference/cluster-properties/#audit_enabled) cluster property to enable audit logging. When you set this to `true`, Redpanda checks for an existing topic named `_redpanda.audit_log`. If none is found, Redpanda automatically creates one for you. Default: `false`.

-   `auditLogging.partitions`: Sets the value of the [`audit_log_num_partitions`](https://docs.redpanda.com/streaming/23.3/reference/cluster-properties/#audit_log_num_partitions) cluster property to define the number of partitions used by a newly created audit topic. This configuration applies only to the audit log topic and may be different from the cluster or other topic configurations. This cannot be altered for an existing audit log topic. Default: `12`.

-   `auditLogging.replicationFactor`: Sets the value of the [`audit_log_replication_factor`](https://docs.redpanda.com/streaming/23.3/reference/cluster-properties/#audit_log_replication_factor) cluster property to define the replication factor for a newly created audit log topic. This configuration applies only to the audit log topic and may be different from the cluster or other topic configurations. This cannot be altered for existing audit log topics. If a value is not provided, Redpanda will use the `internal_topic_replication_factor` cluster config value. Default: `null`.

-   `auditLogging.enabledEventTypes`: Sets the value of the [`audit_enabled_event_types`](https://docs.redpanda.com/streaming/23.3/reference/cluster-properties/#audit_enabled_event_types) cluster property. This option is a list of JSON strings identifying the [event types](#audit-logging-event-types) to include in the audit log. Valid values include any of the following - `management`, `produce`, `consume`, `describe`, `heartbeat`, `authenticate`, `schema_registry`, `admin`. Default: `'["management","authenticate","admin"]'`.

-   `auditLogging.excludedTopics`: Sets the value of the [`audit_exclude_topics`](https://docs.redpanda.com/streaming/23.3/reference/cluster-properties/#audit_exclude_topics) cluster property. This option is a list of JSON strings identifying the topics the audit logging system should ignore. This list cannot include the `_redpanda.audit_log` topic. Redpanda will reject the command if you do attempt to include that topic. Default: `null`.

-   `auditLogging.excludedPrincipals`: Sets the value of the [`audit_exclude_principals`](https://docs.redpanda.com/streaming/23.3/reference/cluster-properties/#audit_exclude_principals) cluster property. This option is a list of JSON strings identifying the principals the audit logging system should ignore. Principals can be listed as `User:name` or `name`, both are accepted. Default: `null`.

-   `auditLogging.clientMaxBufferSize`: Sets the value of the [`audit_client_max_buffer_size`](https://docs.redpanda.com/streaming/23.3/reference/cluster-properties/#audit_client_max_buffer_size) cluster property to define the number of bytes allocated by the internal audit client for audit messages. When changing this, you must disable audit logging and then re-enable it for the change to take effect. Consider increasing this if your system generates a very large number of audit records in a short amount of time. Default: `16777216`.

-   `auditLogging.queueDrainIntervalMs`: Sets the value of the [`audit_queue_drain_interval_ms`](https://docs.redpanda.com/streaming/23.3/reference/cluster-properties/#audit_queue_drain_interval_ms) cluster property. Internally, Redpanda batches audit log messages in memory and periodically writes them to the audit log topic. This option defines the period in milliseconds between draining this queue to the audit log topic. Longer intervals may help prevent duplicate messages, especially in high throughput scenarios, but they also increase the risk of data loss during hard shutdowns where the queue is lost. Default: `500`.

-   `auditLogging.queueMaxBufferSizePerShard`: Sets the value of the [`audit_queue_max_buffer_size_per_shard`](https://docs.redpanda.com/streaming/23.3/reference/cluster-properties/#audit_queue_max_buffer_size_per_shard) cluster property to define the maximum amount of memory in bytes used by the audit buffer in each shard. Once this size is reached, requests to log additional audit messages will return a non-retryable error. Default: `1048576`.


Even though audited event messages are stored to a specialized immutable topic, standard topic settings still apply. For example, you can apply the same Tiered Storage, retention time, and replication settings available to normal topics. These particular options are important for controlling the amount of disk space utilized by your audit topics.

> ❗ **IMPORTANT**
>
> You cannot change the values of `auditLogging.partitions` and `auditLogging.replicationFactor` after enabling audit logging because these settings impact the creation of the `_redpanda.audit_log` topic. The Kafka API allows you to add partitions or alter the replication factor after enabling audit logging, but Redpanda prevents you from altering these two configuration values directly.

## [](#audit-logging-event-types)Audit logging event types

Redpanda’s auditable events fall into one of eight different event types. The APIs associated with each event type are as follows.

| Audit event type | Associated APIs |
| --- | --- |
| management | AlterPartitionReassignmentsCreateACLsCreatePartitionsCreateTopicsDeleteAclsDeleteGroupsDeleteRecordsDeleteTopicsIncrementalAlterconfigsOffsetDelete |
| produce | AddPartitionsToTxnEndTxnInitProducerIdProduce |
| consume | AddOffsetsToTxnFetchJoinGroupLeaveGroupListOffsetOffsetCommitSyncGroupTxOffsetCommit |
| describe | DescribeAclsDescribeConfigsDescribeGroupsDescribeLogDirsFindCoordinatorListGroupsListPartitionReassignmentsMetadataOffsetForLeaderEpochDescribeProducersDescribeTransationsListTransactions |
| heartbeat | Heartbeat |
| authenticate | All authentication events |
| schema_registry | All Schema Registry API calls |
| admin | All Admin API calls |

## [](#enable-audit-logging)Enable audit logging

All audit log settings are applied at the cluster level. You can configure audit log settings in the Redpanda Helm chart, using Helm values or the Redpanda resource with the Redpanda Operator.

### Helm + Operator

If you want to manage the audit topic using a [Topic resource](https://docs.redpanda.com/streaming/23.3/manage/kubernetes/k-manage-topics/):

1.  Create the Redpanda resource with SASL enabled:

    `redpanda-cluster.yaml`

    ```yaml
    apiVersion: cluster.redpanda.com/v1alpha1
    kind: Redpanda
    metadata:
      name: redpanda
    spec:
      chartRef: {}
      clusterSpec:
        auth:
          sasl:
            enabled: true
            secretRef: "redpanda-users"
    ```

2.  Create a Secret to hold the password of either a user or a superuser:

    ```bash
    kubectl --namespace <namespace> create secret generic audit-topic-user-password --from-literal=password=<password>
    ```

    > 📝 **NOTE**
    >
    > If you aren’t using a superuser, make sure the user is authorized to manage the `_redpanda.audit_log` topic.

3.  Create the Topic resource, and make sure to set `overwriteTopicName` to `_redpanda.audit_log`:

    ```yaml
    apiVersion: cluster.redpanda.com/v1alpha1
    kind: Topic
    metadata:
      name: audit-topic
    spec:
      partitions: <partitions>
      overwriteTopicName: _redpanda.audit_log
      kafkaApiSpec:
        brokers:
          - "redpanda-0.redpanda.<namespace>.svc.cluster.local:9093"
          - "redpanda-1.redpanda.<namespace>.svc.cluster.local:9093"
          - "redpanda-2.redpanda.<namespace>.svc.cluster.local:9093"
        tls:
          caCertSecretRef:
            name: "redpanda-default-cert"
            key: "ca.crt"
        sasl:
          username: <username>
          mechanism: <mechanism>
          passwordSecretRef:
            name: audit-topic-user-password
            key: password
    ```

    This example configuration contains values for the `kafkaApiSpec.tls` and `kafkaApiSpec.brokers` settings that work with the default Helm values. If you modified these settings in your deployment, make sure to replace those values.

4.  Update the Redpanda resource to enable audit logging:

    `redpanda-cluster.yaml`

    ```yaml
    apiVersion: cluster.redpanda.com/v1alpha1
    kind: Redpanda
    metadata:
      name: redpanda
    spec:
      chartRef: {}
      clusterSpec:
        auth:
          sasl:
            enabled: true
            secretRef: "redpanda-users"
        auditLogging:
          enabled: true
    ```


If you don’t want to use the Topic resource, you can enable audit logging and Redpanda will create the audit topic for you:

`redpanda-cluster.yaml`

```yaml
apiVersion: cluster.redpanda.com/v1alpha1
kind: Redpanda
metadata:
  name: redpanda
spec:
  chartRef: {}
  clusterSpec:
    auth:
      sasl:
        enabled: true
        secretRef: "redpanda-users"
    auditLogging:
      enabled: true
```

```bash
kubectl apply -f redpanda-cluster.yaml --namespace <namespace>
```

### Helm

#### --values

`audit-logging.yaml`

```yaml
auth:
  sasl:
    enabled: true
    secretRef: "redpanda-users"
auditLogging:
  enabled: true
```

```bash
helm upgrade --install redpanda redpanda/redpanda --namespace <namespace> --create-namespace \
--values audit-logging.yaml --reuse-values
```

#### --set

```bash
helm upgrade --install redpanda redpanda/redpanda --namespace <namespace> --create-namespace \
  --set auth.sasl.enabled=true \
  --set auth.sasl.secretRef="redpanda-users" \
  --set auditLogging.enabled=true
```

-   `auth.sasl.enabled`: To enable audit logging, you must have:

    -   SASL authentication enabled.

    -   At least one Kafka listener that uses the SASL authentication mechanism. By default, the internal Kafka listener is used (`listeners.kafka`). To use a different listener, see [Audit log configuration options](#audit-log-configuration-options).

    -   At least one superuser.


    For details, see [Configure Authentication for Redpanda in Kubernetes](https://docs.redpanda.com/streaming/23.3/manage/kubernetes/security/authentication/k-authentication/).

-   `auditLogging.enabled`: Enable audit logging. When you set this to `true`, Redpanda checks for an existing topic named `_redpanda.audit_log`. If the topic is not found, Redpanda automatically creates one for you. Default: `false`.


## [](#optimize-costs-for-audit-logging)Optimize costs for audit logging

When enabled, audit logging can quickly generate a very large amount of data, especially if all event types are selected. Proper configuration of audit logging is critical to avoid filling your disk or using excess Tiered Storage. The configuration options available help ensure your audit logs contain only the volume of data necessary to meeting your regulatory or legal requirements.

With audit logging, the pattern of message generation may be very different from your typical sources of data. These messages reflect usage of your system as opposed to the operational data your topics typically process. As a result, your retention, replication, and Tiered Storage requirements may differ from your other topics.

A typical scenario with audit logging is to route the messages to an analytics platform like Splunk. If your retention period is too long, you will find that you are storing excessive amounts of replicated messages in both Redpanda and in your analytics suite. Identifying the right balance of retention and replication settings minimizes this duplication while retaining your data in a system that provides actionable intelligence.

Assess the retention needs for your audit logs. You may not need to keep the logs around for the default seven days. This is controlled by setting [`retention.ms`](https://docs.redpanda.com/streaming/23.3/reference/topic-properties/#retentionms) for the `_redpanda.audit_log` topic or by setting [`delete_retention_ms`](https://docs.redpanda.com/streaming/23.3/reference/cluster-properties/#delete_retention_ms) at the cluster level.

## [](#next-steps)Next steps

[See samples of audit log messages](https://docs.redpanda.com/streaming/23.3/manage/audit-logging/audit-log-samples/).

## [](#suggested-reading)Suggested reading

-   [Topic Configuration Properties](https://docs.redpanda.com/streaming/23.3/reference/topic-properties/)

-   [Manage Topics](https://docs.redpanda.com/streaming/23.3/develop/config-topics/)


## Suggested labs

-   [Enable Unified Identity with Azure Entra ID for Redpanda and Redpanda Console](https://docs.redpanda.com/labs/docker-compose/oidc/)
-   [Migrate Data with Redpanda Migrator](https://docs.redpanda.com/labs/docker-compose/redpanda-migrator/)

[Search all labs](https://docs.redpanda.com/labs)