Audit Logging

Many scenarios for streaming data include the need for fine-grained auditing of user activity related to the system. This is especially true for regulated industries such as finance, healthcare, and the public sector. Complying with PCI DSS v4 standards, for example, requires verbose and detailed activity auditing, alerting, and analysis capabilities.

Redpanda’s auditing capabilities support recording both administrative and operational interactions with topics and with users. Redpanda complies with the Open Cybersecurity Schema Framework (OCSF), providing a predictable and extensible solution that works seamlessly with industry standard tools.

With audit logging enabled, there should be no noticeable changes in performance other than slightly elevated CPU usage.

Audit log flow

The Redpanda audit log mechanism functions similar to the Kafka flow. When a user interacts with another user or with a topic, Redpanda writes an event to a specialized audit topic. The audit topic is immutable. Only Redpanda can write to it. Users are prevented from writing to the audit topic directly and the Kafka API cannot create or delete it.

Audit log flow

By default, any management and authentication actions performed on the cluster yield messages written to the audit log topic that are retained for seven days. Interactions with all topics by all principals are audited. Actions performed using the Kafka API and Admin API are all audited, as are actions performed directly through rpk.

Messages recorded to the audit log topic comply with the open cybersecurity schema framework. Any number of analytics frameworks, such as Splunk or Sumo Logic, can receive and process these messages. Using an open standard ensures Redpanda’s audit logs coexist with those produced by other IT assets, powering holistic monitoring and analysis of your assets.

Audit log configuration options

Redpanda’s audit logging mechanism supports several options to control the volume and availability of audit records. Configuration is applied at the cluster level.

To configure audit logging, see Configure Cluster Properties.

  • audit_enabled: Boolean value to enable audit logging. When you set this to true, Redpanda checks for an existing topic named _redpanda.audit_log. If none is found, Redpanda automatically creates one for you. Default: true.

  • audit_enabled_event_types: List of strings in JSON style identifying the event types to include in the audit log. This may include any of the following: management, produce, consume, describe, heartbeat, authenticate, schema_registry, admin. Default: '["management","authenticate","admin"]'.

  • audit_excluded_principals: List of strings in JSON style identifying the principals the audit logging system should ignore. Principals can be listed as User:name or name, both are accepted. Default: null.

Enable audit logging

Audit logging is enabled by default. Cluster administrators can configure the audited topics and principals. However, only the Redpanda team can configure the type of audited events. For more information or support, contact your Redpanda account team.

Configure retention for audit logs

Assess the retention needs for your audit logs. You may not need to keep the logs for the default seven days. This is controlled by setting the retention.ms property for the _redpanda.audit_log topic.

Next steps

See samples of audit log messages