Monitor Data Transforms

Set up monitoring for your cluster.

You can identify performance bottlenecks by monitoring latency and CPU usage:

If latency is high, investigate the transform logic for inefficiencies or consider scaling the resources. High CPU usage might indicate the need for optimization in the code or an increase in allocated CPU resources.

Tracking execution errors and error states helps in maintaining the reliability of your data transforms:

Make sure to implement robust error handling and logging within your transform functions to help with troubleshooting.

Monitoring memory usage metrics and total execution time ensures that the Wasm engine does not exceed allocated resources, helping in efficient resource management:

If memory usage is consistently high or exceeds the maximum allocated memory:

Keeping track of read and write bytes and processor lag helps in understanding the data flow through your transforms, enabling better capacity planning and scaling:

If there is a significant lag or low throughput, investigate potential bottlenecks in the data flow or consider scaling your infrastructure to handle higher throughput.

Runtime logs for transform functions are written to an internal topic called _redpanda.transform_logs. You can read these logs by using the rpk transform logs command.

Replace <transform-name> with the configured name of the transform function.

By default, Redpanda provides several settings to manage logging for data transforms, such as buffer capacity, flush interval, and maximum log line length. These settings ensure that logging operates efficiently without overwhelming the system. However, you may need to adjust these settings based on your specific requirements and workloads. For information on how to configure logging, see the Configure transform logging section of the configuration guide.