# Configuration > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [connect-full.txt](https://docs.redpanda.com/connect-full.txt) --- title: Configuration latest-connect-version: 4.93.0 latest-operator-version: v26.1.4 latest-console-tag: v3.7.3 latest-redpanda-tag: v26.1.9 docname: about page-component-name: connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: about.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/configuration/pages/about.adoc description: Learn about different options for configuring Redpanda Connect. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-05-26" --- Redpanda Connect pipelines are configured in a YAML file that consists of a number of root sections, arranged like so: #### Common ```yaml input: kafka: addresses: [ TODO ] topics: [ foo, bar ] consumer_group: foogroup pipeline: processors: - mapping: | root.message = this root.meta.link_count = this.links.length() output: aws_s3: bucket: TODO path: '${! meta("kafka_topic") }/${! json("message.id") }.json' ``` #### Full ```yaml http: address: 0.0.0.0:4195 debug_endpoints: false input: kafka: addresses: [ TODO ] topics: [ foo, bar ] consumer_group: foogroup buffer: none: {} pipeline: processors: - mapping: | root.message = this root.meta.link_count = this.links.length() output: aws_s3: bucket: TODO path: '${! meta("kafka_topic") }/${! json("message.id") }.json' input_resources: [] cache_resources: [] processor_resources: [] rate_limit_resources: [] output_resources: [] logger: level: INFO static_fields: '@service': benthos metrics: prometheus: {} tracer: none: {} shutdown_timeout: 20s shutdown_delay: "" ``` Most sections represent a component type, which you can read about in more detail in [this document](https://docs.redpanda.com/connect/components/about/). These types are hierarchical. For example, an `input` can have a list of child `processor` types attached to it, which in turn can have their own `processor` children. This is powerful but can potentially lead to large and cumbersome configuration files. This document outlines tooling provided by Redpanda Connect to help with writing and managing these more complex configuration files. ## [](#testing)Testing For guidance on how to write and run unit tests for your configuration files read [this guide](https://docs.redpanda.com/connect/configuration/unit_testing/). ## [](#customizing-your-configuration)Customizing your configuration Sometimes it’s useful to write a configuration where certain fields can be defined during deployment. For this purpose Redpanda Connect supports [environment variable interpolation](https://docs.redpanda.com/connect/configuration/interpolation/), allowing you to set fields in your config with environment variables like so: ```yaml input: kafka: addresses: - ${KAFKA_BROKER:localhost:9092} topics: - ${KAFKA_TOPIC:default-topic} ``` This is very useful for sharing configuration files across different deployment environments. ## [](#labels)Labels Labels are unique, user-defined identifiers used throughout Redpanda Connect configurations. They serve two purposes: - **Reference:** Allow different parts of your pipeline to refer to specific components or resources. - **Readability:** Make your configuration more understandable for humans, especially in complex deployments. You can assign labels to most pipeline components, including resources, inputs, outputs, processors, and entire pipelines. Using clear, descriptive labels improves both maintainability and clarity. Labels are commonly applied to the following components: ### [](#resources)Resources Labels identify [reusable resources](#reuse) such as processors, caches, and rate limiters, making them easy to reference elsewhere in your pipeline. ```yaml processor_resources: - label: my-transformer # Processor resource label mapping: 'root = content().uppercase()' cache_resources: - label: user-cache # Cache resource label memory: default_ttl: 300s rate_limit_resources: - label: api-limiter # Rate limiter resource label local: count: 100 interval: 1m ``` ### [](#pipelines-streams-mode)Pipelines (streams mode) When running in [streams mode](https://docs.redpanda.com/connect/guides/streams_mode/about/) with the Streams API, labels serve as pipeline names. This enables you to create and reference pipelines by their label through the API. ```bash # Create a pipeline labeled "data-processor" curl -X POST http://localhost:4195/streams/data-processor \ -H "Content-Type: application/yaml" \ -d @pipeline-config.yaml # Reference a pipeline by its label curl http://localhost:4195/streams/data-processor ``` ### [](#component-labeling-for-clarity)Component labeling for clarity You can also use labels on inputs, outputs, processors, and other components to improve the human-readability of your configuration and make troubleshooting easier. For example: ```yaml input: label: ingest_api http_server: {} pipeline: label: user_data_ingest processors: - label: sanitize_fields mapping: 'root = this.trim()' - resource: my-transformer ``` ## [](#label-naming-requirements)Label naming requirements Labels must meet the following criteria: - **Length**: 3-128 characters - **Allowed characters**: Alphanumeric, hyphens, and underscores (`A-Za-z0-9-_`) - **Case sensitivity**: Labels are case-sensitive Example valid labels my-processor data\_transformer\_01 UserAnalytics-v2 Example invalid labels ab // Too short (less than 3 characters) my.processor // Invalid character: period my processor // Invalid character: space ## [](#reuse)Reusing configuration snippets Sometimes it’s necessary to use a rather large component multiple times. Instead of copy/pasting the configuration or using YAML anchors you can define your component [as a resource](https://docs.redpanda.com/connect/configuration/resources/). In the following example we want to make an HTTP request with our payloads. Occasionally the payload might get rejected due to garbage within its contents, and so we catch these rejected requests, attempt to "cleanse" the contents and try to make the same HTTP request again. Since the HTTP request component is quite large (and likely to change over time) we make sure to avoid duplicating it by defining it as a resource `get_foo`: ```yaml pipeline: processors: - resource: get_foo - catch: - mapping: | root = this root.content = this.content.strip_html() - resource: get_foo processor_resources: - label: get_foo http: url: http://example.com/foo verb: POST headers: SomeThing: "set-to-this" SomeThingElse: "set-to-something-else" ``` ### [](#feature-toggles)Feature toggles Resources can be imported separately to your config file with the cli flag `-r` or `-resources`, which is a useful way to switch out resources with common names based on your chosen environment. For example, with a main configuration file `config.yaml`: ```yaml pipeline: processors: - resource: get_foo ``` And then two resource files, one stored at the path `./staging/request.yaml`: ```yaml processor_resources: - label: get_foo http: url: http://example.com/foo verb: POST headers: SomeThing: "set-to-this" SomeThingElse: "set-to-something-else" ``` And another stored at the path `./production/request.yaml`: ```yaml processor_resources: - label: get_foo http: url: http://example.com/bar verb: PUT headers: Desires: "are-empty" ``` We can select our chosen resource by changing which file we import, either running: ```bash rpk connect run -r ./staging/request.yaml ./config.yaml ``` Or: ```bash rpk connect run -r ./production/request.yaml ./config.yaml ``` These flags also support wildcards, which allows you to import an entire directory of resource files like `rpk connect run -r "./staging/*.yaml" ./config.yaml`. You can find out more about configuration resources in the [resources document](https://docs.redpanda.com/connect/configuration/resources/). ### [](#templating)Templating Resources can only be instantiated with a single configuration, which means they aren’t suitable for cases where the configuration is required in multiple places but with slightly different parameters. Redpanda Connect has a (currently experimental) alternative feature called templates, with which it’s possible to define a custom configuration schema and a template for building a configuration from that schema. You can read more about templates [in this guide](https://docs.redpanda.com/connect/configuration/templating/). ## [](#reloading)Reloading It’s possible to have a running instance of Redpanda Connect reload configurations, including resource files imported with `-r`/`--resources`, automatically when the files are updated without needing to manually restart the service. This is done by specifying the `-w`/`--watcher` flag when running Redpanda Connect in normal mode or in streams mode: ```bash # Normal mode rpk connect run -w -r ./production/request.yaml ./config.yaml ``` ```bash # Streams mode rpk connect streams -w -r ./production/request.yaml ./stream_configs/*.yaml ``` If a file update results in configuration parsing or linting errors then the change is ignored (with logs informing you of the problem) and the previous configuration will continue to be run (until the issues are fixed). ## [](#enabling-discovery)Enabling discovery The discoverability of configuration fields is a common headache with any configuration driven application. The classic solution is to provide curated documentation that is often hosted on a dedicated site. However, a user often only needs to get their hands on a short, runnable example config file for their use case. They just need to see the format and field names as the fields themselves are usually self explanatory. Forcing such a user to navigate a website, scrolling through paragraphs of text, seems inefficient when all they actually needed to see was something like: ```yaml input: amqp_0_9: urls: [ amqp://guest:guest@localhost:5672/ ] consumer_tag: benthos-consumer queue: benthos-queue prefetch_count: 10 prefetch_size: 0 output: stdout: {} ``` In order to make this process easier Redpanda Connect is able to generate usable configuration examples for any types, and you can do this from the binary using the `create` subcommand. If, for example, we wanted to generate a config with a websocket input, a Kafka output and a [`mapping` processor](https://docs.redpanda.com/connect/components/processors/mapping/) in the middle, we could do it with the following command: ```bash rpk connect create websocket/mapping/kafka ``` > 💡 **TIP** > > To see which components Redpanda Connect offers, use `rpk connect list`. All of these generated configuration examples also include other useful config sections such as `metrics`, `logging`, etc with sensible defaults. For more information read the output from `rpk connect create --help`. ## [](#help-with-debugging)Help with debugging Once you have a config written you now move onto the next headache of proving that it works, and understanding why it doesn’t. Redpanda Connect, like most good config driven services, performs validation on configs and tries to provide sensible error messages. However, with validation it can be hard to capture all problems, and the user usually understands their intentions better than the service. In order to help expose and diagnose config errors Redpanda Connect provides two mechanisms, linting and echoing. ### [](#linting)Linting If you attempt to run a config that has linting errors Redpanda Connect will print the errors and halt execution. If, however, you want to test your configs before deployment you can do so with the `lint` subcommand: For example, imagine we have a config `foo.yaml`, where we intend to read from AMQP, but there is a typo in our config struct: ```text input: amqp_0_9: yourl: amqp://guest:guest@rabbitmqserver:5672/ ``` We can catch this error before attempting to run the config: ```bash rpk connect lint ./foo.yaml ./foo.yaml: line 3: field yourl not recognized ``` For more information read the output from `rpk connect lint --help`. ### [](#echoing)Echoing Echoing is where Redpanda Connect can print back your configuration _after_ it has been parsed. It is done with the `echo` subcommand, which is able to show you a normalized version of your config, allowing you to see how it was interpreted: ```bash rpk connect echo ./your-config.yaml ``` You can check the output of the above command to see if certain sections are missing or fields are incorrect, which allows you to pinpoint typos in the config. ## [](#shutting-down)Shutting down Under normal operating conditions, the Redpanda Connect process will shut down when there are no more messages produced by inputs and the final message has been processed. The shutdown procedure can also be initiated by sending the process a interrupt (`SIGINT`) or termination (`SIGTERM`) signal. There are two top-level configuration options that control the shutdown behavior: `shutdown_timeout` and `shutdown_delay`. ### [](#shutdown-delay)Shutdown delay The `shutdown_delay` option can be used to delay the start of the shutdown procedure. This is useful for pipelines that need a short grace period to have their metrics and traces scraped. While the shutdown delay is in effect, the HTTP metrics endpoint continues to be available for scraping and any active tracers are free to flush remaining traces. The shutdown delay can be interrupted by sending the Redpanda Connect process a second OS interrupt or termination signal. ### [](#shutdown-timeout)Shutdown timeout The `shutdown_timeout` option sets a hard deadline for Redpanda Connect process to gracefully terminate. If this duration is exceeded then the process is forcefully terminated and any messages that were in-flight will be dropped. This option takes effect after the `shutdown_delay` duration has passed if that is enabled.