Secrets

This topic outlines how to add secrets to a Redpanda Connect configuration without exposing them.

Store secrets in environment variables

A common way to securely pass secrets to a service is to use environment variables. Redpanda Connect allows you to inject the values of environment variables into a configuration with the interpolation syntax ${SECRET}. For example:

thing:
  super_secret: "${SECRET}"
Use quotes

It is valid to have super_secret: ${SECRET} above (without the quotes), but if SECRET is unset then the configuration becomes structurally different. Therefore, it’s always best to wrap environment variable interpolations within quotes so that when the variable is unset you still have a valid configuration (with an empty string).

For more information about this syntax, see the interpolation field page.

Look up secrets on a remote system at runtime

Starting with version 4.39.0, you can use the rpk connect CLI flag --secrets to look up secrets values on a remote system at runtime (for example, in your secrets management solution). This means that Redpanda Connect resolves interpolations in your configuration at runtime without setting environment variables.

For example, you could run the following command to retrieve the value for "${SECRET}", when the secret is stored on a Redis server.

rpk connect run ./config.yaml --secrets redis://secret:redis@example.com

The command tries to load the secret value from the specified Uniform Resource Name (URN) using the format scheme://secret:server_address.

You can specify multiple URNs separated by commas, which are tried in turn until a secrets value is successfully returned.

Supported remote systems

You can retrieve secrets from all of the following remote systems.

Remote system URN format

AWS Secret Manager

aws://region/prefix
For example: aws://eu-west-1/redpanda/

Azure Key Vault

az://vault-uri/prefix. The vault-uri value should not contain a https:// prefix.

GCP Secret Manager

gcp://projectID/prefix
For example: gcp://project-id/redpanda-

Redis

redis://secret:redis@example.com

Set secrets using shell commands

You can set specific fields within a configuration using the CLI flag --set, where the syntax is a <path>=<value> pair:

For example, to write data to a AWS DynamoDB table using the `aws_dynamodb`output connector, your configuration might look like this:

output:
  label: ""
  aws_dynamodb:
    table: connect-db-test
    string_columns:
      id: ${!json("id")}
      full_content: ${!content()}

    batching:
      count: 5
      period: 2m

    credentials:
      id: ACCESS_KEY_ID
      secret: SECRET_ACCESS_KEY

You could set your credentials to values stored within something like Hashicorp Vault by running the configuration with the --set flag and backticks to execute shell commands for the values:

rpk connect run ./config.yaml \
  --set "output.aws_dynamodb.credentials.id=`vault kv get -mount=secret access_key_id`" \
  --set "output.aws_dynamodb.credentials.secret=`vault kv get -mount=secret secret_access_key`"

Using this method lets you inject the secret into the configuration without leaking it into an environment variable.

Avoid leaked secrets

There are a few ways in which configurations parsed by Redpanda Connect can be exported back out of the service. In all of these cases Redpanda Connect will attempt to scrub any field values within the config that are known secrets (any field marked as a secret in the docs).

However, if you’re embedding secrets within a configuration outside of the value of secret fields, maybe as part of a Bloblang mapping, then care should be made to avoid exposing the resulting configuration. This specifically means you should not enable debug HTTP endpoints when the port is exposed, and don’t use the rpk connect echo subcommand on configurations containing secrets unless you’re printing to a secure pipe.