Secrets

This topic outlines how to add secrets to a Redpanda Connect configuration without exposing them.

Using environment variables

One of the most prolific approaches to providing secrets to a service is via environment variables. Redpanda Connect allows you to inject the values of environment variables into a configuration with the interpolation syntax ${FOO}, within a configuration it looks like this:

thing:
  super_secret: "${SECRET}"
Use quotes

Note that it would be valid to have super_secret: ${SECRET} above (without the quotes), but if SECRET is unset then the configuration becomes structurally different. Therefore, it’s always best to wrap environment variable interpolations with quotes so that when the variable is unset you still have a valid configuration (with an empty string).

More information about this syntax can be found on the interpolation field page.

Using CLI flags

As an alternative to environment variables, you can set specific fields within a configuration using the CLI flag --set, where the syntax is a <path>=<value> pair:

For example, to write data to a AWS DynamoDB table using the `aws_dynamodb`output connector, your configuration might look like this:

output:
  label: ""
  aws_dynamodb:
    table: connect-db-test
    string_columns:
      id: ${!json("id")}
      full_content: ${!content()}

    batching:
      count: 5
      period: 2m

    credentials:
      id: ACCESS_KEY_ID
      secret: SECRET_ACCESS_KEY

You could set your credentials to values stored within something like Hashicorp Vault by running the configuration with the --set flag and backticks to execute shell commands for the values:

rpk connect run ./config.yaml \
  --set "output.aws_dynamodb.credentials.id=`vault kv get -mount=secret access_key_id`" \
  --set "output.aws_dynamodb.credentials.secret=`vault kv get -mount=secret secret_access_key`"

Using this method lets you inject the secret into the configuration without leaking it into an environment variable.

Avoiding leaked secrets

There are a few ways in which configurations parsed by Redpanda Connect can be exported back out of the service. In all of these cases Redpanda Connect will attempt to scrub any field values within the config that are known secrets (any field marked as a secret in the docs).

However, if you’re embedding secrets within a configuration outside of the value of secret fields, maybe as part of a Bloblang mapping, then care should be made to avoid exposing the resulting configuration. This specifically means you should not enable debug HTTP endpoints when the port is exposed, and don’t use the rpk connect echo subcommand on configurations containing secrets unless you’re printing to a secure pipe.