24.1 23.3 23.2

rpk topic produce

Produce records to a topic.

Producing records reads from STDIN, parses input according to --format, and produce records to Redpanda. The input formatter understands a wide variety of formats.

Parsing input operates on either sizes or on delimiters, both of which can be specified in the same formatting options. If using sizes to specify something, the size must come before what it is specifying. Delimiters match on an exact text basis. This command will quit with an error if any input fails to match your specified format.

Format input

Slashes can be used for common escapes:

    \t \n \r \\ \xNN

matches tabs, newlines, carriage returns, slashes, and hex encoded characters.

Percent encoding reads into specific values of a record:

    %t    topic
    %T    topic length
    %k    key
    %K    key length
    %v    value
    %V    value length
    %h    begin the header specification
    %H    number of headers
    %p    partition (if using the --partition flag)

Three escapes exist to parse characters that are used to modify the previous escapes:

    %%    percent sign
    %{    left brace
    %}    right brace

Modifiers

Text and numbers can be read in multiple formats, and the default format can be changed within brace modifiers. %v reads a value, while %v{hex} reads a value and then hex decodes it before producing. %T reads the length of a topic from the input, while %T{3} reads exactly three bytes for a topic from the input.

All modifiers go within braces following a percent-escape.

Numbers

Reading number values can have the following modifiers:

     ascii       parse numeric digits until a non-numeric (default)

     hex64       sixteen hex characters
     hex32       eight hex characters
     hex16       four hex characters
     hex8        two hex characters
     hex4        one hex character

     big64       eight byte big endian number
     big32       four byte big endian number
     big16       two byte big endian number
     big8        alias for byte

     little64    eight byte little endian number
     little32    four byte little endian number
     little16    two byte little endian number
     little8     alias for byte

     byte        one byte number
     <digits>    directly specify the length as this many digits
     bool        read "true" as 1, "false" as 0

When reading number sizes, the size corresponds to the size of the encoded values, not the decoded values. "%T{6}%t{hex}" will read six hex bytes and decode into three.

Text

Reading text values can have the following modifiers:

    hex       read text then hex decode it
    base64    read text then std-encoding base64 decode it
    re        read text matching a regular expression

Headers

Headers are parsed with an internal key / value specifier format. For example, the following will read three headers that begin and end with a space and are separated by an equal:

    %H{3}%h{ %k=%v }

Examples

In the below examples, we can parse many records at once. The produce command reads input and tokenizes based on your specified format. Every time the format is completely matched, a record is produced and parsing begins anew.

  • A key and value, separated by a space and ending in newline: -f '%k %v\n'

  • A four byte topic, four byte key, and four byte value: -f '%T{4}%K{4}%V{4}%t%k%v'

  • A value to a specific partition, if using a non-negative --partition flag: -f '%p %v\n'

  • A big-endian uint16 key size, the text " foo ", and then that key: -f '%K{big16} foo %k'

  • A value that can be two or three characters followed by a newline: -f '%v{re#...?#}\n'

Miscellaneous

Producing requires a topic to produce to. The topic can be specified either directly on as an argument, or in the input text through %t. A parsed topic takes precedence over the default passed in topic. If no topic is specified directly and no topic is parsed, this command will quit with an error.

The input format can parse partitions to produce directly to with %p. Doing so requires specifying a non-negative --partition flag. Any parsed partition takes precedence over the --partition flag; specifying the flag is the main requirement for being able to directly control which partition to produce to.

You can also specify an output format to write when a record is produced successfully. The output format follows the same formatting rules as the topic consume command. See that command’s help text for a detailed description.

Usage

rpk topic produce [TOPIC] [flags]

Flags

Value Type Description

--acks

int

Number of acks required for producing (-1=all, 0=none, 1=leader) (default -1).

--allow-auto-topic-creation

-

Auto-create non-existent topics; requires auto_create_topics_enabled on the broker.

-z, --compression

string

Compression to use for producing batches (none, gzip, snappy, lz4, zstd) (default "snappy").

--delivery-timeout

duration

Per-record delivery timeout, if non-zero, min 1s.

-f, --format

string

Input record format (default "%v\n").

-H, --header

stringArray

Headers in format key:value to add to each record (repeatable).

-h, --help

-

Help for produce.

-k, --key

string

A fixed key to use for each record (parsed input keys take precedence).

--max-message-bytes

int32

If non-negative, maximum size of a record batch before compression (default -1).

-o, --output-format

string

what to write to stdout when a record is successfully produced (default "Produced to partition %p at offset %o with timestamp %d.\n").

-p, --partition

int32

Partition to directly produce to, if non-negative (also allows %p parsing to set partitions) (default -1).

-Z, --tombstone

-

Produce empty values as tombstones.

--config

string

Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD, and /etc/redpanda/redpanda.yaml.

-X, --config-opt

stringArray

Override rpk configuration settings; '-X help' for detail or '-X list' for terser detail.

--profile

string

rpk profile to use.

-v, --verbose

-

Enable verbose logging.