rpk topic produce
Produce records to a topic.
Producing records reads from STDIN
, parses input according to --format
, and
produce records to Redpanda. The input formatter understands a wide variety of
formats.
Parsing input operates on either sizes or on delimiters, both of which can be specified in the same formatting options. If using sizes to specify something, the size must come before what it is specifying. Delimiters match on an exact text basis. This command will quit with an error if any input fails to match your specified format.
Formatting
Formatting is based on percent escapes and modifiers. Slashes can be used for common escapes:
Escape | Description |
---|---|
|
Tabs |
|
Newlines |
|
Carriage returns |
|
Slashes |
|
Hex encoded characters |
The percent encodings are represented like this:
Percent encoding | Description |
---|---|
|
Topic |
|
Topic length |
|
Key |
|
Key length |
|
Value |
|
Value length |
|
Begin the header specification |
|
Number of headers |
|
Partition |
|
Offset |
|
Leader epoch |
|
Timestamp (formatting described below) |
|
Producer ID |
|
Producer epoch |
|
Partition log start offset |
|
Partition last stable offset |
|
Partition high watermark |
|
Record attributes (formatting described below) |
|
Percent sign |
|
Left brace |
|
Right brace |
|
Number of records formatted |
Modifiers
Text and numbers can be formatted in many different ways, and the default
format can be changed within brace modifiers. %v
prints a value, while %v{hex}
prints the value hex encoded. %T
prints the length of a topic in ASCII, while
%T{big8}
prints the length of the topic as an eight byte big endian.
All modifiers go within braces following a percent-escape.
Numbers
Formatting number values can have the following modifiers:
Format | Description |
---|---|
|
Print the number as ASCII (default) |
|
Sixteen hex characters |
|
Eight hex characters |
|
Four hex characters |
|
Two hex characters |
|
One hex character |
|
Eight byte big endian number |
|
Four byte big endian number |
|
Two byte big endian number |
|
Alias for byte |
|
Eight byte little endian number |
|
Four byte little endian number |
|
Two byte little endian number |
|
Alias for byte |
|
One byte number |
|
|
All numbers are truncated as necessary per the modifier. Printing %V{byte}
for
a length 256 value prints a single null, whereas printing %V{big8}
prints the bytes 1 and 0.
When writing number sizes, the size corresponds to the size of the raw values,
not the size of encoded values. %T% t{hex}
for the topic foo
prints
3 666f6f
, not 6 666f6f
.
Timestamps
By default, the timestamp field is printed as a millisecond number value. In
addition to the number modifiers above, timestamps can be printed with either
Go
formatting:
%d{go[2006-01-02T15:04:05Z07:00]}
Or strftime
formatting:
%d{strftime[%F]}
An arbitrary amount of brackets (or braces, or # symbols) can wrap your date formatting:
%d{strftime=== [%F] ===}
This prints [YYYY-MM-DD]
, while the surrounding three # on each
side are used to wrap the formatting.
For more information on Go time formatting, see the Go documentation.
For more information on strftime
formatting, run man strftime
.
Schema registry
Records can be encoded using a specified schema from our schema registry. Use the --schema-id
or --schema-key-id
flags to define the schema ID, rpk
will retrieve the schemas and encode the record accordingly.
Additionally, utilizing topic
in the mentioned flags allows for the use of the Topic Name Strategy. This strategy identifies a schema subject name based on the topic itself. For example:
Produce to foo
, encode using the latest schema in the subject foo-value
:
rpk topic produce foo --schema-id=topic
For protobuf schemas, you can specify the fully qualified name of the message you want the record to be encoded with. Use the schema-type
flag or schema-key-type
. If the schema contains only one message, specifying the message name is unnecessary. For example:
Produce to foo
, using schema ID 1, message FQN Person.Name:
rpk topic produce foo --schema-id 1 --schema-type Person.Name
Tombstones
By default, records produced without a value will have an empty-string value, ""
. The below example produces a record with the key not_a_tombstone_record
and the value ""
:
rpk topic produce foo -k not_a_tombstone_record
[Enter]
Tombstone records (records with a null
value) can be produced by using the -Z
flag and creating empty-string value records. Using the same example from above, but adding the -Z
flag will produce a record with the key tombstone_record
and the value null
:
rpk topic produce foo -k tombstone_record -Z
[Enter]
It is important to note that records produced with values of string "null"
are not considered tombstones by Redpanda.
Examples
In the below examples, we can parse many records at once. The produce command reads input and tokenizes based on your specified format. Every time the format is completely matched, a record is produced and parsing begins anew.
-
A key and value, separated by a space and ending in newline:
-f '%k %v\n'
-
A four byte topic, four byte key, and four byte value:
-f '%T{4}%K{4}%V{4}%t%k%v'
-
A value to a specific partition, if using a non-negative --partition flag:
-f '%p %v\n'
-
A big-endian uint16 key size, the text " foo ", and then that key:
-f '%K{big16} foo %k'
-
A value that can be two or three characters followed by a newline:
-f '%v{re#...?#}\n'
-
A key and a json value, separated by a space:
-f '%k %v{json}'
Miscellaneous
Producing requires a topic to produce to. The topic can be specified either directly on as an argument, or in the input text through %t. A parsed topic takes precedence over the default passed in topic. If no topic is specified directly and no topic is parsed, this command will quit with an error.
The input format can parse partitions to produce directly to with %p. Doing so requires specifying a non-negative --partition flag. Any parsed partition takes precedence over the --partition flag; specifying the flag is the main requirement for being able to directly control which partition to produce to.
You can also specify an output format to write when a record is produced successfully. The output format follows the same formatting rules as the topic consume command. See that command’s help text for a detailed description.
Flags
Value | Type | Description |
---|---|---|
|
int |
Number of acks required for producing (-1=all, 0=none, 1=leader) (default -1). |
|
- |
Auto-create non-existent topics; requires auto_create_topics_enabled on the broker. |
|
string |
Compression to use for producing batches (none, gzip, snappy, lz4, zstd) (default "snappy"). |
|
duration |
Per-record delivery timeout, if non-zero, min 1s. |
|
string |
Input record format (default "%v\n"). |
|
stringArray |
Headers in format key:value to add to each record (repeatable). |
|
- |
Help for produce. |
|
string |
A fixed key to use for each record (parsed input keys take precedence). |
|
int32 |
If non-negative, maximum size of a record batch before compression (default -1). |
|
string |
what to write to stdout when a record is successfully produced (default "Produced to partition %p at offset %o with timestamp %d.\n"). |
|
int32 |
Partition to directly produce to, if non-negative (also allows %p parsing to set partitions) (default -1). |
|
string |
Schema ID to encode the record value with, use |
|
string |
Schema ID to encode the record key with, use |
|
string |
Name of the protobuf message type to be used to encode the record key using schema registry. |
|
string |
Name of the protobuf message type to be used to encode the record value using schema registry. |
|
- |
Produce empty values as tombstones. |
|
string |
Redpanda or |
|
stringArray |
Override |
|
string |
Profile to use. See |
|
- |
Enable verbose logging. |