rpk topic

You can use rpk topic to manage your topics, create new messages and consume them.

Formatting

Formatting is based on percent escapes and modifiers. Slashes can be used for common escapes:

Escape	Description
\t	Tabs.
\n	Newlines
\r	Carriage returns.
\\	Slashes.
\xNN	Hex encoded characters.

The percent encodings are represented like this:

Percent encoding	Description
%t	Topic.
%T	Topic length.
%k	Key.
%K	Key length.
%v	Value.
%V	Value length.
%h	Begin the header specification.
%H	Number of headers.
%p	Partition.
%o	Offset.
%e	Leader epoch.
%d	Timestamp (formatting described below).
%x	Producer id.
%y	Producer epoch.
%[	Partition log start offset.
%|	Partition last stable offset.
%]	Partition high watermark.
%%	Record attributes (formatting described below).
%a	Percent sign
%{	Left brace.
%}	Right brace.
%i	The number of records formatted.

Modifiers

Text and numbers can be formatted in many different ways, and the default format can be changed within brace modifiers. %v prints a value, while %v{hex} prints the value hex encoded. %T prints the length of a topic in ascii, while %T{big8} prints the length of the topic as an eight byte big endian.

All modifiers go within braces following a percent-escape.

Numbers

Formatting number values can have the following modifiers:

Format	Description
ascii	Print the number as ascii (default).
hex64	Sixteen hex characters.
hex32	Eight hex characters.
hex16	Four hex characters.
hex8	Two hex characters.
hex4	One hex character.
big64	Eight byte big endian number.
big32	Four byte big endian number.
big16	Two byte big endian number.
big8	Alias for byte.
little64	Eight byte little endian number
little32	Four byte little endian number.
little16	Two byte little endian number.
little8	Alias for byte.
byte	One byte number.
bool	true if the number is non-zero, false if the number is zero

All numbers are truncated as necessary per the modifier. Printing %V{byte} for a length 256 value will print a single null, whereas printing %V{big8} would print the bytes 1 and 0.

When writing number sizes, the size corresponds to the size of the raw values, not the size of encoded values. %T% t{hex} for the topic foo will print 3 666f6f, not 6 666f6f.

Timestamps

By default, the timestamp field is printed as a millisecond number value. In addition to the number modifiers above, timestamps can be printed with either Go formatting:

%d{go[2006-01-02T15:04:05Z07:00]}

Or strftime formatting:

%d{strftime[%F]}

An arbitrary amount of brackets (or braces, or # symbols) can wrap your date formatting:

%d{strftime### [%F] ###}

The above will print [YYYY-MM-DD], while the surrounding three # on each side are used to wrap the formatting.

Further details on Go time formatting can be found at Go's time documentation website.

Further details on strftime formatting can be read by running man strftime.

Attributes

Each record (or batch of records) has a set of possible attributes. Internally, these are packed into bit flags. Printing an attribute requires first selecting which attribute you want to print, and then optionally specifying how you want it to be printed:

  %a{compression}
  %a{compression;number}
  %a{compression;big64}
  %a{compression;hex8}

Compression is by default printed as text (none, gzip, ...). Compression can be printed as a number with ;number, where number is any number formatting option described above. No compression is 0, gzip is 1, etc.

  %a{timestamp-type}
  %a{timestamp-type;big64}

The record's timestamp type is printed as:

• -1 for very old records (before timestamps existed)
• 0 for client generated timestamps
• 1 for broker generated timestamps.

note

Number formatting can be controlled with ;number.

%a{transactional-bit}
%a{transactional-bit;bool}

Prints 1 if the record a part of a transaction or 0 if it is not.

  %a{control-bit}
  %a{control-bit;bool}

Prints 1 if the record is a commit marker or 0 if it is not.

Text

Text fields without modifiers default to writing the raw bytes. Alternatively, there are the following modifiers:

Modifier	Description
%t{hex}	Hex encoding
%k{base64}	Base64 encoding
%k{base64raw}	Base64 encoding raw
%v{unpack[<bBhH>iIqQc.$]}	The unpack modifier has a further internal specification, similar to timestamps above.

The hex modifier hex encodes the text, the base64 modifier base64 encodes the text with standard encoding, and the base64raw modifier encodes the text with raw standard encoding. The unpack modifier has a further internal specification, similar to timestamps above:

Character	Description
x	Pad character (does not parse input).
<	Switch what follows to little endian.
>	Switch what follows to big endian.
b	Signed byte.
B	Unsigned byte.
h	int16 ("half word").
H	uint16 ("half word").
i	int32.
I	uint32.
q	int64 ("quad word").
Q	uint64 ("quad word").
c	Any character.
.	Alias for c.
s	Consume the rest of the input as a string
$	Match the end of the line (append error string if anything remains).

Unpacking text can allow translating binary input into readable output. If a value is a big-endian uint32, %v will print the raw four bytes, while %v{unpack[>I]} will print the number in as ascii. If unpacking exhausts the input before something is unpacked fully, an error message is appended to the output.

Headers

Headers are formatted with percent encoding inside of the modifier:

%h{%k=%v{hex}}

will print all headers with a space before the key and after the value, an equals sign between the key and value, and with the value hex encoded. Header formatting actually just parses the internal format as a record format, so all of the above rules about %K, %V, text, and numbers apply.

Examples

A key and value, separated by a space and ending in newline:

-f '%k %v\n'

A key length as four big endian bytes, and the key as hex:

-f '%K{big32}%k{hex}'

A little endian uint32 and a string unpacked from a value:

-f '%v{unpack[is$]}'

Offsets

The --offset flag allows for specifying where to begin consuming, and optionally, where to stop consuming. The literal words start and end specify consuming from the start and the end.

Offset	Description
start	Consume from the beginning.
end	Consume from the end.
:end	Consume until the current end.
+oo	Consume oo after the current start offset.
-oo	Consume oo before the current end offset.
oo	Consume after an exact offset.
oo:	Alias for oo.
:oo	Consume until an exact offset.
o1:o2	Consume from exact offset o1 until exact offset o2.
@t	Consume starting from a given timestamp.
@t:	alias for @t.
@:t	Consume until a given timestamp.
@t1:t2	Consume from timestamp t1 until timestamp t2.

There are a few options for timestamps, with each option being evaluated until one succeeds:

Timestamp	Description
13 digits	Parsed as a unix millisecond.
9 digits	Parsed as a unix second.
YYYY-MM-DD	Parsed as a day, UTC.
YYYY-MM-DDTHH:MM:SSZ	Parsed as RFC3339, UTC; fractional seconds optional (.MMM).
-dur	Duration ago; from now (as t1) or from t1 (as t2).
dur	For t2 in @t1:t2, relative duration from t1.
end	For t2 in @t1:t2, the current end of the partition.

Durations are parsed simply:

3ms    three milliseconds
10s    ten seconds
9m     nine minutes
1h     one hour
1m3ms  one minute and three milliseconds

For example:

-o @2022-02-14:1h   consume 1h of time on Valentine's Day 2022
-o @-48h:-24h       consume from 2 days ago to 1 day ago
-o @-1m:end         consume from 1m ago until now
-o @:-1hr           consume from the start until an hour ago

Misc

Producing requires a topic to produce to. The topic can be specified either directly on as an argument, or in the input text through %t. A parsed topic takes precedence over the default passed in topic. If no topic is specified directly and no topic is parsed, this command will quit with an error.

The input format can parse partitions to produce directly to with %p. Doing so requires specifying a non-negative --partition flag. Any parsed partition takes precedence over the --partition flag; specifying the flag is the main requirement for being able to directly control which partition to produce to.

You can also specify an output format to write when a record is produced successfully. The output format follows the same formatting rules as the topic consume command. See that command's help text for a detailed description.

rpk topic

Create, delete, produce to and consume from Redpanda topics.

Usage

rpk topic [command]

Flags

Value	Type	Description
--brokers	strings	Comma-separated list of broker <ip>:<port> pairs (for example,--brokers '192.168.78.34:9092,192.168.78.35:9092,192.179.23.54:9092'). Alternatively, you may set the REDPANDA_BROKERSenvironment variable with the comma-separated list of broker addresses.
--config	string	Redpanda config file, if not set the file will be searched for in the default locations.
-h, --help	-	Help for topic.
--password	string	SASL password to be used for authentication.
--sasl-mechanism	string	The authentication mechanism to use. Supported values:SCRAM-SHA-256, SCRAM-SHA-512.
--tls-cert	string	The certificate to be used for TLS authentication with the broker.
--tls-enabled	-	Enable TLS for the Kafka API (not necessary if specifying custom certs).
--tls-key	string	The certificate key to be used for TLS authentication with the broker.
--tls-truststore	string	The truststore to be used for TLS communication with the broker.
--user	string	SASL user to be used for authentication.
-v, --verbose	-	Enable verbose logging (default false).