rpk cluster self-test status

Returns the status of the current running tests or the cached results of the last completed run.

Use this command after invoking rpk cluster self-test start to determine the status of the jobs launched. Possible results are:

  • One or more jobs still running

    • Returns the IDs of Redpanda brokers (nodes) still running self-tests. Example:

Node 1 is still running net self test

  • No jobs running

    • Returns the cached results for all brokers of the last completed test.

Test results are grouped by broker ID. Each test returns the following:

  • Name: Description of the test.

  • Info: Details about the test run attached by Redpanda.

  • Type: Either disk, network, or cloud test.

  • Test Id: Unique identifier given to jobs of a run. All IDs in a test should match. If they don’t match, then newer and/or older test results have been included erroneously.

  • Timeouts: Number of timeouts incurred during the test.

  • Start time: Time that the test started, in UTC.

  • End time: Time that the test ended, in UTC.

  • Avg Duration: Duration of the test.

  • IOPS: Number of operations per second. For disk, it’s seastar::dma_read and seastar::dma_write. For network, it’s rpc.send().

  • Throughput: For disk, throughput rate is in bytes per second. For network, throughput rate is in bits per second. Note that GiB vs. Gib is the correct notation displayed by the UI.

  • Latency: 50th, 90th, etc. percentiles of operation latency, reported in microseconds (μs). Represented as P50, P90, P99, P999, and MAX respectively.

If Tiered Storage is not enabled, then cloud storage tests do not run, and a warning displays: "Cloud storage is not enabled." All results are shown as 0.

Usage

rpk cluster self-test status [flags]

Flags

Value Type Description

--format

string

Output format. Possible values: json, text. Default: text.

-h, --help

-

Help for status.

--config

string

Redpanda or rpk config file; default search paths are /var/lib/redpanda/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml.

-X, --config-opt

stringArray

Override rpk configuration settings. See rpk -X or execute rpk -X help for inline detail or rpk -X list for terser detail.

--profile

string

Profile to use. See rpk profile for more details.

-v, --verbose

-

Enable verbose logging.

Example

Example input:

rpk cluster self-test status
Example output 
$ rpk cluster self-test status
NODE ID: 0 | STATUS: IDLE
=========================
NAME          512KB sequential r/w
INFO          write run (iodepth: 4, dsync: true)
TYPE          disk
TEST ID       21c5a3de-c75b-480c-8a3d-0cbb63228cb1
TIMEOUTS      0
START TIME    Fri Jul 19 15:02:45 UTC 2024
END TIME      Fri Jul 19 15:03:15 UTC 2024
AVG DURATION  30002ms
IOPS          1182 req/sec
THROUGHPUT    591.4MiB/sec
LATENCY       P50     P90     P99     P999     MAX
              3199us  3839us  9727us  12799us  21503us

NAME          512KB sequential r/w
INFO          read run
TYPE          disk
TEST ID       21c5a3de-c75b-480c-8a3d-0cbb63228cb1
TIMEOUTS      0
START TIME    Fri Jul 19 15:03:15 UTC 2024
END TIME      Fri Jul 19 15:03:45 UTC 2024
AVG DURATION  30000ms
IOPS          6652 req/sec
THROUGHPUT    3.248GiB/sec
LATENCY       P50    P90    P99    P999   MAX
              607us  671us  831us  991us  2431us

NAME          4KB sequential r/w, low io depth
INFO          write run (iodepth: 1, dsync: true)
TYPE          disk
TEST ID       21c5a3de-c75b-480c-8a3d-0cbb63228cb1
TIMEOUTS      0
START TIME    Fri Jul 19 15:03:45 UTC 2024
END TIME      Fri Jul 19 15:04:15 UTC 2024
AVG DURATION  30000ms
IOPS          406 req/sec
THROUGHPUT    1.59MiB/sec
LATENCY       P50     P90     P99     P999    MAX
              2431us  2559us  2815us  5887us  9215us

NAME          4KB sequential r/w, low io depth
INFO          read run
TYPE          disk
TEST ID       21c5a3de-c75b-480c-8a3d-0cbb63228cb1
TIMEOUTS      0
START TIME    Fri Jul 19 15:04:15 UTC 2024
END TIME      Fri Jul 19 15:04:45 UTC 2024
AVG DURATION  30000ms
IOPS          430131 req/sec
THROUGHPUT    1.641GiB/sec
LATENCY       P50   P90   P99   P999  MAX
              1us   2us   12us  28us  511us

NAME          4KB sequential write, medium io depth
INFO          write run (iodepth: 8, dsync: true)
TYPE          disk
TEST ID       21c5a3de-c75b-480c-8a3d-0cbb63228cb1
TIMEOUTS      0
START TIME    Fri Jul 19 15:04:45 UTC 2024
END TIME      Fri Jul 19 15:05:15 UTC 2024
AVG DURATION  30013ms
IOPS          513 req/sec
THROUGHPUT    2.004MiB/sec
LATENCY       P50      P90      P99      P999     MAX
              15871us  16383us  21503us  32767us  40959us

NAME          4KB sequential write, high io depth
INFO          write run (iodepth: 64, dsync: true)
TYPE          disk
TEST ID       21c5a3de-c75b-480c-8a3d-0cbb63228cb1
TIMEOUTS      0
START TIME    Fri Jul 19 15:05:15 UTC 2024
END TIME      Fri Jul 19 15:05:45 UTC 2024
AVG DURATION  30114ms
IOPS          550 req/sec
THROUGHPUT    2.151MiB/sec
LATENCY       P50       P90       P99       P999      MAX
              118783us  118783us  147455us  180223us  180223us

NAME          4KB sequential write, very high io depth
INFO          write run (iodepth: 256, dsync: true)
TYPE          disk
TEST ID       21c5a3de-c75b-480c-8a3d-0cbb63228cb1
TIMEOUTS      0
START TIME    Fri Jul 19 15:05:45 UTC 2024
END TIME      Fri Jul 19 15:06:16 UTC 2024
AVG DURATION  30460ms
IOPS          558 req/sec
THROUGHPUT    2.183MiB/sec
LATENCY       P50       P90       P99       P999      MAX
              475135us  491519us  507903us  524287us  524287us

NAME          4KB sequential write, no dsync
INFO          write run (iodepth: 64, dsync: false)
TYPE          disk
TEST ID       21c5a3de-c75b-480c-8a3d-0cbb63228cb1
TIMEOUTS      0
START TIME    Fri Jul 19 15:06:16 UTC 2024
END TIME      Fri Jul 19 15:06:46 UTC 2024
AVG DURATION  30000ms
IOPS          424997 req/sec
THROUGHPUT    1.621GiB/sec
LATENCY       P50    P90    P99    P999   MAX
              135us  183us  303us  543us  9727us

NAME          16KB sequential r/w, high io depth
INFO          write run (iodepth: 64, dsync: false)
TYPE          disk
TEST ID       21c5a3de-c75b-480c-8a3d-0cbb63228cb1
TIMEOUTS      0
START TIME    Fri Jul 19 15:06:46 UTC 2024
END TIME      Fri Jul 19 15:07:16 UTC 2024
AVG DURATION  30000ms
IOPS          103047 req/sec
THROUGHPUT    1.572GiB/sec
LATENCY       P50    P90     P99     P999    MAX
              511us  1087us  1343us  1471us  15871us

NAME          16KB sequential r/w, high io depth
INFO          read run
TYPE          disk
TEST ID       21c5a3de-c75b-480c-8a3d-0cbb63228cb1
TIMEOUTS      0
START TIME    Fri Jul 19 15:07:16 UTC 2024
END TIME      Fri Jul 19 15:07:46 UTC 2024
AVG DURATION  30000ms
IOPS          193966 req/sec
THROUGHPUT    2.96GiB/sec
LATENCY       P50    P90    P99    P999    MAX
              319us  383us  735us  1023us  6399us

NAME        8K Network Throughput Test
INFO        Test performed against node: 1
TYPE        network
TEST ID     5e4052f3-b828-4c0d-8fd0-b34ff0b6c35d
TIMEOUTS    0
DURATION    5000ms
IOPS        61612 req/sec
THROUGHPUT  3.76Gib/sec
LATENCY     P50    P90    P99    P999   MAX
            159us  207us  303us  431us  1151us

NAME        8K Network Throughput Test
INFO        Test performed against node: 2
TYPE        network
TEST ID     5e4052f3-b828-4c0d-8fd0-b34ff0b6c35d
TIMEOUTS    0
DURATION    5000ms
IOPS        60306 req/sec
THROUGHPUT  3.68Gib/sec
LATENCY     P50    P90    P99    P999   MAX
            159us  215us  351us  495us  11263us

NAME          Cloud Storage Test
INFO          Put
TYPE          cloud
TEST ID       a349685a-ee49-4141-8390-c302975db3a5
TIMEOUTS      0
START TIME    Tue Jul 16 18:06:30 UTC 2024
END TIME      Tue Jul 16 18:06:30 UTC 2024
AVG DURATION  8ms

NAME          Cloud Storage Test
INFO          List
TYPE          cloud
TEST ID       a349685a-ee49-4141-8390-c302975db3a5
TIMEOUTS      0
START TIME    Tue Jul 16 18:06:30 UTC 2024
END TIME      Tue Jul 16 18:06:30 UTC 2024
AVG DURATION  1ms

NAME          Cloud Storage Test
INFO          Get
TYPE          cloud
TEST ID       a349685a-ee49-4141-8390-c302975db3a5
TIMEOUTS      0
START TIME    Tue Jul 16 18:06:30 UTC 2024
END TIME      Tue Jul 16 18:06:30 UTC 2024
AVG DURATION  1ms

NAME          Cloud Storage Test
INFO          Head
TYPE          cloud
TEST ID       a349685a-ee49-4141-8390-c302975db3a5
TIMEOUTS      0
START TIME    Tue Jul 16 18:06:30 UTC 2024
END TIME      Tue Jul 16 18:06:30 UTC 2024
AVG DURATION  0ms

NAME          Cloud Storage Test
INFO          Delete
TYPE          cloud
TEST ID       a349685a-ee49-4141-8390-c302975db3a5
TIMEOUTS      0
START TIME    Tue Jul 16 18:06:30 UTC 2024
END TIME      Tue Jul 16 18:06:30 UTC 2024
AVG DURATION  1ms

NAME          Cloud Storage Test
INFO          Plural Delete
TYPE          cloud
TEST ID       a349685a-ee49-4141-8390-c302975db3a5
TIMEOUTS      0
START TIME    Tue Jul 16 18:06:30 UTC 2024
END TIME      Tue Jul 16 18:06:30 UTC 2024
AVG DURATION  47ms
If self-test returns write results that are unexpectedly and significantly lower than read results, it may be because the Redpanda rpk client hardcodes the DSync option to true. When DSync is enabled, files are opened with the O_DSYNC flag set, and this represents the actual setting that Redpanda uses when it writes to disk.