cache

Type:

Available in: Cloud, Self-Managed

Stores each message in a cache.

  • Common

  • Advanced

# Common config fields, showing default values
output:
  label: ""
  cache:
    target: "" # No default (required)
    key: ${!count("items")}-${!timestamp_unix_nano()}
    max_in_flight: 64
# All config fields, showing default values
output:
  label: ""
  cache:
    target: "" # No default (required)
    key: ${!count("items")}-${!timestamp_unix_nano()}
    ttl: 60s # No default (optional)
    max_in_flight: 64

Caches are configured as resources, where there’s a wide variety to choose from.

The target field must reference a configured cache resource label like follows:

output:
  cache:
    target: foo
    key: ${!json("document.id")}

cache_resources:
  - label: foo
    memcached:
      addresses:
        - localhost:11211
      default_ttl: 60s

In order to create a unique key value per item you should use function interpolations described in Bloblang queries.

Performance

This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field max_in_flight.

Fields

target

The target cache to store messages in.

Type: string

key

The key to store messages by, function interpolation should be used in order to derive a unique key for each message. This field supports interpolation functions.

Type: string

Default: "${!count(\"items\")}-${!timestamp_unix_nano()}"

# Examples

key: ${!count("items")}-${!timestamp_unix_nano()}

key: ${!json("doc.id")}

key: ${!meta("kafka_key")}

ttl

The TTL of each individual item as a duration string. After this period an item will be eligible for removal during the next compaction. Not all caches support per-key TTLs, and those that do not will fall back to their generally configured TTL setting. This field supports interpolation functions.

Type: string

# Examples

ttl: 60s

ttl: 5m

ttl: 36h

max_in_flight

The maximum number of messages to have in flight at a given time. Increase this to improve throughput.

Type: int

Default: 64