grok

Available in: Self-Managed

Parses messages into a structured format by attempting to apply a list of Grok expressions, the first expression to result in at least one value replaces the original message with a JSON object containing the values.

  • Common

  • Advanced

# Common config fields, showing default values
label: ""
grok:
  expressions: [] # No default (required)
  pattern_definitions: {}
  pattern_paths: []
# All config fields, showing default values
label: ""
grok:
  expressions: [] # No default (required)
  pattern_definitions: {}
  pattern_paths: []
  named_captures_only: true
  use_default_patterns: true
  remove_empty_values: true

Type hints within patterns are respected, therefore with the pattern %\{WORD:first},%{INT:second:int} and a payload of foo,1 the resulting payload would be \{"first":"foo","second":1}.

Performance

This processor currently uses the Go RE2 regular expression engine, which is guaranteed to run in time linear to the size of the input. However, this property often makes it less performant than PCRE based implementations of grok. For more information, see https://swtch.com/~rsc/regexp/regexp1.html.

Examples

  • VPC Flow Logs

Grok can be used to parse unstructured logs such as VPC flow logs that look like this:

2 123456789010 eni-1235b8ca123456789 172.31.16.139 172.31.16.21 20641 22 6 20 4249 1418530010 1418530070 ACCEPT OK

Into structured objects that look like this:

{"accountid":"123456789010","action":"ACCEPT","bytes":4249,"dstaddr":"172.31.16.21","dstport":22,"end":1418530070,"interfaceid":"eni-1235b8ca123456789","logstatus":"OK","packets":20,"protocol":6,"srcaddr":"172.31.16.139","srcport":20641,"start":1418530010,"version":2}

With the following config:

pipeline:
  processors:
    - grok:
        expressions:
          - '%{VPCFLOWLOG}'
        pattern_definitions:
          VPCFLOWLOG: '%{NUMBER:version:int} %{NUMBER:accountid} %{NOTSPACE:interfaceid} %{NOTSPACE:srcaddr} %{NOTSPACE:dstaddr} %{NOTSPACE:srcport:int} %{NOTSPACE:dstport:int} %{NOTSPACE:protocol:int} %{NOTSPACE:packets:int} %{NOTSPACE:bytes:int} %{NUMBER:start:int} %{NUMBER:end:int} %{NOTSPACE:action} %{NOTSPACE:logstatus}'

Fields

expressions

One or more Grok expressions to attempt against incoming messages. The first expression to match at least one value will be used to form a result.

Type: array

pattern_definitions

A map of pattern definitions that can be referenced within patterns.

Type: object

Default: {}

pattern_paths

A list of paths to load Grok patterns from. This field supports wildcards, including super globs (double star).

Type: array

Default: []

named_captures_only

Whether to only capture values from named patterns.

Type: bool

Default: true

use_default_patterns

Whether to use a default set of patterns.

Type: bool

Default: true

remove_empty_values

Whether to remove values that are empty from the resulting structure.

Type: bool

Default: true

Default patterns

For summary of the default patterns on offer, see https://github.com/Jeffail/grok/blob/master/patterns.go#L5.