mongodb

Type:

Available in: Self-Managed

Inserts items into a MongoDB collection.

Introduced in version 3.43.0.

  • Common

  • Advanced

# Common configuration fields, showing default values
output:
  label: ""
  mongodb:
    url: mongodb://localhost:27017 # No default (required)
    database: "" # No default (required)
    username: "" # No default (optional)
    password: "" # No default (optional)
    collection: "" # No default (required)
    operation: update-one
    write_concern:
      w: "" # No default (optional)
      j: false
      w_timeout: "" # No default (optional)
    document_map: "" # No default (optional)
    filter_map: "" # No default (optional)
    hint_map: "" # No default (optional)
    upsert: false
    max_in_flight: 64
    batching:
      count: 0
      byte_size: 0
      period: "" # No default (optional)
      check: "" # No default (optional)
# All configuration fields, showing default values
output:
  label: ""
  mongodb:
    url: mongodb://localhost:27017 # No default (required)
    database: "" # No default (required)
    username: "" # No default (optional)
    password: "" # No default (optional)
    app_name: benthos
    collection: "" # No default (required)
    operation: update-one
    write_concern:
      w: "" # No default (optional)
      j: false
      w_timeout: "" # No default (optional)
    document_map: "" # No default (optional)
    filter_map: "" # No default (optional)
    hint_map: "" # No default (optional)
    upsert: false
    max_in_flight: 64
    batching:
      count: 0
      byte_size: 0
      period: "" # No default (optional)
      check: "" # No default (optional)
      processors: [] # No default (optional)

Performance

This output benefits from sending multiple messages in flight, in parallel, for improved performance. You can tune the maximum number of in flight messages (or message batches) using the max_in_flight field.

This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. For more information, see Message Batching.

Fields

url

The URL of the target MongoDB server.

Type: string

# Examples

url: mongodb://localhost:27017

database

The name of the target MongoDB database.

Type: string

username

The username required to connect to the database.

Type: string

Default: ""

password

The password required to connect to the database.

This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see Secrets.

Type: string

Default: ""

app_name

The client application name.

Type: string

Default: benthos

collection

The name of the target collection. This field supports interpolation functions.

Type: string

operation

The MongoDB database operation to perform.

Type: string

Default: update-one

Options: insert-one , delete-one , delete-many , replace-one , update-one

write_concern

The write concern settings for the MongoDB connection.

Type: object

write_concern.w

The w requests acknowledgement, which write operations propagate to the specified number of MongoDB instances.

Type: string

Default: ""

write_concern.j

The j requests acknowledgement from MongoDB, which is created when write operations are written to the journal.

Type: bool

Default: false

write_concern.w_timeout

The write concern timeout.

Type: string

Default: ""

document_map

A Bloblang map that represents a document to store in MongoDB, expressed as extended JSON in canonical form. The document_map parameter is required for the following database operations: insert-one, replace-one, and update-one.

Type: string

Default: ""

# Examples

document_map: |-
  root.a = this.foo
  root.b = this.bar

filter_map

A Bloblang map that represents a filter for a MongoDB command, expressed as extended JSON in canonical form. The filter_map parameter is required for all database operations except insert-one.

This output uses filter_map to find documents for the specified operation. For example, for a delete-one operation, the filter map should include the fields required to locate the document for deletion.

Type: string

Default: ""

# Examples

filter_map: |-
  root.a = this.foo
  root.b = this.bar

hint_map

A Bloblang map that represents a hint or index for a MongoDB command to use, expressed as extended JSON in canonical form. This map is optional, and is used with all operations except insert-one.

Define a hint_map to improve performance when finding documents in the MongoDB database.

Type: string

Default: ""

# Examples

hint_map: |-
  root.a = this.foo
  root.b = this.bar

upsert

The upsert parameter is optional, and only applies for update-one and replace-one operations. If the filter specified in filter_map matches an existing document, this operation updates or replaces the document, otherwise a new document is created.

Type: bool

Default: false

Requires version 3.60.0 or newer

max_in_flight

The maximum number of messages to have in flight at a given time. Increase this number to improve throughput.

Type: int

Default: 64

batching

Allows you to configure a batching policy.

Type: object

# Examples

batching:
  byte_size: 5000
  count: 0
  period: 1s

batching:
  count: 10
  period: 1s

batching:
  check: this.contains("END BATCH")
  count: 0
  period: 1m

batching.count

The number of messages after which the batch is flushed. Set to 0 to disable count-based batching.

Type: int

Default: 0

batching.byte_size

The number of bytes at which the batch is flushed. Set to 0 to disable size-based batching.

Type: int

Default: 0

batching.period

The period after which an incomplete batch is flushed regardless of its size.

Type: string

Default: ""

# Examples

period: 1s

period: 1m

period: 500ms

batching.check

A Bloblang query that should return a boolean value indicating whether a message should end a batch.

Type: string

Default: ""

# Examples

check: this.type == "end_of_transaction"

batching.processors

For aggregating and archiving message batches, you can add a list of processors to apply to a batch as it is flushed. All resulting messages are flushed as a single batch even when you configure processors to split the batch into smaller batches.

Type: array

# Examples

processors:
  - archive:
      format: concatenate

processors:
  - archive:
      format: lines

processors:
  - archive:
      format: json_array