Bloblang Methods
Methods provide most of the power in Bloblang as they allow you to augment values and can be added to any expression (including other methods):
root.doc.id = this.thing.id.string().catch(uuid_v4())
root.doc.reduced_nums = this.thing.nums.map_each(num -> if num < 10 {
deleted()
} else {
num - 10
})
root.has_good_taste = ["pikachu","mewtwo","magmar"].contains(this.user.fav_pokemon)
# In: {"thing":{"id":123,"nums":[5,12,18,7,25]},"user":{"fav_pokemon":"pikachu"}}
Methods support both named and nameless style arguments:
root.foo_one = this.(bar | baz).trim().replace_all(old: "dog", new: "cat")
root.foo_two = this.(bar | baz).trim().replace_all("dog", "cat")
# In: {"bar":" I love my dog "}
General
apply
Apply a declared mapping to a target value.
Examples
map thing {
root.inner = this.first
}
root.foo = this.doc.apply("thing")
# In: {"doc":{"first":"hello world"}}
# Out: {"foo":{"inner":"hello world"}}
map create_foo {
root.name = "a foo"
root.purpose = "to be a foo"
}
root = this
root.foo = null.apply("create_foo")
# In: {"id":"1234"}
# Out: {"foo":{"name":"a foo","purpose":"to be a foo"},"id":"1234"}
catch
If the result of a target query fails (due to incorrect types, failed parsing, etc) the argument is returned instead.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
A value to yield, or query to execute, if the target query fails. |
Examples
root.doc.id = this.thing.id.string().catch(uuid_v4())
The fallback argument can be a mapping, allowing you to capture the error string and yield structured data back:
root.url = this.url.parse_url().catch(err -> {"error":err,"input":this.url})
# In: {"url":"invalid %&# url"}
# Out: {"url":{"error":"field `this.url`: parse \"invalid %&\": invalid URL escape \"%&\"","input":"invalid %&# url"}}
When the input document is not structured attempting to reference structured fields with this will result in an error. Therefore, a convenient way to delete non-structured data is with a catch:
root = this.catch(deleted())
# In: {"doc":{"foo":"bar"}}
# Out: {"doc":{"foo":"bar"}}
# In: not structured data
# Out: <Message deleted>
from
Modifies a target query such that certain functions are executed from the perspective of another message in the batch. This allows you to mutate events based on the contents of other messages. Functions that support this behavior are content, json and meta.
from_all
Modifies a target query such that certain functions are executed from the perspective of each message in the batch, and returns the set of results as an array. Functions that support this behavior are content, json and meta.
map
Executes a query on the target value, allowing you to transform or extract data from the current context.
not
Returns the logical NOT (negation) of a boolean value. Converts true to false and false to true.
or
If the result of the target query fails or resolves to null, returns the argument instead. This is an explicit method alternative to the coalesce pipe operator |.
Encoding and encryption
compress
Compresses a string or byte array using the specified compression algorithm. Returns compressed data as bytes. Useful for reducing payload size before transmission or storage.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The compression algorithm: |
|
|
Compression level (default: -1 for default compression). Higher values increase compression ratio but use more CPU. Range and effect varies by algorithm. |
Examples
Compress and encode for safe transmission:
root.compressed = content().bytes().compress("gzip").encode("base64")
# In: {"message":"hello world I love space"}
# Out: {"compressed":"H4sIAAAJbogA/wAmANn/eyJtZXNzYWdlIjoiaGVsbG8gd29ybGQgSSBsb3ZlIHNwYWNlIn0DAHEvdwomAAAA"}
Compare compression ratios across algorithms:
root.original_size = content().length()
root.gzip_size = content().compress("gzip").length()
root.lz4_size = content().compress("lz4").length()
# In: The quick brown fox jumps over the lazy dog. The quick brown fox jumps over the lazy dog.
# Out: {"gzip_size":114,"lz4_size":85,"original_size":89}
decompress
Decompresses a byte array using the specified decompression algorithm. Returns decompressed data as bytes. Use with data that was previously compressed using the corresponding algorithm.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The decompression algorithm: |
Examples
Decompress base64-encoded compressed data:
root = this.compressed.decode("base64").decompress("gzip")
# In: {"compressed":"H4sIAN12MWkAA8tIzcnJVyjPL8pJUfBUyMkvS1UoLkhMTgUAQpDxbxgAAAA="}
# Out: hello world I love space
Convert decompressed bytes to string for JSON output:
root.message = this.compressed.decode("base64").decompress("gzip").string()
# In: {"compressed":"H4sIAN12MWkAA8tIzcnJVyjPL8pJUfBUyMkvS1UoLkhMTgUAQpDxbxgAAAA="}
# Out: {"message":"hello world I love space"}
decrypt_aes
Decrypts an AES-encrypted string or byte array.
encrypt_aes
Encrypts a string or byte array using AES encryption.
hash
Hashes a string or byte array using a specified algorithm.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The hashing algorithm to use. |
|
|
An optional key to use. |
|
|
An optional polynomial key to use when selecting the |
Examples
root.h1 = this.value.hash("sha1").encode("hex")
root.h2 = this.value.hash("hmac_sha1","static-key").encode("hex")
# In: {"value":"hello world"}
# Out: {"h1":"2aae6c35c94fcfb415dbe95f408b9ce91ee846ed","h2":"d87e5f068fa08fe90bb95bc7c8344cb809179d76"}
The crc32 algorithm supports options for the polynomial:
root.h1 = this.value.hash(algorithm: "crc32", polynomial: "Castagnoli").encode("hex")
root.h2 = this.value.hash(algorithm: "crc32", polynomial: "Koopman").encode("hex")
# In: {"value":"hello world"}
# Out: {"h1":"c99465aa","h2":"df373d3c"}
GeoIP
geoip_anonymous_ip
Looks up an IP address against a MaxMind database file and, if found, returns an object describing the anonymous IP associated with it.
geoip_asn
Looks up an IP address against a MaxMind database file and, if found, returns an object describing the ASN associated with it.
geoip_city
Looks up an IP address against a MaxMind database file and, if found, returns an object describing the city associated with it.
geoip_connection_type
Looks up an IP address against a MaxMind database file and, if found, returns an object describing the connection type associated with it.
geoip_country
Looks up an IP address against a MaxMind database file and, if found, returns an object describing the country associated with it.
geoip_domain
Looks up an IP address against a MaxMind database file and, if found, returns an object describing the domain associated with it.
geoip_enterprise
Looks up an IP address against a MaxMind database file and, if found, returns an object describing the enterprise associated with it.
geoip_isp
Looks up an IP address against a MaxMind database file and, if found, returns an object describing the ISP associated with it.
JSON web tokens
parse_jwt_es256
Parses a claims object from a JWT string encoded with ES256. This method does not validate JWT claims.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The ES256 secret that was used for signing the token. |
Examples
root.claims = this.signed.parse_jwt_es256("""-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEGtLqIBePHmIhQcf0JLgc+F/4W/oI
dp0Gta53G35VerNDgUUXmp78J2kfh4qLdh0XtmOMI587tCaqjvDAXfs//w==
-----END PUBLIC KEY-----""")
# In: {"signed":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.GIRajP9JJbpTlqSCdNEz4qpQkRvzX4Q51YnTwVyxLDM9tKjR_a8ggHWn9CWj7KG0x8J56OWtmUxn112SRTZVhQ"}
# Out: {"claims":{"iat":1516239022,"mood":"Disdainful","sub":"1234567890"}}
parse_jwt_es384
Parses a claims object from a JWT string encoded with ES384. This method does not validate JWT claims.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The ES384 secret that was used for signing the token. |
Examples
root.claims = this.signed.parse_jwt_es384("""-----BEGIN PUBLIC KEY-----
MHYwEAYHKoZIzj0CAQYFK4EEACIDYgAERoz74/B6SwmLhs8X7CWhnrWyRrB13AuU
8OYeqy0qHRu9JWNw8NIavqpTmu6XPT4xcFanYjq8FbeuM11eq06C52mNmS4LLwzA
2imlFEgn85bvJoC3bnkuq4mQjwt9VxdH
-----END PUBLIC KEY-----""")
# In: {"signed":"eyJhbGciOiJFUzM4NCIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.H2HBSlrvQBaov2tdreGonbBexxtQB-xzaPL4-tNQZ6TVh7VH8VBcSwcWHYa1lBAHmdsKOFcB2Wk0SB7QWeGT3ptSgr-_EhDMaZ8bA5spgdpq5DsKfaKHrd7DbbQlmxNq"}
# Out: {"claims":{"iat":1516239022,"mood":"Disdainful","sub":"1234567890"}}
parse_jwt_es512
Parses a claims object from a JWT string encoded with ES512. This method does not validate JWT claims.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The ES512 secret that was used for signing the token. |
Examples
root.claims = this.signed.parse_jwt_es512("""-----BEGIN PUBLIC KEY-----
MIGbMBAGByqGSM49AgEGBSuBBAAjA4GGAAQAkHLdts9P56fFkyhpYQ31M/Stwt3w
vpaxhlfudxnXgTO1IP4RQRgryRxZ19EUzhvWDcG3GQIckoNMY5PelsnCGnIBT2Xh
9NQkjWF5K6xS4upFsbGSAwQ+GIyyk5IPJ2LHgOyMSCVh5gRZXV3CZLzXujx/umC9
UeYyTt05zRRWuD+p5bY=
-----END PUBLIC KEY-----""")
# In: {"signed":"eyJhbGciOiJFUzUxMiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.ACrpLuU7TKpAnncDCpN9m85nkL55MJ45NFOBl6-nEXmNT1eIxWjiP4pwWVbFH9et_BgN14119jbL_KqEJInPYc9nAXC6dDLq0aBU-dalvNl4-O5YWpP43-Y-TBGAsWnbMTrchILJ4-AEiICe73Ck5yWPleKg9c3LtkEFWfGs7BoPRguZ"}
# Out: {"claims":{"iat":1516239022,"mood":"Disdainful","sub":"1234567890"}}
parse_jwt_hs256
Parses a claims object from a JWT string encoded with HS256. This method does not validate JWT claims.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The HS256 secret that was used for signing the token. |
Examples
root.claims = this.signed.parse_jwt_hs256("""dont-tell-anyone""")
# In: {"signed":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.YwXOM8v3gHVWcQRRRQc_zDlhmLnM62fwhFYGpiA0J1A"}
# Out: {"claims":{"iat":1516239022,"mood":"Disdainful","sub":"1234567890"}}
parse_jwt_hs384
Parses a claims object from a JWT string encoded with HS384. This method does not validate JWT claims.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The HS384 secret that was used for signing the token. |
Examples
root.claims = this.signed.parse_jwt_hs384("""dont-tell-anyone""")
# In: {"signed":"eyJhbGciOiJIUzM4NCIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.2Y8rf_ijwN4t8hOGGViON_GrirLkCQVbCOuax6EoZ3nluX0tCGezcJxbctlIfsQ2"}
# Out: {"claims":{"iat":1516239022,"mood":"Disdainful","sub":"1234567890"}}
parse_jwt_hs512
Parses a claims object from a JWT string encoded with HS512. This method does not validate JWT claims.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The HS512 secret that was used for signing the token. |
Examples
root.claims = this.signed.parse_jwt_hs512("""dont-tell-anyone""")
# In: {"signed":"eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.utRb0urG6LGGyranZJVo5Dk0Fns1QNcSUYPN0TObQ-YzsGGB8jrxHwM5NAJccjJZzKectEUqmmKCaETZvuX4Fg"}
# Out: {"claims":{"iat":1516239022,"mood":"Disdainful","sub":"1234567890"}}
parse_jwt_rs256
Parses a claims object from a JWT string encoded with RS256. This method does not validate JWT claims.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The RS256 secret that was used for signing the token. |
Examples
root.claims = this.signed.parse_jwt_rs256("""-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAs/ibN8r68pLMR6gRzg4S
8v8l6Q7yi8qURjkEbcNeM1rkokC7xh0I4JVTwxYSVv/JIW8qJdyspl5NIfuAVi32
WfKvSAs+NIs+DMsNPYw3yuQals4AX8hith1YDvYpr8SD44jxhz/DR9lYKZFGhXGB
+7NqQ7vpTWp3BceLYocazWJgusZt7CgecIq57ycM5hjM93BvlrUJ8nQ1a46wfL/8
Cy4P0et70hzZrsjjN41KFhKY0iUwlyU41yEiDHvHDDsTMBxAZosWjSREGfJL6Mfp
XOInTHs/Gg6DZMkbxjQu6L06EdJ+Q/NwglJdAXM7Zo9rNELqRig6DdvG5JesdMsO
+QIDAQAB
-----END PUBLIC KEY-----""")
# In: {"signed":"eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.b0lH3jEupZZ4zoaly4Y_GCvu94HH6UKdKY96zfGNsIkPZpQLHIkZ7jMWlLlNOAd8qXlsBGP_i8H2qCKI4zlWJBGyPZgxXDzNRPVrTDfFpn4t4nBcA1WK2-ntXP3ehQxsaHcQU8Z_nsogId7Pme5iJRnoHWEnWtbwz5DLSXL3ZZNnRdrHM9MdI7QSDz9mojKDCaMpGN9sG7Xl-tGdBp1XzXuUOzG8S03mtZ1IgVR1uiBL2N6oohHIAunk8DIAmNWI-zgycTgzUGU7mvPkKH43qO8Ua1-13tCUBKKa8VxcotZ67Mxm1QAvBGoDnTKwWMwghLzs6d6WViXQg6eWlJcpBA"}
# Out: {"claims":{"iat":1516239022,"mood":"Disdainful","sub":"1234567890"}}
parse_jwt_rs384
Parses a claims object from a JWT string encoded with RS384. This method does not validate JWT claims.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The RS384 secret that was used for signing the token. |
Examples
root.claims = this.signed.parse_jwt_rs384("""-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAs/ibN8r68pLMR6gRzg4S
8v8l6Q7yi8qURjkEbcNeM1rkokC7xh0I4JVTwxYSVv/JIW8qJdyspl5NIfuAVi32
WfKvSAs+NIs+DMsNPYw3yuQals4AX8hith1YDvYpr8SD44jxhz/DR9lYKZFGhXGB
+7NqQ7vpTWp3BceLYocazWJgusZt7CgecIq57ycM5hjM93BvlrUJ8nQ1a46wfL/8
Cy4P0et70hzZrsjjN41KFhKY0iUwlyU41yEiDHvHDDsTMBxAZosWjSREGfJL6Mfp
XOInTHs/Gg6DZMkbxjQu6L06EdJ+Q/NwglJdAXM7Zo9rNELqRig6DdvG5JesdMsO
+QIDAQAB
-----END PUBLIC KEY-----""")
# In: {"signed":"eyJhbGciOiJSUzM4NCIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.orcXYBcjVE5DU7mvq4KKWFfNdXR4nEY_xupzWoETRpYmQZIozlZnM_nHxEk2dySvpXlAzVm7kgOPK2RFtGlOVaNRIa3x-pMMr-bhZTno4L8Hl4sYxOks3bWtjK7wql4uqUbqThSJB12psAXw2-S-I_FMngOPGIn4jDT9b802ottJSvTpXcy0-eKTjrV2PSkRRu-EYJh0CJZW55MNhqlt6kCGhAXfbhNazN3ASX-dmpd_JixyBKphrngr_zRA-FCn_Xf3QQDA-5INopb4Yp5QiJ7UxVqQEKI80X_JvJqz9WE1qiAw8pq5-xTen1t7zTP-HT1NbbD3kltcNa3G8acmNg"}
# Out: {"claims":{"iat":1516239022,"mood":"Disdainful","sub":"1234567890"}}
parse_jwt_rs512
Parses a claims object from a JWT string encoded with RS512. This method does not validate JWT claims.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The RS512 secret that was used for signing the token. |
Examples
root.claims = this.signed.parse_jwt_rs512("""-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAs/ibN8r68pLMR6gRzg4S
8v8l6Q7yi8qURjkEbcNeM1rkokC7xh0I4JVTwxYSVv/JIW8qJdyspl5NIfuAVi32
WfKvSAs+NIs+DMsNPYw3yuQals4AX8hith1YDvYpr8SD44jxhz/DR9lYKZFGhXGB
+7NqQ7vpTWp3BceLYocazWJgusZt7CgecIq57ycM5hjM93BvlrUJ8nQ1a46wfL/8
Cy4P0et70hzZrsjjN41KFhKY0iUwlyU41yEiDHvHDDsTMBxAZosWjSREGfJL6Mfp
XOInTHs/Gg6DZMkbxjQu6L06EdJ+Q/NwglJdAXM7Zo9rNELqRig6DdvG5JesdMsO
+QIDAQAB
-----END PUBLIC KEY-----""")
# In: {"signed":"eyJhbGciOiJSUzUxMiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.rsMp_X5HMrUqKnZJIxo27aAoscovRA6SSQYR9rq7pifIj0YHXxMyNyOBDGnvVALHKTi25VUGHpfNUW0VVMmae0A4t_ObNU6hVZHguWvetKZZq4FZpW1lgWHCMqgPGwT5_uOqwYCH6r8tJuZT3pqXeL0CY4putb1AN2w6CVp620nh3l8d3XWb4jaifycd_4CEVCqHuWDmohfug4VhmoVKlIXZkYoAQowgHlozATDssBSWdYtv107Wd2AzEoiXPu6e3pflsuXULlyqQnS4ELEKPYThFLafh1NqvZDPddqozcPZ-iODBW-xf3A4DYDdivnMYLrh73AZOGHexxu8ay6nDA"}
# Out: {"claims":{"iat":1516239022,"mood":"Disdainful","sub":"1234567890"}}
sign_jwt_es256
Hash and sign an object representing JSON Web Token (JWT) claims using ES256.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The secret to use for signing the token. |
|
|
Optional object of JWT header fields to include in the token. Keys "alg", "typ", "jku", "jwk", "x5u", "x5c", "x5t","x5t#S256" and "crit" will be ignored if provided. |
Examples
root.signed = this.claims.sign_jwt_es256("""-----BEGIN EC PRIVATE KEY-----
... signature data ...
-----END EC PRIVATE KEY-----""")
# In: {"claims":{"sub":"user123"}}
# Out: {"signed":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.-8LrOdkEiv_44ADWW08lpbq41ZmHCel58NMORPq1q4Dyw0zFhqDVLrRoSvCvuyyvgXAFb9IHfR-9MlJ_2ShA9A"}
root.signed = this.claims.sign_jwt_es256(signing_secret: """-----BEGIN EC PRIVATE KEY-----
... signature data ...
-----END EC PRIVATE KEY-----""", headers: {"kid": "my-key", "x": "y"})
# In: {"claims":{"sub":"user123"}}
# Out: {"signed":"<signed JWT token>"}
sign_jwt_es384
Hash and sign an object representing JSON Web Token (JWT) claims using ES384.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The secret to use for signing the token. |
|
|
Optional object of JWT header fields to include in the token. Keys "alg", "typ", "jku", "jwk", "x5u", "x5c", "x5t","x5t#S256" and "crit" will be ignored if provided. |
Examples
root.signed = this.claims.sign_jwt_es384("""-----BEGIN EC PRIVATE KEY-----
... signature data ...
-----END EC PRIVATE KEY-----""")
# In: {"claims":{"sub":"user123"}}
# Out: {"signed":"eyJhbGciOiJFUzM4NCIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJ1c2VyMTIzIn0.8FmTKH08dl7dyxrNu0rmvhegiIBCy-O9cddGco2e9lpZtgv5mS5qHgPkgBC5eRw1d7SRJsHwHZeehzdqT5Ba7aZJIhz9ds0sn37YQ60L7jT0j2gxCzccrt4kECHnUnLw"}
root.signed = this.claims.sign_jwt_es384(signing_secret: """-----BEGIN EC PRIVATE KEY-----
... signature data ...
-----END EC PRIVATE KEY-----""", headers: {"kid": "my-key", "x": "y"})
# In: {"claims":{"sub":"user123"}}
# Out: {"signed":"<signed JWT token>"}
sign_jwt_es512
Hash and sign an object representing JSON Web Token (JWT) claims using ES512.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The secret to use for signing the token. |
|
|
Optional object of JWT header fields to include in the token. Keys "alg", "typ", "jku", "jwk", "x5u", "x5c", "x5t","x5t#S256" and "crit" will be ignored if provided. |
Examples
root.signed = this.claims.sign_jwt_es512("""-----BEGIN EC PRIVATE KEY-----
... signature data ...
-----END EC PRIVATE KEY-----""")
# In: {"claims":{"sub":"user123"}}
# Out: {"signed":"eyJhbGciOiJFUzUxMiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJ1c2VyMTIzIn0.AQbEWymoRZxDJEJtKSFFG2k2VbDCTYSuBwAZyMqexCspr3If8aERTVGif8HXG3S7TzMBCCzxkcKr3eIU441l3DlpAMNfQbkcOlBqMvNBn-CX481WyKf3K5rFHQ-6wRonz05aIsWAxCDvAozI_9J0OWllxdQ2MBAuTPbPJ38OqXsYkCQs"}
root.signed = this.claims.sign_jwt_es512(signing_secret: """-----BEGIN EC PRIVATE KEY-----
... signature data ...
-----END EC PRIVATE KEY-----""", headers: {"kid": "my-key", "x": "y"})
# In: {"claims":{"sub":"user123"}}
# Out: {"signed":"<signed JWT token>"}
sign_jwt_hs256
Hash and sign an object representing JSON Web Token (JWT) claims using HS256.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The secret to use for signing the token. |
|
|
Optional object of JWT header fields to include in the token. Keys "alg", "typ", "jku", "jwk", "x5u", "x5c", "x5t","x5t#S256" and "crit" will be ignored if provided. |
Examples
root.signed = this.claims.sign_jwt_hs256("""dont-tell-anyone""")
# In: {"claims":{"sub":"user123"}}
# Out: {"signed":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJ1c2VyMTIzIn0.hUl-nngPMY_3h9vveWJUPsCcO5PeL6k9hWLnMYeFbFQ"}
root.signed = this.claims.sign_jwt_hs256(signing_secret: """dont-tell-anyone""", headers: {"kid": "my-key", "x": "y"})
# In: {"claims":{"sub":"user123"}}
# Out: {"signed":"<signed JWT token>"}
sign_jwt_hs384
Hash and sign an object representing JSON Web Token (JWT) claims using HS384.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The secret to use for signing the token. |
|
|
Optional object of JWT header fields to include in the token. Keys "alg", "typ", "jku", "jwk", "x5u", "x5c", "x5t","x5t#S256" and "crit" will be ignored if provided. |
Examples
root.signed = this.claims.sign_jwt_hs384("""dont-tell-anyone""")
# In: {"claims":{"sub":"user123"}}
# Out: {"signed":"eyJhbGciOiJIUzM4NCIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJ1c2VyMTIzIn0.zGYLr83aToon1efUNq-hw7XgT20lPvZb8sYei8x6S6mpHwb433SJdXJXx0Oio8AZ"}
root.signed = this.claims.sign_jwt_hs384(signing_secret: """dont-tell-anyone""", headers: {"kid": "my-key", "x": "y"})
# In: {"claims":{"sub":"user123"}}
# Out: {"signed":"<signed JWT token>"}
sign_jwt_hs512
Hash and sign an object representing JSON Web Token (JWT) claims using HS512.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The secret to use for signing the token. |
|
|
Optional object of JWT header fields to include in the token. Keys "alg", "typ", "jku", "jwk", "x5u", "x5c", "x5t","x5t#S256" and "crit" will be ignored if provided. |
Examples
root.signed = this.claims.sign_jwt_hs512("""dont-tell-anyone""")
# In: {"claims":{"sub":"user123"}}
# Out: {"signed":"eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJ1c2VyMTIzIn0.zBNR9o_6EDwXXKkpKLNJhG26j8Dc-mV-YahBwmEdCrmiWt5les8I9rgmNlWIowpq6Yxs4kLNAdFhqoRz3NXT3w"}
root.signed = this.claims.sign_jwt_hs512(signing_secret: """dont-tell-anyone""", headers: {"kid": "my-key", "x": "y"})
# In: {"claims":{"sub":"user123"}}
# Out: {"signed":"<signed JWT token>"}
sign_jwt_rs256
Hash and sign an object representing JSON Web Token (JWT) claims using RS256.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The secret to use for signing the token. |
|
|
Optional object of JWT header fields to include in the token. Keys "alg", "typ", "jku", "jwk", "x5u", "x5c", "x5t","x5t#S256" and "crit" will be ignored if provided. |
Examples
root.signed = this.claims.sign_jwt_rs256("""-----BEGIN RSA PRIVATE KEY-----
... signature data ...
-----END RSA PRIVATE KEY-----""")
# In: {"claims":{"sub":"user123"}}
# Out: {"signed":"eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.b0lH3jEupZZ4zoaly4Y_GCvu94HH6UKdKY96zfGNsIkPZpQLHIkZ7jMWlLlNOAd8qXlsBGP_i8H2qCKI4zlWJBGyPZgxXDzNRPVrTDfFpn4t4nBcA1WK2-ntXP3ehQxsaHcQU8Z_nsogId7Pme5iJRnoHWEnWtbwz5DLSXL3ZZNnRdrHM9MdI7QSDz9mojKDCaMpGN9sG7Xl-tGdBp1XzXuUOzG8S03mtZ1IgVR1uiBL2N6oohHIAunk8DIAmNWI-zgycTgzUGU7mvPkKH43qO8Ua1-13tCUBKKa8VxcotZ67Mxm1QAvBGoDnTKwWMwghLzs6d6WViXQg6eWlJcpBA"}
root.signed = this.claims.sign_jwt_rs256(signing_secret: """-----BEGIN RSA PRIVATE KEY-----
... signature data ...
-----END RSA PRIVATE KEY-----""", headers: {"kid": "my-key", "x": "y"})
# In: {"claims":{"sub":"user123"}}
# Out: {"signed":"<signed JWT token>"}
sign_jwt_rs384
Hash and sign an object representing JSON Web Token (JWT) claims using RS384.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The secret to use for signing the token. |
|
|
Optional object of JWT header fields to include in the token. Keys "alg", "typ", "jku", "jwk", "x5u", "x5c", "x5t","x5t#S256" and "crit" will be ignored if provided. |
Examples
root.signed = this.claims.sign_jwt_rs384("""-----BEGIN RSA PRIVATE KEY-----
... signature data ...
-----END RSA PRIVATE KEY-----""")
# In: {"claims":{"sub":"user123"}}
# Out: {"signed":"eyJhbGciOiJSUzM4NCIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.orcXYBcjVE5DU7mvq4KKWFfNdXR4nEY_xupzWoETRpYmQZIozlZnM_nHxEk2dySvpXlAzVm7kgOPK2RFtGlOVaNRIa3x-pMMr-bhZTno4L8Hl4sYxOks3bWtjK7wql4uqUbqThSJB12psAXw2-S-I_FMngOPGIn4jDT9b802ottJSvTpXcy0-eKTjrV2PSkRRu-EYJh0CJZW55MNhqlt6kCGhAXfbhNazN3ASX-dmpd_JixyBKphrngr_zRA-FCn_Xf3QQDA-5INopb4Yp5QiJ7UxVqQEKI80X_JvJqz9WE1qiAw8pq5-xTen1t7zTP-HT1NbbD3kltcNa3G8acmNg"}
root.signed = this.claims.sign_jwt_rs384(signing_secret: """-----BEGIN RSA PRIVATE KEY-----
... signature data ...
-----END RSA PRIVATE KEY-----""", headers: {"kid": "my-key", "x": "y"})
# In: {"claims":{"sub":"user123"}}
# Out: {"signed":"<signed JWT token>"}
sign_jwt_rs512
Hash and sign an object representing JSON Web Token (JWT) claims using RS512.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The secret to use for signing the token. |
|
|
Optional object of JWT header fields to include in the token. Keys "alg", "typ", "jku", "jwk", "x5u", "x5c", "x5t","x5t#S256" and "crit" will be ignored if provided. |
Examples
root.signed = this.claims.sign_jwt_rs512("""-----BEGIN RSA PRIVATE KEY-----
... signature data ...
-----END RSA PRIVATE KEY-----""")
# In: {"claims":{"sub":"user123"}}
# Out: {"signed":"eyJhbGciOiJSUzUxMiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.rsMp_X5HMrUqKnZJIxo27aAoscovRA6SSQYR9rq7pifIj0YHXxMyNyOBDGnvVALHKTi25VUGHpfNUW0VVMmae0A4t_ObNU6hVZHguWvetKZZq4FZpW1lgWHCMqgPGwT5_uOqwYCH6r8tJuZT3pqXeL0CY4putb1AN2w6CVp620nh3l8d3XWb4jaifycd_4CEVCqHuWDmohfug4VhmoVKlIXZkYoAQowgHlozATDssBSWdYtv107Wd2AzEoiXPu6e3pflsuXULlyqQnS4ELEKPYThFLafh1NqvZDPddqozcPZ-iODBW-xf3A4DYDdivnMYLrh73AZOGHexxu8ay6nDA"}
root.signed = this.claims.sign_jwt_rs512(signing_secret: """-----BEGIN RSA PRIVATE KEY-----
... signature data ...
-----END RSA PRIVATE KEY-----""", headers: {"kid": "my-key", "x": "y"})
# In: {"claims":{"sub":"user123"}}
# Out: {"signed":"<signed JWT token>"}
Number manipulation
abs
Returns the absolute value of an int64 or float64 number. As a special case, when an integer is provided that is the minimum value it is converted to the maximum value.
bitwise_xor
Performs a bitwise XOR (exclusive OR) operation between the integer and the specified value.
ceil
Rounds a number up to the nearest integer. Returns an integer if the result fits in 64-bit, otherwise returns a float.
float32
Converts a numerical type into a 32-bit floating point number, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver).
If the value is a string then an attempt will be made to parse it as a 32-bit floating point number. Please refer to the strconv.ParseFloat documentation for details regarding the supported formats.
float64
Converts a numerical type into a 64-bit floating point number, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver).
If the value is a string then an attempt will be made to parse it as a 64-bit floating point number. Please refer to the strconv.ParseFloat documentation for details regarding the supported formats.
floor
Rounds a number down to the nearest integer. Returns an integer if the result fits in 64-bit, otherwise returns a float.
int16
Converts a numerical type into a 16-bit signed integer, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver).
If the value is a string then an attempt will be made to parse it as a 16-bit signed integer. If the target value exceeds the capacity of an integer or contains decimal values then this method will throw an error. In order to convert a floating point number containing decimals first use .round() on the value. Please refer to the strconv.ParseInt documentation for details regarding the supported formats.
int32
Converts a numerical type into a 32-bit signed integer, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver).
If the value is a string then an attempt will be made to parse it as a 32-bit signed integer. If the target value exceeds the capacity of an integer or contains decimal values then this method will throw an error. In order to convert a floating point number containing decimals first use .round() on the value. Please refer to the strconv.ParseInt documentation for details regarding the supported formats.
int64
Converts a numerical type into a 64-bit signed integer, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver).
If the value is a string then an attempt will be made to parse it as a 64-bit signed integer. If the target value exceeds the capacity of an integer or contains decimal values then this method will throw an error. In order to convert a floating point number containing decimals first use .round() on the value. Please refer to the strconv.ParseInt documentation for details regarding the supported formats.
int8
Converts a numerical type into a 8-bit signed integer, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver).
If the value is a string then an attempt will be made to parse it as a 8-bit signed integer. If the target value exceeds the capacity of an integer or contains decimal values then this method will throw an error. In order to convert a floating point number containing decimals first use .round() on the value. Please refer to the strconv.ParseInt documentation for details regarding the supported formats.
max
Returns the largest number from an array. All elements must be numbers and the array cannot be empty.
min
Returns the smallest number from an array. All elements must be numbers and the array cannot be empty.
round
Rounds a number to the nearest integer. Values at .5 round away from zero. Returns an integer if the result fits in 64-bit, otherwise returns a float.
uint16
Converts a numerical type into a 16-bit unsigned integer, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver).
If the value is a string then an attempt will be made to parse it as a 16-bit unsigned integer. If the target value exceeds the capacity of an integer or contains decimal values then this method will throw an error. In order to convert a floating point number containing decimals first use .round() on the value. Please refer to the strconv.ParseInt documentation for details regarding the supported formats.
uint32
Converts a numerical type into a 32-bit unsigned integer, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver).
If the value is a string then an attempt will be made to parse it as a 32-bit unsigned integer. If the target value exceeds the capacity of an integer or contains decimal values then this method will throw an error. In order to convert a floating point number containing decimals first use .round() on the value. Please refer to the strconv.ParseInt documentation for details regarding the supported formats.
uint64
Converts a numerical type into a 64-bit unsigned integer, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver).
If the value is a string then an attempt will be made to parse it as a 64-bit unsigned integer. If the target value exceeds the capacity of an integer or contains decimal values then this method will throw an error. In order to convert a floating point number containing decimals first use .round() on the value. Please refer to the strconv.ParseInt documentation for details regarding the supported formats.
uint8
Converts a numerical type into a 8-bit unsigned integer, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver).
If the value is a string then an attempt will be made to parse it as a 8-bit unsigned integer. If the target value exceeds the capacity of an integer or contains decimal values then this method will throw an error. In order to convert a floating point number containing decimals first use .round() on the value. Please refer to the strconv.ParseInt documentation for details regarding the supported formats.
Object & array manipulation
all
Tests whether all elements in an array satisfy a condition. Returns true only if the query evaluates to true for every element. Returns false for empty arrays.
Examples
root.all_over_21 = this.patrons.all(patron -> patron.age >= 21)
# In: {"patrons":[{"id":"1","age":18},{"id":"2","age":23}]}
# Out: {"all_over_21":false}
# In: {"patrons":[{"id":"1","age":45},{"id":"2","age":23}]}
# Out: {"all_over_21":true}
root.all_positive = this.values.all(v -> v > 0)
# In: {"values":[1,2,3,4,5]}
# Out: {"all_positive":true}
# In: {"values":[1,-2,3,4,5]}
# Out: {"all_positive":false}
any
Tests whether at least one element in an array satisfies a condition. Returns true if the query evaluates to true for any element. Returns false for empty arrays.
Examples
root.any_over_21 = this.patrons.any(patron -> patron.age >= 21)
# In: {"patrons":[{"id":"1","age":18},{"id":"2","age":23}]}
# Out: {"any_over_21":true}
# In: {"patrons":[{"id":"1","age":10},{"id":"2","age":12}]}
# Out: {"any_over_21":false}
root.has_errors = this.results.any(r -> r.status == "error")
# In: {"results":[{"status":"ok"},{"status":"error"},{"status":"ok"}]}
# Out: {"has_errors":true}
# In: {"results":[{"status":"ok"},{"status":"ok"}]}
# Out: {"has_errors":false}
append
Adds one or more elements to the end of an array and returns the new array. The original array is not modified.
assign
Merges two objects or arrays with override behavior. For objects, source values replace destination values on key conflicts. Arrays are concatenated. To preserve both values on conflict, use the merge method instead.
Examples
root = this.foo.assign(this.bar)
# In: {"foo":{"first_name":"fooer","likes":"bars"},"bar":{"second_name":"barer","likes":"foos"}}
# Out: {"first_name":"fooer","likes":"foos","second_name":"barer"}
Override defaults with user settings:
root.config = this.defaults.assign(this.user_settings)
# In: {"defaults":{"timeout":30,"retries":3},"user_settings":{"timeout":60}}
# Out: {"config":{"retries":3,"timeout":60}}
collapse
Flattens a nested structure into a flat object with dot-notation keys.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
Whether to include empty objects and arrays in the resulting object. |
Examples
root.result = this.collapse()
# In: {"foo":[{"bar":"1"},{"bar":{}},{"bar":"2"},{"bar":[]}]}
# Out: {"result":{"foo.0.bar":"1","foo.2.bar":"2"}}
Set include_empty to true to preserve empty objects and arrays in the output:
root.result = this.collapse(include_empty: true)
# In: {"foo":[{"bar":"1"},{"bar":{}},{"bar":"2"},{"bar":[]}]}
# Out: {"result":{"foo.0.bar":"1","foo.1.bar":{},"foo.2.bar":"2","foo.3.bar":[]}}
contains
Tests if an array or object contains a value.
Examples
root.has_foo = this.thing.contains("foo")
# In: {"thing":["this","foo","that"]}
# Out: {"has_foo":true}
# In: {"thing":["this","bar","that"]}
# Out: {"has_foo":false}
root.has_bar = this.thing.contains(20)
# In: {"thing":[10.3,20.0,"huh",3]}
# Out: {"has_bar":true}
# In: {"thing":[2,3,40,67]}
# Out: {"has_bar":false}
root.has_foo = this.thing.contains("foo")
# In: {"thing":"this foo that"}
# Out: {"has_foo":true}
# In: {"thing":"this bar that"}
# Out: {"has_foo":false}
diff
Compares the current value with another value and returns a detailed changelog describing all differences. The changelog contains operations (create, update, delete) with their paths and values, enabling you to track changes between data versions, implement audit logs, or synchronize data between systems.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The value to compare against the current value. Can be any structured data (object or array). |
Examples
Compare two objects to track field changes:
root.changes = this.before.diff(this.after)
# In: {"before":{"name":"Alice","age":30},"after":{"name":"Alice","age":31,"city":"NYC"}}
# Out: {"changes":[{"From":30,"Path":["age"],"To":31,"Type":"update"},{"From":null,"Path":["city"],"To":"NYC","Type":"create"}]}
Detect deletions in configuration changes:
root.changelog = this.old_config.diff(this.new_config)
# In: {"old_config":{"debug":true,"timeout":30},"new_config":{"timeout":60}}
# Out: {"changelog":[{"From":true,"Path":["debug"],"To":null,"Type":"delete"},{"From":30,"Path":["timeout"],"To":60,"Type":"update"}]}
enumerated
Transforms an array into an array of objects with index and value fields, making it easy to access both the position and content of each element.
Examples
root.foo = this.foo.enumerated()
# In: {"foo":["bar","baz"]}
# Out: {"foo":[{"index":0,"value":"bar"},{"index":1,"value":"baz"}]}
Useful for filtering by index position:
root.first_two = this.items.enumerated().filter(item -> item.index < 2).map_each(item -> item.value)
# In: {"items":["a","b","c","d"]}
# Out: {"first_two":["a","b"]}
exists
Checks whether a field exists at the specified dot path within an object. Returns true if the field is present (even if null), false otherwise.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
A dot path to a field. |
Examples
root.result = this.foo.exists("bar.baz")
# In: {"foo":{"bar":{"baz":"yep, I exist"}}}
# Out: {"result":true}
# In: {"foo":{"bar":{}}}
# Out: {"result":false}
# In: {"foo":{}}
# Out: {"result":false}
Also returns true for null values if the field exists:
root.has_field = this.data.exists("optional_field")
# In: {"data":{"optional_field":null}}
# Out: {"has_field":true}
# In: {"data":{}}
# Out: {"has_field":false}
explode
Expands a nested field into multiple documents.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
A dot path to a field to explode. |
Examples
On arrays
When exploding an array, each element becomes a separate document with the array element replacing the original field:
root = this.explode("value")
# In: {"id":1,"value":["foo","bar","baz"]}
# Out: [{"id":1,"value":"foo"},{"id":1,"value":"bar"},{"id":1,"value":"baz"}]
On objects
When exploding an object, the output keys match the nested object’s keys, with values being the full document where the target field is replaced by each nested value:
root = this.explode("value")
# In: {"id":1,"value":{"foo":2,"bar":[3,4],"baz":{"bev":5}}}
# Out: {"bar":{"id":1,"value":[3,4]},"baz":{"id":1,"value":{"bev":5}},"foo":{"id":1,"value":2}}
filter
Filters array or object elements based on a condition.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
A query to apply to each element, if this query resolves to any value other than a boolean |
Examples
root.new_nums = this.nums.filter(num -> num > 10)
# In: {"nums":[3,11,4,17]}
# Out: {"new_nums":[11,17]}
On objects
When filtering objects, the query receives a context with key and value fields for each entry:
root.new_dict = this.dict.filter(item -> item.value.contains("foo"))
# In: {"dict":{"first":"hello foo","second":"world","third":"this foo is great"}}
# Out: {"new_dict":{"first":"hello foo","third":"this foo is great"}}
find
Searches an array for a matching value and returns the index of the first occurrence. Returns -1 if no match is found. Numeric types are compared by value regardless of representation.
find_all
Searches an array for all occurrences of a value and returns an array of matching indexes. Returns an empty array if no matches are found. Numeric types are compared by value regardless of representation.
find_all_by
Searches an array for all elements that satisfy a condition and returns an array of their indexes. Returns an empty array if no elements match.
Examples
root.index = this.find_all_by(v -> v != "bar")
# In: ["foo", "bar", "baz"]
# Out: {"index":[0,2]}
Find all indexes matching criteria:
root.error_indexes = this.logs.find_all_by(log -> log.level == "error")
# In: {"logs":[{"level":"info"},{"level":"error"},{"level":"warn"},{"level":"error"}]}
# Out: {"error_indexes":[1,3]}
find_by
Searches an array for the first element that satisfies a condition and returns its index. Returns -1 if no element matches the query.
Examples
root.index = this.find_by(v -> v != "bar")
# In: ["foo", "bar", "baz"]
# Out: {"index":0}
Find first object matching criteria:
root.first_adult = this.users.find_by(u -> u.age >= 18)
# In: {"users":[{"name":"Alice","age":15},{"name":"Bob","age":22},{"name":"Carol","age":19}]}
# Out: {"first_adult":1}
flatten
Flattens an array by one level, expanding nested arrays into the parent array. Only the first level of nesting is removed.
fold
Reduces an array to a single value by iteratively applying a function. Also known as reduce or aggregate. The query receives an accumulator (tally) and current element (value) for each iteration.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The initial value to start the fold with. For example, an empty object |
|
|
A query to apply for each element. The query is provided an object with two fields; |
Examples
Sum numbers in an array:
root.sum = this.foo.fold(0, item -> item.tally + item.value)
# In: {"foo":[3,8,11]}
# Out: {"sum":22}
Concatenate strings:
root.result = this.foo.fold("", item -> "%v%v".format(item.tally, item.value))
# In: {"foo":["hello ", "world"]}
# Out: {"result":"hello world"}
Merge an array of objects into a single object:
root.smoothie = this.fruits.fold({}, item -> item.tally.merge(item.value))
# In: {"fruits":[{"apple":5},{"banana":3},{"orange":8}]}
# Out: {"smoothie":{"apple":5,"banana":3,"orange":8}}
get
Extract a field value, identified via a dot path, from an object.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
A dot path identifying a field to obtain. |
index
Extract an element from an array by an index. The index can be negative, and if so the element will be selected from the end counting backwards starting from -1. E.g. an index of -1 returns the last element, an index of -2 returns the element before the last, and so on.
Examples
root.last_name = this.names.index(-1)
# In: {"names":["rachel","stevens"]}
# Out: {"last_name":"stevens"}
It is also possible to use this method on byte arrays, in which case the selected element will be returned as an integer:
root.last_byte = this.name.bytes().index(-1)
# In: {"name":"foobar bazson"}
# Out: {"last_byte":110}
join
Joins an array of strings with an optional delimiter.
json_path
Executes the given JSONPath expression on an object or array and returns the result. The JSONPath expression syntax can be found at https://goessner.net/articles/JsonPath/. For more complex logic, you can use Gval expressions (https://github.com/PaesslerAG/gval).
Examples
root.all_names = this.json_path("$..name")
# In: {"name":"alice","foo":{"name":"bob"}}
# Out: {"all_names":["alice","bob"]}
# In: {"thing":["this","bar",{"name":"alice"}]}
# Out: {"all_names":["alice"]}
root.text_objects = this.json_path("$.body[?(@.type=='text')]")
# In: {"body":[{"type":"image","id":"foo"},{"type":"text","id":"bar"}]}
# Out: {"text_objects":[{"id":"bar","type":"text"}]}
json_schema
Checks a JSON schema against a value and returns the value if it matches or throws and error if it does not.
Examples
root = this.json_schema("""{
"type":"object",
"properties":{
"foo":{
"type":"string"
}
}
}""")
# In: {"foo":"bar"}
# Out: {"foo":"bar"}
# In: {"foo":5}
# Out: Error("failed assignment (line 1): field `this`: foo invalid type. expected: string, given: integer")
In order to load a schema from a file use the file function:
root = this.json_schema(file(env("BENTHOS_TEST_BLOBLANG_SCHEMA_FILE")))
key_values
Converts an object into an array of key-value pair objects. Each element has a 'key' field and a 'value' field. Order is not guaranteed unless sorted.
Examples
root.foo_key_values = this.foo.key_values().sort_by(pair -> pair.key)
# In: {"foo":{"bar":1,"baz":2}}
# Out: {"foo_key_values":[{"key":"bar","value":1},{"key":"baz","value":2}]}
Filter object entries by value:
root.large_items = this.items.key_values().filter(pair -> pair.value > 15).map_each(pair -> pair.key)
# In: {"items":{"a":5,"b":15,"c":20,"d":3}}
# Out: {"large_items":["c"]}
map_each
Applies a mapping to each element of an array or object.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
A query that will be used to map each element. |
map_each_key
Transforms object keys using a mapping query.
Examples
root.new_dict = this.dict.map_each_key(key -> key.uppercase())
# In: {"dict":{"keya":"hello","keyb":"world"}}
# Out: {"new_dict":{"KEYA":"hello","KEYB":"world"}}
Conditionally transform keys:
root = this.map_each_key(key -> if key.contains("kafka") { "_" + key })
# In: {"amqp_key":"foo","kafka_key":"bar","kafka_topic":"baz"}
# Out: {"_kafka_key":"bar","_kafka_topic":"baz","amqp_key":"foo"}
merge
Combines two objects or arrays. When merging objects, conflicting keys create arrays containing both values. Arrays are concatenated. For key override behavior instead, use the assign method.
Examples
root = this.foo.merge(this.bar)
# In: {"foo":{"first_name":"fooer","likes":"bars"},"bar":{"second_name":"barer","likes":"foos"}}
# Out: {"first_name":"fooer","likes":["bars","foos"],"second_name":"barer"}
Merge arrays:
root.combined = this.list1.merge(this.list2)
# In: {"list1":["a","b"],"list2":["c","d"]}
# Out: {"combined":["a","b","c","d"]}
patch
Applies a changelog (created by the diff method) to the current value, transforming it according to the specified operations. This enables you to synchronize data, replay changes, or implement event sourcing patterns by applying recorded changes to reconstruct state.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The changelog array to apply. Should be in the format returned by the diff method, containing Type, Path, From, and To fields for each change. |
Examples
Apply recorded changes to update an object:
root.updated = this.current.patch(this.changelog)
# In: {"current":{"name":"Alice","age":30},"changelog":[{"Type":"update","Path":["age"],"From":30,"To":31},{"Type":"create","Path":["city"],"From":null,"To":"NYC"}]}
# Out: {"updated":{"age":31,"city":"NYC","name":"Alice"}}
Restore previous state by applying inverse changes:
root.restored = this.modified.patch(this.reverse_changelog)
# In: {"modified":{"timeout":60},"reverse_changelog":[{"Type":"create","Path":["debug"],"From":null,"To":true},{"Type":"update","Path":["timeout"],"From":60,"To":30}]}
# Out: {"restored":{"debug":true,"timeout":30}}
slice
Extracts a portion of an array or string.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The low bound, which is the first element of the selection, or if negative selects from the end. |
|
|
An optional high bound. |
Examples
root.beginning = this.value.slice(0, 2)
root.end = this.value.slice(4)
# In: {"value":"foo bar"}
# Out: {"beginning":"fo","end":"bar"}
A negative low index can be used, indicating an offset from the end of the sequence. If the low index is greater than the length of the sequence then an empty result is returned:
root.last_chunk = this.value.slice(-4)
root.the_rest = this.value.slice(0, -4)
# In: {"value":"foo bar"}
# Out: {"last_chunk":" bar","the_rest":"foo"}
root.beginning = this.value.slice(0, 2)
root.end = this.value.slice(4)
# In: {"value":["foo","bar","baz","buz","bev"]}
# Out: {"beginning":["foo","bar"],"end":["bev"]}
A negative low index can be used, indicating an offset from the end of the sequence. If the low index is greater than the length of the sequence then an empty result is returned:
root.last_chunk = this.value.slice(-2)
root.the_rest = this.value.slice(0, -2)
# In: {"value":["foo","bar","baz","buz","bev"]}
# Out: {"last_chunk":["buz","bev"],"the_rest":["foo","bar","baz"]}
sort
Sorts array elements in ascending order.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
An optional query that should explicitly compare elements |
Examples
root.sorted = this.foo.sort()
# In: {"foo":["bbb","ccc","aaa"]}
# Out: {"sorted":["aaa","bbb","ccc"]}
Custom comparison for complex objects - return true if left < right:
root.sorted = this.foo.sort(item -> item.left.v < item.right.v)
# In: {"foo":[{"id":"foo","v":"bbb"},{"id":"bar","v":"ccc"},{"id":"baz","v":"aaa"}]}
# Out: {"sorted":[{"id":"baz","v":"aaa"},{"id":"foo","v":"bbb"},{"id":"bar","v":"ccc"}]}
sort_by
Sorts array elements by a specified field or expression.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
A query to apply to each element that yields a value used for sorting. |
Examples
root.sorted = this.foo.sort_by(ele -> ele.id)
# In: {"foo":[{"id":"bbb","message":"bar"},{"id":"aaa","message":"foo"},{"id":"ccc","message":"baz"}]}
# Out: {"sorted":[{"id":"aaa","message":"foo"},{"id":"bbb","message":"bar"},{"id":"ccc","message":"baz"}]}
Sort by numeric field:
root.sorted = this.items.sort_by(item -> item.priority)
# In: {"items":[{"name":"low","priority":3},{"name":"high","priority":1},{"name":"med","priority":2}]}
# Out: {"sorted":[{"name":"high","priority":1},{"name":"med","priority":2},{"name":"low","priority":3}]}
squash
Squashes an array of objects into a single object, where key collisions result in the values being merged (following similar rules as the .merge() method).
Examples
root.locations = this.locations.map_each(loc -> {loc.state: [loc.name]}).squash()
# In: {"locations":[{"name":"Seattle","state":"WA"},{"name":"New York","state":"NY"},{"name":"Bellevue","state":"WA"},{"name":"Olympia","state":"WA"}]}
# Out: {"locations":{"NY":["New York"],"WA":["Seattle","Bellevue","Olympia"]}}
unique
Returns an array with duplicate elements removed.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
An optional query that can be used in order to yield a value for each element to determine uniqueness. |
Examples
root.uniques = this.foo.unique()
# In: {"foo":["a","b","a","c"]}
# Out: {"uniques":["a","b","c"]}
Use a query to determine uniqueness by a field:
root.unique_users = this.users.unique(u -> u.id)
# In: {"users":[{"id":1,"name":"Alice"},{"id":2,"name":"Bob"},{"id":1,"name":"Alice Duplicate"}]}
# Out: {"unique_users":[{"id":1,"name":"Alice"},{"id":2,"name":"Bob"}]}
with
Returns an object where all but one or more field path arguments are removed. Each path specifies a specific field to be retained from the input object, allowing for nested fields.
If a key within a nested path does not exist then it is ignored.
without
Returns an object with specified keys removed.
Examples
root = this.without("inner.a","inner.c","d")
# In: {"inner":{"a":"first","b":"second","c":"third"},"d":"fourth","e":"fifth"}
# Out: {"e":"fifth","inner":{"b":"second"}}
Remove sensitive fields:
root = this.without("password","ssn","creditCard")
# In: {"username":"alice","password":"secret","email":"alice@example.com","ssn":"123-45-6789"}
# Out: {"email":"alice@example.com","username":"alice"}
Parsing
bloblang
Executes an argument Bloblang mapping on the target. This method can be used in order to execute dynamic mappings. Imports and functions that interact with the environment, such as file and env, or that access message information directly, such as content or json, are not enabled for dynamic Bloblang mappings.
format_json
Formats a value as a JSON string.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
Indentation string. Each element in a JSON object or array will begin on a new, indented line followed by one or more copies of indent according to the indentation nesting. |
|
|
Disable indentation. |
|
|
Escape problematic HTML characters. |
Examples
root = this.doc.format_json()
# In: {"doc":{"foo":"bar"}}
# Out: {
"foo": "bar"
}
Pass a string to the indent parameter in order to customise the indentation:
root = this.format_json(" ")
# In: {"doc":{"foo":"bar"}}
# Out: {
"doc": {
"foo": "bar"
}
}
Use the .string() method in order to coerce the result into a string:
root.doc = this.doc.format_json().string()
# In: {"doc":{"foo":"bar"}}
# Out: {"doc":"{\n \"foo\": \"bar\"\n}"}
Set the no_indent parameter to true to disable indentation. The result is equivalent to calling bytes():
root = this.doc.format_json(no_indent: true)
# In: {"doc":{"foo":"bar"}}
# Out: {"foo":"bar"}
Escapes problematic HTML characters:
root = this.doc.format_json()
# In: {"doc":{"email":"foo&bar@benthos.dev","name":"foo>bar"}}
# Out: {
"email": "foo\u0026bar@benthos.dev",
"name": "foo\u003ebar"
}
Set the escape_html parameter to false to disable escaping of problematic HTML characters:
root = this.doc.format_json(escape_html: false)
# In: {"doc":{"email":"foo&bar@benthos.dev","name":"foo>bar"}}
# Out: {
"email": "foo&bar@benthos.dev",
"name": "foo>bar"
}
format_msgpack
Serializes structured data into MessagePack binary format. MessagePack is a compact binary serialization that is faster and more space-efficient than JSON, making it ideal for network transmission and storage of structured data. Returns a byte array that can be further encoded as needed.
Examples
Serialize object to MessagePack and encode as hex for transmission:
root = this.format_msgpack().encode("hex")
# In: {"foo":"bar"}
# Out: 81a3666f6fa3626172
Serialize data to MessagePack and base64 encode for embedding in JSON:
root.msgpack_payload = this.data.format_msgpack().encode("base64")
# In: {"data":{"foo":"bar"}}
# Out: {"msgpack_payload":"gaNmb2+jYmFy"}
format_xml
Serializes an object into an XML document. Converts structured data to XML format with support for attributes (prefixed with hyphen), custom indentation, and configurable root element. Returns XML as a byte array.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
String to use for each level of indentation (default is 4 spaces). Each nested XML element will be indented by this string. |
|
|
Disable indentation and newlines to produce compact XML on a single line. |
|
|
Custom name for the root XML element. By default, the root element name is derived from the first key in the object. |
Examples
Serialize object to pretty-printed XML with default indentation:
root = this.format_xml()
# In: {"foo":{"bar":{"baz":"foo bar baz"}}}
# Out: <foo>
<bar>
<baz>foo bar baz</baz>
</bar>
</foo>
Create compact XML without indentation for smaller message size:
root = this.format_xml(no_indent: true)
# In: {"foo":{"bar":{"baz":"foo bar baz"}}}
# Out: <foo><bar><baz>foo bar baz</baz></bar></foo>
infer_schema
Attempt to infer the schema of a given value. The resulting schema can then be used as an input to schema conversion and enforcement methods.
parse_csv
Parses CSV data into an array.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
Whether to reference the first row as a header row. If set to true the output structure for messages will be an object where field keys are determined by the header row. Otherwise, the output will be an array of row arrays. |
|
|
The delimiter to use for splitting values in each record. It must be a single character. |
|
|
If set to |
Examples
Parses CSV data with a header row:
root.orders = this.orders.parse_csv()
# In: {"orders":"foo,bar\nfoo 1,bar 1\nfoo 2,bar 2"}
# Out: {"orders":[{"bar":"bar 1","foo":"foo 1"},{"bar":"bar 2","foo":"foo 2"}]}
Parses CSV data without a header row:
root.orders = this.orders.parse_csv(false)
# In: {"orders":"foo 1,bar 1\nfoo 2,bar 2"}
# Out: {"orders":[["foo 1","bar 1"],["foo 2","bar 2"]]}
Parses CSV data delimited by dots:
root.orders = this.orders.parse_csv(delimiter:".")
# In: {"orders":"foo.bar\nfoo 1.bar 1\nfoo 2.bar 2"}
# Out: {"orders":[{"bar":"bar 1","foo":"foo 1"},{"bar":"bar 2","foo":"foo 2"}]}
Parses CSV data containing a quote in an unquoted field:
root.orders = this.orders.parse_csv(lazy_quotes:true)
# In: {"orders":"foo,bar\nfoo 1,bar 1\nfoo\" \"2,bar\" \"2"}
# Out: {"orders":[{"bar":"bar 1","foo":"foo 1"},{"bar":"bar\" \"2","foo":"foo\" \"2"}]}
parse_form_url_encoded
Attempts to parse a url-encoded query string (from an x-www-form-urlencoded request body) and returns a structured result.
parse_json
Parses a JSON string into a structured value.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
An optional flag that when set makes parsing numbers as json.Number instead of the default float64. |
Examples
root.doc = this.doc.parse_json()
# In: {"doc":"{\"foo\":\"bar\"}"}
# Out: {"doc":{"foo":"bar"}}
root.doc = this.doc.parse_json(use_number: true)
# In: {"doc":"{\"foo\":\"11380878173205700000000000000000000000000000000\"}"}
# Out: {"doc":{"foo":"11380878173205700000000000000000000000000000000"}}
parse_msgpack
Parses MessagePack binary data into a structured object. MessagePack is an efficient binary serialization format that is more compact than JSON while maintaining similar data structures. Commonly used for high-performance APIs and data interchange between microservices.
Examples
Parse MessagePack data from hex-encoded content:
root = content().decode("hex").parse_msgpack()
# In: 81a3666f6fa3626172
# Out: {"foo":"bar"}
Parse MessagePack from base64-encoded field:
root.decoded = this.msgpack_data.decode("base64").parse_msgpack()
# In: {"msgpack_data":"gaNmb2+jYmFy"}
# Out: {"decoded":{"foo":"bar"}}
parse_parquet
Parses Apache Parquet binary data into an array of objects. Parquet is a columnar storage format optimized for analytics, commonly used with big data systems like Apache Spark, Hive, and cloud data warehouses. Each row in the Parquet file becomes an object in the output array.
parse_url
Attempts to parse a URL from a string value, returning a structured result that describes the various facets of the URL. The fields returned within the structured result roughly follow https://pkg.go.dev/net/url#URL, and may be expanded in future in order to present more information.
Examples
root.foo_url = this.foo_url.parse_url()
# In: {"foo_url":"https://docs.redpanda.com/redpanda-connect/guides/bloblang/about/"}
# Out: {"foo_url":{"fragment":"","host":"docs.redpanda.com","opaque":"","path":"/redpanda-connect/guides/bloblang/about/","raw_fragment":"","raw_path":"","raw_query":"","scheme":"https"}}
root.username = this.url.parse_url().user.name | "unknown"
# In: {"url":"amqp://foo:bar@127.0.0.1:5672/"}
# Out: {"username":"foo"}
# In: {"url":"redis://localhost:6379"}
# Out: {"username":"unknown"}
parse_xml
Parses an XML document into a structured object. Converts XML elements to JSON-like objects following these rules:
-
Element attributes are prefixed with a hyphen (e.g.,
-idfor anidattribute) -
Elements with both attributes and text content store the text in a
#textfield -
Repeated elements become arrays
-
XML comments, directives, and processing instructions are ignored
-
Optionally cast numeric and boolean strings to their proper types.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
Whether to automatically cast numeric and boolean string values to their proper types. When false, all values remain as strings. |
Examples
Parse XML document into object structure:
root.doc = this.doc.parse_xml()
# In: {"doc":"<root><title>This is a title</title><content>This is some content</content></root>"}
# Out: {"doc":{"root":{"content":"This is some content","title":"This is a title"}}}
Parse XML with type casting enabled to convert strings to numbers and booleans:
root.doc = this.doc.parse_xml(cast: true)
# In: {"doc":"<root><title>This is a title</title><number id=\"99\">123</number><bool>True</bool></root>"}
# Out: {"doc":{"root":{"bool":true,"number":{"#text":123,"-id":99},"title":"This is a title"}}}
Regular expressions
re_find_all_object
Finds all regex matches as objects with named groups.
Examples
root.matches = this.value.re_find_all_object("a(?P<foo>x*)b")
# In: {"value":"-axxb-ab-"}
# Out: {"matches":[{"0":"axxb","foo":"xx"},{"0":"ab","foo":""}]}
root.matches = this.value.re_find_all_object("(?m)(?P<key>\\w+):\\s+(?P<value>\\w+)$")
# In: {"value":"option1: value1\noption2: value2\noption3: value3"}
# Out: {"matches":[{"0":"option1: value1","key":"option1","value":"value1"},{"0":"option2: value2","key":"option2","value":"value2"},{"0":"option3: value3","key":"option3","value":"value3"}]}
re_find_all_submatch
Finds all regex matches with capture groups.
Examples
root.matches = this.value.re_find_all_submatch("a(x*)b")
# In: {"value":"-axxb-ab-"}
# Out: {"matches":[["axxb","xx"],["ab",""]]}
root.emails = this.text.re_find_all_submatch("(\\w+)@(\\w+\\.\\w+)")
# In: {"text":"Contact: alice@example.com or bob@test.org"}
# Out: {"emails":[["alice@example.com","alice","example.com"],["bob@test.org","bob","test.org"]]}
re_find_object
Finds the first regex match as an object with named groups.
Examples
root.matches = this.value.re_find_object("a(?P<foo>x*)b")
# In: {"value":"-axxb-ab-"}
# Out: {"matches":{"0":"axxb","foo":"xx"}}
root.matches = this.value.re_find_object("(?P<key>\\w+):\\s+(?P<value>\\w+)")
# In: {"value":"option1: value1"}
# Out: {"matches":{"0":"option1: value1","key":"option1","value":"value1"}}
re_replace
Replaces all regex matches with a replacement string that can reference capture groups using $1, $2, etc. Use for pattern-based transformations or data reformatting.
SQL
vector
Converts an array of numbers into a vector type suitable for insertion into SQL databases with vector/embedding support. This is commonly used with PostgreSQL’s pgvector extension for storing and querying machine learning embeddings, enabling similarity search and vector operations in your database.
String manipulation
compare_argon2
Checks whether a string matches a hashed secret using Argon2.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The hashed secret to compare with the input. This must be a fully-qualified string which encodes the Argon2 options used to generate the hash. |
Examples
root.match = this.secret.compare_argon2("$argon2id$v=19$m=4096,t=3,p=1$c2FsdHktbWNzYWx0ZmFjZQ$RMUMwgtS32/mbszd+ke4o4Ej1jFpYiUqY6MHWa69X7Y")
# In: {"secret":"there-are-many-blobs-in-the-sea"}
# Out: {"match":true}
root.match = this.secret.compare_argon2("$argon2id$v=19$m=4096,t=3,p=1$c2FsdHktbWNzYWx0ZmFjZQ$RMUMwgtS32/mbszd+ke4o4Ej1jFpYiUqY6MHWa69X7Y")
# In: {"secret":"will-i-ever-find-love"}
# Out: {"match":false}
compare_bcrypt
Checks whether a string matches a hashed secret using bcrypt.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The hashed secret value to compare with the input. |
Examples
root.match = this.secret.compare_bcrypt("$2y$10$Dtnt5NNzVtMCOZONT705tOcS8It6krJX8bEjnDJnwxiFKsz1C.3Ay")
# In: {"secret":"there-are-many-blobs-in-the-sea"}
# Out: {"match":true}
root.match = this.secret.compare_bcrypt("$2y$10$Dtnt5NNzVtMCOZONT705tOcS8It6krJX8bEjnDJnwxiFKsz1C.3Ay")
# In: {"secret":"will-i-ever-find-love"}
# Out: {"match":false}
contains
Tests if an array or object contains a value.
Examples
root.has_foo = this.thing.contains("foo")
# In: {"thing":["this","foo","that"]}
# Out: {"has_foo":true}
# In: {"thing":["this","bar","that"]}
# Out: {"has_foo":false}
root.has_bar = this.thing.contains(20)
# In: {"thing":[10.3,20.0,"huh",3]}
# Out: {"has_bar":true}
# In: {"thing":[2,3,40,67]}
# Out: {"has_bar":false}
root.has_foo = this.thing.contains("foo")
# In: {"thing":"this foo that"}
# Out: {"has_foo":true}
# In: {"thing":"this bar that"}
# Out: {"has_foo":false}
format
Formats a value using a specified format string.
Examples
root.foo = "%s(%v): %v".format(this.name, this.age, this.fingers)
# In: {"name":"lance","age":37,"fingers":13}
# Out: {"foo":"lance(37): 13"}
root.message = "User %s has %v points".format(this.username, this.score)
# In: {"username":"alice","score":100}
# Out: {"message":"User alice has 100 points"}
replace
Replaces all occurrences of a substring with another string. Use for text transformation, cleaning data, or normalizing strings.
replace_all
Replaces all occurrences of a substring with another.
replace_all_many
Performs multiple find-and-replace operations in sequence.
replace_many
Performs multiple find-and-replace operations in sequence using an array of [old, new] pairs. More efficient than chaining multiple replace_all calls. Use for bulk text transformations.
slice
Extracts a portion of an array or string.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The low bound, which is the first element of the selection, or if negative selects from the end. |
|
|
An optional high bound. |
Examples
root.beginning = this.value.slice(0, 2)
root.end = this.value.slice(4)
# In: {"value":"foo bar"}
# Out: {"beginning":"fo","end":"bar"}
A negative low index can be used, indicating an offset from the end of the sequence. If the low index is greater than the length of the sequence then an empty result is returned:
root.last_chunk = this.value.slice(-4)
root.the_rest = this.value.slice(0, -4)
# In: {"value":"foo bar"}
# Out: {"last_chunk":" bar","the_rest":"foo"}
root.beginning = this.value.slice(0, 2)
root.end = this.value.slice(4)
# In: {"value":["foo","bar","baz","buz","bev"]}
# Out: {"beginning":["foo","bar"],"end":["bev"]}
A negative low index can be used, indicating an offset from the end of the sequence. If the low index is greater than the length of the sequence then an empty result is returned:
root.last_chunk = this.value.slice(-2)
root.the_rest = this.value.slice(0, -2)
# In: {"value":["foo","bar","baz","buz","bev"]}
# Out: {"last_chunk":["buz","bev"],"the_rest":["foo","bar","baz"]}
slug
Converts a string into a URL-friendly slug by replacing spaces with hyphens, removing special characters, and converting to lowercase. Supports multiple languages for proper transliteration of non-ASCII characters.
Examples
Create a URL-friendly slug from a string with special characters:
root.slug = this.title.slug()
# In: {"title":"Hello World! Welcome to Redpanda Connect"}
# Out: {"slug":"hello-world-welcome-to-redpanda-connect"}
Create a slug preserving French language rules:
root.slug = this.title.slug("fr")
# In: {"title":"Café & Restaurant"}
# Out: {"slug":"cafe-et-restaurant"}
split
Splits a string into an array of substrings.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The delimiter to split with. |
|
|
To treat empty substrings as null values |
Examples
root.new_value = this.value.split(",")
# In: {"value":"foo,bar,baz"}
# Out: {"new_value":["foo","bar","baz"]}
root.new_value = this.value.split(",", true)
# In: {"value":"foo,,qux"}
# Out: {"new_value":["foo",null,"qux"]}
root.words = this.sentence.split(" ")
# In: {"sentence":"hello world from bloblang"}
# Out: {"words":["hello","world","from","bloblang"]}
strip_html
Removes HTML tags from a string, returning only the text content. Useful for extracting plain text from HTML documents, sanitizing user input, or preparing content for text analysis. Optionally preserves specific HTML elements while stripping all others.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
Optional array of HTML element names to preserve (e.g., ["strong", "em", "a"]). All other HTML tags will be removed. |
Examples
Extract plain text from HTML content:
root.plain_text = this.html_content.strip_html()
# In: {"html_content":"<p>Welcome to <strong>Redpanda Connect</strong>!</p>"}
# Out: {"plain_text":"Welcome to Redpanda Connect!"}
Preserve specific HTML elements while removing others:
root.sanitized = this.html.strip_html(["strong", "em"])
# In: {"html":"<div><p>Some <strong>bold</strong> and <em>italic</em> text with a <script>alert('xss')</script></p></div>"}
# Out: {"sanitized":"Some <strong>bold</strong> and <em>italic</em> text with a "}
trim
Removes leading and trailing characters from a string.
trim_prefix
Removes a specified prefix from the beginning of a string.
trim_suffix
Removes a specified suffix from the end of a string.
unicode_segments
Splits text into segments based on Unicode text segmentation rules. Returns an array of strings representing individual graphemes (visual characters), words (including punctuation and whitespace), or sentences. Handles complex Unicode correctly, including emoji with skin tone modifiers and zero-width joiners.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
Type of segmentation: "grapheme", "word", or "sentence" |
Examples
Split text into sentences (preserves trailing spaces):
root.sentences = this.text.unicode_segments("sentence")
# In: {"text":"Hello world. How are you?"}
# Out: {"sentences":["Hello world. ","How are you?"]}
Split text into grapheme clusters (handles complex emoji correctly):
root.graphemes = this.emoji.unicode_segments("grapheme")
# In: {"emoji":"👨👩👧👦❤️"}
# Out: {"graphemes":["👨👩👧👦","❤️"]}
Timestamp manipulation
parse_duration
Parses a Go-style duration string into nanoseconds. A duration string is a signed sequence of decimal numbers with unit suffixes like "300ms", "-1.5h", or "2h45m". Valid units: "ns", "us" (or "µs"), "ms", "s", "m", "h".
parse_duration_iso8601
Parses an ISO 8601 duration string into nanoseconds. Format: "P[n]Y[n]M[n]DT[n]H[n]M[n]S" or "P[n]W". Example: "P3Y6M4DT12H30M5S" means 3 years, 6 months, 4 days, 12 hours, 30 minutes, 5 seconds. Supports fractional seconds with full precision (not just one decimal place).
Examples
Parse complex ISO 8601 duration to nanoseconds:
root.delay_for_ns = this.delay_for.parse_duration_iso8601()
# In: {"delay_for":"P3Y6M4DT12H30M5S"}
# Out: {"delay_for_ns":110839937000000000}
Parse hours to seconds:
root.delay_for_s = this.delay_for.parse_duration_iso8601() / 1000000000
# In: {"delay_for":"PT2H"}
# Out: {"delay_for_s":7200}
ts_add_iso8601
Adds an ISO 8601 duration to a timestamp with calendar-aware precision for years, months, and days. Useful when you need to add durations that account for variable month lengths or leap years.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
Duration in ISO 8601 format (e.g., "P1Y2M3D" for 1 year, 2 months, 3 days) |
Examples
Add one year to a timestamp:
root.next_year = this.created_at.ts_add_iso8601("P1Y")
# In: {"created_at":"2020-08-14T05:54:23Z"}
# Out: {"next_year":"2021-08-14T05:54:23Z"}
Add a complex duration with multiple units:
root.future_date = this.created_at.ts_add_iso8601("P1Y2M3DT4H5M6S")
# In: {"created_at":"2020-01-01T00:00:00Z"}
# Out: {"future_date":"2021-03-04T04:05:06Z"}
ts_format
Formats a timestamp as a string using Go’s reference time format. Defaults to RFC 3339 if no format specified. The format uses "Mon Jan 2 15:04:05 -0700 MST 2006" as a reference. Accepts unix timestamps (with decimal precision) or RFC 3339 strings. Use ts_strftime for strftime-style formats.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The output format using Go’s reference time. |
|
|
Optional timezone (e.g., 'UTC', 'America/New_York'). Defaults to input timezone or local time for unix timestamps. |
Examples
Format timestamp with custom format:
root.something_at = this.created_at.ts_format("2006-Jan-02 15:04:05")
# In: {"created_at":"2020-08-14T11:50:26.371Z"}
# Out: {"something_at":"2020-Aug-14 11:50:26"}
Format unix timestamp with timezone specification:
root.something_at = this.created_at.ts_format(format: "2006-Jan-02 15:04:05", tz: "UTC")
# In: {"created_at":1597405526}
# Out: {"something_at":"2020-Aug-14 11:45:26"}
ts_parse
Parses a timestamp string using Go’s reference time format and outputs a timestamp object. The format uses "Mon Jan 2 15:04:05 -0700 MST 2006" as a reference - show how this reference time would appear in your format. Use ts_strptime for strftime-style formats instead.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The format of the input string using Go’s reference time. |
Examples
Parse a date with abbreviated month name:
root.doc.timestamp = this.doc.timestamp.ts_parse("2006-Jan-02")
# In: {"doc":{"timestamp":"2020-Aug-14"}}
# Out: {"doc":{"timestamp":"2020-08-14T00:00:00Z"}}
Parse a custom datetime format:
root.parsed = this.timestamp.ts_parse("Jan 2, 2006 at 3:04pm (MST)")
# In: {"timestamp":"Aug 14, 2020 at 5:54am (UTC)"}
# Out: {"parsed":"2020-08-14T05:54:00Z"}
ts_round
Rounds a timestamp to the nearest multiple of the specified duration. Halfway values round up. Accepts unix timestamps (seconds with optional decimal precision) or RFC 3339 formatted strings.
Examples
Round timestamp to the nearest hour:
root.created_at_hour = this.created_at.ts_round("1h".parse_duration())
# In: {"created_at":"2020-08-14T05:54:23Z"}
# Out: {"created_at_hour":"2020-08-14T06:00:00Z"}
Round timestamp to the nearest minute:
root.created_at_minute = this.created_at.ts_round("1m".parse_duration())
# In: {"created_at":"2020-08-14T05:54:23Z"}
# Out: {"created_at_minute":"2020-08-14T05:54:00Z"}
ts_strftime
Formats a timestamp as a string using strptime format specifiers (like %Y, %m, %d). Accepts unix timestamps (with decimal precision) or RFC 3339 strings. Supports %f for microseconds. Use ts_format for Go-style reference time formats.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The output format using strptime specifiers. |
|
|
Optional timezone. Defaults to input timezone or local time for unix timestamps. |
Examples
Format timestamp with strftime specifiers:
root.something_at = this.created_at.ts_strftime("%Y-%b-%d %H:%M:%S")
# In: {"created_at":"2020-08-14T11:50:26.371Z"}
# Out: {"something_at":"2020-Aug-14 11:50:26"}
Format with microseconds using %f directive:
root.something_at = this.created_at.ts_strftime("%Y-%b-%d %H:%M:%S.%f", "UTC")
# In: {"created_at":"2020-08-14T11:50:26.371Z"}
# Out: {"something_at":"2020-Aug-14 11:50:26.371000"}
ts_strptime
Parses a timestamp string using strptime format specifiers (like %Y, %m, %d) and outputs a timestamp object. Use ts_parse for Go-style reference time formats instead.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The format string using strptime specifiers (e.g., %Y-%m-%d). |
Examples
Parse date with abbreviated month using strptime format:
root.doc.timestamp = this.doc.timestamp.ts_strptime("%Y-%b-%d")
# In: {"doc":{"timestamp":"2020-Aug-14"}}
# Out: {"doc":{"timestamp":"2020-08-14T00:00:00Z"}}
Parse datetime with microseconds using %f directive:
root.doc.timestamp = this.doc.timestamp.ts_strptime("%Y-%b-%d %H:%M:%S.%f")
# In: {"doc":{"timestamp":"2020-Aug-14 11:50:26.371000"}}
# Out: {"doc":{"timestamp":"2020-08-14T11:50:26.371Z"}}
ts_sub
Calculates the duration in nanoseconds between two timestamps (t1 - t2). Returns a signed integer: positive if t1 is after t2, negative if t1 is before t2. Use .abs() for absolute duration.
Examples
Calculate absolute duration between two timestamps:
root.between = this.started_at.ts_sub("2020-08-14T05:54:23Z").abs()
# In: {"started_at":"2020-08-13T05:54:23Z"}
# Out: {"between":86400000000000}
Calculate signed duration (can be negative):
root.duration_ns = this.end_time.ts_sub(this.start_time)
# In: {"start_time":"2020-08-14T10:00:00Z","end_time":"2020-08-14T11:30:00Z"}
# Out: {"duration_ns":5400000000000}
ts_sub_iso8601
Subtracts an ISO 8601 duration from a timestamp with calendar-aware precision for years, months, and days. Useful when you need to subtract durations that account for variable month lengths or leap years.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
Duration in ISO 8601 format (e.g., "P1Y2M3D" for 1 year, 2 months, 3 days) |
Examples
Subtract one year from a timestamp:
root.last_year = this.created_at.ts_sub_iso8601("P1Y")
# In: {"created_at":"2020-08-14T05:54:23Z"}
# Out: {"last_year":"2019-08-14T05:54:23Z"}
Subtract a complex duration with multiple units:
root.past_date = this.created_at.ts_sub_iso8601("P1Y2M3DT4H5M6S")
# In: {"created_at":"2021-03-04T04:05:06Z"}
# Out: {"past_date":"2020-01-01T00:00:00Z"}
ts_tz
Converts a timestamp to a different timezone while preserving the moment in time. Accepts unix timestamps (seconds with optional decimal precision) or RFC 3339 formatted strings.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The timezone to change to. Use "UTC" for UTC, "Local" for local timezone, or an IANA Time Zone database location name like "America/New_York". |
Examples
Convert timestamp to UTC timezone:
root.created_at_utc = this.created_at.ts_tz("UTC")
# In: {"created_at":"2021-02-03T17:05:06+01:00"}
# Out: {"created_at_utc":"2021-02-03T16:05:06Z"}
Convert timestamp to a specific timezone:
root.created_at_ny = this.created_at.ts_tz("America/New_York")
# In: {"created_at":"2021-02-03T16:05:06Z"}
# Out: {"created_at_ny":"2021-02-03T11:05:06-05:00"}
ts_unix
Converts a timestamp to a unix timestamp (seconds since epoch). Accepts unix timestamps or RFC 3339 strings. Returns an integer representing seconds.
Examples
Convert RFC 3339 timestamp to unix seconds:
root.created_at_unix = this.created_at.ts_unix()
# In: {"created_at":"2009-11-10T23:00:00Z"}
# Out: {"created_at_unix":1257894000}
Unix timestamp passthrough returns same value:
root.timestamp = this.ts.ts_unix()
# In: {"ts":1257894000}
# Out: {"timestamp":1257894000}
ts_unix_micro
Converts a timestamp to a unix timestamp with microsecond precision (microseconds since epoch). Accepts unix timestamps or RFC 3339 strings. Returns an integer representing microseconds.
Examples
Convert timestamp to microseconds since epoch:
root.created_at_unix = this.created_at.ts_unix_micro()
# In: {"created_at":"2009-11-10T23:00:00Z"}
# Out: {"created_at_unix":1257894000000000}
Preserve microsecond precision from timestamp:
root.precise_time = this.timestamp.ts_unix_micro()
# In: {"timestamp":"2020-08-14T11:45:26.123456Z"}
# Out: {"precise_time":1597405526123456}
ts_unix_milli
Converts a timestamp to a unix timestamp with millisecond precision (milliseconds since epoch). Accepts unix timestamps or RFC 3339 strings. Returns an integer representing milliseconds.
Examples
Convert timestamp to milliseconds since epoch:
root.created_at_unix = this.created_at.ts_unix_milli()
# In: {"created_at":"2009-11-10T23:00:00Z"}
# Out: {"created_at_unix":1257894000000}
Useful for JavaScript timestamp compatibility:
root.js_timestamp = this.event_time.ts_unix_milli()
# In: {"event_time":"2020-08-14T11:45:26.123Z"}
# Out: {"js_timestamp":1597405526123}
ts_unix_nano
Converts a timestamp to a unix timestamp with nanosecond precision (nanoseconds since epoch). Accepts unix timestamps or RFC 3339 strings. Returns an integer representing nanoseconds.
Examples
Convert timestamp to nanoseconds since epoch:
root.created_at_unix = this.created_at.ts_unix_nano()
# In: {"created_at":"2009-11-10T23:00:00Z"}
# Out: {"created_at_unix":1257894000000000000}
Preserve full nanosecond precision:
root.precise_time = this.timestamp.ts_unix_nano()
# In: {"timestamp":"2020-08-14T11:45:26.123456789Z"}
# Out: {"precise_time":1597405526123456789}
Type coercion
bool
Converts a value to a boolean with optional fallback.
not_empty
Ensures a value is not empty.
Examples
root.a = this.a.not_empty()
# In: {"a":"foo"}
# Out: {"a":"foo"}
# In: {"a":""}
# Out: Error("failed assignment (line 1): field `this.a`: string value is empty")
# In: {"a":["foo","bar"]}
# Out: {"a":["foo","bar"]}
# In: {"a":[]}
# Out: Error("failed assignment (line 1): field `this.a`: array value is empty")
# In: {"a":{"b":"foo","c":"bar"}}
# Out: {"a":{"b":"foo","c":"bar"}}
# In: {"a":{}}
# Out: Error("failed assignment (line 1): field `this.a`: object value is empty")
number
Converts a value to a number with optional fallback.
timestamp
Converts a value to a timestamp with optional fallback.
type
Returns the type of a value as a string.
Examples
root.bar_type = this.bar.type()
root.foo_type = this.foo.type()
# In: {"bar":10,"foo":"is a string"}
# Out: {"bar_type":"number","foo_type":"string"}
root.type = this.type()
# In: "foobar"
# Out: {"type":"string"}
# In: 666
# Out: {"type":"number"}
# In: false
# Out: {"type":"bool"}
# In: ["foo", "bar"]
# Out: {"type":"array"}
# In: {"foo": "bar"}
# Out: {"type":"object"}
# In: null
# Out: {"type":"null"}
root.type = content().type()
# In: foobar
# Out: {"type":"bytes"}
root.type = this.ts_parse("2006-01-02").type()
# In: "2022-06-06"
# Out: {"type":"timestamp"}
Deprecated
format_timestamp
|
This method is deprecated and will be removed in a future version. |
Formats a timestamp as a string using Go’s reference time format. Defaults to RFC 3339 if no format specified. The format uses "Mon Jan 2 15:04:05 -0700 MST 2006" as a reference. Accepts unix timestamps (with decimal precision) or RFC 3339 strings. Use ts_strftime for strftime-style formats.
format_timestamp_strftime
|
This method is deprecated and will be removed in a future version. |
Formats a timestamp as a string using strptime format specifiers (like %Y, %m, %d). Accepts unix timestamps (with decimal precision) or RFC 3339 strings. Supports %f for microseconds. Use ts_format for Go-style reference time formats.
format_timestamp_unix
|
This method is deprecated and will be removed in a future version. |
Converts a timestamp to a unix timestamp (seconds since epoch). Accepts unix timestamps or RFC 3339 strings. Returns an integer representing seconds.
format_timestamp_unix_micro
|
This method is deprecated and will be removed in a future version. |
Converts a timestamp to a unix timestamp with microsecond precision (microseconds since epoch). Accepts unix timestamps or RFC 3339 strings. Returns an integer representing microseconds.
format_timestamp_unix_milli
|
This method is deprecated and will be removed in a future version. |
Converts a timestamp to a unix timestamp with millisecond precision (milliseconds since epoch). Accepts unix timestamps or RFC 3339 strings. Returns an integer representing milliseconds.
format_timestamp_unix_nano
|
This method is deprecated and will be removed in a future version. |
Converts a timestamp to a unix timestamp with nanosecond precision (nanoseconds since epoch). Accepts unix timestamps or RFC 3339 strings. Returns an integer representing nanoseconds.
parse_timestamp
|
This method is deprecated and will be removed in a future version. |
Parses a timestamp string using Go’s reference time format and outputs a timestamp object. The format uses "Mon Jan 2 15:04:05 -0700 MST 2006" as a reference - show how this reference time would appear in your format. Use ts_strptime for strftime-style formats instead.