Bloblang Methods
Methods provide most of the power in Bloblang as they allow you to augment values and can be added to any expression (including other methods):
root.doc.id = this.thing.id.string().catch(uuid_v4())
root.doc.reduced_nums = this.thing.nums.map_each(num -> if num < 10 {
deleted()
} else {
num - 10
})
root.has_good_taste = ["pikachu","mewtwo","magmar"].contains(this.user.fav_pokemon)
# In: {"thing":{"id":123,"nums":[5,12,18,7,25]},"user":{"fav_pokemon":"pikachu"}}
Methods support both named and nameless style arguments:
root.foo_one = this.(bar | baz).trim().replace_all(old: "dog", new: "cat")
root.foo_two = this.(bar | baz).trim().replace_all("dog", "cat")
# In: {"bar":" I love my dog "}
General
apply
Apply a declared mapping to a target value.
Examples
map thing {
root.inner = this.first
}
root.foo = this.doc.apply("thing")
# In: {"doc":{"first":"hello world"}}
# Out: {"foo":{"inner":"hello world"}}
map create_foo {
root.name = "a foo"
root.purpose = "to be a foo"
}
root = this
root.foo = null.apply("create_foo")
# In: {"id":"1234"}
# Out: {"foo":{"name":"a foo","purpose":"to be a foo"},"id":"1234"}
catch
If the result of a target query fails (due to incorrect types, failed parsing, etc) the argument is returned instead.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
A value to yield, or query to execute, if the target query fails. |
Examples
root.doc.id = this.thing.id.string().catch(uuid_v4())
The fallback argument can be a mapping, allowing you to capture the error string and yield structured data back:
root.url = this.url.parse_url().catch(err -> {"error":err,"input":this.url})
# In: {"url":"invalid %&# url"}
# Out: {"url":{"error":"field `this.url`: parse \"invalid %&\": invalid URL escape \"%&\"","input":"invalid %&# url"}}
When the input document is not structured attempting to reference structured fields with this will result in an error. Therefore, a convenient way to delete non-structured data is with a catch:
root = this.catch(deleted())
# In: {"doc":{"foo":"bar"}}
# Out: {"doc":{"foo":"bar"}}
# In: not structured data
# Out: <Message deleted>
from
Modifies a target query such that certain functions are executed from the perspective of another message in the batch. This allows you to mutate events based on the contents of other messages. Functions that support this behavior are content, json and meta.
from_all
Modifies a target query such that certain functions are executed from the perspective of each message in the batch, and returns the set of results as an array. Functions that support this behavior are content, json and meta.
map
Executes a query on the target value, allowing you to transform or extract data from the current context.
not
Returns the logical NOT (negation) of a boolean value. Converts true to false and false to true.
or
If the result of the target query fails or resolves to null, returns the argument instead. This is an explicit method alternative to the coalesce pipe operator |.
Encoding and encryption
compress
Compresses a string or byte array using the specified compression algorithm. Returns compressed data as bytes. Useful for reducing payload size before transmission or storage.
Introduced in version 4.12.0.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The compression algorithm: |
|
|
Compression level (default: -1 for default compression). Higher values increase compression ratio but use more CPU. Range and effect varies by algorithm. Default: |
Examples
Compress and encode for safe transmission:
root.compressed = content().bytes().compress("gzip").encode("base64")
# In: {"message":"hello world I love space"}
# Out: {"compressed":"H4sIAAAJbogA/wAmANn/eyJtZXNzYWdlIjoiaGVsbG8gd29ybGQgSSBsb3ZlIHNwYWNlIn0DAHEvdwomAAAA"}
Compare compression ratios across algorithms:
root.original_size = content().length()
root.gzip_size = content().compress("gzip").length()
root.lz4_size = content().compress("lz4").length()
# In: The quick brown fox jumps over the lazy dog. The quick brown fox jumps over the lazy dog.
# Out: {"gzip_size":114,"lz4_size":85,"original_size":89}
decode
Decodes an encoded string target according to a chosen scheme and returns the result as a byte array. When mapping the result to a JSON field the value should be cast to a string using the method string, or encoded using the method encode, otherwise it will be base64 encoded by default.
Available schemes are: base64, base64url (RFC 4648 with padding characters), base64rawurl (RFC 4648 without padding characters), hex, ascii85.
decompress
Decompresses a byte array using the specified decompression algorithm. Returns decompressed data as bytes. Use with data that was previously compressed using the corresponding algorithm.
Introduced in version 4.12.0.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The decompression algorithm: |
Examples
Decompress base64-encoded compressed data:
root = this.compressed.decode("base64").decompress("gzip")
# In: {"compressed":"H4sIAN12MWkAA8tIzcnJVyjPL8pJUfBUyMkvS1UoLkhMTgUAQpDxbxgAAAA="}
# Out: hello world I love space
Convert decompressed bytes to string for JSON output:
root.message = this.compressed.decode("base64").decompress("gzip").string()
# In: {"compressed":"H4sIAN12MWkAA8tIzcnJVyjPL8pJUfBUyMkvS1UoLkhMTgUAQpDxbxgAAAA="}
# Out: {"message":"hello world I love space"}
decrypt_aes
Decrypts an encrypted string or byte array target according to a chosen AES encryption method and returns the result as a byte array. The algorithms require a key and an initialization vector / nonce. Available schemes are: ctr, gcm, ofb, cbc.
encode
Encodes a string or byte array target according to a chosen scheme and returns a string result. Available schemes are: base64, base64url (RFC 4648 with padding characters), base64rawurl (RFC 4648 without padding characters), hex, ascii85.
encrypt_aes
Encrypts a string or byte array target according to a chosen AES encryption method and returns a string result. The algorithms require a key and an initialization vector / nonce. Available schemes are: ctr, gcm, ofb, cbc.
hash
Hashes a string or byte array according to a chosen algorithm and returns the result as a byte array. When mapping the result to a JSON field the value should be cast to a string using the method string, or encoded using the method encode, otherwise it will be base64 encoded by default.
Available algorithms are: hmac_sha1, hmac_sha256, hmac_sha512, md5, sha1, sha256, sha512, sha3_256, sha3_512, xxhash64, crc32, fnv32.
The following algorithms require a key, which is specified as a second argument: hmac_sha1, hmac_sha256, hmac_sha512.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The hashing algorithm to use. |
|
|
An optional key to use. |
|
|
An optional polynomial key to use when selecting the |
Examples
root.h1 = this.value.hash("sha1").encode("hex")
root.h2 = this.value.hash("hmac_sha1","static-key").encode("hex")
# In: {"value":"hello world"}
# Out: {"h1":"2aae6c35c94fcfb415dbe95f408b9ce91ee846ed","h2":"d87e5f068fa08fe90bb95bc7c8344cb809179d76"}
The crc32 algorithm supports options for the polynomial:
root.h1 = this.value.hash(algorithm: "crc32", polynomial: "Castagnoli").encode("hex")
root.h2 = this.value.hash(algorithm: "crc32", polynomial: "Koopman").encode("hex")
# In: {"value":"hello world"}
# Out: {"h1":"c99465aa","h2":"df373d3c"}
GeoIP
geoip_anonymous_ip
Looks up an IP address against a MaxMind database file and, if found, returns an object describing the anonymous IP associated with it.
geoip_asn
Looks up an IP address against a MaxMind database file and, if found, returns an object describing the ASN associated with it.
geoip_city
Looks up an IP address against a MaxMind database file and, if found, returns an object describing the city associated with it.
geoip_connection_type
Looks up an IP address against a MaxMind database file and, if found, returns an object describing the connection type associated with it.
geoip_country
Looks up an IP address against a MaxMind database file and, if found, returns an object describing the country associated with it.
geoip_domain
Looks up an IP address against a MaxMind database file and, if found, returns an object describing the domain associated with it.
geoip_enterprise
Looks up an IP address against a MaxMind database file and, if found, returns an object describing the enterprise associated with it.
geoip_isp
Looks up an IP address against a MaxMind database file and, if found, returns an object describing the ISP associated with it.
JSON web tokens
parse_jwt_es256
Parses a claims object from a JWT string encoded with ES256. This method does not validate JWT claims.
Introduced in version 4.20.0.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The ES256 secret that was used for signing the token. |
Examples
root.claims = this.signed.parse_jwt_es256("""-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEGtLqIBePHmIhQcf0JLgc+F/4W/oI
dp0Gta53G35VerNDgUUXmp78J2kfh4qLdh0XtmOMI587tCaqjvDAXfs//w==
-----END PUBLIC KEY-----""")
# In: {"signed":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.GIRajP9JJbpTlqSCdNEz4qpQkRvzX4Q51YnTwVyxLDM9tKjR_a8ggHWn9CWj7KG0x8J56OWtmUxn112SRTZVhQ"}
# Out: {"claims":{"iat":1516239022,"mood":"Disdainful","sub":"1234567890"}}
parse_jwt_es384
Parses a claims object from a JWT string encoded with ES384. This method does not validate JWT claims.
Introduced in version 4.20.0.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The ES384 secret that was used for signing the token. |
Examples
root.claims = this.signed.parse_jwt_es384("""-----BEGIN PUBLIC KEY-----
MHYwEAYHKoZIzj0CAQYFK4EEACIDYgAERoz74/B6SwmLhs8X7CWhnrWyRrB13AuU
8OYeqy0qHRu9JWNw8NIavqpTmu6XPT4xcFanYjq8FbeuM11eq06C52mNmS4LLwzA
2imlFEgn85bvJoC3bnkuq4mQjwt9VxdH
-----END PUBLIC KEY-----""")
# In: {"signed":"eyJhbGciOiJFUzM4NCIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.H2HBSlrvQBaov2tdreGonbBexxtQB-xzaPL4-tNQZ6TVh7VH8VBcSwcWHYa1lBAHmdsKOFcB2Wk0SB7QWeGT3ptSgr-_EhDMaZ8bA5spgdpq5DsKfaKHrd7DbbQlmxNq"}
# Out: {"claims":{"iat":1516239022,"mood":"Disdainful","sub":"1234567890"}}
parse_jwt_es512
Parses a claims object from a JWT string encoded with ES512. This method does not validate JWT claims.
Introduced in version 4.20.0.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The ES512 secret that was used for signing the token. |
Examples
root.claims = this.signed.parse_jwt_es512("""-----BEGIN PUBLIC KEY-----
MIGbMBAGByqGSM49AgEGBSuBBAAjA4GGAAQAkHLdts9P56fFkyhpYQ31M/Stwt3w
vpaxhlfudxnXgTO1IP4RQRgryRxZ19EUzhvWDcG3GQIckoNMY5PelsnCGnIBT2Xh
9NQkjWF5K6xS4upFsbGSAwQ+GIyyk5IPJ2LHgOyMSCVh5gRZXV3CZLzXujx/umC9
UeYyTt05zRRWuD+p5bY=
-----END PUBLIC KEY-----""")
# In: {"signed":"eyJhbGciOiJFUzUxMiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.ACrpLuU7TKpAnncDCpN9m85nkL55MJ45NFOBl6-nEXmNT1eIxWjiP4pwWVbFH9et_BgN14119jbL_KqEJInPYc9nAXC6dDLq0aBU-dalvNl4-O5YWpP43-Y-TBGAsWnbMTrchILJ4-AEiICe73Ck5yWPleKg9c3LtkEFWfGs7BoPRguZ"}
# Out: {"claims":{"iat":1516239022,"mood":"Disdainful","sub":"1234567890"}}
parse_jwt_hs256
Parses a claims object from a JWT string encoded with HS256. This method does not validate JWT claims.
Introduced in version 4.12.0.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The HS256 secret that was used for signing the token. |
Examples
root.claims = this.signed.parse_jwt_hs256("""dont-tell-anyone""")
# In: {"signed":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.YwXOM8v3gHVWcQRRRQc_zDlhmLnM62fwhFYGpiA0J1A"}
# Out: {"claims":{"iat":1516239022,"mood":"Disdainful","sub":"1234567890"}}
parse_jwt_hs384
Parses a claims object from a JWT string encoded with HS384. This method does not validate JWT claims.
Introduced in version 4.12.0.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The HS384 secret that was used for signing the token. |
Examples
root.claims = this.signed.parse_jwt_hs384("""dont-tell-anyone""")
# In: {"signed":"eyJhbGciOiJIUzM4NCIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.2Y8rf_ijwN4t8hOGGViON_GrirLkCQVbCOuax6EoZ3nluX0tCGezcJxbctlIfsQ2"}
# Out: {"claims":{"iat":1516239022,"mood":"Disdainful","sub":"1234567890"}}
parse_jwt_hs512
Parses a claims object from a JWT string encoded with HS512. This method does not validate JWT claims.
Introduced in version 4.12.0.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The HS512 secret that was used for signing the token. |
Examples
root.claims = this.signed.parse_jwt_hs512("""dont-tell-anyone""")
# In: {"signed":"eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.utRb0urG6LGGyranZJVo5Dk0Fns1QNcSUYPN0TObQ-YzsGGB8jrxHwM5NAJccjJZzKectEUqmmKCaETZvuX4Fg"}
# Out: {"claims":{"iat":1516239022,"mood":"Disdainful","sub":"1234567890"}}
parse_jwt_rs256
Parses a claims object from a JWT string encoded with RS256. This method does not validate JWT claims.
Introduced in version 4.20.0.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The RS256 secret that was used for signing the token. |
Examples
root.claims = this.signed.parse_jwt_rs256("""-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAs/ibN8r68pLMR6gRzg4S
8v8l6Q7yi8qURjkEbcNeM1rkokC7xh0I4JVTwxYSVv/JIW8qJdyspl5NIfuAVi32
WfKvSAs+NIs+DMsNPYw3yuQals4AX8hith1YDvYpr8SD44jxhz/DR9lYKZFGhXGB
+7NqQ7vpTWp3BceLYocazWJgusZt7CgecIq57ycM5hjM93BvlrUJ8nQ1a46wfL/8
Cy4P0et70hzZrsjjN41KFhKY0iUwlyU41yEiDHvHDDsTMBxAZosWjSREGfJL6Mfp
XOInTHs/Gg6DZMkbxjQu6L06EdJ+Q/NwglJdAXM7Zo9rNELqRig6DdvG5JesdMsO
+QIDAQAB
-----END PUBLIC KEY-----""")
# In: {"signed":"eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.b0lH3jEupZZ4zoaly4Y_GCvu94HH6UKdKY96zfGNsIkPZpQLHIkZ7jMWlLlNOAd8qXlsBGP_i8H2qCKI4zlWJBGyPZgxXDzNRPVrTDfFpn4t4nBcA1WK2-ntXP3ehQxsaHcQU8Z_nsogId7Pme5iJRnoHWEnWtbwz5DLSXL3ZZNnRdrHM9MdI7QSDz9mojKDCaMpGN9sG7Xl-tGdBp1XzXuUOzG8S03mtZ1IgVR1uiBL2N6oohHIAunk8DIAmNWI-zgycTgzUGU7mvPkKH43qO8Ua1-13tCUBKKa8VxcotZ67Mxm1QAvBGoDnTKwWMwghLzs6d6WViXQg6eWlJcpBA"}
# Out: {"claims":{"iat":1516239022,"mood":"Disdainful","sub":"1234567890"}}
parse_jwt_rs384
Parses a claims object from a JWT string encoded with RS384. This method does not validate JWT claims.
Introduced in version 4.20.0.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The RS384 secret that was used for signing the token. |
Examples
root.claims = this.signed.parse_jwt_rs384("""-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAs/ibN8r68pLMR6gRzg4S
8v8l6Q7yi8qURjkEbcNeM1rkokC7xh0I4JVTwxYSVv/JIW8qJdyspl5NIfuAVi32
WfKvSAs+NIs+DMsNPYw3yuQals4AX8hith1YDvYpr8SD44jxhz/DR9lYKZFGhXGB
+7NqQ7vpTWp3BceLYocazWJgusZt7CgecIq57ycM5hjM93BvlrUJ8nQ1a46wfL/8
Cy4P0et70hzZrsjjN41KFhKY0iUwlyU41yEiDHvHDDsTMBxAZosWjSREGfJL6Mfp
XOInTHs/Gg6DZMkbxjQu6L06EdJ+Q/NwglJdAXM7Zo9rNELqRig6DdvG5JesdMsO
+QIDAQAB
-----END PUBLIC KEY-----""")
# In: {"signed":"eyJhbGciOiJSUzM4NCIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.orcXYBcjVE5DU7mvq4KKWFfNdXR4nEY_xupzWoETRpYmQZIozlZnM_nHxEk2dySvpXlAzVm7kgOPK2RFtGlOVaNRIa3x-pMMr-bhZTno4L8Hl4sYxOks3bWtjK7wql4uqUbqThSJB12psAXw2-S-I_FMngOPGIn4jDT9b802ottJSvTpXcy0-eKTjrV2PSkRRu-EYJh0CJZW55MNhqlt6kCGhAXfbhNazN3ASX-dmpd_JixyBKphrngr_zRA-FCn_Xf3QQDA-5INopb4Yp5QiJ7UxVqQEKI80X_JvJqz9WE1qiAw8pq5-xTen1t7zTP-HT1NbbD3kltcNa3G8acmNg"}
# Out: {"claims":{"iat":1516239022,"mood":"Disdainful","sub":"1234567890"}}
parse_jwt_rs512
Parses a claims object from a JWT string encoded with RS512. This method does not validate JWT claims.
Introduced in version 4.20.0.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The RS512 secret that was used for signing the token. |
Examples
root.claims = this.signed.parse_jwt_rs512("""-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAs/ibN8r68pLMR6gRzg4S
8v8l6Q7yi8qURjkEbcNeM1rkokC7xh0I4JVTwxYSVv/JIW8qJdyspl5NIfuAVi32
WfKvSAs+NIs+DMsNPYw3yuQals4AX8hith1YDvYpr8SD44jxhz/DR9lYKZFGhXGB
+7NqQ7vpTWp3BceLYocazWJgusZt7CgecIq57ycM5hjM93BvlrUJ8nQ1a46wfL/8
Cy4P0et70hzZrsjjN41KFhKY0iUwlyU41yEiDHvHDDsTMBxAZosWjSREGfJL6Mfp
XOInTHs/Gg6DZMkbxjQu6L06EdJ+Q/NwglJdAXM7Zo9rNELqRig6DdvG5JesdMsO
+QIDAQAB
-----END PUBLIC KEY-----""")
# In: {"signed":"eyJhbGciOiJSUzUxMiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.rsMp_X5HMrUqKnZJIxo27aAoscovRA6SSQYR9rq7pifIj0YHXxMyNyOBDGnvVALHKTi25VUGHpfNUW0VVMmae0A4t_ObNU6hVZHguWvetKZZq4FZpW1lgWHCMqgPGwT5_uOqwYCH6r8tJuZT3pqXeL0CY4putb1AN2w6CVp620nh3l8d3XWb4jaifycd_4CEVCqHuWDmohfug4VhmoVKlIXZkYoAQowgHlozATDssBSWdYtv107Wd2AzEoiXPu6e3pflsuXULlyqQnS4ELEKPYThFLafh1NqvZDPddqozcPZ-iODBW-xf3A4DYDdivnMYLrh73AZOGHexxu8ay6nDA"}
# Out: {"claims":{"iat":1516239022,"mood":"Disdainful","sub":"1234567890"}}
sign_jwt_es256
Hash and sign an object representing JSON Web Token (JWT) claims using ES256.
Introduced in version 4.20.0.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The secret to use for signing the token. |
|
|
Optional object of JWT header fields to include in the token. Keys "alg", "typ", "jku", "jwk", "x5u", "x5c", "x5t","x5t#S256" and "crit" will be ignored if provided. |
Examples
root.signed = this.claims.sign_jwt_es256("""-----BEGIN EC PRIVATE KEY-----
... signature data ...
-----END EC PRIVATE KEY-----""")
# In: {"claims":{"sub":"user123"}}
# Out: {"signed":"<signed JWT token>"}
root.signed = this.claims.sign_jwt_es256(signing_secret: """-----BEGIN EC PRIVATE KEY-----
... signature data ...
-----END EC PRIVATE KEY-----""", headers: {"kid": "my-key", "x": "y"})
# In: {"claims":{"sub":"user123"}}
# Out: {"signed":"<signed JWT token>"}
sign_jwt_es384
Hash and sign an object representing JSON Web Token (JWT) claims using ES384.
Introduced in version 4.20.0.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The secret to use for signing the token. |
|
|
Optional object of JWT header fields to include in the token. Keys "alg", "typ", "jku", "jwk", "x5u", "x5c", "x5t","x5t#S256" and "crit" will be ignored if provided. |
Examples
root.signed = this.claims.sign_jwt_es384("""-----BEGIN EC PRIVATE KEY-----
... signature data ...
-----END EC PRIVATE KEY-----""")
# In: {"claims":{"sub":"user123"}}
# Out: {"signed":"<signed JWT token>"}
root.signed = this.claims.sign_jwt_es384(signing_secret: """-----BEGIN EC PRIVATE KEY-----
... signature data ...
-----END EC PRIVATE KEY-----""", headers: {"kid": "my-key", "x": "y"})
# In: {"claims":{"sub":"user123"}}
# Out: {"signed":"<signed JWT token>"}
sign_jwt_es512
Hash and sign an object representing JSON Web Token (JWT) claims using ES512.
Introduced in version 4.20.0.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The secret to use for signing the token. |
|
|
Optional object of JWT header fields to include in the token. Keys "alg", "typ", "jku", "jwk", "x5u", "x5c", "x5t","x5t#S256" and "crit" will be ignored if provided. |
Examples
root.signed = this.claims.sign_jwt_es512("""-----BEGIN EC PRIVATE KEY-----
... signature data ...
-----END EC PRIVATE KEY-----""")
# In: {"claims":{"sub":"user123"}}
# Out: {"signed":"<signed JWT token>"}
root.signed = this.claims.sign_jwt_es512(signing_secret: """-----BEGIN EC PRIVATE KEY-----
... signature data ...
-----END EC PRIVATE KEY-----""", headers: {"kid": "my-key", "x": "y"})
# In: {"claims":{"sub":"user123"}}
# Out: {"signed":"<signed JWT token>"}
sign_jwt_hs256
Hash and sign an object representing JSON Web Token (JWT) claims using HS256.
Introduced in version 4.12.0.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The secret to use for signing the token. |
|
|
Optional object of JWT header fields to include in the token. Keys "alg", "typ", "jku", "jwk", "x5u", "x5c", "x5t","x5t#S256" and "crit" will be ignored if provided. |
Examples
root.signed = this.claims.sign_jwt_hs256("""dont-tell-anyone""")
# In: {"claims":{"sub":"user123"}}
# Out: {"signed":"<signed JWT token>"}
root.signed = this.claims.sign_jwt_hs256(signing_secret: """dont-tell-anyone""", headers: {"kid": "my-key", "x": "y"})
# In: {"claims":{"sub":"user123"}}
# Out: {"signed":"<signed JWT token>"}
sign_jwt_hs384
Hash and sign an object representing JSON Web Token (JWT) claims using HS384.
Introduced in version 4.12.0.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The secret to use for signing the token. |
|
|
Optional object of JWT header fields to include in the token. Keys "alg", "typ", "jku", "jwk", "x5u", "x5c", "x5t","x5t#S256" and "crit" will be ignored if provided. |
Examples
root.signed = this.claims.sign_jwt_hs384("""dont-tell-anyone""")
# In: {"claims":{"sub":"user123"}}
# Out: {"signed":"<signed JWT token>"}
root.signed = this.claims.sign_jwt_hs384(signing_secret: """dont-tell-anyone""", headers: {"kid": "my-key", "x": "y"})
# In: {"claims":{"sub":"user123"}}
# Out: {"signed":"<signed JWT token>"}
sign_jwt_hs512
Hash and sign an object representing JSON Web Token (JWT) claims using HS512.
Introduced in version 4.12.0.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The secret to use for signing the token. |
|
|
Optional object of JWT header fields to include in the token. Keys "alg", "typ", "jku", "jwk", "x5u", "x5c", "x5t","x5t#S256" and "crit" will be ignored if provided. |
Examples
root.signed = this.claims.sign_jwt_hs512("""dont-tell-anyone""")
# In: {"claims":{"sub":"user123"}}
# Out: {"signed":"<signed JWT token>"}
root.signed = this.claims.sign_jwt_hs512(signing_secret: """dont-tell-anyone""", headers: {"kid": "my-key", "x": "y"})
# In: {"claims":{"sub":"user123"}}
# Out: {"signed":"<signed JWT token>"}
sign_jwt_rs256
Hash and sign an object representing JSON Web Token (JWT) claims using RS256.
Introduced in version 4.18.0.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The secret to use for signing the token. |
|
|
Optional object of JWT header fields to include in the token. Keys "alg", "typ", "jku", "jwk", "x5u", "x5c", "x5t","x5t#S256" and "crit" will be ignored if provided. |
Examples
root.signed = this.claims.sign_jwt_rs256("""-----BEGIN RSA PRIVATE KEY-----
... signature data ...
-----END RSA PRIVATE KEY-----""")
# In: {"claims":{"sub":"user123"}}
# Out: {"signed":"<signed JWT token>"}
root.signed = this.claims.sign_jwt_rs256(signing_secret: """-----BEGIN RSA PRIVATE KEY-----
... signature data ...
-----END RSA PRIVATE KEY-----""", headers: {"kid": "my-key", "x": "y"})
# In: {"claims":{"sub":"user123"}}
# Out: {"signed":"<signed JWT token>"}
sign_jwt_rs384
Hash and sign an object representing JSON Web Token (JWT) claims using RS384.
Introduced in version 4.18.0.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The secret to use for signing the token. |
|
|
Optional object of JWT header fields to include in the token. Keys "alg", "typ", "jku", "jwk", "x5u", "x5c", "x5t","x5t#S256" and "crit" will be ignored if provided. |
Examples
root.signed = this.claims.sign_jwt_rs384("""-----BEGIN RSA PRIVATE KEY-----
... signature data ...
-----END RSA PRIVATE KEY-----""")
# In: {"claims":{"sub":"user123"}}
# Out: {"signed":"<signed JWT token>"}
root.signed = this.claims.sign_jwt_rs384(signing_secret: """-----BEGIN RSA PRIVATE KEY-----
... signature data ...
-----END RSA PRIVATE KEY-----""", headers: {"kid": "my-key", "x": "y"})
# In: {"claims":{"sub":"user123"}}
# Out: {"signed":"<signed JWT token>"}
sign_jwt_rs512
Hash and sign an object representing JSON Web Token (JWT) claims using RS512.
Introduced in version 4.18.0.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The secret to use for signing the token. |
|
|
Optional object of JWT header fields to include in the token. Keys "alg", "typ", "jku", "jwk", "x5u", "x5c", "x5t","x5t#S256" and "crit" will be ignored if provided. |
Examples
root.signed = this.claims.sign_jwt_rs512("""-----BEGIN RSA PRIVATE KEY-----
... signature data ...
-----END RSA PRIVATE KEY-----""")
# In: {"claims":{"sub":"user123"}}
# Out: {"signed":"<signed JWT token>"}
root.signed = this.claims.sign_jwt_rs512(signing_secret: """-----BEGIN RSA PRIVATE KEY-----
... signature data ...
-----END RSA PRIVATE KEY-----""", headers: {"kid": "my-key", "x": "y"})
# In: {"claims":{"sub":"user123"}}
# Out: {"signed":"<signed JWT token>"}
Number manipulation
abs
Returns the absolute value of an int64 or float64 number. As a special case, when an integer is provided that is the minimum value it is converted to the maximum value.
bitwise_xor
Performs a bitwise XOR (exclusive OR) operation between the integer and the specified value.
ceil
Rounds a number up to the nearest integer. Returns an integer if the result fits in 64-bit, otherwise returns a float.
float32
Converts a numerical type into a 32-bit floating point number, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver).
If the value is a string then an attempt will be made to parse it as a 32-bit floating point number. Please refer to the strconv.ParseFloat documentation for details regarding the supported formats.
Introduced in version 4.19.0.
float64
Converts a numerical type into a 64-bit floating point number, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver).
If the value is a string then an attempt will be made to parse it as a 64-bit floating point number. Please refer to the strconv.ParseFloat documentation for details regarding the supported formats.
Introduced in version 4.19.0.
floor
Rounds a number down to the nearest integer. Returns an integer if the result fits in 64-bit, otherwise returns a float.
int16
Converts a numerical type into a 16-bit signed integer, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver).
If the value is a string then an attempt will be made to parse it as a 16-bit signed integer. If the target value exceeds the capacity of an integer or contains decimal values then this method will throw an error. In order to convert a floating point number containing decimals first use .round() on the value. Please refer to the strconv.ParseInt documentation for details regarding the supported formats.
int32
Converts a numerical type into a 32-bit signed integer, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver).
If the value is a string then an attempt will be made to parse it as a 32-bit signed integer. If the target value exceeds the capacity of an integer or contains decimal values then this method will throw an error. In order to convert a floating point number containing decimals first use .round() on the value. Please refer to the strconv.ParseInt documentation for details regarding the supported formats.
int64
Converts a numerical type into a 64-bit signed integer, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver).
If the value is a string then an attempt will be made to parse it as a 64-bit signed integer. If the target value exceeds the capacity of an integer or contains decimal values then this method will throw an error. In order to convert a floating point number containing decimals first use .round() on the value. Please refer to the strconv.ParseInt documentation for details regarding the supported formats.
int8
Converts a numerical type into a 8-bit signed integer, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver).
If the value is a string then an attempt will be made to parse it as a 8-bit signed integer. If the target value exceeds the capacity of an integer or contains decimal values then this method will throw an error. In order to convert a floating point number containing decimals first use .round() on the value. Please refer to the strconv.ParseInt documentation for details regarding the supported formats.
max
Returns the largest number from an array. All elements must be numbers and the array cannot be empty.
min
Returns the smallest number from an array. All elements must be numbers and the array cannot be empty.
round
Rounds a number to the nearest integer. Values at .5 round away from zero. Returns an integer if the result fits in 64-bit, otherwise returns a float.
uint16
Converts a numerical type into a 16-bit unsigned integer, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver).
If the value is a string then an attempt will be made to parse it as a 16-bit unsigned integer. If the target value exceeds the capacity of an integer or contains decimal values then this method will throw an error. In order to convert a floating point number containing decimals first use .round() on the value. Please refer to the strconv.ParseInt documentation for details regarding the supported formats.
uint32
Converts a numerical type into a 32-bit unsigned integer, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver).
If the value is a string then an attempt will be made to parse it as a 32-bit unsigned integer. If the target value exceeds the capacity of an integer or contains decimal values then this method will throw an error. In order to convert a floating point number containing decimals first use .round() on the value. Please refer to the strconv.ParseInt documentation for details regarding the supported formats.
uint64
Converts a numerical type into a 64-bit unsigned integer, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver).
If the value is a string then an attempt will be made to parse it as a 64-bit unsigned integer. If the target value exceeds the capacity of an integer or contains decimal values then this method will throw an error. In order to convert a floating point number containing decimals first use .round() on the value. Please refer to the strconv.ParseInt documentation for details regarding the supported formats.
uint8
Converts a numerical type into a 8-bit unsigned integer, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver).
If the value is a string then an attempt will be made to parse it as a 8-bit unsigned integer. If the target value exceeds the capacity of an integer or contains decimal values then this method will throw an error. In order to convert a floating point number containing decimals first use .round() on the value. Please refer to the strconv.ParseInt documentation for details regarding the supported formats.
Object & array manipulation
all
Tests whether all elements in an array satisfy a condition. Returns true only if the query evaluates to true for every element. Returns false for empty arrays.
Examples
root.all_over_21 = this.patrons.all(patron -> patron.age >= 21)
# In: {"patrons":[{"id":"1","age":18},{"id":"2","age":23}]}
# Out: {"all_over_21":false}
# In: {"patrons":[{"id":"1","age":45},{"id":"2","age":23}]}
# Out: {"all_over_21":true}
root.all_positive = this.values.all(v -> v > 0)
# In: {"values":[1,2,3,4,5]}
# Out: {"all_positive":true}
# In: {"values":[1,-2,3,4,5]}
# Out: {"all_positive":false}
any
Tests whether at least one element in an array satisfies a condition. Returns true if the query evaluates to true for any element. Returns false for empty arrays.
Examples
root.any_over_21 = this.patrons.any(patron -> patron.age >= 21)
# In: {"patrons":[{"id":"1","age":18},{"id":"2","age":23}]}
# Out: {"any_over_21":true}
# In: {"patrons":[{"id":"1","age":10},{"id":"2","age":12}]}
# Out: {"any_over_21":false}
root.has_errors = this.results.any(r -> r.status == "error")
# In: {"results":[{"status":"ok"},{"status":"error"},{"status":"ok"}]}
# Out: {"has_errors":true}
# In: {"results":[{"status":"ok"},{"status":"ok"}]}
# Out: {"has_errors":false}
append
Adds one or more elements to the end of an array and returns the new array. The original array is not modified.
assign
Merges two objects or arrays with override behavior. For objects, source values replace destination values on key conflicts. Arrays are concatenated. To preserve both values on conflict, use the merge method instead.
Examples
root = this.foo.assign(this.bar)
# In: {"foo":{"first_name":"fooer","likes":"bars"},"bar":{"second_name":"barer","likes":"foos"}}
# Out: {"first_name":"fooer","likes":"foos","second_name":"barer"}
Override defaults with user settings:
root.config = this.defaults.assign(this.user_settings)
# In: {"defaults":{"timeout":30,"retries":3},"user_settings":{"timeout":60}}
# Out: {"config":{"retries":3,"timeout":60}}
collapse
Flattens a nested structure into a single-level object with dot-notation keys representing the full path to each value. Empty arrays and objects are excluded by default.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
Whether to include empty objects and arrays in the resulting object. Default: |
Examples
root.result = this.collapse()
# In: {"foo":[{"bar":"1"},{"bar":{}},{"bar":"2"},{"bar":[]}]}
# Out: {"result":{"foo.0.bar":"1","foo.2.bar":"2"}}
Set include_empty to true to preserve empty objects and arrays in the output:
root.result = this.collapse(include_empty: true)
# In: {"foo":[{"bar":"1"},{"bar":{}},{"bar":"2"},{"bar":[]}]}
# Out: {"result":{"foo.0.bar":"1","foo.1.bar":{},"foo.2.bar":"2","foo.3.bar":[]}}
contains
Checks whether an array contains an element matching the argument, or an object contains a value matching the argument, and returns a boolean result. Numerical comparisons are made irrespective of the representation type (float versus integer).
Examples
root.has_foo = this.thing.contains("foo")
# In: {"thing":["this","foo","that"]}
# Out: {"has_foo":true}
# In: {"thing":["this","bar","that"]}
# Out: {"has_foo":false}
root.has_bar = this.thing.contains(20)
# In: {"thing":[10.3,20.0,"huh",3]}
# Out: {"has_bar":true}
# In: {"thing":[2,3,40,67]}
# Out: {"has_bar":false}
diff
Compares the current value with another value and returns a detailed changelog describing all differences. The changelog contains operations (create, update, delete) with their paths and values, enabling you to track changes between data versions, implement audit logs, or synchronize data between systems.
Introduced in version 4.25.0.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The value to compare against the current value. Can be any structured data (object or array). |
Examples
Compare two objects to track field changes:
root.changes = this.before.diff(this.after)
# In: {"before":{"name":"Alice","age":30},"after":{"name":"Alice","age":31,"city":"NYC"}}
# Out: {"changes":[{"From":30,"Path":["age"],"To":31,"Type":"update"},{"From":null,"Path":["city"],"To":"NYC","Type":"create"}]}
Detect deletions in configuration changes:
root.changelog = this.old_config.diff(this.new_config)
# In: {"old_config":{"debug":true,"timeout":30},"new_config":{"timeout":60}}
# Out: {"changelog":[{"From":true,"Path":["debug"],"To":null,"Type":"delete"},{"From":30,"Path":["timeout"],"To":60,"Type":"update"}]}
enumerated
Transforms an array into an array of objects with index and value fields, making it easy to access both the position and content of each element.
Examples
root.foo = this.foo.enumerated()
# In: {"foo":["bar","baz"]}
# Out: {"foo":[{"index":0,"value":"bar"},{"index":1,"value":"baz"}]}
Useful for filtering by index position:
root.first_two = this.items.enumerated().filter(item -> item.index < 2).map_each(item -> item.value)
# In: {"items":["a","b","c","d"]}
# Out: {"first_two":["a","b"]}
exists
Checks whether a field exists at the specified dot path within an object. Returns true if the field is present (even if null), false otherwise.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
A dot path to a field. |
Examples
root.result = this.foo.exists("bar.baz")
# In: {"foo":{"bar":{"baz":"yep, I exist"}}}
# Out: {"result":true}
# In: {"foo":{"bar":{}}}
# Out: {"result":false}
# In: {"foo":{}}
# Out: {"result":false}
Also returns true for null values if the field exists:
root.has_field = this.data.exists("optional_field")
# In: {"data":{"optional_field":null}}
# Out: {"has_field":true}
# In: {"data":{}}
# Out: {"has_field":false}
explode
Expands a nested array or object field into multiple documents, distributing elements while preserving the surrounding structure. Useful for denormalizing data.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
A dot path to a field to explode. |
Examples
On arrays
When exploding an array, each element becomes a separate document with the array element replacing the original field:
root = this.explode("value")
# In: {"id":1,"value":["foo","bar","baz"]}
# Out: [{"id":1,"value":"foo"},{"id":1,"value":"bar"},{"id":1,"value":"baz"}]
On objects
When exploding an object, the output keys match the nested object’s keys, with values being the full document where the target field is replaced by each nested value:
root = this.explode("value")
# In: {"id":1,"value":{"foo":2,"bar":[3,4],"baz":{"bev":5}}}
# Out: {"bar":{"id":1,"value":[3,4]},"baz":{"id":1,"value":{"bev":5}},"foo":{"id":1,"value":2}}
filter
Returns a new array or object containing only elements that satisfy the provided condition. Elements for which the query returns true are kept, all others are removed.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
A query to apply to each element, if this query resolves to any value other than a boolean |
Examples
root.new_nums = this.nums.filter(num -> num > 10)
# In: {"nums":[3,11,4,17]}
# Out: {"new_nums":[11,17]}
On objects
When filtering objects, the query receives a context with key and value fields for each entry:
root.new_dict = this.dict.filter(item -> item.value.contains("foo"))
# In: {"dict":{"first":"hello foo","second":"world","third":"this foo is great"}}
# Out: {"new_dict":{"first":"hello foo","third":"this foo is great"}}
find
|
This method is in beta. Its behavior might change in a future release. |
Searches an array for a matching value and returns the index of the first occurrence. Returns -1 if no match is found. Numeric types are compared by value regardless of representation.
find_all
|
This method is in beta. Its behavior might change in a future release. |
Searches an array for all occurrences of a value and returns an array of matching indexes. Returns an empty array if no matches are found. Numeric types are compared by value regardless of representation.
find_all_by
|
This method is in beta. Its behavior might change in a future release. |
Searches an array for all elements that satisfy a condition and returns an array of their indexes. Returns an empty array if no elements match.
Examples
root.index = this.find_all_by(v -> v != "bar")
# In: ["foo", "bar", "baz"]
# Out: {"index":[0,2]}
Find all indexes matching criteria:
root.error_indexes = this.logs.find_all_by(log -> log.level == "error")
# In: {"logs":[{"level":"info"},{"level":"error"},{"level":"warn"},{"level":"error"}]}
# Out: {"error_indexes":[1,3]}
find_by
|
This method is in beta. Its behavior might change in a future release. |
Searches an array for the first element that satisfies a condition and returns its index. Returns -1 if no element matches the query.
Examples
root.index = this.find_by(v -> v != "bar")
# In: ["foo", "bar", "baz"]
# Out: {"index":0}
Find first object matching criteria:
root.first_adult = this.users.find_by(u -> u.age >= 18)
# In: {"users":[{"name":"Alice","age":15},{"name":"Bob","age":22},{"name":"Carol","age":19}]}
# Out: {"first_adult":1}
flatten
Flattens an array by one level, expanding nested arrays into the parent array. Only the first level of nesting is removed.
fold
Reduces an array to a single value by iteratively applying a function. Also known as reduce or aggregate. The query receives an accumulator (tally) and current element (value) for each iteration.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The initial value to start the fold with. For example, an empty object |
|
|
A query to apply for each element. The query is provided an object with two fields; |
Examples
Sum numbers in an array:
root.sum = this.foo.fold(0, item -> item.tally + item.value)
# In: {"foo":[3,8,11]}
# Out: {"sum":22}
Concatenate strings:
root.result = this.foo.fold("", item -> "%v%v".format(item.tally, item.value))
# In: {"foo":["hello ", "world"]}
# Out: {"result":"hello world"}
Merge an array of objects into a single object:
root.smoothie = this.fruits.fold({}, item -> item.tally.merge(item.value))
# In: {"fruits":[{"apple":5},{"banana":3},{"orange":8}]}
# Out: {"smoothie":{"apple":5,"banana":3,"orange":8}}
get
Extract a field value, identified via a dot path, from an object.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
A dot path identifying a field to obtain. |
index
Extract an element from an array by an index. The index can be negative, and if so the element will be selected from the end counting backwards starting from -1. E.g. an index of -1 returns the last element, an index of -2 returns the element before the last, and so on.
Examples
root.last_name = this.names.index(-1)
# In: {"names":["rachel","stevens"]}
# Out: {"last_name":"stevens"}
It is also possible to use this method on byte arrays, in which case the selected element will be returned as an integer:
root.last_byte = this.name.bytes().index(-1)
# In: {"name":"foobar bazson"}
# Out: {"last_byte":110}
join
Concatenates an array of strings into a single string with an optional delimiter between elements. Use for building CSV strings, URLs, or combining text fragments.
json_path
Executes the given JSONPath expression on an object or array and returns the result. The JSONPath expression syntax can be found at https://goessner.net/articles/JsonPath/. For more complex logic, you can use Gval expressions (https://github.com/PaesslerAG/gval).
Introduced in version 4.6.0.
Examples
root.all_names = this.json_path("$..name")
# In: {"name":"alice","foo":{"name":"bob"}}
# Out: {"all_names":["alice","bob"]}
# In: {"thing":["this","bar",{"name":"alice"}]}
# Out: {"all_names":["alice"]}
root.text_objects = this.json_path("$.body[?(@.type=='text')]")
# In: {"body":[{"type":"image","id":"foo"},{"type":"text","id":"bar"}]}
# Out: {"text_objects":[{"id":"bar","type":"text"}]}
json_schema
|
This method is in beta. Its behavior might change in a future release. |
Checks a JSON schema against a value and returns the value if it matches or throws an error if it does not. To load the schema from a file, pass it the result of the file function, which isn’t available in Redpanda Cloud.
key_values
Converts an object into an array of key-value pair objects. Each element has a 'key' field and a 'value' field. Order is not guaranteed unless sorted.
Examples
root.foo_key_values = this.foo.key_values().sort_by(pair -> pair.key)
# In: {"foo":{"bar":1,"baz":2}}
# Out: {"foo_key_values":[{"key":"bar","value":1},{"key":"baz","value":2}]}
Filter object entries by value:
root.large_items = this.items.key_values().filter(pair -> pair.value > 15).map_each(pair -> pair.key)
# In: {"items":{"a":5,"b":15,"c":20,"d":3}}
# Out: {"large_items":["c"]}
map_each
Applies a mapping query to each element of an array or each value in an object. Returns a new collection with the transformed values.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
A query that will be used to map each element. |
map_each_key
Transforms object keys using a query. The query receives each key as a string and must return a new string key. Use this to rename or transform keys while preserving values.
Examples
root.new_dict = this.dict.map_each_key(key -> key.uppercase())
# In: {"dict":{"keya":"hello","keyb":"world"}}
# Out: {"new_dict":{"KEYA":"hello","KEYB":"world"}}
Conditionally transform keys:
root = this.map_each_key(key -> if key.contains("kafka") { "_" + key })
# In: {"amqp_key":"foo","kafka_key":"bar","kafka_topic":"baz"}
# Out: {"_kafka_key":"bar","_kafka_topic":"baz","amqp_key":"foo"}
merge
Combines two objects or arrays. When merging objects, conflicting keys create arrays containing both values. Arrays are concatenated. For key override behavior instead, use the assign method.
Examples
root = this.foo.merge(this.bar)
# In: {"foo":{"first_name":"fooer","likes":"bars"},"bar":{"second_name":"barer","likes":"foos"}}
# Out: {"first_name":"fooer","likes":["bars","foos"],"second_name":"barer"}
Merge arrays:
root.combined = this.list1.merge(this.list2)
# In: {"list1":["a","b"],"list2":["c","d"]}
# Out: {"combined":["a","b","c","d"]}
patch
Applies a changelog (created by the diff method) to the current value, transforming it according to the specified operations. This enables you to synchronize data, replay changes, or implement event sourcing patterns by applying recorded changes to reconstruct state.
Introduced in version 4.25.0.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The changelog array to apply. Should be in the format returned by the diff method, containing Type, Path, From, and To fields for each change. |
Examples
Apply recorded changes to update an object:
root.updated = this.current.patch(this.changelog)
# In: {"current":{"name":"Alice","age":30},"changelog":[{"Type":"update","Path":["age"],"From":30,"To":31},{"Type":"create","Path":["city"],"From":null,"To":"NYC"}]}
# Out: {"updated":{"age":31,"city":"NYC","name":"Alice"}}
Restore previous state by applying inverse changes:
root.restored = this.modified.patch(this.reverse_changelog)
# In: {"modified":{"timeout":60},"reverse_changelog":[{"Type":"create","Path":["debug"],"From":null,"To":true},{"Type":"update","Path":["timeout"],"From":60,"To":30}]}
# Out: {"restored":{"debug":true,"timeout":30}}
slice
Extract a slice from an array by specifying two indices, a low and high bound, which selects a half-open range that includes the first element, but excludes the last one. If the second index is omitted then it defaults to the length of the input sequence.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The low bound, which is the first element of the selection, or if negative selects from the end. |
|
|
An optional high bound. |
Examples
root.beginning = this.value.slice(0, 2)
root.end = this.value.slice(4)
# In: {"value":["foo","bar","baz","buz","bev"]}
# Out: {"beginning":["foo","bar"],"end":["bev"]}
A negative low index can be used, indicating an offset from the end of the sequence. If the low index is greater than the length of the sequence then an empty result is returned:
root.last_chunk = this.value.slice(-2)
root.the_rest = this.value.slice(0, -2)
# In: {"value":["foo","bar","baz","buz","bev"]}
# Out: {"last_chunk":["buz","bev"],"the_rest":["foo","bar","baz"]}
sort
Sorts an array in ascending order. Works with strings and numbers. For custom sorting logic, provide a comparison query that receives 'left' and 'right' elements.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
An optional query that should explicitly compare elements |
Examples
root.sorted = this.foo.sort()
# In: {"foo":["bbb","ccc","aaa"]}
# Out: {"sorted":["aaa","bbb","ccc"]}
Custom comparison for complex objects - return true if left < right:
root.sorted = this.foo.sort(item -> item.left.v < item.right.v)
# In: {"foo":[{"id":"foo","v":"bbb"},{"id":"bar","v":"ccc"},{"id":"baz","v":"aaa"}]}
# Out: {"sorted":[{"id":"baz","v":"aaa"},{"id":"foo","v":"bbb"},{"id":"bar","v":"ccc"}]}
sort_by
Sorts an array by a value extracted from each element using a query. The extracted values determine sort order and must all be strings or numbers.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
A query to apply to each element that yields a value used for sorting. |
Examples
root.sorted = this.foo.sort_by(ele -> ele.id)
# In: {"foo":[{"id":"bbb","message":"bar"},{"id":"aaa","message":"foo"},{"id":"ccc","message":"baz"}]}
# Out: {"sorted":[{"id":"aaa","message":"foo"},{"id":"bbb","message":"bar"},{"id":"ccc","message":"baz"}]}
Sort by numeric field:
root.sorted = this.items.sort_by(item -> item.priority)
# In: {"items":[{"name":"low","priority":3},{"name":"high","priority":1},{"name":"med","priority":2}]}
# Out: {"sorted":[{"name":"high","priority":1},{"name":"med","priority":2},{"name":"low","priority":3}]}
squash
Squashes an array of objects into a single object, where key collisions result in the values being merged (following similar rules as the .merge() method).
Introduced in version 4.6.0.
Examples
root.locations = this.locations.map_each(loc -> {loc.state: [loc.name]}).squash()
# In: {"locations":[{"name":"Seattle","state":"WA"},{"name":"New York","state":"NY"},{"name":"Bellevue","state":"WA"},{"name":"Olympia","state":"WA"}]}
# Out: {"locations":{"NY":["New York"],"WA":["Seattle","Bellevue","Olympia"]}}
unique
Removes duplicate values from an array, keeping the first occurrence of each unique value. Strings and numbers are treated as distinct types ("5" differs from 5).
Parameters
| Name | Type | Description |
|---|---|---|
|
|
An optional query that can be used in order to yield a value for each element to determine uniqueness. |
Examples
root.uniques = this.foo.unique()
# In: {"foo":["a","b","a","c"]}
# Out: {"uniques":["a","b","c"]}
Use a query to determine uniqueness by a field:
root.unique_users = this.users.unique(u -> u.id)
# In: {"users":[{"id":1,"name":"Alice"},{"id":2,"name":"Bob"},{"id":1,"name":"Alice Duplicate"}]}
# Out: {"unique_users":[{"id":1,"name":"Alice"},{"id":2,"name":"Bob"}]}
values
Extracts all values from an object and returns them as an array. Order is not guaranteed unless the result is sorted.
with
Returns an object where all but one or more field path arguments are removed. Each path specifies a specific field to be retained from the input object, allowing for nested fields.
If a key within a nested path does not exist then it is ignored.
Introduced in version 4.10.0.
without
Removes specified fields from an object using dot-notation paths. Returns a new object with the fields removed. Non-existent paths are safely ignored.
Examples
root = this.without("inner.a","inner.c","d")
# In: {"inner":{"a":"first","b":"second","c":"third"},"d":"fourth","e":"fifth"}
# Out: {"e":"fifth","inner":{"b":"second"}}
Remove sensitive fields:
root = this.without("password","ssn","creditCard")
# In: {"username":"alice","password":"secret","email":"alice@example.com","ssn":"123-45-6789"}
# Out: {"email":"alice@example.com","username":"alice"}
Parsing
bloblang
|
This method is in beta. Its behavior might change in a future release. |
Executes an argument Bloblang mapping on the target. This method can be used in order to execute dynamic mappings. Imports and functions that interact with the environment, such as file and env, or that access message information directly, such as content or json, are not enabled for dynamic Bloblang mappings.
format_json
|
This method is in beta. Its behavior might change in a future release. |
Serializes a target value into a pretty-printed JSON byte array (with 4 space indentation by default).
Parameters
| Name | Type | Description |
|---|---|---|
|
|
Indentation string. Each element in a JSON object or array will begin on a new, indented line followed by one or more copies of indent according to the indentation nesting. Default: |
|
|
Disable indentation. Default: |
|
|
Escape problematic HTML characters. Default: |
Examples
root = this.doc.format_json()
# In: {"doc":{"foo":"bar"}}
# Out: {
"foo": "bar"
}
Pass a string to the indent parameter in order to customise the indentation:
root = this.format_json(" ")
# In: {"doc":{"foo":"bar"}}
# Out: {
"doc": {
"foo": "bar"
}
}
Use the .string() method in order to coerce the result into a string:
root.doc = this.doc.format_json().string()
# In: {"doc":{"foo":"bar"}}
# Out: {"doc":"{\n \"foo\": \"bar\"\n}"}
Set the no_indent parameter to true to disable indentation. The result is equivalent to calling bytes():
root = this.doc.format_json(no_indent: true)
# In: {"doc":{"foo":"bar"}}
# Out: {"foo":"bar"}
Escapes problematic HTML characters:
root = this.doc.format_json()
# In: {"doc":{"email":"foo&bar@benthos.dev","name":"foo>bar"}}
# Out: {
"email": "foo\u0026bar@benthos.dev",
"name": "foo\u003ebar"
}
Set the escape_html parameter to false to disable escaping of problematic HTML characters:
root = this.doc.format_json(escape_html: false)
# In: {"doc":{"email":"foo&bar@benthos.dev","name":"foo>bar"}}
# Out: {
"email": "foo&bar@benthos.dev",
"name": "foo>bar"
}
format_msgpack
Serializes structured data into MessagePack binary format. MessagePack is a compact binary serialization that is faster and more space-efficient than JSON, making it ideal for network transmission and storage of structured data. Returns a byte array that can be further encoded as needed.
Examples
Serialize object to MessagePack and encode as hex for transmission:
root = this.format_msgpack().encode("hex")
# In: {"foo":"bar"}
# Out: 81a3666f6fa3626172
Serialize data to MessagePack and base64 encode for embedding in JSON:
root.msgpack_payload = this.data.format_msgpack().encode("base64")
# In: {"data":{"foo":"bar"}}
# Out: {"msgpack_payload":"gaNmb2+jYmFy"}
format_xml
Serializes an object into an XML document. Converts structured data to XML format with support for attributes (prefixed with hyphen), custom indentation, and configurable root element. Returns XML as a byte array.
Introduced in version 4.11.0.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
String to use for each level of indentation (default is 4 spaces). Each nested XML element will be indented by this string. Default: |
|
|
Disable indentation and newlines to produce compact XML on a single line. Default: |
|
|
Custom name for the root XML element. By default, the root element name is derived from the first key in the object. |
Examples
Serialize object to pretty-printed XML with default indentation:
root = this.format_xml()
# In: {"foo":{"bar":{"baz":"foo bar baz"}}}
# Out: <foo>
<bar>
<baz>foo bar baz</baz>
</bar>
</foo>
Create compact XML without indentation for smaller message size:
root = this.format_xml(no_indent: true)
# In: {"foo":{"bar":{"baz":"foo bar baz"}}}
# Out: <foo><bar><baz>foo bar baz</baz></bar></foo>
infer_schema
Attempt to infer the schema of a given value. The resulting schema can then be used as an input to schema conversion and enforcement methods.
Introduced in version 4.62.0.
parse_csv
Attempts to parse a string into an array of objects by following the CSV format described in RFC 4180.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
Whether to reference the first row as a header row. If set to true the output structure for messages will be an object where field keys are determined by the header row. Otherwise, the output will be an array of row arrays. Default: |
|
|
The delimiter to use for splitting values in each record. It must be a single character. Default: |
|
|
If set to |
Examples
Parses CSV data with a header row:
root.orders = this.orders.parse_csv()
# In: {"orders":"foo,bar\nfoo 1,bar 1\nfoo 2,bar 2"}
# Out: {"orders":[{"bar":"bar 1","foo":"foo 1"},{"bar":"bar 2","foo":"foo 2"}]}
Parses CSV data without a header row:
root.orders = this.orders.parse_csv(false)
# In: {"orders":"foo 1,bar 1\nfoo 2,bar 2"}
# Out: {"orders":[["foo 1","bar 1"],["foo 2","bar 2"]]}
Parses CSV data delimited by dots:
root.orders = this.orders.parse_csv(delimiter:".")
# In: {"orders":"foo.bar\nfoo 1.bar 1\nfoo 2.bar 2"}
# Out: {"orders":[{"bar":"bar 1","foo":"foo 1"},{"bar":"bar 2","foo":"foo 2"}]}
Parses CSV data containing a quote in an unquoted field:
root.orders = this.orders.parse_csv(lazy_quotes:true)
# In: {"orders":"foo,bar\nfoo 1,bar 1\nfoo\" \"2,bar\" \"2"}
# Out: {"orders":[{"bar":"bar 1","foo":"foo 1"},{"bar":"bar\" \"2","foo":"foo\" \"2"}]}
parse_form_url_encoded
Attempts to parse a url-encoded query string (from an x-www-form-urlencoded request body) and returns a structured result.
Introduced in version 4.5.0.
parse_json
Attempts to parse a string as a JSON document and returns the result.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
An optional flag that when set makes parsing numbers as json.Number instead of the default float64. |
Examples
root.doc = this.doc.parse_json()
# In: {"doc":"{\"foo\":\"bar\"}"}
# Out: {"doc":{"foo":"bar"}}
root.doc = this.doc.parse_json(use_number: true)
# In: {"doc":"{\"foo\":\"11380878173205700000000000000000000000000000000\"}"}
# Out: {"doc":{"foo":"11380878173205700000000000000000000000000000000"}}
parse_logfmt
Attempts to parse a logfmt encoded string into an object. A logfmt string contains key=value pairs separated by spaces, where values can optionally be quoted.
parse_msgpack
Parses MessagePack binary data into a structured object. MessagePack is an efficient binary serialization format that is more compact than JSON while maintaining similar data structures. Commonly used for high-performance APIs and data interchange between microservices.
Examples
Parse MessagePack data from hex-encoded content:
root = content().decode("hex").parse_msgpack()
# In: 81a3666f6fa3626172
# Out: {"foo":"bar"}
Parse MessagePack from base64-encoded field:
root.decoded = this.msgpack_data.decode("base64").parse_msgpack()
# In: {"msgpack_data":"gaNmb2+jYmFy"}
# Out: {"decoded":{"foo":"bar"}}
parse_parquet
Parses Apache Parquet binary data into an array of objects. Parquet is a columnar storage format optimized for analytics, commonly used with big data systems like Apache Spark, Hive, and cloud data warehouses. Each row in the Parquet file becomes an object in the output array.
Introduced in version 4.4.0.
parse_url
Attempts to parse a URL from a string value, returning a structured result that describes the various facets of the URL. The fields returned within the structured result roughly follow https://pkg.go.dev/net/url#URL, and may be expanded in future in order to present more information.
Examples
root.foo_url = this.foo_url.parse_url()
# In: {"foo_url":"https://docs.redpanda.com/redpanda-connect/guides/bloblang/about/"}
# Out: {"foo_url":{"fragment":"","host":"docs.redpanda.com","opaque":"","path":"/redpanda-connect/guides/bloblang/about/","raw_fragment":"","raw_path":"","raw_query":"","scheme":"https"}}
root.username = this.url.parse_url().user.name | "unknown"
# In: {"url":"amqp://foo:bar@127.0.0.1:5672/"}
# Out: {"username":"foo"}
# In: {"url":"redis://localhost:6379"}
# Out: {"username":"unknown"}
parse_xml
Parses an XML document into a structured object. Converts XML elements to JSON-like objects following these rules:
-
Element attributes are prefixed with a hyphen (e.g.,
-idfor anidattribute) -
Elements with both attributes and text content store the text in a
#textfield -
Repeated elements become arrays
-
XML comments, directives, and processing instructions are ignored
-
Optionally cast numeric and boolean strings to their proper types.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
Whether to automatically cast numeric and boolean string values to their proper types. When false, all values remain as strings. Default: |
Examples
Parse XML document into object structure:
root.doc = this.doc.parse_xml()
# In: {"doc":"<root><title>This is a title</title><content>This is some content</content></root>"}
# Out: {"doc":{"root":{"content":"This is some content","title":"This is a title"}}}
Parse XML with type casting enabled to convert strings to numbers and booleans:
root.doc = this.doc.parse_xml(cast: true)
# In: {"doc":"<root><title>This is a title</title><number id=\"99\">123</number><bool>True</bool></root>"}
# Out: {"doc":{"root":{"bool":true,"number":{"#text":123,"-id":99},"title":"This is a title"}}}
Regular expressions
re_find_all
Finds all matches of a regular expression in a string and returns them as an array. Use for extracting multiple patterns or validating repeating structures.
re_find_all_object
Finds all regex matches and returns an array of objects with named capture groups as keys. Each object represents one match with its captured groups. Use for parsing multiple structured records from text.
Examples
root.matches = this.value.re_find_all_object("a(?P<foo>x*)b")
# In: {"value":"-axxb-ab-"}
# Out: {"matches":[{"0":"axxb","foo":"xx"},{"0":"ab","foo":""}]}
root.matches = this.value.re_find_all_object("(?m)(?P<key>\\w+):\\s+(?P<value>\\w+)$")
# In: {"value":"option1: value1\noption2: value2\noption3: value3"}
# Out: {"matches":[{"0":"option1: value1","key":"option1","value":"value1"},{"0":"option2: value2","key":"option2","value":"value2"},{"0":"option3: value3","key":"option3","value":"value3"}]}
re_find_all_submatch
Finds all regex matches and their capture groups, returning an array of arrays where each inner array contains the full match and captured subgroups. Use for extracting structured data with capture groups.
Examples
root.matches = this.value.re_find_all_submatch("a(x*)b")
# In: {"value":"-axxb-ab-"}
# Out: {"matches":[["axxb","xx"],["ab",""]]}
root.emails = this.text.re_find_all_submatch("(\\w+)@(\\w+\\.\\w+)")
# In: {"text":"Contact: alice@example.com or bob@test.org"}
# Out: {"emails":[["alice@example.com","alice","example.com"],["bob@test.org","bob","test.org"]]}
re_find_object
Finds the first regex match and returns an object with named capture groups as keys (or numeric indices for unnamed groups). The key "0" contains the full match. Use for parsing structured text into fields.
Examples
root.matches = this.value.re_find_object("a(?P<foo>x*)b")
# In: {"value":"-axxb-ab-"}
# Out: {"matches":{"0":"axxb","foo":"xx"}}
root.matches = this.value.re_find_object("(?P<key>\\w+):\\s+(?P<value>\\w+)")
# In: {"value":"option1: value1"}
# Out: {"matches":{"0":"option1: value1","key":"option1","value":"value1"}}
re_match
Tests if a regular expression matches anywhere in a string, returning true or false. Use for validation or conditional routing based on patterns.
re_replace
Replaces all regex matches with a replacement string that can reference capture groups using $1, $2, etc. Use for pattern-based transformations or data reformatting.
SQL
vector
Converts an array of numbers into a vector type suitable for insertion into SQL databases with vector/embedding support. This is commonly used with PostgreSQL’s pgvector extension for storing and querying machine learning embeddings, enabling similarity search and vector operations in your database.
Introduced in version 4.33.0.
String manipulation
capitalize
Converts the first letter of each word in a string to uppercase (title case). Useful for formatting names, titles, and headings.
compare_argon2
Checks whether a string matches a hashed secret using Argon2.
Introduced in version 4.12.0.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The hashed secret to compare with the input. This must be a fully-qualified string which encodes the Argon2 options used to generate the hash. |
Examples
root.match = this.secret.compare_argon2("$argon2id$v=19$m=4096,t=3,p=1$c2FsdHktbWNzYWx0ZmFjZQ$RMUMwgtS32/mbszd+ke4o4Ej1jFpYiUqY6MHWa69X7Y")
# In: {"secret":"there-are-many-blobs-in-the-sea"}
# Out: {"match":true}
root.match = this.secret.compare_argon2("$argon2id$v=19$m=4096,t=3,p=1$c2FsdHktbWNzYWx0ZmFjZQ$RMUMwgtS32/mbszd+ke4o4Ej1jFpYiUqY6MHWa69X7Y")
# In: {"secret":"will-i-ever-find-love"}
# Out: {"match":false}
compare_bcrypt
Checks whether a string matches a hashed secret using bcrypt.
Introduced in version 4.12.0.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The hashed secret value to compare with the input. |
Examples
root.match = this.secret.compare_bcrypt("$2y$10$Dtnt5NNzVtMCOZONT705tOcS8It6krJX8bEjnDJnwxiFKsz1C.3Ay")
# In: {"secret":"there-are-many-blobs-in-the-sea"}
# Out: {"match":true}
root.match = this.secret.compare_bcrypt("$2y$10$Dtnt5NNzVtMCOZONT705tOcS8It6krJX8bEjnDJnwxiFKsz1C.3Ay")
# In: {"secret":"will-i-ever-find-love"}
# Out: {"match":false}
escape_html
Escapes special HTML characters (<, >, &, ', ") to make a string safe for HTML output. Use when embedding untrusted text in HTML to prevent XSS vulnerabilities.
escape_url_path
Encodes a string for safe use in URL path segments using percent-encoding. Unlike escape_url_query, spaces are encoded as %20 instead of +. Use when building URL paths with dynamic segments.
escape_url_query
Encodes a string for safe use in URL query parameters. Converts spaces to + and special characters to percent-encoded values. Use when building URLs with dynamic query parameters.
filepath_join
Combines an array of path components into a single OS-specific file path using the correct separator (/ on Unix, \ on Windows). Use for constructing file paths from components.
filepath_split
Separates a file path into directory and filename components, returning a two-element array [directory, filename]. Use for extracting the filename or directory from a full path.
format
Formats a string using Go’s printf-style formatting with the string as the format template. Supports all Go format verbs (%s, %d, %v, etc.). Use for building formatted strings from dynamic values.
Examples
root.foo = "%s(%v): %v".format(this.name, this.age, this.fingers)
# In: {"name":"lance","age":37,"fingers":13}
# Out: {"foo":"lance(37): 13"}
root.message = "User %s has %v points".format(this.username, this.score)
# In: {"username":"alice","score":100}
# Out: {"message":"User alice has 100 points"}
has_prefix
Tests if a string starts with a specified prefix. Returns true if the string begins with the prefix, false otherwise. Use for conditional logic based on string patterns.
has_suffix
Tests if a string ends with a specified suffix. Returns true if the string ends with the suffix, false otherwise. Use for filtering or routing based on file extensions or string patterns.
index_of
Finds the position of a substring within a string. Returns the zero-based index of the first occurrence, or -1 if not found. Useful for searching and string manipulation.
lowercase
Converts all letters in a string to lowercase. Use for case-insensitive comparisons, normalization, or formatting output.
quote
Wraps a string in double quotes and escapes special characters (newlines, tabs, etc.) using Go escape sequences. Use for generating string literals or preparing strings for JSON-like formats.
repeat
Creates a new string by repeating the input string a specified number of times. Use for generating padding, separators, or test data.
replace
Replaces all occurrences of a substring with another string. Use for text transformation, cleaning data, or normalizing strings.
replace_all
Replaces all occurrences of a substring with another string. Use for text transformation, cleaning data, or normalizing strings.
replace_all_many
Performs multiple find-and-replace operations in sequence using an array of [old, new] pairs. More efficient than chaining multiple replace_all calls. Use for bulk text transformations.
replace_many
Performs multiple find-and-replace operations in sequence using an array of [old, new] pairs. More efficient than chaining multiple replace_all calls. Use for bulk text transformations.
reverse
Reverses the order of characters in a string. Unicode-aware for proper handling of multi-byte characters. Use for creating palindrome checks or reversing text data.
slice
Extract a slice from a string by specifying two indices, a low and high bound, which selects a half-open range that includes the first character, but excludes the last one. If the second index is omitted then it defaults to the length of the input sequence.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The low bound, which is the first element of the selection, or if negative selects from the end. |
|
|
An optional high bound. |
Examples
root.beginning = this.value.slice(0, 2)
root.end = this.value.slice(4)
# In: {"value":"foo bar"}
# Out: {"beginning":"fo","end":"bar"}
A negative low index can be used, indicating an offset from the end of the sequence. If the low index is greater than the length of the sequence then an empty result is returned:
root.last_chunk = this.value.slice(-4)
root.the_rest = this.value.slice(0, -4)
# In: {"value":"foo bar"}
# Out: {"last_chunk":" bar","the_rest":"foo"}
slug
Converts a string into a URL-friendly slug by replacing spaces with hyphens, removing special characters, and converting to lowercase. Supports multiple languages for proper transliteration of non-ASCII characters.
Introduced in version 4.2.0.
Examples
Create a URL-friendly slug from a string with special characters:
root.slug = this.title.slug()
# In: {"title":"Hello World! Welcome to Redpanda Connect"}
# Out: {"slug":"hello-world-welcome-to-redpanda-connect"}
Create a slug preserving French language rules:
root.slug = this.title.slug("fr")
# In: {"title":"Café & Restaurant"}
# Out: {"slug":"cafe-et-restaurant"}
split
Splits a string into an array of substrings using a delimiter. Use for parsing CSV-like data, splitting paths, or breaking text into tokens.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The delimiter to split with. |
|
|
To treat empty substrings as null values. Default: |
Examples
root.new_value = this.value.split(",")
# In: {"value":"foo,bar,baz"}
# Out: {"new_value":["foo","bar","baz"]}
root.new_value = this.value.split(",", true)
# In: {"value":"foo,,qux"}
# Out: {"new_value":["foo",null,"qux"]}
root.words = this.sentence.split(" ")
# In: {"sentence":"hello world from bloblang"}
# Out: {"words":["hello","world","from","bloblang"]}
strip_html
Removes HTML tags from a string, returning only the text content. Useful for extracting plain text from HTML documents, sanitizing user input, or preparing content for text analysis. Optionally preserves specific HTML elements while stripping all others.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
Optional array of HTML element names to preserve (e.g., ["strong", "em", "a"]). All other HTML tags will be removed. |
Examples
Extract plain text from HTML content:
root.plain_text = this.html_content.strip_html()
# In: {"html_content":"<p>Welcome to <strong>Redpanda Connect</strong>!</p>"}
# Out: {"plain_text":"Welcome to Redpanda Connect!"}
Preserve specific HTML elements while removing others:
root.sanitized = this.html.strip_html(["strong", "em"])
# In: {"html":"<div><p>Some <strong>bold</strong> and <em>italic</em> text with a <script>alert('xss')</script></p></div>"}
# Out: {"sanitized":"Some <strong>bold</strong> and <em>italic</em> text with a "}
trim
Removes leading and trailing characters from a string. Without arguments, removes whitespace. With a cutset argument, removes any characters in the cutset. Use for cleaning user input or normalizing strings.
trim_prefix
Removes a specified prefix from the beginning of a string if present. If the string doesn’t start with the prefix, returns the string unchanged. Use for stripping known prefixes from identifiers or paths.
Introduced in version 4.12.0.
trim_suffix
Removes a specified suffix from the end of a string if present. If the string doesn’t end with the suffix, returns the string unchanged. Use for stripping file extensions or known suffixes.
Introduced in version 4.12.0.
unescape_html
Converts HTML entities back to their original characters. Handles named entities (&, <), decimal (á), and hexadecimal (&xE1;) formats. Use for processing HTML content or decoding HTML-escaped data.
unescape_url_path
Decodes URL path percent-encoding, converting %20 to spaces and other percent-encoded characters to their original values. Use for parsing URL path segments.
unescape_url_query
Decodes URL query parameter encoding, converting + to spaces and percent-encoded characters to their original values. Use for parsing URL query parameters.
unicode_segments
Splits text into segments based on Unicode text segmentation rules. Returns an array of strings representing individual graphemes (visual characters), words (including punctuation and whitespace), or sentences. Handles complex Unicode correctly, including emoji with skin tone modifiers and zero-width joiners.
Introduced in version 4.50.0.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
Type of segmentation: "grapheme", "word", or "sentence" |
Examples
Split text into sentences (preserves trailing spaces):
root.sentences = this.text.unicode_segments("sentence")
# In: {"text":"Hello world. How are you?"}
# Out: {"sentences":["Hello world. ","How are you?"]}
Split text into grapheme clusters (handles complex emoji correctly):
root.graphemes = this.emoji.unicode_segments("grapheme")
# In: {"emoji":"👨👩👧👦❤️"}
# Out: {"graphemes":["👨👩👧👦","❤️"]}
unquote
Removes surrounding quotes and interprets escape sequences (\n, \t, etc.) to their literal characters. Use for parsing quoted string literals.
Timestamp manipulation
parse_duration
Parses a Go-style duration string into nanoseconds. A duration string is a signed sequence of decimal numbers with unit suffixes like "300ms", "-1.5h", or "2h45m". Valid units: "ns", "us" (or "µs"), "ms", "s", "m", "h".
parse_duration_iso8601
|
This method is in beta. Its behavior might change in a future release. |
Parses an ISO 8601 duration string into nanoseconds. Format: "P[n]Y[n]M[n]DT[n]H[n]M[n]S" or "P[n]W". Example: "P3Y6M4DT12H30M5S" means 3 years, 6 months, 4 days, 12 hours, 30 minutes, 5 seconds. Supports fractional seconds with full precision (not just one decimal place).
Examples
Parse complex ISO 8601 duration to nanoseconds:
root.delay_for_ns = this.delay_for.parse_duration_iso8601()
# In: {"delay_for":"P3Y6M4DT12H30M5S"}
# Out: {"delay_for_ns":110839937000000000}
Parse hours to seconds:
root.delay_for_s = this.delay_for.parse_duration_iso8601() / 1000000000
# In: {"delay_for":"PT2H"}
# Out: {"delay_for_s":7200}
ts_add_iso8601
|
This method is in beta. Its behavior might change in a future release. |
Adds an ISO 8601 duration to a timestamp with calendar-aware precision for years, months, and days. Useful when you need to add durations that account for variable month lengths or leap years.
Introduced in version 4.14.0.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
Duration in ISO 8601 format (e.g., "P1Y2M3D" for 1 year, 2 months, 3 days) |
Examples
Add one year to a timestamp:
root.next_year = this.created_at.ts_add_iso8601("P1Y")
# In: {"created_at":"2020-08-14T05:54:23Z"}
# Out: {"next_year":"2021-08-14T05:54:23Z"}
Add a complex duration with multiple units:
root.future_date = this.created_at.ts_add_iso8601("P1Y2M3DT4H5M6S")
# In: {"created_at":"2020-01-01T00:00:00Z"}
# Out: {"future_date":"2021-03-04T04:05:06Z"}
ts_format
|
This method is in beta. Its behavior might change in a future release. |
Formats a timestamp as a string using Go’s reference time format. Defaults to RFC 3339 if no format specified. The format uses "Mon Jan 2 15:04:05 -0700 MST 2006" as a reference. Accepts unix timestamps (with decimal precision) or RFC 3339 strings. Use ts_strftime for strftime-style formats.
Introduced in version 4.2.0.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The output format using Go’s reference time. Default: |
|
|
Optional timezone (e.g., 'UTC', 'America/New_York'). Defaults to input timezone or local time for unix timestamps. |
Examples
Format timestamp with custom format:
root.something_at = this.created_at.ts_format("2006-Jan-02 15:04:05")
# In: {"created_at":"2020-08-14T11:50:26.371Z"}
# Out: {"something_at":"2020-Aug-14 11:50:26"}
Format unix timestamp with timezone specification:
root.something_at = this.created_at.ts_format(format: "2006-Jan-02 15:04:05", tz: "UTC")
# In: {"created_at":1597405526}
# Out: {"something_at":"2020-Aug-14 11:45:26"}
ts_parse
|
This method is in beta. Its behavior might change in a future release. |
Parses a timestamp string using Go’s reference time format and outputs a timestamp object. The format uses "Mon Jan 2 15:04:05 -0700 MST 2006" as a reference - show how this reference time would appear in your format. Use ts_strptime for strftime-style formats instead.
Introduced in version 4.2.0.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The format of the input string using Go’s reference time. |
Examples
Parse a date with abbreviated month name:
root.doc.timestamp = this.doc.timestamp.ts_parse("2006-Jan-02")
# In: {"doc":{"timestamp":"2020-Aug-14"}}
# Out: {"doc":{"timestamp":"2020-08-14T00:00:00Z"}}
Parse a custom datetime format:
root.parsed = this.timestamp.ts_parse("Jan 2, 2006 at 3:04pm (MST)")
# In: {"timestamp":"Aug 14, 2020 at 5:54am (UTC)"}
# Out: {"parsed":"2020-08-14T05:54:00Z"}
ts_round
|
This method is in beta. Its behavior might change in a future release. |
Rounds a timestamp to the nearest multiple of the specified duration. Halfway values round up. Accepts unix timestamps (seconds with optional decimal precision) or RFC 3339 formatted strings.
Introduced in version 4.2.0.
Examples
Round timestamp to the nearest hour:
root.created_at_hour = this.created_at.ts_round("1h".parse_duration())
# In: {"created_at":"2020-08-14T05:54:23Z"}
# Out: {"created_at_hour":"2020-08-14T06:00:00Z"}
Round timestamp to the nearest minute:
root.created_at_minute = this.created_at.ts_round("1m".parse_duration())
# In: {"created_at":"2020-08-14T05:54:23Z"}
# Out: {"created_at_minute":"2020-08-14T05:54:00Z"}
ts_strftime
|
This method is in beta. Its behavior might change in a future release. |
Formats a timestamp as a string using strptime format specifiers (like %Y, %m, %d). Accepts unix timestamps (with decimal precision) or RFC 3339 strings. Supports %f for microseconds. Use ts_format for Go-style reference time formats.
Introduced in version 4.2.0.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The output format using strptime specifiers. |
|
|
Optional timezone. Defaults to input timezone or local time for unix timestamps. |
Examples
Format timestamp with strftime specifiers:
root.something_at = this.created_at.ts_strftime("%Y-%b-%d %H:%M:%S")
# In: {"created_at":"2020-08-14T11:50:26.371Z"}
# Out: {"something_at":"2020-Aug-14 11:50:26"}
Format with microseconds using %f directive:
root.something_at = this.created_at.ts_strftime("%Y-%b-%d %H:%M:%S.%f", "UTC")
# In: {"created_at":"2020-08-14T11:50:26.371Z"}
# Out: {"something_at":"2020-Aug-14 11:50:26.371000"}
ts_strptime
|
This method is in beta. Its behavior might change in a future release. |
Parses a timestamp string using strptime format specifiers (like %Y, %m, %d) and outputs a timestamp object. Use ts_parse for Go-style reference time formats instead.
Introduced in version 4.2.0.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The format string using strptime specifiers (e.g., %Y-%m-%d). |
Examples
Parse date with abbreviated month using strptime format:
root.doc.timestamp = this.doc.timestamp.ts_strptime("%Y-%b-%d")
# In: {"doc":{"timestamp":"2020-Aug-14"}}
# Out: {"doc":{"timestamp":"2020-08-14T00:00:00Z"}}
Parse datetime with microseconds using %f directive:
root.doc.timestamp = this.doc.timestamp.ts_strptime("%Y-%b-%d %H:%M:%S.%f")
# In: {"doc":{"timestamp":"2020-Aug-14 11:50:26.371000"}}
# Out: {"doc":{"timestamp":"2020-08-14T11:50:26.371Z"}}
ts_sub
|
This method is in beta. Its behavior might change in a future release. |
Calculates the duration in nanoseconds between two timestamps (t1 - t2). Returns a signed integer: positive if t1 is after t2, negative if t1 is before t2. Use .abs() for absolute duration.
Introduced in version 4.23.0.
Examples
Calculate absolute duration between two timestamps:
root.between = this.started_at.ts_sub("2020-08-14T05:54:23Z").abs()
# In: {"started_at":"2020-08-13T05:54:23Z"}
# Out: {"between":86400000000000}
Calculate signed duration (can be negative):
root.duration_ns = this.end_time.ts_sub(this.start_time)
# In: {"start_time":"2020-08-14T10:00:00Z","end_time":"2020-08-14T11:30:00Z"}
# Out: {"duration_ns":5400000000000}
ts_sub_iso8601
|
This method is in beta. Its behavior might change in a future release. |
Subtracts an ISO 8601 duration from a timestamp with calendar-aware precision for years, months, and days. Useful when you need to subtract durations that account for variable month lengths or leap years.
Introduced in version 4.14.0.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
Duration in ISO 8601 format (e.g., "P1Y2M3D" for 1 year, 2 months, 3 days) |
Examples
Subtract one year from a timestamp:
root.last_year = this.created_at.ts_sub_iso8601("P1Y")
# In: {"created_at":"2020-08-14T05:54:23Z"}
# Out: {"last_year":"2019-08-14T05:54:23Z"}
Subtract a complex duration with multiple units:
root.past_date = this.created_at.ts_sub_iso8601("P1Y2M3DT4H5M6S")
# In: {"created_at":"2021-03-04T04:05:06Z"}
# Out: {"past_date":"2020-01-01T00:00:00Z"}
ts_tz
|
This method is in beta. Its behavior might change in a future release. |
Converts a timestamp to a different timezone while preserving the moment in time. Accepts unix timestamps (seconds with optional decimal precision) or RFC 3339 formatted strings.
Introduced in version 4.3.0.
Parameters
| Name | Type | Description |
|---|---|---|
|
|
The timezone to change to. Use "UTC" for UTC, "Local" for local timezone, or an IANA Time Zone database location name like "America/New_York". |
Examples
Convert timestamp to UTC timezone:
root.created_at_utc = this.created_at.ts_tz("UTC")
# In: {"created_at":"2021-02-03T17:05:06+01:00"}
# Out: {"created_at_utc":"2021-02-03T16:05:06Z"}
Convert timestamp to a specific timezone:
root.created_at_ny = this.created_at.ts_tz("America/New_York")
# In: {"created_at":"2021-02-03T16:05:06Z"}
# Out: {"created_at_ny":"2021-02-03T11:05:06-05:00"}
ts_unix
|
This method is in beta. Its behavior might change in a future release. |
Converts a timestamp to a unix timestamp (seconds since epoch). Accepts unix timestamps or RFC 3339 strings. Returns an integer representing seconds.
Introduced in version 4.2.0.
Examples
Convert RFC 3339 timestamp to unix seconds:
root.created_at_unix = this.created_at.ts_unix()
# In: {"created_at":"2009-11-10T23:00:00Z"}
# Out: {"created_at_unix":1257894000}
Unix timestamp passthrough returns same value:
root.timestamp = this.ts.ts_unix()
# In: {"ts":1257894000}
# Out: {"timestamp":1257894000}
ts_unix_micro
|
This method is in beta. Its behavior might change in a future release. |
Converts a timestamp to a unix timestamp with microsecond precision (microseconds since epoch). Accepts unix timestamps or RFC 3339 strings. Returns an integer representing microseconds.
Introduced in version 4.12.0.
Examples
Convert timestamp to microseconds since epoch:
root.created_at_unix = this.created_at.ts_unix_micro()
# In: {"created_at":"2009-11-10T23:00:00Z"}
# Out: {"created_at_unix":1257894000000000}
Preserve microsecond precision from timestamp:
root.precise_time = this.timestamp.ts_unix_micro()
# In: {"timestamp":"2020-08-14T11:45:26.123456Z"}
# Out: {"precise_time":1597405526123456}
ts_unix_milli
|
This method is in beta. Its behavior might change in a future release. |
Converts a timestamp to a unix timestamp with millisecond precision (milliseconds since epoch). Accepts unix timestamps or RFC 3339 strings. Returns an integer representing milliseconds.
Introduced in version 4.12.0.
Examples
Convert timestamp to milliseconds since epoch:
root.created_at_unix = this.created_at.ts_unix_milli()
# In: {"created_at":"2009-11-10T23:00:00Z"}
# Out: {"created_at_unix":1257894000000}
Useful for JavaScript timestamp compatibility:
root.js_timestamp = this.event_time.ts_unix_milli()
# In: {"event_time":"2020-08-14T11:45:26.123Z"}
# Out: {"js_timestamp":1597405526123}
ts_unix_nano
|
This method is in beta. Its behavior might change in a future release. |
Converts a timestamp to a unix timestamp with nanosecond precision (nanoseconds since epoch). Accepts unix timestamps or RFC 3339 strings. Returns an integer representing nanoseconds.
Introduced in version 4.2.0.
Examples
Convert timestamp to nanoseconds since epoch:
root.created_at_unix = this.created_at.ts_unix_nano()
# In: {"created_at":"2009-11-10T23:00:00Z"}
# Out: {"created_at_unix":1257894000000000000}
Preserve full nanosecond precision:
root.precise_time = this.timestamp.ts_unix_nano()
# In: {"timestamp":"2020-08-14T11:45:26.123456789Z"}
# Out: {"precise_time":1597405526123456789}
Type coercion
array
Return an array containing the target value. If the value is already an array it is unchanged.
bool
Attempt to parse a value into a boolean. An optional argument can be provided, in which case if the value cannot be parsed the argument will be returned instead. If the value is a number then any non-zero value will resolve to true, if the value is a string then any of the following values are considered valid: 1, t, T, TRUE, true, True, 0, f, F, FALSE.
not_empty
Ensures that the given string, array or object value is not empty, and if so returns it, otherwise an error is returned.
Examples
root.a = this.a.not_empty()
# In: {"a":"foo"}
# Out: {"a":"foo"}
# In: {"a":""}
# Out: Error("failed assignment (line 1): field `this.a`: string value is empty")
# In: {"a":["foo","bar"]}
# Out: {"a":["foo","bar"]}
# In: {"a":[]}
# Out: Error("failed assignment (line 1): field `this.a`: array value is empty")
# In: {"a":{"b":"foo","c":"bar"}}
# Out: {"a":{"b":"foo","c":"bar"}}
# In: {"a":{}}
# Out: Error("failed assignment (line 1): field `this.a`: object value is empty")
not_null
Ensures that the given value is not null, and if so returns it, otherwise an error is returned.
number
Attempt to parse a value into a number. An optional argument can be provided, in which case if the value cannot be parsed into a number the argument will be returned instead.
string
Converts any value to its string representation. Numbers, booleans, and objects are converted to strings; existing strings are unchanged. Use for type coercion or creating string representations.
timestamp
Attempt to parse a value into a timestamp. An optional argument can be provided, in which case if the value cannot be parsed into a timestamp the argument will be returned instead.
type
Returns the type of a value as a string, providing one of the following values: string, bytes, number, bool, timestamp, array, object or null.
Examples
root.bar_type = this.bar.type()
root.foo_type = this.foo.type()
# In: {"bar":10,"foo":"is a string"}
# Out: {"bar_type":"number","foo_type":"string"}
root.type = this.type()
# In: "foobar"
# Out: {"type":"string"}
# In: 666
# Out: {"type":"number"}
# In: false
# Out: {"type":"bool"}
# In: ["foo", "bar"]
# Out: {"type":"array"}
# In: {"foo": "bar"}
# Out: {"type":"object"}
# In: null
# Out: {"type":"null"}
root.type = content().type()
# In: foobar
# Out: {"type":"bytes"}
root.type = this.ts_parse("2006-01-02").type()
# In: "2022-06-06"
# Out: {"type":"timestamp"}
Deprecated
format_timestamp
|
This method is deprecated and will be removed in a future version. |
Formats a timestamp as a string using Go’s reference time format. Defaults to RFC 3339 if no format specified. The format uses "Mon Jan 2 15:04:05 -0700 MST 2006" as a reference. Accepts unix timestamps (with decimal precision) or RFC 3339 strings. Use ts_strftime for strftime-style formats.
format_timestamp_strftime
|
This method is deprecated and will be removed in a future version. |
Formats a timestamp as a string using strptime format specifiers (like %Y, %m, %d). Accepts unix timestamps (with decimal precision) or RFC 3339 strings. Supports %f for microseconds. Use ts_format for Go-style reference time formats.
format_timestamp_unix
|
This method is deprecated and will be removed in a future version. |
Converts a timestamp to a unix timestamp (seconds since epoch). Accepts unix timestamps or RFC 3339 strings. Returns an integer representing seconds.
format_timestamp_unix_micro
|
This method is deprecated and will be removed in a future version. |
Converts a timestamp to a unix timestamp with microsecond precision (microseconds since epoch). Accepts unix timestamps or RFC 3339 strings. Returns an integer representing microseconds.
Introduced in version 4.12.0.
format_timestamp_unix_milli
|
This method is deprecated and will be removed in a future version. |
Converts a timestamp to a unix timestamp with millisecond precision (milliseconds since epoch). Accepts unix timestamps or RFC 3339 strings. Returns an integer representing milliseconds.
Introduced in version 4.12.0.
format_timestamp_unix_nano
|
This method is deprecated and will be removed in a future version. |
Converts a timestamp to a unix timestamp with nanosecond precision (nanoseconds since epoch). Accepts unix timestamps or RFC 3339 strings. Returns an integer representing nanoseconds.
parse_timestamp
|
This method is deprecated and will be removed in a future version. |
Parses a timestamp string using Go’s reference time format and outputs a timestamp object. The format uses "Mon Jan 2 15:04:05 -0700 MST 2006" as a reference - show how this reference time would appear in your format. Use ts_strptime for strftime-style formats instead.