VRL example reference

Here you’ll find a comprehensive list of all VRL program examples. These examples demonstrate the breadth of the language and its observability-focused facilities.

Real world examples

Parse Syslog logs

Given this Vector log event...
{
 "message": "\u003c102\u003e1 2020-12-22T15:22:31.111Z vector-user.biz su 2666 ID389 - Something went wrong"
}
...and this VRL program...
. |= parse_syslog!(.message)
...the following log event is output:
{
 "log": {
  "appname": "su",
  "facility": "ntp",
  "hostname": "vector-user.biz",
  "message": "Something went wrong",
  "msgid": "ID389",
  "procid": 2666,
  "severity": "info",
  "timestamp": "2020-12-22T15:22:31.111Z",
  "version": 1
 }
}

Parse key/value (logfmt) logs

Given this Vector log event...
{
 "message": "@timestamp=\"Sun Jan 10 16:47:39 EST 2021\" level=info msg=\"Stopping all fetchers\" tag#production=stopping_fetchers id=ConsumerFetcherManager-1382721708341 module=kafka.consumer.ConsumerFetcherManager"
}
...and this VRL program...
. = parse_key_value!(.message)
...the following log event is output:
{
 "log": {
  "@timestamp": "Sun Jan 10 16:47:39 EST 2021",
  "id": "ConsumerFetcherManager-1382721708341",
  "level": "info",
  "module": "kafka.consumer.ConsumerFetcherManager",
  "msg": "Stopping all fetchers",
  "tag#production": "stopping_fetchers"
 }
}

Parse custom logs

Given this Vector log event...
{
 "message": "2021/01/20 06:39:15 +0000 [error] 17755#17755: *3569904 open() \"/usr/share/nginx/html/test.php\" failed (2: No such file or directory), client: xxx.xxx.xxx.xxx, server: localhost, request: \"GET /test.php HTTP/1.1\", host: \"yyy.yyy.yyy.yyy\""
}
...and this VRL program...
. |= parse_regex!(.message, r'^(?P<timestamp>\d+/\d+/\d+ \d+:\d+:\d+ \+\d+) \[(?P<severity>\w+)\] (?P<pid>\d+)#(?P<tid>\d+):(?: \*(?P<connid>\d+))? (?P<message>.*)$')

# Coerce parsed fields
.timestamp = parse_timestamp(.timestamp, "%Y/%m/%d %H:%M:%S %z") ?? now()
.pid = to_int!(.pid)
.tid = to_int!(.tid)

# Extract structured data
message_parts = split(.message, ", ", limit: 2)
structured = parse_key_value(message_parts[1], key_value_delimiter: ":", field_delimiter: ",") ?? {}
.message = message_parts[0]
. = merge(., structured)
...the following log event is output:
{
 "log": {
  "client": "xxx.xxx.xxx.xxx",
  "connid": "3569904",
  "host": "yyy.yyy.yyy.yyy",
  "message": "open() \"/usr/share/nginx/html/test.php\" failed (2: No such file or directory)",
  "pid": 17755,
  "request": "GET /test.php HTTP/1.1",
  "server": "localhost",
  "severity": "error",
  "tid": 17755,
  "timestamp": "2021-01-20T06:39:15Z"
 }
}

Multiple parsing strategies

Given this Vector log event...
{
 "message": "\u003c102\u003e1 2020-12-22T15:22:31.111Z vector-user.biz su 2666 ID389 - Something went wrong"
}
...and this VRL program...
structured =
  parse_syslog(.message) ??
  parse_common_log(.message) ??
  parse_regex!(.message, r'^(?P<timestamp>\d+/\d+/\d+ \d+:\d+:\d+) \[(?P<severity>\w+)\] (?P<pid>\d+)#(?P<tid>\d+):(?: \*(?P<connid>\d+))? (?P<message>.*)$')
. = merge(., structured)
...the following log event is output:
{
 "log": {
  "appname": "su",
  "facility": "ntp",
  "hostname": "vector-user.biz",
  "message": "Something went wrong",
  "msgid": "ID389",
  "procid": 2666,
  "severity": "info",
  "timestamp": "2020-12-22T15:22:31.111Z",
  "version": 1
 }
}

Modify metric tags

Given this Vector metric event...
{
 "counter": {
  "value": 102
 },
 "kind": "incremental",
 "name": "user_login_total",
 "tags": {
  "email": "vic@vector.dev",
  "host": "my.host.com",
  "instance_id": "abcd1234"
 }
}
...and this VRL program...
.tags.environment = get_env_var!("ENV") # add
.tags.hostname = del(.tags.host) # rename
del(.tags.email)
...the following metric event is output:
{
 "metric": {
  "counter": {
   "value": 102
  },
  "kind": "incremental",
  "name": "user_login_total",
  "tags": {
   "environment": "production",
   "hostname": "my.host.com",
   "instance_id": "abcd1234"
  }
 }
}

Emitting multiple logs from JSON

Given this Vector log event...
{
 "message": "[{\"message\": \"first_log\"}, {\"message\": \"second_log\"}]"
}
...and this VRL program...
. = parse_json!(.message) # sets `.` to an array of objects
...the following log event is output:
[
 {
  "log": {
   "message": "first_log"
  }
 },
 {
  "log": {
   "message": "second_log"
  }
 }
]

Emitting multiple non-object logs from JSON

Given this Vector log event...
{
 "message": "[5, true, \"hello\"]"
}
...and this VRL program...
. = parse_json!(.message) # sets `.` to an array
...the following log event is output:
[
 {
  "log": {
   "message": 5
  }
 },
 {
  "log": {
   "message": true
  }
 },
 {
  "log": {
   "message": "hello"
  }
 }
]

Invalid argument type

Given this Vector log event...
{
 "not_a_string": 1
}
...and this VRL program...
upcase(42)
...you should see this error:
error[E110]: invalid argument type
  ┌─ :1:8
  
1  upcase(42)
          ^^
          
          this expression resolves to the exact type "integer"
          but the parameter "value" expects the exact type "string"
  
  = try: ensuring an appropriate type at runtime
  =
  =     42 = string!(42)
  =     upcase(42)
  =
  = try: coercing to an appropriate type and specifying a default value as a fallback in case coercion fails
  =
  =     42 = to_string(42) ?? "default"
  =     upcase(42)
  =
  = see documentation about error handling at https://errors.vrl.dev/#handling
  = learn more about error code 110 at https://errors.vrl.dev/110
  = see language documentation at https://vrl.dev

Unhandled fallible assignment

Given this Vector log event...
{
 "message": "key1=value1 key2=value2"
}
...and this VRL program...
structured = parse_key_value(.message)
...you should see this error:
error[E103]: unhandled fallible assignment
  ┌─ :1:14
  
1  structured = parse_key_value(.message)
   ------------ ^^^^^^^^^^^^^^^^^^^^^^^^^
               
               this expression is fallible
               update the expression to be infallible
   or change this to an infallible assignment:
   structured, err = parse_key_value(.message)
  
  = see documentation about error handling at https://errors.vrl.dev/#handling
  = learn more about error code 103 at https://errors.vrl.dev/103
  = see language documentation at https://vrl.dev

Array examples

append

Appends each item in the items array to the end of the value array.

Append to an array

Source
append([1, 2], [3, 4])
Return
[1,2,3,4]
Learn more about the append function

push

Adds the item to the end of the value array.

Push an item onto an array

Source
push([1, 2], 3)
Return
[1,2,3]
Learn more about the push function

Codec examples

decode_base64

Decodes the value (a Base64 string) into its original string.

Decode Base64 data (default)

Source
decode_base64!("eW91IGhhdmUgc3VjY2Vzc2Z1bGx5IGRlY29kZWQgbWU=")
Return
you have successfully decoded me
Learn more about the decode_base64 function

Decode Base64 data (URL safe)

Source
decode_base64!("eW91IGNhbid0IG1ha2UgeW91ciBoZWFydCBmZWVsIHNvbWV0aGluZyBpdCB3b24ndA==", charset: "url_safe")
Return
you can't make your heart feel something it won't
Learn more about the decode_base64 function

decode_percent

Decodes a percent-encoded value like a URL.

Percent decode a value

Source
decode_percent("foo%20bar%3F")
Return
foo bar?
Learn more about the decode_percent function

encode_base64

Encodes the value to Base64.

Encode to Base64 (default)

Source
encode_base64("please encode me")
Return
cGxlYXNlIGVuY29kZSBtZQ==
Learn more about the encode_base64 function

Encode to Base64 (without padding)

Source
encode_base64("please encode me, no padding though", padding: false)
Return
cGxlYXNlIGVuY29kZSBtZSwgbm8gcGFkZGluZyB0aG91Z2g
Learn more about the encode_base64 function

Encode to Base64 (URL safe)

Source
encode_base64("please encode me, but safe for URLs", charset: "url_safe")
Return
cGxlYXNlIGVuY29kZSBtZSwgYnV0IHNhZmUgZm9yIFVSTHM=
Learn more about the encode_base64 function

encode_json

Encodes the value to JSON.

Encode to JSON

Source
.payload = encode_json({"hello": "world"})
Return
{"hello":"world"}
Learn more about the encode_json function

encode_key_value

Encodes the value to in key/value format with customizable delimiters. Default delimiters match the logfmt format.

Encode with default delimiters (no ordering)

Source
encode_key_value({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info"})
Return
lvl=info msg="This is a message" ts=2021-06-05T17:20:00Z
Learn more about the encode_key_value function

Encode with default delimiters (fields ordering)

Source
encode_key_value!({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info", "log_id": 12345}, ["ts", "lvl", "msg"])
Return
ts=2021-06-05T17:20:00Z lvl=info msg="This is a message" log_id=12345
Learn more about the encode_key_value function

Encode with default delimiters (nested fields)

Source
encode_key_value({"agent": {"name": "vector"}, "log": {"file": {"path": "my.log"}}, "event": "log"})
Return
agent.name=vector event=log log.file.path=my.log
Learn more about the encode_key_value function

Encode with default delimiters (nested fields ordering)

Source
encode_key_value!({"agent": {"name": "vector"}, "log": {"file": {"path": "my.log"}}, "event": "log"}, ["event", "log.file.path", "agent.name"])
Return
event=log log.file.path=my.log agent.name=vector
Learn more about the encode_key_value function

Encode with custom delimiters (no ordering)

Source
encode_key_value(
	{"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info"},
	field_delimiter: ",",
	key_value_delimiter: ":"
)
Return
lvl:info,msg:"This is a message",ts:2021-06-05T17:20:00Z
Learn more about the encode_key_value function

Encode with custom delimiters and flatten boolean

Source
encode_key_value(
	{"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info", "beta": true, "dropped": false},
	field_delimiter: ",",
	key_value_delimiter: ":",
	flatten_boolean: true
)
Return
beta,lvl:info,msg:"This is a message",ts:2021-06-05T17:20:00Z
Learn more about the encode_key_value function

encode_logfmt

Encodes the value to logfmt.

Encode to logfmt (no ordering)

Source
encode_logfmt({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info"})
Return
lvl=info msg="This is a message" ts=2021-06-05T17:20:00Z
Learn more about the encode_logfmt function

Encode to logfmt (fields ordering)

Source
encode_logfmt!({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info", "log_id": 12345}, ["ts", "lvl", "msg"])
Return
ts=2021-06-05T17:20:00Z lvl=info msg="This is a message" log_id=12345
Learn more about the encode_logfmt function

Encode to logfmt (nested fields)

Source
encode_logfmt({"agent": {"name": "vector"}, "log": {"file": {"path": "my.log"}}, "event": "log"})
Return
agent.name=vector event=log log.file.path=my.log
Learn more about the encode_logfmt function

Encode to logfmt (nested fields ordering)

Source
encode_logfmt!({"agent": {"name": "vector"}, "log": {"file": {"path": "my.log"}}, "event": "log"}, ["event", "log.file.path", "agent.name"])
Return
event=log log.file.path=my.log agent.name=vector
Learn more about the encode_logfmt function

encode_percent

Encodes a value with percent encoding to safely be used in URLs.

Percent encode all non-alphanumeric characters (default)

Source
encode_percent("foo bar?")
Return
foo%20bar%3F
Learn more about the encode_percent function

Percent encode only control characters

Source
encode_percent("foo 	bar", ascii_set: "CONTROLS")
Return
foo %09bar
Learn more about the encode_percent function

Coerce examples

to_bool

Coerces the value into a boolean.

Coerce to a Boolean (string)

Source
to_bool!("yes")
Return
true
Learn more about the to_bool function

Coerce to a Boolean (float)

Source
to_bool(0.0)
Learn more about the to_bool function

Coerce to a Boolean (int)

Source
to_bool(0)
Learn more about the to_bool function

Coerce to a Boolean (null)

Source
to_bool(null)
Learn more about the to_bool function

Coerce to a Boolean (Boolean)

Source
to_bool(true)
Return
true
Learn more about the to_bool function

to_float

Coerces the value into a float.

Coerce to a float

Source
to_float!("3.145")
Return
3.145
Learn more about the to_float function

to_int

Coerces the value into an integer.

Coerce to an int (string)

Source
to_int!("2")
Return
2
Learn more about the to_int function

Coerce to an int (timestamp)

Source
to_int(t'2020-12-30T22:20:53.824727Z')
Return
1609366853
Learn more about the to_int function

to_regex

Coerces the value into a regex.

Coerce to a regex

Source
to_regex("^foo$") ?? r''
Return
^foo$
Learn more about the to_regex function

to_string

Coerces the value into a string.

Coerce to a string (Boolean)

Source
to_string(true)
Return
true
Learn more about the to_string function

Coerce to a string (int)

Source
to_string(52)
Return
52
Learn more about the to_string function

Coerce to a string (float)

Source
to_string(52.2)
Return
52.2
Learn more about the to_string function

to_timestamp

Coerces the value into a timestamp.

Coerce to a timestamp

Source
to_timestamp!("2020-10-21T16:00:00Z")
Return
2020-10-21T16:00:00Z
Learn more about the to_timestamp function

Convert examples

to_syslog_facility

Converts the value, a Syslog facility code, into its corresponding Syslog keyword. i.e. 0 into "kern", 1 into "user", etc.

Coerce to a Syslog facility

Source
to_syslog_facility!(4)
Return
auth
Learn more about the to_syslog_facility function

to_syslog_level

Converts the value, a Syslog severity level, into its corresponding keyword, i.e. 0 into "emerg", 1 into "alert", etc.

Coerce to a Syslog level

Source
to_syslog_level!(5)
Return
notice
Learn more about the to_syslog_level function

to_syslog_severity

Converts the value, a Syslog log level keyword, into a Syslog integer severity level (0 to 7).

Coerce to Syslog severity

Source
to_syslog_severity!("alert")
Return
1
Learn more about the to_syslog_severity function

to_unix_timestamp

Converts the value timestamp into a Unix timestamp.

Returns the number of seconds since the Unix epoch by default, but milliseconds or nanoseconds can also be specified by unit.

Convert to a Unix timestamp (seconds)

Source
to_unix_timestamp(t'2021-01-01T00:00:00+00:00')
Return
1609459200
Learn more about the to_unix_timestamp function

Convert to a Unix timestamp (milliseconds)

Source
to_unix_timestamp(t'2021-01-01T00:00:00Z', unit: "milliseconds")
Return
1609459200000
Learn more about the to_unix_timestamp function

Convert to a Unix timestamp (nanoseconds)

Source
to_unix_timestamp(t'2021-01-01T00:00:00Z', unit: "nanoseconds")
Return
1609459200000000000
Learn more about the to_unix_timestamp function

Debug examples

assert

Asserts the condition, which must be a Boolean expression. The program is aborted with message if the condition evaluates to false.

Assertion (true)

Source
ok, err = assert("foo" == "foo", message: "\"foo\" must be \"foo\"!")
Return
true
Learn more about the assert function

Assertion (false)

Source
assert!("foo" == "bar", message: "\"foo\" must be \"foo\"!")
Learn more about the assert function

assert_eq

Asserts that two expressions, left and right, have the same value. The program is aborted with the message if they are unequal.

Successful assertion

Source
assert_eq!(1, 1)
Return
true
Learn more about the assert_eq function

Unsuccessful assertion

Source
assert_eq!(127, [1, 2, 3])
Learn more about the assert_eq function

Unsuccessful assertion with custom log message

Source
 assert_eq!(1, 0, message: "Unequal integers")
Learn more about the assert_eq function

log

Logs the value to Vector’s stdout at the specified level.

Log a message

Source
log("Hello, World!", level: "info", rate_limit_secs: 60)
Learn more about the log function

Log an error

Source
ts, err = to_int(.field)
if err != null {
	log(err, level: "error")
}
Learn more about the log function

Enumerate examples

compact

Compacts the value by removing “empty” values, where emptiness is defined using the available parameters.

Compact an array

Source
compact(["foo", "bar", "", null, [], "buzz"], string: true, array: true, null: true)
Return
["foo","bar","buzz"]
Learn more about the compact function

Compact an object

Source
compact({"field1": 1, "field2": "", "field3": [], "field4": null}, string: true, array: true, null: true)
Return
{
 "field1": 1
}
Learn more about the compact function

flatten

Flattens the value into a single-level representation.

Flatten array

Source
flatten([1, [2, 3, 4], [5, [6, 7], 8], 9])
Return
[1,2,3,4,5,6,7,8,9]
Learn more about the flatten function

Flatten object

Source
flatten({
	"parent1": {
		"child1": 1,
		"child2": 2
	},
	"parent2": {
		"child3": 3
	}
})
Return
{
 "parent1.child1": 1,
 "parent1.child2": 2,
 "parent2.child3": 3
}
Learn more about the flatten function

includes

Determines whether the value array includes the specified item.

Array includes

Source
includes(["apple", "orange", "banana"], "banana")
Return
true
Learn more about the includes function

length

Returns the length of the value.

Length (object)

Source
length({
	"portland": "Trail Blazers",
	"seattle":  "Supersonics"
})
Return
2
Learn more about the length function

Length (nested object)

Source
length({
	"home": {
		"city":  "Portland",
		"state": "Oregon"
	},
	"name": "Trail Blazers",
	"mascot": {
		"name": "Blaze the Trail Cat"
	}
})
Return
3
Learn more about the length function

Length (array)

Source
length(["Trail Blazers", "Supersonics", "Grizzlies"])
Return
3
Learn more about the length function

Length (string)

Source
length("The Planet of the Apes Musical")
Return
30
Learn more about the length function

match_array

Determines whether the elements in the value array matches the pattern - by default it checks at least one element matches, but can be set to determine if all the elements match.

Match at least one element

Source
	match_array(["foobar", "bazqux"], r'foo')
Return
true
Learn more about the match_array function

Match all elements

Source
	match_array(["foo", "foobar", "barfoo"], r'foo', all: true)
Return
true
Learn more about the match_array function

No matches

Source
	match_array(["bazqux", "xyz"], r'foo')
Learn more about the match_array function

Not all elements match

Source
	match_array(["foo", "foobar", "baz"], r'foo', all: true)
Learn more about the match_array function

unique

Returns unique values for an array.

The first occurrence of each element is kept.

Unique

Source
unique(["foo", "bar", "foo", "baz"])
Return
["foo","bar","baz"]
Learn more about the unique function

Event examples

del

Removes the field specified by the path from the current event object.

Delete a field

Source
del(.field1)
Learn more about the del function

Rename a field

Source
.new_field = del(.old_field)
Learn more about the del function

exists

Checks whether the path exists for the current event.

Exists (field)

Source
exists(.field)
Return
true
Learn more about the exists function

Exists (array element)

Source
exists(.array[2])
Return
true
Learn more about the exists function

Hash examples

md5

Calculates an md5 hash of the value.

Create md5 hash

Source
md5("foo")
Return
acbd18db4cc2f85cedef654fccc4a4d8
Learn more about the md5 function

sha1

Calculates a SHA-1 hash of the value.

Calculate sha1 hash

Source
sha1("foo")
Return
0beec7b5ea3f0fdbc95d0dd47f3c5bc275da8a33
Learn more about the sha1 function

sha2

Calculates a SHA-2 hash of the value.

Calculate sha2 hash

Source
sha2("foo", variant: "SHA-512/224")
Return
d68f258d37d670cfc1ec1001a0394784233f88f056994f9a7e5e99be
Learn more about the sha2 function

sha3

Calculates a SHA-3 hash of the value.

Calculate sha3 hash

Source
sha3("foo", variant: "SHA3-224")
Return
f4f6779e153c391bbd29c95e72b0708e39d9166c7cea51d1f10ef58a
Learn more about the sha3 function

IP examples

ip_aton

Converts IPv4 address in numbers-and-dots notation into network-order bytes represented as an integer.

This behavior mimics inet_aton.

IPv4 to integer

Source
ip_aton!("1.2.3.4")
Return
16909060
Learn more about the ip_aton function

ip_cidr_contains

Determines whether the ip is contained in the block referenced by the cidr.

IPv4 contains CIDR

Source
ip_cidr_contains!("192.168.0.0/16", "192.168.10.32")
Return
true
Learn more about the ip_cidr_contains function

IPv6 contains CIDR

Source
ip_cidr_contains!("2001:4f8:4:ba::/64", "2001:4f8:4:ba:2e0:81ff:fe22:d1f1")
Return
true
Learn more about the ip_cidr_contains function

ip_ntoa

Converts numeric representation of IPv4 address in network-order bytes to numbers-and-dots notation..

This behavior mimics inet_ntoa.

Integer to IPv4

Source
ip_ntoa!(16909060)
Return
1.2.3.4
Learn more about the ip_ntoa function

ip_subnet

Extracts the subnet address from the ip using the supplied subnet.

IPv4 subnet

Source
ip_subnet!("192.168.10.32", "255.255.255.0")
Return
192.168.10.0
Learn more about the ip_subnet function

IPv6 subnet

Source
ip_subnet!("2404:6800:4003:c02::64", "/32")
Return
2404:6800::
Learn more about the ip_subnet function

ip_to_ipv6

Converts the ip to an IPv6 address.

IPv4 to IPv6

Source
ip_to_ipv6!("192.168.10.32")
Return
::ffff:192.168.10.32
Learn more about the ip_to_ipv6 function

ipv6_to_ipv4

Converts the ip to an IPv4 address. ip is returned unchanged if it’s already an IPv4 address. If ip is currently an IPv6 address then it needs to be IPv4 compatible, otherwise an error is thrown.

IPv6 to IPv4

Source
ipv6_to_ipv4!("::ffff:192.168.0.1")
Return
192.168.0.1
Learn more about the ipv6_to_ipv4 function

Number examples

ceil

Rounds the value up to the specified precision.

Round a number up (without precision)

Source
ceil(4.345)
Return
5
Learn more about the ceil function

Round a number up (with precision)

Source
ceil(4.345, precision: 2)
Return
4.35
Learn more about the ceil function

floor

Rounds the value down to the specified precision.

Round a number down (without precision)

Source
floor(4.345)
Return
4
Learn more about the floor function

Round a number down (with precision)

Source
floor(4.345, precision: 2)
Return
4.34
Learn more about the floor function

format_int

Formats the integer value into a string representation using the given base/radix.

Format as a hexadecimal integer

Source
format_int!(42, 16)
Return
2a
Learn more about the format_int function

Format as a negative hexadecimal integer

Source
format_int!(-42, 16)
Return
-2a
Learn more about the format_int function

format_number

Formats the value into a string representation of the number.

Format a number (3 decimals)

Source
format_number(1234567.89, 3, decimal_separator: ".", grouping_separator: ",")
Return
1,234,567.890
Learn more about the format_number function

round

Rounds the value to the specified precision.

Round a number (without precision)

Source
round(4.345)
Return
4
Learn more about the round function

Round a number (with precision)

Source
round(4.345, precision: 2)
Return
4.35
Learn more about the round function

Object examples

match_datadog_query

Matches an object against a Datadog Search Syntax query.

OR query

Source
match_datadog_query({"message": "contains this and that"}, "this OR that")
Return
true
Learn more about the match_datadog_query function

AND query

Source
match_datadog_query({"message": "contains only this"}, "this AND that")
Learn more about the match_datadog_query function

Facet wildcard

Source
match_datadog_query({"custom": {"name": "vector"}}, "@name:vec*")
Return
true
Learn more about the match_datadog_query function

Tag range

Source
match_datadog_query({"tags": ["a:x", "b:y", "c:z"]}, s'b:["x" TO "z"]')
Return
true
Learn more about the match_datadog_query function

merge

Merges the from object into the to object.

Object merge (shallow)

Source
merge(
	{
		"parent1": {
			"child1": 1,
			"child2": 2
		},
		"parent2": {
			"child3": 3
		}
	},
	{
		"parent1": {
			"child2": 4,
			"child5": 5
		}
	}
)
Return
{
 "parent1": {
  "child2": 4,
  "child5": 5
 },
 "parent2": {
  "child3": 3
 }
}
Learn more about the merge function

Object merge (deep)

Source
merge(
	{
		"parent1": {
			"child1": 1,
			"child2": 2
		},
		"parent2": {
			"child3": 3
		}
	},
	{
		"parent1": {
			"child2": 4,
			"child5": 5
		}
	},
	deep: true
)
Return
{
 "parent1": {
  "child1": 1,
  "child2": 4,
  "child5": 5
 },
 "parent2": {
  "child3": 3
 }
}
Learn more about the merge function

unnest

Unnest an array field from an object to create an array of objects using that field; keeping all other fields.

Assigning the array result of this to . will result in multiple events being emitted from remap. See the remap transform docs for more details.

This is also referred to as explodeing in some languages.

Unnest an array field

Source
. = unnest!(.messages)
Learn more about the unnest function

Unnest nested an array field

Source
. = unnest!(.event.messages)
Learn more about the unnest function

Parse examples

parse_apache_log

Parses Apache access and error log lines. Lines can be in common, combined, or default error format.

Parse via Apache log format (common)

Source
parse_apache_log!("127.0.0.1 bob frank [10/Oct/2000:13:55:36 -0700] \"GET /apache_pb.gif HTTP/1.0\" 200 2326", format: "common")
Return
{
 "host": "127.0.0.1",
 "identity": "bob",
 "message": "GET /apache_pb.gif HTTP/1.0",
 "method": "GET",
 "path": "/apache_pb.gif",
 "protocol": "HTTP/1.0",
 "size": 2326,
 "status": 200,
 "timestamp": "2000-10-10T20:55:36Z",
 "user": "frank"
}
Learn more about the parse_apache_log function

Parse via Apache log format (combined)

Source
parse_apache_log!(
	s'127.0.0.1 bob frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 "http://www.seniorinfomediaries.com/vertical/channels/front-end/bandwidth" "Mozilla/5.0 (X11; Linux i686; rv:5.0) Gecko/1945-10-12 Firefox/37.0"',
	"combined",
)
Return
{
 "agent": "Mozilla/5.0 (X11; Linux i686; rv:5.0) Gecko/1945-10-12 Firefox/37.0",
 "host": "127.0.0.1",
 "identity": "bob",
 "message": "GET /apache_pb.gif HTTP/1.0",
 "method": "GET",
 "path": "/apache_pb.gif",
 "protocol": "HTTP/1.0",
 "referrer": "http://www.seniorinfomediaries.com/vertical/channels/front-end/bandwidth",
 "size": 2326,
 "status": 200,
 "timestamp": "2000-10-10T20:55:36Z",
 "user": "frank"
}
Learn more about the parse_apache_log function

Parse via Apache log format (error)

Source
parse_apache_log!(
	s'[01/Mar/2021:12:00:19 +0000] [ab:alert] [pid 4803:tid 3814] [client 147.159.108.175:24259] I will bypass the haptic COM bandwidth, that should matrix the CSS driver!',
	"error"
)
Return
{
 "client": "147.159.108.175",
 "message": "I will bypass the haptic COM bandwidth, that should matrix the CSS driver!",
 "module": "ab",
 "pid": 4803,
 "port": 24259,
 "severity": "alert",
 "thread": "3814",
 "timestamp": "2021-03-01T12:00:19Z"
}
Learn more about the parse_apache_log function

parse_aws_alb_log

Parse AWS ALB log

Source
parse_aws_alb_log!(
	"http 2018-11-30T22:23:00.186641Z app/my-loadbalancer/50dc6c495c0c9188 192.168.131.39:2817 - 0.000 0.001 0.000 200 200 34 366 \"GET http://www.example.com:80/ HTTP/1.1\" \"curl/7.46.0\" - - arn:aws:elasticloadbalancing:us-east-2:123456789012:targetgroup/my-targets/73e2d6bc24d8a067 \"Root=1-58337364-23a8c76965a2ef7629b185e3\" \"-\" \"-\" 0 2018-11-30T22:22:48.364000Z \"forward\" \"-\" \"-\" \"-\" \"-\" \"-\" \"-\""
)
Return
{
 "actions_executed": "forward",
 "chosen_cert_arn": null,
 "classification": null,
 "classification_reason": null,
 "client_host": "192.168.131.39:2817",
 "domain_name": null,
 "elb": "app/my-loadbalancer/50dc6c495c0c9188",
 "elb_status_code": "200",
 "error_reason": null,
 "matched_rule_priority": "0",
 "received_bytes": 34,
 "redirect_url": null,
 "request_creation_time": "2018-11-30T22:22:48.364000Z",
 "request_method": "GET",
 "request_processing_time": 0,
 "request_protocol": "HTTP/1.1",
 "request_url": "http://www.example.com:80/",
 "response_processing_time": 0,
 "sent_bytes": 366,
 "ssl_cipher": null,
 "ssl_protocol": null,
 "target_group_arn": "arn:aws:elasticloadbalancing:us-east-2:123456789012:targetgroup/my-targets/73e2d6bc24d8a067",
 "target_host": null,
 "target_port_list": [],
 "target_processing_time": 0.001,
 "target_status_code": "200",
 "target_status_code_list": [],
 "timestamp": "2018-11-30T22:23:00.186641Z",
 "trace_id": "Root=1-58337364-23a8c76965a2ef7629b185e3",
 "type": "http",
 "user_agent": "curl/7.46.0"
}
Learn more about the parse_aws_alb_log function

parse_aws_cloudwatch_log_subscription_message

Parses AWS CloudWatch Logs events (configured through AWS Cloudwatch subscriptions) from the aws_kinesis_firehose source.

Parse AWS Cloudwatch Log subscription message

Source
parse_aws_cloudwatch_log_subscription_message!(.message)
Return
{
 "log_events": [
  {
   "id": "35683658089614582423604394983260738922885519999578275840",
   "message": "{\"bytes\":26780,\"datetime\":\"14/Sep/2020:11:45:41 -0400\",\"host\":\"157.130.216.193\",\"method\":\"PUT\",\"protocol\":\"HTTP/1.0\",\"referer\":\"https://www.principalcross-platform.io/markets/ubiquitous\",\"request\":\"/expedite/convergence\",\"source_type\":\"stdin\",\"status\":301,\"user-identifier\":\"-\"}",
   "timestamp": "2020-09-14T19:09:29.039Z"
  }
 ],
 "log_group": "test",
 "log_stream": "test",
 "message_type": "DATA_MESSAGE",
 "owner": "111111111111",
 "subscription_filters": [
  "Destination"
 ]
}
Learn more about the parse_aws_cloudwatch_log_subscription_message function

parse_aws_vpc_flow_log

Parses value in the VPC Flow Logs format.

Parse AWS VPC Flow log (default format)

Source
parse_aws_vpc_flow_log!("2 123456789010 eni-1235b8ca123456789 - - - - - - - 1431280876 1431280934 - NODATA")
Return
{
 "account_id": 123456789010,
 "action": null,
 "bytes": null,
 "dstaddr": null,
 "dstport": null,
 "end": 1431280934,
 "interface_id": "eni-1235b8ca123456789",
 "log_status": "NODATA",
 "packets": null,
 "protocol": null,
 "srcaddr": null,
 "srcport": null,
 "start": 1431280876,
 "version": 2
}
Learn more about the parse_aws_vpc_flow_log function

Parse AWS VPC Flow log (custom format)

Source
parse_aws_vpc_flow_log!(
	"- eni-1235b8ca123456789 10.0.1.5 10.0.0.220 10.0.1.5 203.0.113.5",
	"instance_id interface_id srcaddr dstaddr pkt_srcaddr pkt_dstaddr"
)
Return
{
 "dstaddr": "10.0.0.220",
 "instance_id": null,
 "interface_id": "eni-1235b8ca123456789",
 "pkt_dstaddr": "203.0.113.5",
 "pkt_srcaddr": "10.0.1.5",
 "srcaddr": "10.0.1.5"
}
Learn more about the parse_aws_vpc_flow_log function

parse_common_log

Parses the value using the Common Log Format (CLF).

Parse via Common Log Format (with default timestamp format)

Source
parse_common_log!("127.0.0.1 bob frank [10/Oct/2000:13:55:36 -0700] \"GET /apache_pb.gif HTTP/1.0\" 200 2326")
Return
{
 "host": "127.0.0.1",
 "identity": "bob",
 "message": "GET /apache_pb.gif HTTP/1.0",
 "method": "GET",
 "path": "/apache_pb.gif",
 "protocol": "HTTP/1.0",
 "size": 2326,
 "status": 200,
 "timestamp": "2000-10-10T20:55:36Z",
 "user": "frank"
}
Learn more about the parse_common_log function

Parse via Common Log Format (with custom timestamp format)

Source
parse_common_log!(
	"127.0.0.1 bob frank [2000-10-10T20:55:36Z] \"GET /apache_pb.gif HTTP/1.0\" 200 2326",
	"%+"
)
Return
{
 "host": "127.0.0.1",
 "identity": "bob",
 "message": "GET /apache_pb.gif HTTP/1.0",
 "method": "GET",
 "path": "/apache_pb.gif",
 "protocol": "HTTP/1.0",
 "size": 2326,
 "status": 200,
 "timestamp": "2000-10-10T20:55:36Z",
 "user": "frank"
}
Learn more about the parse_common_log function

parse_csv

Parses a single CSV formatted row. Only the first row is parsed in case of multiline input value.

Parse a single CSV formatted row

Source
parse_csv!("foo,bar,\"foo \"\", bar\"")
Return
["foo","bar","foo \", bar"]
Learn more about the parse_csv function

Parse a single CSV formatted row with custom delimiter

Source
parse_csv!("foo bar", delimiter: " ")
Return
["foo","bar"]
Learn more about the parse_csv function

parse_duration

Parses the value into a human-readable duration format specified by unit.

Parse duration (milliseconds)

Source
parse_duration!("1005ms", unit: "s")
Return
1.005
Learn more about the parse_duration function

parse_glog

Parses the value using the glog (Google Logging Library) format.

Parse via glog

Source
parse_glog!("I20210131 14:48:54.411655 15520 main.c++:9] Hello world!")
Return
{
 "file": "main.c++",
 "id": 15520,
 "level": "info",
 "line": 9,
 "message": "Hello world!",
 "timestamp": "2021-01-31T14:48:54.411655Z"
}
Learn more about the parse_glog function

parse_grok

Parses the value using the grok format. All patterns listed here are supported.

Parse using Grok

Source
parse_grok!(
	"2020-10-02T23:22:12.223222Z info Hello world",
	"%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{GREEDYDATA:message}"
)
Return
{
 "level": "info",
 "message": "Hello world",
 "timestamp": "2020-10-02T23:22:12.223222Z"
}
Learn more about the parse_grok function

parse_int

Parses the string value representing a number in an optional base/radix to an integer.

Parse decimal

Source
parse_int!("-42")
Return
-42
Learn more about the parse_int function

Parse binary

Source
parse_int!("0b1001")
Return
9
Learn more about the parse_int function

Parse octal

Source
parse_int!("0o42")
Return
34
Learn more about the parse_int function

Parse hexadecimal

Source
parse_int!("0x2a")
Return
42
Learn more about the parse_int function

Parse explicit base

Source
parse_int!("2a", 17)
Return
44
Learn more about the parse_int function

parse_json

Parses the value as JSON.

Parse JSON

Source
parse_json!("{\"key\": \"val\"}")
Return
{
 "key": "val"
}
Learn more about the parse_json function

parse_key_value

Parses the value in key/value format. Also known as logfmt.

  • Keys and values can be wrapped with ".
  • " characters can be escaped using \.

Parse logfmt log

Source
parse_key_value!(
	"@timestamp=\"Sun Jan 10 16:47:39 EST 2021\" level=info msg=\"Stopping all fetchers\" tag#production=stopping_fetchers id=ConsumerFetcherManager-1382721708341 module=kafka.consumer.ConsumerFetcherManager"
)
Return
{
 "@timestamp": "Sun Jan 10 16:47:39 EST 2021",
 "id": "ConsumerFetcherManager-1382721708341",
 "level": "info",
 "module": "kafka.consumer.ConsumerFetcherManager",
 "msg": "Stopping all fetchers",
 "tag#production": "stopping_fetchers"
}
Learn more about the parse_key_value function

Parse comma delimited log

Source
parse_key_value!(
	"path:\"/cart_link\", host:store.app.com, fwd: \"102.30.171.16\", dyno: web.1, connect:0ms, service:87ms, status:304, bytes:632, protocol:https",
	field_delimiter: ",",
	key_value_delimiter: ":"
)
Return
{
 "bytes": "632",
 "connect": "0ms",
 "dyno": "web.1",
 "fwd": "102.30.171.16",
 "host": "store.app.com",
 "path": "/cart_link",
 "protocol": "https",
 "service": "87ms",
 "status": "304"
}
Learn more about the parse_key_value function

Parse comma delimited log with standalone keys

Source
parse_key_value!(
	"env:prod,service:backend,region:eu-east1,beta",
	field_delimiter: ",",
	key_value_delimiter: ":",
)
Return
{
 "beta": true,
 "env": "prod",
 "region": "eu-east1",
 "service": "backend"
}
Learn more about the parse_key_value function

parse_klog

Parses the value using the klog format used by Kubernetes components.

Parse via klog

Source
parse_klog!("I0505 17:59:40.692994   28133 klog.go:70] hello from klog")
Return
{
 "file": "klog.go",
 "id": 28133,
 "level": "info",
 "line": 70,
 "message": "hello from klog",
 "timestamp": "2021-05-05T17:59:40.692994Z"
}
Learn more about the parse_klog function

parse_linux_authorization

Parses Linux authorization logs usually found under either /var/log/auth.log (for Debian-based systems) or /var/log/secure (for RedHat-based systems) according to Syslog format.

Parse Linux authorization event

Source
parse_linux_authorization!(
	s'Mar 23 01:49:58 localhost sshd[1111]: Accepted publickey for eng from 10.1.1.1 port 8888 ssh2: RSA SHA256:foobar'
)
Return
{
 "appname": "sshd",
 "hostname": "localhost",
 "message": "Accepted publickey for eng from 10.1.1.1 port 8888 ssh2: RSA SHA256:foobar",
 "procid": 1111,
 "timestamp": "2021-03-23T01:49:58Z"
}
Learn more about the parse_linux_authorization function

parse_logfmt

Parses the value in logfmt.

  • Keys and values can be wrapped using the " character.
  • " characters can be escaped by the \ character.
  • As per this logfmt specification, the parse_logfmt function accepts standalone keys and assigns them a Boolean value of true.

Parse logfmt log

Source
parse_logfmt!(
	"@timestamp=\"Sun Jan 10 16:47:39 EST 2021\" level=info msg=\"Stopping all fetchers\" tag#production=stopping_fetchers id=ConsumerFetcherManager-1382721708341 module=kafka.consumer.ConsumerFetcherManager"
)
Return
{
 "@timestamp": "Sun Jan 10 16:47:39 EST 2021",
 "id": "ConsumerFetcherManager-1382721708341",
 "level": "info",
 "module": "kafka.consumer.ConsumerFetcherManager",
 "msg": "Stopping all fetchers",
 "tag#production": "stopping_fetchers"
}
Learn more about the parse_logfmt function

parse_nginx_log

Parses Nginx access and error log lines. Lines can be in combined, or error format.

Parse via Nginx log format (combined)

Source
parse_nginx_log!(
    s'172.17.0.1 alice - [01/Apr/2021:12:02:31 +0000] "POST /not-found HTTP/1.1" 404 153 "http://localhost/somewhere" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36" "2.75"',
    "combined",
)
Return
{
 "agent": "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36",
 "client": "172.17.0.1",
 "compression": "2.75",
 "method": "POST",
 "path": "/not-found",
 "protocol": "HTTP/1.1",
 "referer": "http://localhost/somewhere",
 "request": "POST /not-found HTTP/1.1",
 "size": 153,
 "status": 404,
 "timestamp": "2021-04-01T12:02:31Z",
 "user": "alice"
}
Learn more about the parse_nginx_log function

Parse via Nginx log format (error)

Source
parse_nginx_log!(
    s'2021/04/01 13:02:31 [error] 31#31: *1 open() "/usr/share/nginx/html/not-found" failed (2: No such file or directory), client: 172.17.0.1, server: localhost, request: "POST /not-found HTTP/1.1", host: "localhost:8081"',
    "error"
)
Return
{
 "cid": 1,
 "client": "172.17.0.1",
 "host": "localhost:8081",
 "message": "open() \"/usr/share/nginx/html/not-found\" failed (2: No such file or directory)",
 "pid": 31,
 "request": "POST /not-found HTTP/1.1",
 "server": "localhost",
 "severity": "error",
 "tid": 31,
 "timestamp": "2021-04-01T13:02:31Z"
}
Learn more about the parse_nginx_log function

parse_query_string

Parses the value as a query string.

Parse query string

Source
parse_query_string("foo=%2B1&bar=2&bar=3&xyz")
Return
{
 "bar": [
  "2",
  "3"
 ],
 "foo": "+1",
 "xyz": ""
}
Learn more about the parse_query_string function

Parse Ruby on Rails' query string

Source
parse_query_string("?foo%5b%5d=1&foo%5b%5d=2")
Return
{
 "foo[]": [
  "1",
  "2"
 ]
}
Learn more about the parse_query_string function

parse_regex

Parses the value via the provided Regex pattern.

This function differs from the parse_regex_all function in that it returns only the first match.

Parse using Regex (with capture groups)

Source
parse_regex!("first group and second group.", r'(?P<number>.*?) group')
Return
{
 "number": "first"
}
Learn more about the parse_regex function

Parse using Regex (without capture groups)

Source
parse_regex!("first group and second group.", r'(\w+) group', numeric_groups: true)
Return
{
 "0": "first group",
 "1": "first"
}
Learn more about the parse_regex function

parse_regex_all

Parses the value via the provided Regex pattern.

This function differs from the parse_regex function in that it returns all matches, not just the first.

Parse using Regex (all matches)

Source
parse_regex_all!("first group and second group.", r'(?P<number>\w+) group', numeric_groups: true)
Return
[{"0":"first group","1":"first","number":"first"},{"0":"second group","1":"second","number":"second"}]
Learn more about the parse_regex_all function

parse_ruby_hash

Parses the value as ruby hash.

Parse ruby hash

Source
parse_ruby_hash!(s'{ "test" => "value", "testNum" => 0.2, "testObj" => { "testBool" => true, "testNull" => nil } }')
Return
{
 "test": "value",
 "testNum": 0.2,
 "testObj": {
  "testBool": true,
  "testNull": null
 }
}
Learn more about the parse_ruby_hash function

parse_syslog

Parses the value in Syslog format.

Parse Syslog log (5424)

Source
parse_syslog!(
	s'<13>1 2020-03-13T20:45:38.119Z dynamicwireless.name non 2426 ID931 [exampleSDID@32473 iut="3" eventSource= "Application" eventID="1011"] Try to override the THX port, maybe it will reboot the neural interface!'
)
Return
{
 "appname": "non",
 "exampleSDID@32473.eventID": "1011",
 "exampleSDID@32473.eventSource": "Application",
 "exampleSDID@32473.iut": "3",
 "facility": "user",
 "hostname": "dynamicwireless.name",
 "message": "Try to override the THX port, maybe it will reboot the neural interface!",
 "msgid": "ID931",
 "procid": 2426,
 "severity": "notice",
 "timestamp": "2020-03-13T20:45:38.119Z",
 "version": 1
}
Learn more about the parse_syslog function

parse_timestamp

Parses the value in strptime format.

Parse timestamp

Source
parse_timestamp!("10-Oct-2020 16:00+00:00", format: "%v %R %:z")
Return
2020-10-10T16:00:00Z
Learn more about the parse_timestamp function

parse_tokens

Parses the value in “token” format. A token is considered to be one of the following:

  • A word surrounded by whitespace.
  • Text delimited by double quotes: "..". Quotes can be included in the token if they are escaped by a backslash (\).
  • Text delimited by square brackets: [..]. Closing square brackets can be included in the token if they are escaped by a backslash (\).

Parse tokens

Source
parse_tokens(
	"A sentence \"with \\\"a\\\" sentence inside\" and [some brackets]"
)
Return
["A","sentence","with \\\"a\\\" sentence inside","and","some brackets"]
Learn more about the parse_tokens function

parse_url

Parses the value in URL format.

Parse URL

Source
parse_url!("ftp://foo:bar@vector.dev:4343/foobar?hello=world#123")
Return
{
 "fragment": "123",
 "host": "vector.dev",
 "password": "bar",
 "path": "/foobar",
 "port": 4343,
 "query": {
  "hello": "world"
 },
 "scheme": "ftp",
 "username": "foo"
}
Learn more about the parse_url function

parse_user_agent

Parses the value as a user agent string. Which has a loosely defined format so this parser only provides best effort guarantee.

Fast mode

Source
parse_user_agent(
	"Mozilla Firefox 1.0.1 Mozilla/5.0 (X11; U; Linux i686; de-DE; rv:1.7.6) Gecko/20050223 Firefox/1.0.1"
)
Return
{
 "browser": {
  "family": "Firefox",
  "version": "1.0.1"
 },
 "device": {
  "category": "pc"
 },
 "os": {
  "family": "Linux",
  "version": null
 }
}
Learn more about the parse_user_agent function

Reliable mode

Source
parse_user_agent(
	"Mozilla/4.0 (compatible; MSIE 7.66; Windows NT 5.1; SV1; .NET CLR 1.1.4322)",
	mode: "reliable"
)
Return
{
 "browser": {
  "family": "Internet Explorer",
  "version": "7.66"
 },
 "device": {
  "category": "pc"
 },
 "os": {
  "family": "Windows XP",
  "version": "NT 5.1"
 }
}
Learn more about the parse_user_agent function

Enriched mode

Source
parse_user_agent(
	"Opera/9.80 (J2ME/MIDP; Opera Mini/4.3.24214; iPhone; CPU iPhone OS 4_2_1 like Mac OS X; AppleWebKit/24.783; U; en) Presto/2.5.25 Version/10.54",
	mode: "enriched"
)
Return
{
 "browser": {
  "family": "Opera Mini",
  "major": "4",
  "minor": "3",
  "patch": "24214",
  "version": "10.54"
 },
 "device": {
  "brand": "Apple",
  "category": "smartphone",
  "family": "iPhone",
  "model": "iPhone"
 },
 "os": {
  "family": "iOS",
  "major": "4",
  "minor": "2",
  "patch": "1",
  "patch_minor": null,
  "version": "4.2.1"
 }
}
Learn more about the parse_user_agent function

parse_xml

Parses the value as XML.

Parse XML

Source
value = s'<book category="CHILDREN"><title lang="en">Harry Potter</title><author>J K. Rowling</author><year>2005</year></book>';

parse_xml!(value, text_key: "value", parse_number: false)
Return
{
 "book": {
  "@category": "CHILDREN",
  "author": "J K. Rowling",
  "title": {
   "@lang": "en",
   "value": "Harry Potter"
  },
  "year": "2005"
 }
}
Learn more about the parse_xml function

Random examples

uuid_v4

Generates a random UUIDv4 string.

Create a UUIDv4

Source
uuid_v4()
Return
1d262f4f-199b-458d-879f-05fd0a5f0683
Learn more about the uuid_v4 function

String examples

contains

Determines whether the value string contains the specified substring.

String contains (case sensitive)

Source
contains("The Needle In The Haystack", "Needle")
Return
true
Learn more about the contains function

String contains (case insensitive)

Source
contains("The Needle In The Haystack", "needle", case_sensitive: false)
Return
true
Learn more about the contains function

downcase

Downcases the value string, where “downcase” is defined according to the terms of the Unicode Derived Core Property Lowercase.

Downcase a string

Source
downcase("Hello, World!")
Return
hello, world!
Learn more about the downcase function

ends_with

Determines whether the value string ends with the specified substring.

String ends with (case sensitive)

Source
ends_with("The Needle In The Haystack", "The Haystack")
Return
true
Learn more about the ends_with function

String ends with (case insensitive)

Source
ends_with("The Needle In The Haystack", "the haystack", case_sensitive: false)
Return
true
Learn more about the ends_with function

find

Determines the start position of the first found element in value, from left to right, that matches the pattern or returns -1 if not found.

Match text

Source
find("foobar", "foo")
Learn more about the find function

Match regex

Source
find("foobar", r'b.r')
Return
3
Learn more about the find function

No matches

Source
find("foobar", "baz")
Return
-1
Learn more about the find function

With an offset

Source
find("foobarfoobarfoo", "bar", 4)
Return
9
Learn more about the find function

join

Joins each string in the value array into a single string, with items optionally separated from one another by a separator.

Join array (no separator)

Source
join!(["bring", "us", "together"])
Return
bringustogether
Learn more about the join function

Join array (comma separator)

Source
join!(["sources", "transforms", "sinks"], separator: ", ")
Return
sources, transforms, sinks
Learn more about the join function

match

Determines whether the value matches the pattern.

Regex match on a string

Source
match("I'm a little teapot", r'teapot')
Return
true
Learn more about the match function

match_any

Determines whether the value matches any the given patterns. All patterns are checked in a single pass over the target string, giving this function a potentially significant performance advantage over multiple calls to match.

Regex match on a string

Source
match_any("I'm a little teapot", [r'frying pan', r'teapot'])
Return
true
Learn more about the match_any function

redact

Redact sensitive data in value such as:

This can help achieve compliance by ensuring sensitive data never leaves your network.

Replace text using a regex

Source
redact("my id is 123456", filters: [r'\d+'])
Return
my id is [REDACTED]
Learn more about the redact function

Replace us social security numbers in any field

Source
redact({ "name": "John Doe", "ssn": "123-12-1234"}, filters: ["us_social_security_number"])
Return
{
 "name": "John Doe",
 "ssn": "[REDACTED]"
}
Learn more about the redact function

replace

Replaces all matching instances of pattern in the value.

Replace literal text

Source
replace("Apples and Bananas", "and", "not")
Return
Apples not Bananas
Learn more about the replace function

Replace via regular expression

Source
replace("Apples and Bananas", r'(?i)bananas', "Pineapples")
Return
Apples and Pineapples
Learn more about the replace function

Replace first instance

Source
replace("Bananas and Bananas", "Bananas", "Pineapples", count: 1)
Return
Pineapples and Bananas
Learn more about the replace function

slice

Returns a slice of the value between the start and end positions.

If the start and end parameters are negative, they refer to positions counting from the right of the string or array. If end refers to a position that is greater than the length of the string or array a slice up to the end of the string or array is returned.

Slice a string (positive index)

Source
slice!("Supercalifragilisticexpialidocious", start: 5, end: 13)
Return
califrag
Learn more about the slice function

Slice a string (negative index)

Source
slice!("Supercalifragilisticexpialidocious", start: 5, end: -14)
Return
califragilistic
Learn more about the slice function

split

Splits the value string via the pattern.

Split a string (no limit)

Source
split("apples and pears and bananas", " and ")
Return
["apples","pears","bananas"]
Learn more about the split function

Split a string (with a limit)

Source
split("apples and pears and bananas", " and ", limit: 2)
Return
["apples","pears and bananas"]
Learn more about the split function

starts_with

Determines whether the value begins with the substring.

String starts with (case sensitive)

Source
starts_with("The Needle In The Haystack", "The Needle")
Return
true
Learn more about the starts_with function

String starts with (case insensitive)

Source
starts_with("The Needle In The Haystack", "the needle", case_sensitive: false)
Return
true
Learn more about the starts_with function

strip_ansi_escape_codes

Strips ANSI escape codes from the value.

Strip ANSI escape codes

Source
strip_ansi_escape_codes("\e[46mfoo\e[0m bar")
Return
foo bar
Learn more about the strip_ansi_escape_codes function

strip_whitespace

Strips whitespace from the start and end of the value, where whitespace is defined by the Unicode White_Space property

Strip whitespace

Source
strip_whitespace("  A sentence.  ")
Return
A sentence.
Learn more about the strip_whitespace function

truncate

Truncates the value string up to the limit number of characters.

Truncate a string

Source
truncate("A rather long sentence.", limit: 11, ellipsis: true)
Return
A rather lo...
Learn more about the truncate function

upcase

Upcases the value, where “upcase” is defined according to the terms of the Unicode Derived Core Property Uppercase.

Upcase a string

Source
upcase("Hello, World!")
Return
HELLO, WORLD!
Learn more about the upcase function

System examples

get_env_var

Returns the value of the environment variable specified by name.

Get an environment variable

Source
get_env_var!("HOME")
Return
/root
Learn more about the get_env_var function

get_hostname

Returns the local system’s hostname.

Get hostname

Source
.hostname = get_hostname!()
Learn more about the get_hostname function

Timestamp examples

format_timestamp

Formats the value into a string representation of the timestamp.

Format a timestamp (ISO8601/RFC 3339)

Source
format_timestamp!(t'2020-10-21T16:00:00Z', format: "%+")
Return
2020-10-21T16:00:00+00:00
Learn more about the format_timestamp function

Format a timestamp (custom)

Source
format_timestamp!(t'2020-10-21T16:00:00Z', format: "%v %R")
Return
21-Oct-2020 16:00
Learn more about the format_timestamp function

now

Returns the current timestamp in the UTC timezone with nanosecond precision.

Generate a current timestamp

Source
now()
Return
2021-03-04T10:51:15.928937Z
Learn more about the now function

Type examples

array

Returns the value if it’s an array and errors otherwise. This enables the type checker to guarantee that the returned value is an array and can be used in any function that expects one.

Declare an array type

Source
array!(.value)
Return
[1,2,3]
Learn more about the array function

bool

Returns the value if it’s a Boolean and errors otherwise. This enables the type checker to guarantee that the returned value is a Boolean and can be used in any function that expects one.

Declare a Boolean type

Source
bool!(.value)
Learn more about the bool function

float

Returns the value if it’s a float and errors otherwise. This enables the type checker to guarantee that the returned value is a float and can be used in any function that expects one.

Declare a float type

Source
float!(.value)
Return
42
Learn more about the float function

int

Returns the value if it’s an integer and errors otherwise. This enables the type checker to guarantee that the returned value is an integer and can be used in any function that expects one.

Declare an integer type

Source
int!(.value)
Return
42
Learn more about the int function

is_array

Check if the type of a value is an array or not.

Valid array

Source
is_array([1, 2, 3])
Return
true
Learn more about the is_array function

Non-matching type

Source
is_array("a string")
Learn more about the is_array function

is_boolean

Check if the type of a value is a boolean or not.

Valid boolean

Source
is_boolean(false)
Return
true
Learn more about the is_boolean function

Non-matching type

Source
is_boolean("a string")
Learn more about the is_boolean function

is_float

Check if the type of a value is a float or not.

Valid float

Source
is_float(0.577)
Return
true
Learn more about the is_float function

Non-matching type

Source
is_float("a string")
Learn more about the is_float function

is_integer

Check if the type of a value is an integer or not.

Valid integer

Source
is_integer(1)
Return
true
Learn more about the is_integer function

Non-matching type

Source
is_integer("a string")
Learn more about the is_integer function

is_null

Check if the type of a value is null or not. For a more relaxed function please check is_nullish.

Null value

Source
is_null(null)
Return
true
Learn more about the is_null function

Non-matching type

Source
is_null("a string")
Learn more about the is_null function

is_nullish

Determines whether the value is “nullish,” where nullish denotes the absence of a meaningful value.

Null detection (blank string)

Source
is_nullish("")
Return
true
Learn more about the is_nullish function

Null detection (dash string)

Source
is_nullish("-")
Return
true
Learn more about the is_nullish function

Null detection (whitespace)

Source
is_nullish("
  
")
Return
true
Learn more about the is_nullish function

is_object

Check if the type of a value is an object or not.

Valid object

Source
is_object({"foo": "bar"})
Return
true
Learn more about the is_object function

Non-matching type

Source
is_object("a string")
Learn more about the is_object function

is_regex

Check if the type of a value is a regex or not.

Valid regex

Source
is_regex(r'pattern')
Return
true
Learn more about the is_regex function

Non-matching type

Source
is_regex("a string")
Learn more about the is_regex function

is_string

Check if the type of a value is a string or not.

Valid string

Source
is_string("a string")
Return
true
Learn more about the is_string function

Non-matching type

Source
is_string([1, 2, 3])
Learn more about the is_string function

is_timestamp

Check if the type of a value is a timestamp or not.

Valid timestamp

Source
is_timestamp(t'2021-03-26T16:00:00Z')
Return
true
Learn more about the is_timestamp function

Non-matching type

Source
is_timestamp("a string")
Learn more about the is_timestamp function

object

Returns the value if it’s an object and errors otherwise. This enables the type checker to guarantee that the returned value is an object and can be used in any function that expects one.

Declare an object type

Source
object!(.value)
Return
{
 "field1": "value1",
 "field2": "value2"
}
Learn more about the object function

string

Returns the value if it’s a string and errors otherwise. This enables the type checker to guarantee that the returned value is a string and can be used in any function that expects one.

Declare a string type

Source
string!(.message)
Return
{"field": "value"}
Learn more about the string function

tag_types_externally

Adds type information to all (nested) scalar values in the provided value.

The type information is added externally, meaning that value has the shape of "type": value after this transformation.

Tag types externally (scalar)

Source
tag_types_externally(123)
Return
{
 "integer": 123
}
Learn more about the tag_types_externally function

Tag types externally (object)

Source
tag_types_externally({
	"message": "Hello world",
	"request": {
		"duration_ms": 67.9
	}
})
Return
{
 "message": {
  "string": "Hello world"
 },
 "request": {
  "duration_ms": {
   "float": 67.9
  }
 }
}
Learn more about the tag_types_externally function

Tag types externally (array)

Source
tag_types_externally(["foo", "bar"])
Return
[{"string":"foo"},{"string":"bar"}]
Learn more about the tag_types_externally function

Tag types externally (null)

Source
tag_types_externally(null)
Learn more about the tag_types_externally function

timestamp

Returns the value if it’s a timestamp and errors otherwise. This enables the type checker to guarantee that the returned value is a timestamp and can be used in any function that expects one.

Declare a timestamp type

Source
ok, err = timestamp(.timestamp)
Return
function call error for "timestamp" at (10:31): expected "timestamp", got "string"
Learn more about the timestamp function