VRL expression reference

Syntax

VRL programs can be constructed with the following syntax rules.

Comment

A comment serves as program documentation and is identified with #. Each line must be preceeded with a # character. VRL currently does not allow for block comments.

Examples

# comment

# multi-line
# comment

Keywords

Keywords are reserved words that are used for primitive language features, such as if, and cannot be used as variable assignments or other custom directives. The following words are reserved:

abort
as
break
continue
else
false
for
if
impl
in
let
loop
null
return
self
std
then
this
true
type
until
use
while

Whitespace

Whitespace is any non-empty string as defined by the Unicode White_Space property.

VRL is a “free-form” language, meaning that all forms of whitespace serve only to separate tokens in the grammar, and have no semantic significance.

Literal expressions

As in most other languages, literals in VRL are values written exactly as they are meant to be interpreted. Literals include things like strings, Booleans, and integers.

Array

An array literal is a comma-delimited set of expressions that represents a contiguous growable array type.

Examples

[]

["first", "second", "third"]

["mixed", 1, 1.0, true, false, {"foo": "bar"}]

["first-level", ["second-level", ["third-level"]]

[.field1, .field2, to_int!("2"), variable_1]

[
  "expressions",
  1 + 2,
  2 == 5,
  true || false
]

Boolean

A Boolean literal represents a binary value which can only be either true or false.

Examples

true

false

Float

A float literal is a decimal representation of a 64-bit floating-point type (specifically, the “binary64” type defined in IEEE 754-2008).

A decimal floating-point literal consists of an integer part (decimal digits), a decimal point, a fractional part (decimal digits).

Examples

1_000_000.01

1000000.01

1.001

Characteristics

Limits

Floats in VRL can range from -1.7976931348623157E+308f64 to 1.7976931348623157E+308f64. Floats outside that range are wrapped.

Underscores

Floats can use underscore (_) characters instead of , to make them human readable. For example, 1_000_000.

Integer

An integer literal is a sequence of digits representing a 64-bit signed integer type.

Examples

1_000_000

Characteristics

Limits

Integers in VRL can range from -9223372036854775807 to 9223372036854775807. Integers outside that range are wrapped.

Underscore

Integers can use underscore (_) characters instead of , to make them human readable. For example, 1_000_000.

Null

A null literal is the absence of a defined value.

Examples

null

Object

An object literal is a growable key/value structure that is syntactically equivalent to a JSON object.

A well-formed JSON document is a valid VRL object.

Examples

{
	"field1": "value1",
	"field2": [ "value2", "value3", "value4" ],
	"field3": { "field4": "value5" }
}

{
	"field1": .some_path,
	"field2": some_variable,
	"field3": { "subfield": "some value" }
}

Characteristics

Ordering

Object fields are ordered alphabetically by the key in ascending order. Therefore, operations like encoding into JSON produce a string with keys that are in ascending alphabetical order.

Regular Expression

A regular expression literal represents a Regular Expression used for string matching and parsing.

Regular expressions are defined by the r sigil and wrapped with single quotes (r'...'). The value between the quotes uses the Rust regex syntax.

Examples

r'^Hello, World!$'

r'(?i)^Hello, World!$'

r'^\d{4}-\d{2}-\d{2}$'

r'(?P<y>\d{4})-(?P<m>\d{2})-(?P<d>\d{2})'

Characteristics

Flags

Regular expressions allow for flags. Flags can be combined, as in r'(?ixm)pattern', r'(?im)pattern', etc.

To learn more about regular expressions in Rust—and by extension in VRL—we strongly recommend the in-browser Rustexp expression editor and tester.

Enum options

Option	Description
`U`	Swap the meaning of x* and x*?
`i`	Case insensitive
`m`	Multi-line mode
`s`	Allow . to match
`u`	Unicode support (enabled by default)
`x`	Ignore whitespace

Named Captures

Regular expressions support named capture groups, allowing extractions to be associated with keys. Named captures should be preceded with a ?P<name> declaration. This regex, for example…

r'(?P<y>\d{4})-(?P<m>\d{2})-(?P<d>\d{2})'

…extracts captures with the y, m, and d keys.

String

A string literal is a UTF-8–encoded string. String literals can be raw or interpreted.

Raw string literals are composed of the uninterpreted (implicitly UTF-8-encoded) characters between single quotes identified with the s sigil and wrapped with single quotes (s'...'); in particular, backslashes have no special meaning and the string may contain newlines.

Interpreted string literals are character sequences between double quotes ("..."). Within the quotes, any character may appear except unescaped newline and unescaped double quote. The text between the quotes forms the result of the literal, with backslash escapes interpreted as defined below. Strings can be templated by enclosing variables in {{..}}. The value of the variables are inserted into the string at that position.

Examples

"Hello, world! 🌎"

"Hello, world! \u1F30E"

"Hello, \
 world!"

"Hello, {{ planet }}!"

s'Hello, world!'

s'{ "foo": "bar" }'

Characteristics

Backslash escapes

Special characters, such as newlines, can be expressed with a backslash escape.

Enum options

Option	Description
`\"`	Double quote
`\'`	Single quote
`\0`	Null
`\\`	Backslash
`\n`	Newline
`\r`	Carriage return
`\t`	Tab
`\u{7FFF}`	24-bit Unicode character code (up to 6 digits)
`\{`	Brace

Concatenation

Strings can be concatenated with the + operator.

Invalid Characters

Invalid UTF-8 sequences are replaced with the � character.

Multiline strings

Long strings can be split over multiple lines by adding a backslash just before the newline. The newline and any whitespace at the start of the ensuing line is not included in the string.

Templates

Strings can be templated by enclosing a variable name with {{..}}. The value of the variable is inserted into the string at this position at runtime. Currently, the variable has to be a string. Only variables are supported, if you want to insert a path from the event you must assign it to a variable first. To insert a {{ into the string it can be escaped with a \ escape: \{{..\}}. We plan to expand this in future to allow paths and format strings to enable non string variables.

Timestamp

A timestamp literal defines a native timestamp expressed in the RFC 3339 format with a nanosecond precision.

Timestamp literals are defined by the t sigil and wrapped with single quotes (t'2021-02-11T10:32:50.553955473Z').

Examples

t'2021-02-11T10:32:50.553955473Z'

t'2021-02-11T10:32:50.553Z'

t'2021-02-11T10:32:50.553-04:00'

Characteristics

Timezones

As defined in RFC 3339 format, timestamp literals support UTC and local offsets.

Dynamic expressions

VRL is an expression-oriented language. A VRL program consists entirely of expressions and every expression returns a value.

Abort

An abort expression causes the VRL program to terminate, aborting any modifications made to the event.

Grammar

abort ~ message?

Argument	Meaning
`message`	`message` is an optional debug message that can be used for diagnostic purposes and is included in a `remap` transform’s dropped event metadata.

Examples

Ignoring invalid events

Vector event

{
  "log": {
    "message": "hello world"
  }
}

VRL program

if contains(string!(.message), "hello") {
	abort
}
.message = "not hello world"

Resulting event

{
  "message": "hello world"
}

Arithmetic

An arithmetic expression performs an operation on two expressions (operands) as defined by the operator.

Although arithmetic is commonly applied to numbers, you can use it with other types as well, such as strings.

Grammar

expression ~ operator ~ expression

Argument	Meaning
`expression`	The `expression` can be any expression that returns a valid type as defined by the `operator`.
`operator`	The `operator` defines the operation performed on the left-hand- and right-hand-side operands.

Examples

Sum (int)

VRL program

1 + 1

Resulting event

Sum (float)

VRL program

1.0 + 1.0

Resulting event

Sum (numeric)

VRL program

1 + 1.0

Resulting event

Sum (string)

VRL program

"Hello" + ", " + "World!"

Resulting event

Hello, World!

Difference (int)

VRL program

2 - 1

Resulting event

Difference (float)

VRL program

2.0 - 1.0

Resulting event

Difference (numeric)

VRL program

2.0 - 1

Resulting event

Multiplication (int)

VRL program

2 * 1

Resulting event

Multiplication (float)

VRL program

2.0 * 1.0

Resulting event

Multiplication (numeric)

VRL program

2.0 * 1

Resulting event

Float division (int)

VRL program

2 / 1

Resulting event

Float division (float)

VRL program

2.0 / 1.0

Resulting event

Float division (numeric)

VRL program

2.0 / 1

Resulting event

Remainder

VRL program

mod(3, 2)

Resulting event

Assignment

An assignment expression assigns the result of the right-hand-side expression to the left-hand-side target (path or variable).

Grammar

target ~ ("," ~ error)? ~ operator ~ expression

Argument	Meaning
`error`	The `error` allows for optional assignment to errors when the right-hand-side expression is fallible. This is commonly used when invoking fallible functions.
`expression`	If the `target` is a variable, the `expression` can be any expression. If the `target` is a path, the `expression` can be any expression that returns a supported object value type (i.e. not a regular expression).
`operator`	The `operator` delimits the `target` and `expression` and defines assignment conditions.
`target`	The `target` must be a path, with an optional second variable for error handling if the right-hand side is fallible.

Examples

Path assignment

VRL program

.message = "Hello, World!"

Nested path assignment

VRL program

.parent.child = "Hello, World!"

Double assignment

VRL program

.first = .second = "Hello, World!"

Array element assignment

VRL program

.array[1] = "Hello, World!"

Variable assignment

VRL program

my_variable = "Hello, World!"

Resulting event

Hello, World!

Object merge assignment

VRL program

my_variable = {"message": "Hello, World!"}
my_variable |= {"level": "info"}

Resulting event

{
  "level": "info",
  "message": "Hello, World!"
}

Fallible assignment (success)

VRL program

.parsed, .err = parse_json("{\"Hello\": \"World!\"}")

Fallible assignment (error)

VRL program

.parsed, .err = parse_json("malformed")

Block

A block expression is a sequence of one or more expressions within matching brace brackets.

Blocks can’t be empty. Instead, empty blocks ({}) are treated as blank objects.

Grammar

"{" ~ NEWLINE* ~ expressions ~ NEWLINE* ~ "}"

Argument	Meaning
`expressions`	One or more expressions.

Examples

Simple block

VRL program

{
	message = "{\"Hello\": \"World!\"}"
	parse_json!(message)
}

Resulting event

{
  "Hello": "World!"
}

Assignment block

VRL program

.structured = {
	message = "{\"Hello\": \"World!\"}"
	parse_json!(message)
}

Coalesce

A coalesce expression is composed of multiple expressions (operands) delimited by a coalesce operator, short-circuiting on the first expression that doesn’t violate the operator condition.

Grammar

expression ~ (operator ~ expression)+

Argument	Meaning
`expression`	The `expression` (operand) can be any expression.
`operator`	The `operator` delimits two or more `expression`s.

Examples

Error coalescing

VRL program

parse_syslog("not syslog") ?? parse_common_log("not common") ?? "malformed"

Resulting event

malformed

Comparison

A comparison expression compares two expressions (operands) and produces a Boolean as defined by the operator.

Grammar

expression ~ operator ~ expression

Argument	Meaning
`expression`	The `expression` (operand) can be any expression that returns a valid type as defined by the `operator`.
`operator`	The `operator` defines the operation performed on the left-hand and right-hand side operations.

Examples

Equal

VRL program

1 == 1

Resulting event

true

Not equal

VRL program

1 != 2

Resulting event

true

Greater than or equal

VRL program

2 >= 2

Resulting event

true

Greater than

VRL program

2 > 1

Resulting event

true

Less than or equal

VRL program

2 <= 2

Resulting event

true

Less than

VRL program

1 < 2

Resulting event

true

Function call

A function call expression invokes built-in VRL functions.

Grammar

function ~ abort? ~ "(" ~ arguments? ~ ")" ~ closure?

Argument	Meaning
`abort`	`abort` represents a literal `!` that can optionally be used with fallible functions to abort the program when the function fails: `result = f!()` Otherwise, errors must be handled: `result, err = f()` Failure to handle errors from fallible functions results in compile-time errors. See the error reference for more info.
`arguments`	The `arguments` are comma-delimited expressions that can optionally be prefixed with the documented name.
`closure`	The `closure` is an optional piece of code resolved by the function call. It is primarily used in functions that iterate over collections. Its syntax is as follows: `for_each([]) -> \|index, value\| { ... }`
`function`	`function` represents the name of the built-in function.

Examples

Positional function invocation

VRL program

split("hello, world!", ", ")

Resulting event

["hello","world!"]

Named function invocation (ordered)

VRL program

split("hello, world!", pattern: ", ")

Resulting event

["hello","world!"]

Named function invocation (unordered)

VRL program

split(pattern: ", ", value: "hello, world!")

Resulting event

["hello","world!"]

Characteristics

Function Fallibility

VRL functions can be marked as “fallible” or “infallible”. When a function is defined as fallible, it can fail at runtime, requiring the error to be handled before the program can be compiled.

If a function is defined as infallible, it means that given the correct function arguments, the function can never fail at runtime, and thus no error handling is needed.

Note that even if a function is defined as infallible, if any of its arguments can fail at runtime, the function is considered to be fallible, and thus the error case needs to be handled in this case.

The VRL compiler ensures all potential errors in a program are handled, so there’s no need to worry about missing any potential runtime failures.

If

An if expression specifies the conditional execution of two branches according to the value of a Boolean expression. If the Boolean expression evaluates to true, the “if” branch is executed, otherwise the “else” branch is executed (if present).

Grammar

"if" ~ predicate ~ block ~ ("else if" ~ predicate ~ block)* ~ ("else" ~ block)?

Argument	Meaning
`predicate`	The `predicate` must be an expression that resolves to a Boolean. If a Boolean isn’t returned, a compile-time error is raised. The predicate can contain multiple expressions. Multiple expression predicates must be wrapped in parentheses. The expressions need to be separated by either a semicolon (`;`) or a new line.

Examples

True if expression

VRL program

if true {
	"Hello, World!"
}

Resulting event

Hello, World!

False if expression

VRL program

if false {
	# not evaluated
	null
}

If/else expression

VRL program

if false {
	# not evaluated
	null
} else {
	"Hello, World!"
}

Resulting event

Hello, World!

If/else if/else expression

VRL program

if false {
	# not evaluated
	null
} else if false {
	# not evaluated
	null
} else {
	"Hello, World!"
}

Resulting event

Hello, World!

Multiline expression

VRL program

x = 3
if (x = x + 1; x == 5) {
	# not evaluated
	null
} else if (
	x = x + 1
	x == 5
) {
	"Hello, World!"
}

Resulting event

Hello, World!

Index

An index expression denotes an element of an array. Array indices in VRL start at zero.

Grammar

"[" ~ index ~ "]"

Argument	Meaning
`index`	The `index` represents the zero-based position of the element.

Examples

Array index expression

Vector event

{
  "log": {
    "array": [
      "first",
      "second"
    ]
  }
}

VRL program

.array[0]

Resulting event

first

Logical

A logical expression compares two expressions (operands), short-circuiting on the last expression evaluated as defined by the operator.

Grammar

expression ~ operator ~ expression

Argument	Meaning
`expression`	The `expression` (operand) can be any expression that returns a valid type as defined by the `operator`.
`operator`	The `operator` defines the operation performed on the left-hand- and right-hand-side operations.

Examples

AND

VRL program

true && true

Resulting event

true

OR (boolean)

VRL program

false || "foo"

Resulting event

foo

OR (null)

VRL program

null || "foo"

Resulting event

foo

NOT

VRL program

!false

Resulting event

true

Path

A path expression is a sequence of period-delimited segments that represent the location of a value within an object.

Grammar

"." ~ path_segments

Argument	Meaning
`"."`	The `"."` character represents the root of the event. Therefore, all paths must begin with the `.` character, and `.` alone is a valid path.
`path_segments`	`path_segments` denote a segment of a nested path. Each segment must be delimited by a `.` character and only contain alpha-numeric characters and `_` (`a-zA-Z0-9_`). Segments that contain characters outside of this range must be quoted.

Examples

Root path

Vector event

{
  "log": {
    "message": "Hello, World!"
  }
}

VRL program

Resulting event

{
  "message": "Hello, World!"
}

Top-level path

Vector event

{
  "log": {
    "message": "Hello, World!"
  }
}

VRL program

.message

Resulting event

Hello, World!

Nested path

Vector event

{
  "log": {
    "parent": {
      "child": "Hello, World!"
    }
  }
}

VRL program

.parent.child

Resulting event

Hello, World!

Nested path coalescing

Vector event

{
  "log": {
    "grand_parent": {
      "parent2": {
        "child": "Hello, World!"
      }
    }
  }
}

VRL program

.grand_parent.(parent1 | parent2).child

Resulting event

Hello, World!

Array element path (first)

Vector event

{
  "log": {
    "array": [
      "first",
      "second"
    ]
  }
}

VRL program

.array[0]

Resulting event

first

Array element path (second)

Vector event

{
  "log": {
    "array": [
      "first",
      "second"
    ]
  }
}

VRL program

.array[1]

Resulting event

second

Quoted path

Vector event

{
  "log": {
    "parent.key.with.special characters": {
      "child": "Hello, World!"
    }
  }
}

VRL program

."parent.key.with.special characters".child

Resulting event

Hello, World!

Variable

A variable expression names variables. A variable is a sequence of one or more letters and digits. The first character in a variable must be a letter.

Grammar

first ~ (trailing)*

Argument	Meaning
`first`	The `first` character can only be an alpha-numeric character (`a-zA-Z0-9`).
`trailing`	The `trailing` characters must only contain ASCII alpha-numeric and underscore characters (`a-zA-Z0-9_`).

Examples

Simple variable

VRL program

my_variable = 1

Resulting event

Variable with path

VRL program

my_object = { "one": 1 }
my_object.one

Resulting event