Vector Remap Language (VRL) is an expression-oriented language designed for transforming observability data (logs and metrics) in a safe and performant manner. It features a simple syntax and a rich set of built-in functions tailored specifically to observability use cases.
All language constructs are contained in the following reference pages. Use these references as you write your VRL programs:
VRL is designed to minimize the learning curve as much as possible. The following resources will help you get acquainted with Vector and the language in just a few minutes:
How It Works
VRL is purpose-built by the Vector team and designed to exploit 2 principles: performance and safety, without compromising on flexibility. This makes VRL ideal for critical, performance-sensitive infrastructure, like observabiity pipelines. To illustrate how we achieve these, below is a VRL feature matrix across these principles:
|Vector & Rust native||✔||✔|
VRL programs are compiled to and run as native Rust code. This has several important implications:
- VRL programs are extremely fast and efficient, with performance characteristics very close to Rust itself
- VRL has no runtime and thus imposes no per-event foreign function interface (FFI) or data conversion costs
- VRL has no garbage collection, which means no GC pauses and no accumulated memory usage across events
Fail safety checks
At compile time, Vector performs fail safety checks to ensure that all errors thrown by fallible functions are handled. If you fail to pass a string to the
parse_syslogfunction, for example, the VRL compiler aborts and provides a helpful error message. Fail safety means that you need to make explicit decisions about how to handle potentially malformed data—a superior alternative to being surprised by such issues when Vector is already handling your data in production.
Type safety checks
At compile time, Vector performs type safety checks to catch runtime errors stemming from type mismatches, for example passing an integer to the
parse_syslogfunction, which can only take a string. VRL essentially forces you to write programs around the assumption that every incoming event could be malformed, which provides a strong bulwark against both human error and also the many potential consequences of malformed data.
VRL is ergonomically safe in that it makes it difficult to create slow or buggy VRL programs. While VRL's compile-time checks prevent runtime errors, they can't prevent some of the more elusive performance and maintainability problems that stem from program complexity—problems that can result in observability pipeline instability and unexpected resource costs. To protect against these more subtle ergonomic problems, VRL is a carefully limited language that offers only those features necessary to transform observability data. Any features that are extraneous to that task or likely to result in degraded ergonomics are omitted from the language by design.
Internal logging limitation
VRL programs do produce internal logs but not a rate that's bound to saturate I/O.
VRL lacks access to system I/O, which tends to be computationally expensive, to require careful caching, and to produce degraded performance.
Lack of recursion
VRL lacks recursion capabilities, making it impossible to create large or infinite loops that could stall VRL programs or needlessly drain memory.
Lack of custom functions
VRL requires you to use only its built-in functions and doesn't enable you to create your own. This keeps VRL programs easy to debug and reason about.
Lack of state
VRL lacks the ability to hold and maintain state across events. This prevents things like unbounded memory growth, hard-to-debug production issues, and unexpected program behavior.
logfunction implements rate limiting by default. This ensures that VRL programs invoking the
logmethod don't accidentally saturate I/O.
Purpose built for observability
VRL is laser focused on observability use cases and only those use cases. This makes many frustration- and complexity-producing constructs you find in other languages completely superfluous. Functions like
parse_key_value, for example, make otherwise complex tasks simple and prevent the need for complex low-level constructs.
VRL programs are fail safe, meaning that a VRL program won't compile unless all errors thrown by fallible functions are handled. This eliminates unexpected runtime errors that often plague production observability pipelines with data loss and downtime. See the error reference for more information on VRL errors.
Logs and metrics
Quality error messages
VRL strives to provide high-quality, helpful error messages, streamling the development and iteration workflow around VRL programs.
This VRL program, for example...
.foo, err = upcase(.foo)
...would result in this error:
error: program aborted┌─ :2:1│2 │ parse_json!(1)│ ^^^^^^^^^^^^^^│ ││ function call error│ unable to parse json: key must be a string at line 1 column 3│= see function documentation at: https://master.vector.dev/docs/reference/vrl/functions/#parse_json= see language documentation at: /docs/reference/vrl/
VRL programs are stateless, operating on a single event at a time. This makes VRL programs simple, fast, and safe. Operations involving state across events, such as deduplication, are delegated to other Vector transforms designed specifically for stateful operations.
VRL's type-safety is progressive, meaning it will implement type-safety for any value for which it knows the type. Because observability data can be quite unpredictable, it's not always known which type a field might be, hence the progressive nature of VRL's type-safety. As VRL scripts are evaluated, type information is built up and used at compile-time to enforce type-safety. Let's look at an example:.foo # any.foo = downcase!(.foo) # string.foo = upcase(.foo) # string
Breaking down the above:
.foofield starts off as an
anytype (AKA unknown).
- The call to the
downcase!function requires error handling (
!) since VRL cannot guarantee that
.foois a string (the only type supported by
- Afterwards, assuming the
downcaseinvocation is successful, VRL knows that
.foois a string, since
downcasecan only return strings.
- Finally, the call to
upcasedoes not require error handling (
!) since VRL knows that
.foois a string, making the
To avoid error handling for argument errors, you can specify the types of your fields at the top of your VRL script:.foo = string!(.foo) # string.foo = downcase(.foo) # string
This is generally good practice, and it provides the ability to opt-into type safety as you see fit, VRL scripts are written once and evaluated many times, therefore the tradeoff for type safety will ensure reliable production execution.
Vector & Rust native
Like Vector, VRL is built with Rust and compiles to native Rust code. Therefore, it inherits Rust's safety and performance characteristics that make it ideal for observability pipelines. And because both VRL and Vector are written in Rust, they are tightly integrated, avoiding communication inefficiencies such as event serialization or foreign function interfaces (FFI). This makes VRL significantly faster than non-Rust alternatives.
Lack of garbage collection
Rust's affine type system avoids the need for garbage collection, making VRL exceptionally fast, memory efficient, and memory safe. Memory is precisely allocated and freed, avoiding the pauses and performance pitfalls associated with garbage collectors.
VRL inherits Rusts's memory safety guarantees, protecting you from common software bugs and security vulnerabilities that stem from improper memory access. This makes VRL ideal for infrastructure use cases, like observability pipelines, where reliability and security are top concerns.
VRL is implemented in the very fast and efficient Rust language and VRL scripts are compiled into Rust code when Vector is started. This means that you can use VRL to transform observability data with a minimal per-event performance penalty vis-à-vis pure Rust. In addition, ergonomic features such as compile-time correctness checks and the lack of language constructs like loops make it difficult to write scripts that are slow or buggy or require optimization.
VRL is a safe language in several senses: VRL scripts have access only to the event data that they handle and not, for example, to the Internet or the host; VRL provides the same strong memory safety guarantees as Rust; and, as mentioned above, compile-time correctness checks prevent VRL scripts from behaving in unexpected or sub-optimal ways. These factors distinguish VRL from other available event data transformation languages and runtimes.