Remap Transform
The Vector remap
transform
is the recommended transform for parsing, shaping, and transforming data in
Vector. It implements the Vector Remap
Language (VRL), an expression-oriented
language designed for processing observability data (logs and metrics) in a
safe and performant manner. Please refer to the VRL
reference when writing VRL scripts.
Configuration
- vector.toml
- vector.yaml
- vector.json
[transforms.my_transform_id]type = "remap" # requiredinputs = ["my-source-or-transform-id", "prefix-*"] # requiredsource = '''. = parse_json(.message).new_field = "new value".status = to_int(.status).duration = parse_duration(.duration, "s").new_name = .old_namedel(.old_name)'''
- commonrequiredstring
source
The Vector Remap Language (VRL) program to execute for each event.
This field accepts a full Vector Remap Language (VRL) program. Please refer to the Vector Remap Language reference for a list of expressions, functions, and examples.
- Syntax:
remap_program
- View examples
- Syntax:
Telemetry
This component provides the following metrics that can be retrieved through
the internal_metrics
source. See the
metrics section in the
monitoring page for more info.
- counter
processing_errors_total
The total number of processing errors encountered by this component. This metric includes the following tags:
component_kind
- The Vector component kind.component_name
- The Vector component ID.component_type
- The Vector component type.error_type
- The type of the errorinstance
- The Vector instance identified by host and port.job
- The name of the job producing Vector metrics.
- counter
events_in_total
The total number of events accepted by this component. This metric includes the following tags:
component_kind
- The Vector component kind.component_name
- The Vector component ID.component_type
- The Vector component type.instance
- The Vector instance identified by host and port.job
- The name of the job producing Vector metrics.
- counter
processed_events_total
The total number of events processed by this component. This metric includes the following tags:
component_kind
- The Vector component kind.component_name
- The Vector component ID.component_type
- The Vector component type.file
- The file that produced the errorinstance
- The Vector instance identified by host and port.job
- The name of the job producing Vector metrics.
- counter
events_out_total
The total number of events emitted by this component. This metric includes the following tags:
component_kind
- The Vector component kind.component_name
- The Vector component ID.component_type
- The Vector component type.instance
- The Vector instance identified by host and port.job
- The name of the job producing Vector metrics.
- counter
processed_bytes_total
The total number of bytes processed by the component. This metric includes the following tags:
component_kind
- The Vector component kind.component_name
- The Vector component ID.component_type
- The Vector component type.instance
- The Vector instance identified by host and port.job
- The name of the job producing Vector metrics.
Examples
Given the following Vector log event:
{"message": "<102>1 2020-12-22T15:22:31.111Z vector-user.biz su 2666 ID389 - Something went wrong"}
And the following configuration:
[transforms.remap]type = "remap"source = """structured = parse_syslog!(.message). = merge(., structured)"""
The following Vector log event will be output:
{"appname": "su","facility": "ntp","hostname": "vector-user.biz","message": "Something went wrong","msgid": "ID389","procid": 2666,"severity": "info","timestamp": "2020-12-22T15:22:31.111Z"}
How It Works
State
This component is stateless, meaning its behavior is consistent across each input.
Vector Remap Language
The Vector Remap Language (VRL) is a restrictive, fast, and safe language we designed specifically for mapping observability data. It avoids the need to chain together many fundamental Vector transforms to accomplish rudimentary reshaping of data.
The intent is to offer the same robustness of full language runtime (ex: Lua) without paying the performance or safety penalty.
Learn more about Vector's Remap Language in the Vector Remap Language reference.