Lua Transform
The Vector lua
transform
transform events with a full embedded Lua 5.4 engine.
Warnings
Configuration
- Common
- Advanced
- vector.toml
- vector.yaml
- vector.json
[transforms.my_transform_id]# Generaltype = "lua" # requiredinputs = ["my-source-or-transform-id", "prefix-*"] # requiredversion = "2" # required# Hookshooks.process = '''function (event, emit)event.log.field = "value" -- set value of a fieldevent.log.another_field = nil -- remove fieldevent.log.first, event.log.second = nil, event.log.first -- rename field-- Very important! Emit the processed event.emit(event)end'''
- commonrequiredtable
hooks
Configures hooks handlers.
- optionalstring
init
A function which is called when the first event comes, before calling
hooks.process
- Syntax:
literal
- View examples
- Syntax:
- commonrequiredstring
process
A function which is called for each incoming event. It can produce new events using
emit
function. See Event Data Model for more info.- Syntax:
literal
- View examples
- Syntax:
- optionalstring
shutdown
A function which is called when Vector is stopped. It can produce new events using
emit
function.- Syntax:
literal
- View examples
- Syntax:
- optional[string]
search_dirs
A list of directories to search when loading a Lua file via the
require
function. If not specified, the modules are looked up in the directories of Vector's configs. See Search Directories for more info.- View examples
- optionalstring
source
The source which is evaluated when the transform is created.
- Syntax:
literal
- View examples
- Syntax:
- optional[table]
timers
Configures timers which are executed periodically at given interval.
- commonrequiredstring
handler
Defines a handler function which is executed periodially at
interval_seconds
. It can produce new events usingemit
function.- Syntax:
literal
- View examples
- Syntax:
- commonrequireduint
interval_seconds
Defines the interval at which the timer handler would be executed.
- View examples
- enumcommonrequiredstring
version
Transform API version. Specifying this version ensures that Vector does not break backward compatibility.
- Syntax:
literal
- Enum, must be one of:
"2"
- View examples
- Syntax:
Telemetry
This component provides the following metrics that can be retrieved through
the internal_metrics
source. See the
metrics section in the
monitoring page for more info.
- gauge
memory_used_bytes
The total memory currently being used by Vector (in bytes). This metric includes the following tags:
instance
- The Vector instance identified by host and port.job
- The name of the job producing Vector metrics.
- counter
events_in_total
The total number of events accepted by this component. This metric includes the following tags:
component_kind
- The Vector component kind.component_name
- The Vector component ID.component_type
- The Vector component type.instance
- The Vector instance identified by host and port.job
- The name of the job producing Vector metrics.
- counter
processed_events_total
The total number of events processed by this component. This metric includes the following tags:
component_kind
- The Vector component kind.component_name
- The Vector component ID.component_type
- The Vector component type.file
- The file that produced the errorinstance
- The Vector instance identified by host and port.job
- The name of the job producing Vector metrics.
- counter
processing_errors_total
The total number of processing errors encountered by this component. This metric includes the following tags:
component_kind
- The Vector component kind.component_name
- The Vector component ID.component_type
- The Vector component type.error_type
- The type of the errorinstance
- The Vector instance identified by host and port.job
- The name of the job producing Vector metrics.
- counter
events_out_total
The total number of events emitted by this component. This metric includes the following tags:
component_kind
- The Vector component kind.component_name
- The Vector component ID.component_type
- The Vector component type.instance
- The Vector instance identified by host and port.job
- The name of the job producing Vector metrics.
- counter
processed_bytes_total
The total number of bytes processed by the component. This metric includes the following tags:
component_kind
- The Vector component kind.component_name
- The Vector component ID.component_type
- The Vector component type.instance
- The Vector instance identified by host and port.job
- The name of the job producing Vector metrics.
Examples
Given the following Vector log event:
{"field_to_rename": "old value","field_to_remove": "remove me"}
And the following configuration:
[transforms.lua]type = "lua"hooks.process = '''function (event, emit)-- Add root level fieldevent.log.field = "new value"-- Add nested fieldevent.log.nested.field = "nested value"-- Rename fieldevent.log.renamed_field = event.log.field_to_renameevent.log.field_to_rename = nil-- Remove fieldsevent.log.field_to_remove = nilemit(event)end'''
The following Vector log event will be output:
{"field": "new value","nested": {"field": "nested value"},"renamed_field": "old value"}
How It Works
Event Data Model
The process
hook takes an event
as its first argument.
Events are represented as tables in Lua
and follow Vector's data model exactly. Please refer to
Vector's data model reference for the event
schema. How Vector's types map to Lua's type are covered below.
Type Mappings
The correspondence between Vector's data types and Lua data type is summarized by the following table:
Vector Type | Lua Type | Comment |
---|---|---|
String | string | |
Integer | integer | |
Float | number | |
Boolean | boolean | |
Timestamp | table | There is no dedicated timestamp type in Lua. Timestamps are represented as tables using the convention defined by os.date and os.time . The table representation of a timestamp contains the fields year , month , day , hour , min , sec , nanosec , yday , wday , and isdst . If such a table is passed from Lua to Vector, the fields yday , wday , and isdst can be omitted. In addition to the os.time representation, Vector supports sub-second resolution with a nanosec field in the table. |
Null | empty string | In Lua setting the value of a table field to nil means deletion of this field. In addition, the length operator # does not work in the expected way with sequences containing nulls. Because of that Null values are encoded as empty strings. |
Map | table | |
Array | sequence | Sequences are a special case of tables. Indexes start from 1, following the Lua convention. |
Learning Lua
In order to write non-trivial transforms in Lua, one has to have basic understanding of Lua. Because Lua is an easy to learn language, reading a few first chapters of the official book or consulting the manual would suffice.
Search Directories
Vector provides a search_dirs
option that allows you to specify
absolute paths that will be searched when using the
Lua require
function. If this option is not
set, the directories of the configuration files will be used instead.
State
This component is stateful, meaning its behavior changes based on previous inputs (events). State is not preserved across restarts, therefore state-dependent behavior will reset between restarts and depend on the inputs (events) received since the most recent restart.