Vector v0.12.0 Release Notes

Vector Remap Language
Feb 11th, 2021 /

The Vector team is pleased to announce 0.12.0. This release introduces the Vector Remap Language, an expression-oriented language designed for transforming observability data (logs and metrics) in a safe and performant manner.

Check out the VRL announcement post for more details as well as the highlights and changelog for a complete list of changes in 0.12.0.

Highlights

Changelog

  • 82 Enhancements

    • socket sourceAdd UNIX datagram mode

    • remapsupport path queries for variables

    • remapadd support for literal arrays

    • remapaccept array of arguments for function parameters

    • kubernetes platformSupport exposing internal mertics out of the box in our Helm charts

    • networkingImplement TCP keepalive `time` configuration

    • docker_logs sourceEnable excluding Docker containers via configuration

    • codecRemove default `encoding.codec` where appropriate

    • prometheus sinkExport metadata to remote_write

    • authAdd support for basic auth credendtials in URL for more sinks

    • remapadd integer division

    • remapallow multiple statements in boolean conditional

    • docker_logs sourceAdd host and TLS configuations

    • remaptrack inner container type definitions

    • remapadd support for literal maps

    • kubernetes platformExpose host metrics in our Helm charts out of the box

    • configAdd global `healthchecks` option

    • remap transformadd `parse_aws_alb_log` function

    • remapimprove error messages, part 1

    • remap transformadd `parse_aws_vpc_flow_log` function

    • elasticsearch sinkadd support for data streams

    • generator sourceEnable format selection for generator source

    • remapAdd includes function for arrays

    • remapAdd to_syslog_level function for parsing Syslog levels into strings

    • remap transformdocs for remap transform metrics

    • remapAdd to_syslog_severity function for Syslog log level keywords

    • codecobservabilityPrint warning on encoding error

    • remapadd encode_json function

    • networkingsecurityhumio sinkSupport TLS configuration for Humio sinks

    • graphql apiFile source metrics

    • docker_logs sourceAdd hostname to Docker source

    • generator sourceAdd JSON format

    • generator sourceAdd RFC 3164 format

    • http sourceAdd `gzip` & `deflate` decompression

    • graphql api`componentByName` query

    • kafka sinkBetter tracing/tests on rdkafka options.

    • kubernetes platformAdds script for generating README.md for Agent role helm-chart

    • ciBuild ARMv7 images for all supported platforms

    • remap transformadd `parse_aws_cloudwatch_log_subscription_message` function

    • remapEnable casting timestamps into integers

    • kafka sinkallow send metrics to kafka

    • remapnon-boolean if conditional compile-time error

    • graphql apiRelay connections (components, initially)

    • graphql apiRelay-compatible connections for sources/transforms/sinks

    • remapAdd is_nullish function to VRL

    • remapMove VRL CLI into the Vector CLI

    • remapbare variable names

    • remapAdd to_syslog_facility function for parsing syslog facilities into strings

    • remapAdd to_unix_timestamp function

    • remapAdd push and append functions

    • remapAdd base64 encoding and decoding functions

    • configremove the check_fields default for conditions

    • kubernetes platformAllow passing component configs as YAML in Helm charts

    • remapOpen docs URL for help in VRL REPL

    • remapPretty print functions list in VRL REPL

    • prometheus_remote_write sinkSupport X-Scope-OrgID header

    • configdeprecate `check_fields` conditions

    • remapAdd charset parameter to encode_base64 function

    • prometheus_exporter sinkadd tls support

    • internal_logs sourceCapture initialization logs

    • remapNew `get_env_var` Remap function

    • graphql apiFiltering in the API

    • remapUpdate VRL CLI behavior

    • remapAdd length function for VRL

    • graphql apiAPI sorting, starting with `components`

    • graphql apiSorting for sources/transforms/sinks

    • file sourceAdd encoding charset and line delimiter configuration

    • postgresql_metrics sourcesupport regexp in filters

    • authaws platformAdd `auth.access_key_id` and `auth.secret_access_key` options

    • remap transformadd `get_hostname` function

    • kubernetes platformadd the ability to configure Vector API at the Helm charts

    • performanceAdd more benchmark comparisons

    • file sourcesplit start_at_beginning into simpler options

    • networkingMake size of socket send/receive buffers configurable

    • remapmaintain type definitions for maps

    • remapAdd charset option to decode_base64 function

    • host_metrics sourceAdd `uptime` & `boot_time` metrics

    • remapAdd join function

    • graphql api`FileSourceMetricFile` sort

    • file sourceLog IO errors when globbing

    • templatingAdd support for metrics in templates

    • releasingchange release profile

  • 22 New Features

    • remapadd `redact` function

    • administrationHelm chart for deploying Vector with an aggregator role

    • remapadd initial "trl" CLI

    • remapsupport multiline expressions

    • new sourceNew internal_logs source

    • remapadd initial CLI REPL support

    • remapadd remap functions to CLI

    • remapadd `ok` error handling function

    • remapenable remap to work with metric events

    • remapadd `parse_regex` and `parse_regex_all` remap functions

    • remapadded `parse_key_value` remap function

    • remapadd error coalescing operator

    • remapsupport "abort on error" bang-function-calls

    • remaperror-return-value error handling pattern

    • file sourceallow user configuration of the `ignore_not_found` option

    • remapdiagnostic error messages

    • remapimprove test harness set-up

    • configadd wildcard expansion to inputs

    • prometheus_remote_write sinkAdd HTTP auth support

    • remap transformAdd `parse_common_log` function

    • remap transformAdd `parse_glog` function

    • kubernetes platformExpose node-port for vector-aggregator

  • 63 Bug Fixes

    • kubernetes platformAdd exclusion label at vector-aggregator

    • configfixing Compression parsing with tests

    • testsrequire features for generate_configfile test

    • remapFix build issue caused by macro removal

    • metricsAdd missing methods to Subscriber forwarder

    • kubernetes platformCorrect the prometheus format parsing at K8s E2E tests for the metrics pipeline

    • kubernetes platformImplement improvements to the Pod recreation on Helm release upgrades

    • observabilityEmit `processed_events_total` in topology for transforms

    • prometheus sinkfix set metric reset on flush period

    • documentationtenant_id is templatable on loki sink

    • remapFix type definition for `includes()`

    • testspodman pod stop/rm don't need a --name option

    • networkingAllow owning the same port with different protocols

    • journald sourcehandle incorrect int sequences as strings

    • journald sourcefix tests for fake journal

    • http sourcelower handle http req log to debug

    • internal_metrics sourceMake internal metrics pass promtools check

    • syslog sourcedo not use static year in tests

    • metricsTrivial fixes for metrics batch buffer

    • internal docsCorrect misspelling

    • depsfix features for lib/shared

    • remapcorrectly handle nested map insertions

    • docker_logs sourceFix HTTP and HTTPS URL

    • testsHandle early return from influxdb create database

    • prometheus_exporter sinkFix prometheus exposition format

    • configAvoid alias `in_flight_limit`

    • remapRename tokenize to parse_tokens

    • remapMake the redact function fallible

    • testsSet `core_threads` for `merge_and_fork`

    • loki sinkFix remove_label_fields

    • influxdb_metrics sinkFix transmission of counter metrics

    • kubernetes platformAdjusting pod security policy to allow vector-agent to read host logs

    • kubernetes platformAdjusting to adhere to linting rules used by ct

    • remapallow comments between multiline statements

    • internal_metrics sourceRework defaults and a minimal scraping interval value

    • buffersTrigger `leveldb` compact after certain amount of deleted bytes

    • kubernetes_logs sourceSupport for collecting logs from static pods

    • prometheus_exporter sinkFix Metrics not Updating

    • remap`ipv6_to_ipv4` function allows ipv4 addresses to pass through

    • remapUpdate VRL docs URLs

    • docker_logs sourcefix and document exclude_self logic

    • aws_cloudwatch_logs sinkRename healthcheck error

    • nats sinkcall ack on any result

    • remapfix field sublocation_type in parse_aws_vpc_flow_log

    • teststest again released version of loki

    • remap`parse_grok` should error when it fails to parse

    • configdisallow non-unique component names

    • metricsFix metrics::tests::test_cardinality_metric

    • testsuse poll backend for watcher

    • loki sinkFix out-of-order error

    • testsrevert poll watcher change

    • metricsRework the metrics batch buffer

    • deployment#5642 - build deb packages that install on Deb 8

    • remapRe-flatten the remap-functions directory tree

    • topologyFix handling of component errors

    • file sourceDeprecate fingerprint.bytes option

    • dockerdependenciesUpdate bollard to 0.9.1 to fix #5937

    • observabilityEmit `processed_events_total` after transform has processed event

    • mongodb_metrics sourcepostgresql_metrics sourceremove connect in SourceConfig::build

    • metricssinksMove normalization out of MetricsBuffer

    • wasm transformFix parse_json Wasm function

    • templatingFix merge error with test cases

    • kubernetes_logs sourceHandle OK response from api_watcher with embedded desync elegantly

  • 2 Perf Improvements

    • Switch to `futures-0.3` channels

    • lua transformFix lua_field_filter/v2 implementation

What's Next

  • Vector's own observability

    We will be working to improve Vector's own observability with high-quality internal metrics, logs, CLI tools, and dashboards.

  • Improved Vector to Vector communication

    The vector source and sink currently leverage the TCP protocol to transmit data. While this has served users well, it presents operational challenges for large-scale deployments of Vector. Because HTTP is widely used and easy to integrate into platforms like Kubernetes we will be moving these components to the HTTP protocol.

  • Schema support

    With the inclusion of type-safety in VRL we will be working to officially support common and custom schemas within Vector. This will be an opt-in feature that will provide Vector with type information about your data, making Vector type safe from end to end and further raising confidence that Vector will operate properly post-deployment.

The Vector roadmap is publicly available and represents forward looking progress. Vector is open-source and community driven. Your feedback is welcome!

Like What You See?