Journald Source
The Vector journald
source
collects logs from
JournalD.
Configuration
- Common
- Advanced
- vector.toml
- vector.yaml
- vector.json
[sources.my_source_id]type = "journald" # requiredcurrent_boot_only = true # optional, defaultexclude_units = [] # optional, defaultinclude_units = [] # optional, default
- optionaluint
batch_size
The systemd journal is read in batches, and a checkpoint is set at the end of each batch. This option limits the size of the batch.
- Default:
16
- Default:
- commonoptionalbool
current_boot_only
Include only entries from the current boot.
- Default:
true
- View examples
- Default:
- optionalstring
data_dir
The directory used to persist file checkpoint positions. By default, the global
data_dir
option is used. Please make sure the Vector project has write permissions to this dir. See Checkpointing for more info.This field accepts a valid file system path.
- Syntax:
file_system_path
- View examples
- Syntax:
- commonoptional[string]
exclude_units
The list of unit names to exclude from monitoring. Unit names lacking a
"."
will have".service"
appended to make them a valid service unit name.- Default:
[]
- View examples
- Default:
- commonoptional[string]
include_units
The list of unit names to monitor. If empty or not present, all units are accepted. Unit names lacking a
"."
will have".service"
appended to make them a valid service unit name.- Default:
[]
- View examples
- Default:
- optionalstring
journalctl_path
The full path of the
journalctl
executable. If not set, Vector will search the path forjournalctl
. See Communication Strategy for more info.- Syntax:
literal
- Default:
"journalctl"
- View examples
- Syntax:
Output
This component outputs log events with the following fields:
{"*" : "/usr/sbin/ntpd","host" : "my-host.local","message" : "53.126.150.246 - - [01/Oct/2020:11:25:58 -0400] \"GET /disintermediate HTTP/2.0\" 401 20308","timestamp" : "2020-10-10T17:07:36+00:00"}
- commonrequiredstring
host
The local hostname, equivalent to the
gethostname
command.- Syntax:
literal
- View examples
- Syntax:
- commonrequiredstring
message
The raw line from the file.
- Syntax:
literal
- View examples
- Syntax:
- commonrequiredtimestamp
timestamp
The exact time the event was ingested into Vector.
- View examples
- optionalstring
*
Any Journald field
- Syntax:
literal
- View examples
- Syntax:
Telemetry
This component provides the following metrics that can be retrieved through
the internal_metrics
source. See the
metrics section in the
monitoring page for more info.
- counter
invalid_record_total
The total number of invalid records that have been discarded. This metric includes the following tags:
component_kind
- The Vector component kind.component_name
- The Vector component ID.component_type
- The Vector component type.instance
- The Vector instance identified by host and port.job
- The name of the job producing Vector metrics.
- counter
invalid_record_bytes_total
The total number of bytes from invalid records that have been discarded. This metric includes the following tags:
component_kind
- The Vector component kind.component_name
- The Vector component ID.component_type
- The Vector component type.instance
- The Vector instance identified by host and port.job
- The name of the job producing Vector metrics.
- counter
processed_bytes_total
The total number of bytes processed by the component. This metric includes the following tags:
component_kind
- The Vector component kind.component_name
- The Vector component ID.component_type
- The Vector component type.instance
- The Vector instance identified by host and port.job
- The name of the job producing Vector metrics.
- counter
events_out_total
The total number of events emitted by this component. This metric includes the following tags:
component_kind
- The Vector component kind.component_name
- The Vector component ID.component_type
- The Vector component type.instance
- The Vector instance identified by host and port.job
- The name of the job producing Vector metrics.
- counter
processed_events_total
The total number of events processed by this component. This metric includes the following tags:
component_kind
- The Vector component kind.component_name
- The Vector component ID.component_type
- The Vector component type.file
- The file that produced the errorinstance
- The Vector instance identified by host and port.job
- The name of the job producing Vector metrics.
Examples
Given the following input:
2019-07-26 20:30:27 reply from 192.168.1.2: offset -0.001791 delay 0.000176, next query 1500s
And the following configuration:
[sources.journald]type = "journald"
The following Vector log event will be output:
[{"timestamp": "2020-10-10T17:07:36.452332Z","message": "reply from 192.168.1.2: offset -0.001791 delay 0.000176, next query 1500s","host": "my-host.local","__REALTIME_TIMESTAMP": "1564173027000443","__MONOTONIC_TIMESTAMP": "98694000446","_BOOT_ID": "124c781146e841ae8d9b4590df8b9231","SYSLOG_FACILITY": "3","_UID": "0","_GID": "0","_CAP_EFFECTIVE": "3fffffffff","_MACHINE_ID": "c36e9ea52800a19d214cb71b53263a28","PRIORITY": "6","_TRANSPORT": "stdout","_STREAM_ID": "92c79f4b45c4457490ebdefece29995e","SYSLOG_IDENTIFIER": "ntpd","_PID": "2156","_COMM": "ntpd","_EXE": "/usr/sbin/ntpd","_CMDLINE": "ntpd: [priv]","_SYSTEMD_CGROUP": "/system.slice/ntpd.service","_SYSTEMD_UNIT": "ntpd.service","_SYSTEMD_SLICE": "system.slice","_SYSTEMD_INVOCATION_ID": "496ad5cd046d48e29f37f559a6d176f8"}]
How It Works
Checkpointing
Vector checkpoints the current read position after each
successful read. This ensures that Vector resumes where it left
off if restarted, preventing data from being read twice. The
checkpoint positions are stored in the data directory which is
specified via the global data_dir
option, but can be overridden
via the data_dir
option in the file source directly.
Communication Strategy
To ensure the journald
source works across all platforms, Vector interacts
with the Systemd journal via the journalctl
command. This is accomplished by
spawning a subprocess that Vector interacts
with. If the journalctl
command is not in the environment path you can
specify the exact location via the journalctl_path
option. For more
information on this communication strategy please see
issue #1473.
Context
By default, the journald
source will augment events with helpful
context keys as shown in the "Output" section.
Non-ASCII Messages
When journald
has stored a message that is not strict ASCII,
journalctl
will output it in an alternate format to prevent data
loss. Vector handles this alternate format by translating such messages
into UTF-8 in "lossy" mode, where characters that are not valid UTF-8
are replaced with the Unicode replacement character, �
.
State
This component is stateless, meaning its behavior is consistent across each input.