collect

Journald Source

The Vector journald source collects logs from JournalD.

Configuration

[sources.my_source_id]
  type = "journald" # required
  current_boot_only = true # optional, default
  exclude_units = [] # optional, default
  include_units = [] # optional, default
6 items
  • optionaluint

    batch_size

    The systemd journal is read in batches, and a checkpoint is set at the end of each batch. This option limits the size of the batch.

    • Default: 16
  • commonoptionalbool

    current_boot_only

    Include only entries from the current boot.

    • Default: true
    • View examples
  • optionalstring

    data_dir

    The directory used to persist file checkpoint positions. By default, the global data_dir option is used. Please make sure the Vector project has write permissions to this dir. See Checkpointing for more info.

    This field accepts a valid file system path.

    • Syntax: file_system_path
    • View examples
  • commonoptional[string]

    exclude_units

    The list of unit names to exclude from monitoring. Unit names lacking a "." will have ".service" appended to make them a valid service unit name.

    • Default: []
    • View examples
  • commonoptional[string]

    include_units

    The list of unit names to monitor. If empty or not present, all units are accepted. Unit names lacking a "." will have ".service" appended to make them a valid service unit name.

    • Default: []
    • View examples
  • optionalstring

    journalctl_path

    The full path of the journalctl executable. If not set, Vector will search the path for journalctl. See Communication Strategy for more info.

    • Syntax: literal
    • Default: "journalctl"
    • View examples

Output

This component outputs log events with the following fields:

{
  "*" : "/usr/sbin/ntpd",
  "host" : "my-host.local",
  "message" : "53.126.150.246 - - [01/Oct/2020:11:25:58 -0400] \"GET /disintermediate HTTP/2.0\" 401 20308",
  "timestamp" : "2020-10-10T17:07:36+00:00"
}
  • commonrequiredstring

    host

    The local hostname, equivalent to the gethostname command.

    • Syntax: literal
    • View examples
  • commonrequiredstring

    message

    The raw line from the file.

    • Syntax: literal
    • View examples
  • commonrequiredtimestamp

    timestamp

    The exact time the event was ingested into Vector.

    • View examples
  • optionalstring

    *

    Any Journald field

    • Syntax: literal
    • View examples

Telemetry

This component provides the following metrics that can be retrieved through the internal_metrics source. See the metrics section in the monitoring page for more info.

  • counter

    invalid_record_total

    The total number of invalid records that have been discarded. This metric includes the following tags:

    • component_kind - The Vector component kind.

    • component_name - The Vector component ID.

    • component_type - The Vector component type.

    • instance - The Vector instance identified by host and port.

    • job - The name of the job producing Vector metrics.

  • counter

    invalid_record_bytes_total

    The total number of bytes from invalid records that have been discarded. This metric includes the following tags:

    • component_kind - The Vector component kind.

    • component_name - The Vector component ID.

    • component_type - The Vector component type.

    • instance - The Vector instance identified by host and port.

    • job - The name of the job producing Vector metrics.

  • counter

    processed_bytes_total

    The total number of bytes processed by the component. This metric includes the following tags:

    • component_kind - The Vector component kind.

    • component_name - The Vector component ID.

    • component_type - The Vector component type.

    • instance - The Vector instance identified by host and port.

    • job - The name of the job producing Vector metrics.

  • counter

    events_out_total

    The total number of events emitted by this component. This metric includes the following tags:

    • component_kind - The Vector component kind.

    • component_name - The Vector component ID.

    • component_type - The Vector component type.

    • instance - The Vector instance identified by host and port.

    • job - The name of the job producing Vector metrics.

  • counter

    processed_events_total

    The total number of events processed by this component. This metric includes the following tags:

    • component_kind - The Vector component kind.

    • component_name - The Vector component ID.

    • component_type - The Vector component type.

    • file - The file that produced the error

    • instance - The Vector instance identified by host and port.

    • job - The name of the job producing Vector metrics.

Examples

Given the following input:

2019-07-26 20:30:27 reply from 192.168.1.2: offset -0.001791 delay 0.000176, next query 1500s

And the following configuration:

vector.toml
[sources.journald]
type = "journald"

The following Vector log event will be output:

[
  {
    "timestamp": "2020-10-10T17:07:36.452332Z",
    "message": "reply from 192.168.1.2: offset -0.001791 delay 0.000176, next query 1500s",
    "host": "my-host.local",
    "__REALTIME_TIMESTAMP": "1564173027000443",
    "__MONOTONIC_TIMESTAMP": "98694000446",
    "_BOOT_ID": "124c781146e841ae8d9b4590df8b9231",
    "SYSLOG_FACILITY": "3",
    "_UID": "0",
    "_GID": "0",
    "_CAP_EFFECTIVE": "3fffffffff",
    "_MACHINE_ID": "c36e9ea52800a19d214cb71b53263a28",
    "PRIORITY": "6",
    "_TRANSPORT": "stdout",
    "_STREAM_ID": "92c79f4b45c4457490ebdefece29995e",
    "SYSLOG_IDENTIFIER": "ntpd",
    "_PID": "2156",
    "_COMM": "ntpd",
    "_EXE": "/usr/sbin/ntpd",
    "_CMDLINE": "ntpd: [priv]",
    "_SYSTEMD_CGROUP": "/system.slice/ntpd.service",
    "_SYSTEMD_UNIT": "ntpd.service",
    "_SYSTEMD_SLICE": "system.slice",
    "_SYSTEMD_INVOCATION_ID": "496ad5cd046d48e29f37f559a6d176f8"
  }
]

How It Works

Checkpointing

Vector checkpoints the current read position after each successful read. This ensures that Vector resumes where it left off if restarted, preventing data from being read twice. The checkpoint positions are stored in the data directory which is specified via the global data_dir option, but can be overridden via the data_dir option in the file source directly.

Communication Strategy

To ensure the journald source works across all platforms, Vector interacts with the Systemd journal via the journalctl command. This is accomplished by spawning a subprocess that Vector interacts with. If the journalctl command is not in the environment path you can specify the exact location via the journalctl_path option. For more information on this communication strategy please see issue #1473.

Context

By default, the journald source will augment events with helpful context keys as shown in the "Output" section.

Non-ASCII Messages

When journald has stored a message that is not strict ASCII, journalctl will output it in an alternate format to prevent data loss. Vector handles this alternate format by translating such messages into UTF-8 in "lossy" mode, where characters that are not valid UTF-8 are replaced with the Unicode replacement character, �.

State

This component is stateless, meaning its behavior is consistent across each input.

Support
Contents
Resources