JournalD
Collect logs from JournalD
Requirements
journalctl
. When installed from a package manager this should be
handled automatically, otherwise ensure the running user is part of the systemd-journal
group.Configuration
Example configurations
{
"sources": {
"my_source_id": {
"type": "journald"
}
}
}
[sources.my_source_id]
type = "journald"
---
sources:
my_source_id:
type: journald
{
"sources": {
"my_source_id": {
"type": "journald",
"batch_size": 16,
"current_boot_only": true,
"data_dir": "/var/lib/vector",
"exclude_matches": {
"_SYSTEMD_UNIT": [
"sshd.service",
"ntpd.service"
],
"_TRANSPORT": [
"kernel"
]
},
"exclude_units": [
"badservice"
],
"include_matches": {
"_SYSTEMD_UNIT": [
"sshd.service",
"ntpd.service"
],
"_TRANSPORT": [
"kernel"
]
},
"include_units": [
"ntpd"
]
}
}
}
[sources.my_source_id]
type = "journald"
batch_size = 16
current_boot_only = true
data_dir = "/var/lib/vector"
exclude_units = [ "badservice" ]
include_units = [ "ntpd" ]
[sources.my_source_id.exclude_matches]
_SYSTEMD_UNIT = [ "sshd.service", "ntpd.service" ]
_TRANSPORT = [ "kernel" ]
[sources.my_source_id.include_matches]
_SYSTEMD_UNIT = [ "sshd.service", "ntpd.service" ]
_TRANSPORT = [ "kernel" ]
---
sources:
my_source_id:
type: journald
batch_size: 16
current_boot_only: true
data_dir: /var/lib/vector
exclude_matches:
_SYSTEMD_UNIT:
- sshd.service
- ntpd.service
_TRANSPORT:
- kernel
exclude_units:
- badservice
include_matches:
_SYSTEMD_UNIT:
- sshd.service
- ntpd.service
_TRANSPORT:
- kernel
include_units:
- ntpd
acknowledgements
optional objectControls how acknowledgements are handled by this source.
This setting is deprecated in favor of enabling acknowledgements
at the global or sink level.
Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
acknowledgements.enabled
optional boolbatch_size
optional uintThe systemd journal is read in batches, and a checkpoint is set at the end of each batch.
This option limits the size of the batch.
16
(events)current_boot_only
optional booltrue
data_dir
optional string literalThe directory used to persist file checkpoint positions.
By default, the global data_dir
option is used. Make sure the running user has write
permissions to this directory.
exclude_matches
optional objectA list of sets of field/value pairs that, if any are present in a journal entry, excludes the entry from this source.
If exclude_units
is specified, it is merged into this list.
exclude_matches.*
required [string]exclude_units
optional [string]A list of unit names to exclude from monitoring.
Unit names lacking a .
have .service
appended to make them a valid service unit
name.
include_matches
optional objectA list of sets of field/value pairs to monitor.
If empty or not present, all journal fields are accepted.
If include_units
is specified, it is merged into this list.
include_matches.*
required [string]include_units
optional [string]A list of unit names to monitor.
If empty or not present, all units are accepted.
Unit names lacking a .
have .service
appended to make them a valid service unit name.
journal_directory
optional string literalThe full path of the journal directory.
If not set, journalctl
uses the default system journal path.
journalctl_path
optional string literalThe full path of the journalctl
executable.
If not set, a search is done for the journalctl
path.
remap_priority
optional boolDeprecated
remap
transform and to_syslog_level
function instead.Enables remapping the PRIORITY
field from an integer to string value.
Has no effect unless the value of the field is already an integer.
false
since_now
optional boolfalse
Outputs
<component_id>
Output Data
Logs
Event
/usr/sbin/ntpd
c36e9ea52800a19d214cb71b53263a28
gethostname
command.my-host.local
53.126.150.246 - - [01/Oct/2020:11:25:58 -0400] "GET /disintermediate HTTP/2.0" 401 20308
journald
2020-10-10T17:07:36.452332Z
Telemetry
Metrics
linkcomponent_discarded_events_total
countercomponent_id
instead. The value is the same as component_id
.component_errors_total
countercomponent_id
instead. The value is the same as component_id
.component_received_bytes_total
countercomponent_id
instead. The value is the same as component_id
.component_received_event_bytes_total
countercomponent_id
instead. The value is the same as component_id
.component_received_events_count
histogramA histogram of the number of events passed in each internal batch in Vector’s internal topology.
Note that this is separate than sink-level batching. It is mostly useful for low level debugging performance issues in Vector due to small internal batches.
component_id
instead. The value is the same as component_id
.component_received_events_total
countercomponent_id
instead. The value is the same as component_id
.component_sent_event_bytes_total
countercomponent_id
instead. The value is the same as component_id
.component_sent_events_total
countercomponent_id
instead. The value is the same as component_id
.invalid_record_bytes_total
countercomponent_id
instead. The value is the same as component_id
.invalid_record_total
countercomponent_id
instead. The value is the same as component_id
.source_lag_time_seconds
histogramcomponent_id
instead. The value is the same as component_id
.Examples
Sample Output
Given this event...2019-07-26 20:30:27 reply from 192.168.1.2: offset -0.001791 delay 0.000176, next query 1500s
[sources.my_source_id]
type = "journald"
---
sources:
my_source_id:
type: journald
{
"sources": {
"my_source_id": {
"type": "journald"
}
}
}
[{"log":{"PRIORITY":"6","SYSLOG_FACILITY":"3","SYSLOG_IDENTIFIER":"ntpd","_BOOT_ID":"124c781146e841ae8d9b4590df8b9231","_CAP_EFFECTIVE":"3fffffffff","_CMDLINE":"ntpd: [priv]","_COMM":"ntpd","_EXE":"/usr/sbin/ntpd","_GID":"0","_MACHINE_ID":"c36e9ea52800a19d214cb71b53263a28","_PID":"2156","_STREAM_ID":"92c79f4b45c4457490ebdefece29995e","_SYSTEMD_CGROUP":"/system.slice/ntpd.service","_SYSTEMD_INVOCATION_ID":"496ad5cd046d48e29f37f559a6d176f8","_SYSTEMD_SLICE":"system.slice","_SYSTEMD_UNIT":"ntpd.service","_TRANSPORT":"stdout","_UID":"0","__MONOTONIC_TIMESTAMP":"98694000446","__REALTIME_TIMESTAMP":"1564173027000443","host":"my-host.local","message":"reply from 192.168.1.2: offset -0.001791 delay 0.000176, next query 1500s","source_type":"journald","timestamp":"2020-10-10T17:07:36.452332Z"}}]
How it works
Checkpointing
data_dir
option, but can be overridden
via the data_dir
option in the file source directly.Communication Strategy
journald
source works across all platforms, Vector interacts
with the systemd journal via the journalctl
command. This is accomplished by
spawning a subprocess that Vector interacts
with. If the journalctl
command is not in the environment path you can
specify the exact location via the journalctl_path
option. For more
information on this communication strategy please see
issue #1473.Non-ASCII Messages
journald
has stored a message that is not strict ASCII,
journalctl
will output it in an alternate format to prevent data
loss. Vector handles this alternate format by translating such messages
into UTF-8 in “lossy” mode, where characters that are not valid UTF-8
are replaced with the Unicode replacement character, �
.