JournalD
Collect logs from JournalD
status: stable
role: daemon
delivery: at-least-once
acknowledgements: yes
egress: batch
state: stateless
output: log
Configuration
Example configurations
{
"sources": {
"my_source_id": {
"type": "journald",
"current_boot_only": true,
"exclude_units": [
"badservice"
],
"exclude_matches": {},
"include_units": [
"ntpd"
],
"include_matches": {}
}
}
}
[sources.my_source_id]
type = "journald"
current_boot_only = true
exclude_units = [ "badservice" ]
exclude_matches = { }
include_units = [ "ntpd" ]
include_matches = { }
---
sources:
my_source_id:
type: journald
current_boot_only: true
exclude_units:
- badservice
exclude_matches: {}
include_units:
- ntpd
include_matches: {}
{
"sources": {
"my_source_id": {
"type": "journald",
"batch_size": 16,
"current_boot_only": true,
"exclude_units": [
"badservice"
],
"exclude_matches": {},
"include_units": [
"ntpd"
],
"include_matches": {},
"journalctl_path": "journalctl",
"data_dir": "/var/lib/vector",
"journal_directory": "/run/log/journal"
}
}
}
[sources.my_source_id]
type = "journald"
batch_size = 16
current_boot_only = true
exclude_units = [ "badservice" ]
exclude_matches = { }
include_units = [ "ntpd" ]
include_matches = { }
journalctl_path = "journalctl"
data_dir = "/var/lib/vector"
journal_directory = "/run/log/journal"
---
sources:
my_source_id:
type: journald
batch_size: 16
current_boot_only: true
exclude_units:
- badservice
exclude_matches: {}
include_units:
- ntpd
include_matches: {}
journalctl_path: journalctl
data_dir: /var/lib/vector
journal_directory: /run/log/journal
acknowledgements
common optional objectControls how acknowledgements are handled by this source. These settings override the global
acknowledgement
settings. This setting is deprecated in favor of enabling acknowledgements
in the destination sink.acknowledgements.enabled
common optional boolControls if the source will wait for destination sinks to deliver the events before acknowledging receipt.
default:
false
batch_size
optional uintThe systemd journal is read in batches, and a checkpoint is set at the end of each batch. This option limits the size of the batch.
default:
16
data_dir
optional string file_system_pathThe directory used to persist file checkpoint positions. By default, the global
data_dir
option is used. Please make sure the Vector project has write permissions to this dir.exclude_matches
common optional objectThis list contains sets of field/value pairs that, if any are present in a journal entry, will cause the entry to be excluded from this source. If
exclude_units
is specified, it will be merged into this list.exclude_matches.*
optional [string]The set of field values to match in journal entries that are to be excluded.
exclude_units
common optional [string]The list of unit names to exclude from monitoring. Unit names lacking a
"."
will have ".service"
appended to make them a valid service unit name.include_matches
common optional objectThis list contains sets of field/value pairs to monitor. If empty or not present, all journal fields are accepted. If
include_units
is specified, it will be merged into this list.include_matches.*
optional [string]The set of field values to match in journal entries that are to be included.
include_units
common optional [string]The list of unit names to monitor. If empty or not present, all units are accepted. Unit names lacking a
"."
will have ".service"
appended to make them a valid service unit name.journal_directory
optional string literalThe full path of the journal directory. If not set,
journalctl
will use the default system journal pathsjournalctl_path
optional string literalThe full path of the
journalctl
executable. If not set, Vector will search the path for journalctl
.default:
journalctl
Outputs
<component_id>
Default output stream of the component. Use this component’s ID as an input to downstream transforms and sinks.
Output Data
Logs
Event
A Journald event
*
optional
string
literal
Any Journald field
Examples
/usr/sbin/ntpd
c36e9ea52800a19d214cb71b53263a28
host
required
string
literal
The local hostname, equivalent to the
gethostname
command.Examples
my-host.local
message
required
string
literal
The raw line from the file.
Examples
53.126.150.246 - - [01/Oct/2020:11:25:58 -0400] "GET /disintermediate HTTP/2.0" 401 20308
source_type
required
string
literal
The name of the source type.
Examples
journald
timestamp
required
timestamp
The exact time the event was ingested into Vector.
Examples
2020-10-10T17:07:36.452332Z
Telemetry
Metrics
linkcomponent_received_bytes_total
counterThe number of raw bytes accepted by this component from source origins.
component_id
The Vector component ID.
component_kind
The Vector component kind.
component_name
Deprecated, use
component_id
instead. The value is the same as component_id
.component_type
The Vector component type.
container_name
optional
The name of the container from which the data originated.
file
optional
The file from which the data originated.
host
optional
The hostname of the system Vector is running on.
mode
optional
The connection mode used by the component.
peer_addr
optional
The IP from which the data originated.
peer_path
optional
The pathname from which the data originated.
pid
optional
The process ID of the Vector instance.
pod_name
optional
The name of the pod from which the data originated.
uri
optional
The sanitized URI from which the data originated.
component_received_event_bytes_total
counterThe number of event bytes accepted by this component either from
tagged origins like file and uri, or cumulatively from other origins.
component_id
The Vector component ID.
component_kind
The Vector component kind.
component_name
Deprecated, use
component_id
instead. The value is the same as component_id
.component_type
The Vector component type.
container_name
optional
The name of the container from which the data originated.
file
optional
The file from which the data originated.
host
optional
The hostname of the system Vector is running on.
mode
optional
The connection mode used by the component.
peer_addr
optional
The IP from which the data originated.
peer_path
optional
The pathname from which the data originated.
pid
optional
The process ID of the Vector instance.
pod_name
optional
The name of the pod from which the data originated.
uri
optional
The sanitized URI from which the data originated.
component_received_events_total
counterThe number of events accepted by this component either from tagged
origins like file and uri, or cumulatively from other origins.
component_id
The Vector component ID.
component_kind
The Vector component kind.
component_name
Deprecated, use
component_id
instead. The value is the same as component_id
.component_type
The Vector component type.
container_name
optional
The name of the container from which the data originated.
file
optional
The file from which the data originated.
host
optional
The hostname of the system Vector is running on.
mode
optional
The connection mode used by the component.
peer_addr
optional
The IP from which the data originated.
peer_path
optional
The pathname from which the data originated.
pid
optional
The process ID of the Vector instance.
pod_name
optional
The name of the pod from which the data originated.
uri
optional
The sanitized URI from which the data originated.
component_sent_event_bytes_total
counterThe total number of event bytes emitted by this component.
component_id
The Vector component ID.
component_kind
The Vector component kind.
component_name
Deprecated, use
component_id
instead. The value is the same as component_id
.component_type
The Vector component type.
host
optional
The hostname of the system Vector is running on.
output
optional
The specific output of the component.
pid
optional
The process ID of the Vector instance.
component_sent_events_total
counterThe total number of events emitted by this component.
component_id
The Vector component ID.
component_kind
The Vector component kind.
component_name
Deprecated, use
component_id
instead. The value is the same as component_id
.component_type
The Vector component type.
host
optional
The hostname of the system Vector is running on.
output
optional
The specific output of the component.
pid
optional
The process ID of the Vector instance.
events_in_total
counterThe number of events accepted by this component either from tagged
origins like file and uri, or cumulatively from other origins.
This metric is deprecated and will be removed in a future version.
Use
component_received_events_total
instead.component_id
The Vector component ID.
component_kind
The Vector component kind.
component_name
Deprecated, use
component_id
instead. The value is the same as component_id
.component_type
The Vector component type.
container_name
optional
The name of the container from which the data originated.
file
optional
The file from which the data originated.
host
optional
The hostname of the system Vector is running on.
mode
optional
The connection mode used by the component.
peer_addr
optional
The IP from which the data originated.
peer_path
optional
The pathname from which the data originated.
pid
optional
The process ID of the Vector instance.
pod_name
optional
The name of the pod from which the data originated.
uri
optional
The sanitized URI from which the data originated.
events_out_total
counterThe total number of events emitted by this component.
This metric is deprecated and will be removed in a future version.
Use
component_sent_events_total
instead.component_id
The Vector component ID.
component_kind
The Vector component kind.
component_name
Deprecated, use
component_id
instead. The value is the same as component_id
.component_type
The Vector component type.
host
optional
The hostname of the system Vector is running on.
output
optional
The specific output of the component.
pid
optional
The process ID of the Vector instance.
invalid_record_bytes_total
counterThe total number of bytes from invalid records that have been discarded.
component_id
The Vector component ID.
component_kind
The Vector component kind.
component_name
Deprecated, use
component_id
instead. The value is the same as component_id
.component_type
The Vector component type.
host
optional
The hostname of the system Vector is running on.
pid
optional
The process ID of the Vector instance.
invalid_record_total
counterThe total number of invalid records that have been discarded.
component_id
The Vector component ID.
component_kind
The Vector component kind.
component_name
Deprecated, use
component_id
instead. The value is the same as component_id
.component_type
The Vector component type.
host
optional
The hostname of the system Vector is running on.
pid
optional
The process ID of the Vector instance.
processed_bytes_total
counterThe number of bytes processed by the component.
component_id
The Vector component ID.
component_kind
The Vector component kind.
component_name
Deprecated, use
component_id
instead. The value is the same as component_id
.component_type
The Vector component type.
container_name
optional
The name of the container from which the bytes originate.
file
optional
The file from which the bytes originate.
host
optional
The hostname of the system Vector is running on.
mode
optional
The connection mode used by the component.
peer_addr
optional
The IP from which the bytes originate.
peer_path
optional
The pathname from which the bytes originate.
pid
optional
The process ID of the Vector instance.
pod_name
optional
The name of the pod from which the bytes originate.
uri
optional
The sanitized URI from which the bytes originate.
processed_events_total
counterThe total number of events processed by this component.
This metric is deprecated in place of using
component_received_events_total
and
component_sent_events_total
metrics.component_id
The Vector component ID.
component_kind
The Vector component kind.
component_name
Deprecated, use
component_id
instead. The value is the same as component_id
.component_type
The Vector component type.
host
optional
The hostname of the system Vector is running on.
pid
optional
The process ID of the Vector instance.
source_lag_time_seconds
histogramThe difference between the timestamp recorded in each event and the time when it was ingested, expressed as fractional seconds.
component_id
The Vector component ID.
component_kind
The Vector component kind.
component_name
Deprecated, use
component_id
instead. The value is the same as component_id
.component_type
The Vector component type.
host
optional
The hostname of the system Vector is running on.
pid
optional
The process ID of the Vector instance.
Examples
Sample Output
Given this event...2019-07-26 20:30:27 reply from 192.168.1.2: offset -0.001791 delay 0.000176, next query 1500s
[sources.my_source_id]
type = "journald"
---
sources:
my_source_id:
type: journald
{
"sources": {
"my_source_id": {
"type": "journald"
}
}
}
[{"log":{"PRIORITY":"6","SYSLOG_FACILITY":"3","SYSLOG_IDENTIFIER":"ntpd","_BOOT_ID":"124c781146e841ae8d9b4590df8b9231","_CAP_EFFECTIVE":"3fffffffff","_CMDLINE":"ntpd: [priv]","_COMM":"ntpd","_EXE":"/usr/sbin/ntpd","_GID":"0","_MACHINE_ID":"c36e9ea52800a19d214cb71b53263a28","_PID":"2156","_STREAM_ID":"92c79f4b45c4457490ebdefece29995e","_SYSTEMD_CGROUP":"/system.slice/ntpd.service","_SYSTEMD_INVOCATION_ID":"496ad5cd046d48e29f37f559a6d176f8","_SYSTEMD_SLICE":"system.slice","_SYSTEMD_UNIT":"ntpd.service","_TRANSPORT":"stdout","_UID":"0","__MONOTONIC_TIMESTAMP":"98694000446","__REALTIME_TIMESTAMP":"1564173027000443","host":"my-host.local","message":"reply from 192.168.1.2: offset -0.001791 delay 0.000176, next query 1500s","source_type":"journald","timestamp":"2020-10-10T17:07:36.452332Z"}}]
How it works
Checkpointing
Vector checkpoints the current read position after each
successful read. This ensures that Vector resumes where it left
off if restarted, preventing data from being read twice. The
checkpoint positions are stored in the data directory which is
specified via the global
data_dir
option, but can be overridden
via the data_dir
option in the file source directly.Communication Strategy
To ensure the
journald
source works across all platforms, Vector interacts
with the Systemd journal via the journalctl
command. This is accomplished by
spawning a subprocess that Vector interacts
with. If the journalctl
command is not in the environment path you can
specify the exact location via the journalctl_path
option. For more
information on this communication strategy please see
issue #1473.Non-ASCII Messages
When
journald
has stored a message that is not strict ASCII,
journalctl
will output it in an alternate format to prevent data
loss. Vector handles this alternate format by translating such messages
into UTF-8 in “lossy” mode, where characters that are not valid UTF-8
are replaced with the Unicode replacement character, �
.