Kubernetes logs

Collect logs from Kubernetes Nodes

status: stable role: daemon delivery: best effort egress: stream state: stateless output: log
Page source Edit this page
Collects all log data for Kubernetes Nodes, automatically enriching data with Kubernetes metadata via the Kubernetes API.

Requirements

Kubernetes version >= 1.15 is required.

Configuration

Example configurations 

{
  "sources": {
    "my_source_id": {
      "type": "kubernetes_logs"
    }
  }
}
[sources.my_source_id]
type = "kubernetes_logs"

---
sources:
  my_source_id:
    type: kubernetes_logs

{
  "sources": {
    "my_source_id": {
      "type": "kubernetes_logs",
      "auto_partial_merge": true,
      "ingestion_timestamp_field": null,
      "kube_config_file": null,
      "self_node_name": "${VECTOR_SELF_NODE_NAME}",
      "exclude_paths_glob_patterns": [
        "**/exclude/**"
      ],
      "extra_field_selector": "metadata.name!=pod-name-to-exclude",
      "extra_label_selector": "my_custom_label!=my_value",
      "max_read_bytes": 2048,
      "max_line_bytes": 32768,
      "fingerprint_lines": 1,
      "glob_minimum_cooldown_ms": 60000,
      "delay_deletion_ms": 60000,
      "data_dir": "/var/lib/vector",
      "timezone": "local"
    }
  }
}
[sources.my_source_id]
type = "kubernetes_logs"
auto_partial_merge = true
self_node_name = "${VECTOR_SELF_NODE_NAME}"
exclude_paths_glob_patterns = [ "**/exclude/**" ]
extra_field_selector = "metadata.name!=pod-name-to-exclude"
extra_label_selector = "my_custom_label!=my_value"
max_read_bytes = 2_048
max_line_bytes = 32_768
fingerprint_lines = 1
glob_minimum_cooldown_ms = 60_000
delay_deletion_ms = 60_000
data_dir = "/var/lib/vector"
timezone = "local"

---
sources:
  my_source_id:
    type: kubernetes_logs
    pod_annotation_fields: null
    namespace_annotation_fields: null
    auto_partial_merge: true
    ingestion_timestamp_field: null
    kube_config_file: null
    self_node_name: ${VECTOR_SELF_NODE_NAME}
    exclude_paths_glob_patterns:
      - "**/exclude/**"
    extra_field_selector: metadata.name!=pod-name-to-exclude
    extra_label_selector: my_custom_label!=my_value
    max_read_bytes: 2048
    max_line_bytes: 32768
    fingerprint_lines: 1
    glob_minimum_cooldown_ms: 60000
    delay_deletion_ms: 60000
    data_dir: /var/lib/vector
    timezone: local

auto_partial_merge

optional bool
Automatically merge partial messages into a single event. Partial here is in respect to messages that were split by the Kubernetes Container Runtime log driver.
default: true

data_dir

optional string file_system_path
The directory used to persist file checkpoint positions. By default, the global data_dir option is used. Please make sure the Vector project has write permissions to this dir.
Examples 
"/var/lib/vector"

delay_deletion_ms

optional uint
Delay between receiving a DELETE event and removing any related metadata Vector has stored. This controls how quickly Vector will remove metadata for resources that have been removed from Kubernetes, a longer delay will allow Vector to continue processing and enriching logs after the source Pod has been deleted. If Vector tries to process logs from a Pod which has already had its metadata removed from the local cache, it will fail to enrich the event with metadata and log a warning.
default: 60000 (milliseconds)

exclude_paths_glob_patterns

optional [string]
A list of glob patterns to exclude from reading the files.
Array string literal
Examples 
[
  "**/exclude/**"
]
default: [**/*.gz **/*.tmp]

extra_field_selector

optional string literal
Specifies the field selector to filter Pods with, to be used in addition to the built-in Node filter. The name of the Kubernetes Node this Vector instance runs at. Configured to use an env var by default, to be evaluated to a value provided by Kubernetes at Pod deploy time.
Examples 
"metadata.name!=pod-name-to-exclude"
"metadata.name!=pod-name-to-exclude,metadata.name=mypod"

extra_label_selector

optional string literal
Specifies the label selector to filter Pods with, to be used in addition to the built-in vector.dev/exclude filter.
Examples 
"my_custom_label!=my_value"
"my_custom_label!=my_value,my_other_custom_label=my_value"

fingerprint_lines

optional uint
The number of lines to read when generating a unique fingerprint of a log file. This is helpful when some containers share common first log lines. WARNING: If the file has less than this amount of lines then it won’t be read at all. This is important since container logs are broken up into several files, so the greater lines value is, the greater the chance of it not reading the last file/logs of the container.
default: 1 (lines)

glob_minimum_cooldown_ms

optional uint
Delay between file discovery calls. This controls the interval at which Vector searches for files within a single pod.
default: 60000 (milliseconds)

ingestion_timestamp_field

optional string literal
The exact time the event was ingested into Vector.

kube_config_file

optional string literal
Optional path to a kubeconfig file readable by Vector. If not set, Vector will try to connect to Kubernetes using in-cluster configuration.

max_line_bytes

optional uint
The maximum number of a bytes a line can contain before being discarded. This protects against malformed lines or tailing incorrect files.
default: 32768 (bytes)

max_read_bytes

optional uint
An approximate limit on the amount of data read from a single pod log file at a given time.
Examples 
2048
default: 2048 (bytes)

namespace_annotation_fields

optional object
Configuration for how the events are annotated with Namespace metadata.

namespace_annotation_fields.namespace_labels

optional string literal
Event field for Namespace labels.
default: kubernetes.namespace_labels

pod_annotation_fields

optional object
Configuration for how the events are annotated with Pod metadata.

pod_annotation_fields.container_image

optional string literal
Event field for Container image.
default: kubernetes.container_image

pod_annotation_fields.container_name

optional string literal
Event field for Container name.
default: kubernetes.container_name

pod_annotation_fields.pod_annotations

optional string literal
Event field for Pod annotations.
default: kubernetes.pod_annotations

pod_annotation_fields.pod_ip

optional string literal
Event field for Pod IPv4 Address.
default: kubernetes.pod_ip

pod_annotation_fields.pod_ips

optional string literal
Event field for Pod IPv4 and IPv6 Addresses.
default: kubernetes.pod_ips

pod_annotation_fields.pod_labels

optional string literal
Event field for Pod labels.
default: kubernetes.pod_labels

pod_annotation_fields.pod_name

optional string literal
Event field for Pod name.
default: kubernetes.pod_name

pod_annotation_fields.pod_namespace

optional string literal
Event field for Pod namespace.
default: kubernetes.pod_namespace

pod_annotation_fields.pod_node_name

optional string literal
Event field for Pod node_name.
default: kubernetes.pod_node_name

pod_annotation_fields.pod_owner

optional string literal
Event field for Pod owner reference.
default: kubernetes.pod_owner

pod_annotation_fields.pod_uid

optional string literal
Event field for Pod uid.
default: kubernetes.pod_uid

self_node_name

optional string literal
The name of the Kubernetes Node this Vector instance runs at. Configured to use an env var by default, to be evaluated to a value provided by Kubernetes at Pod deploy time.
default: ${VECTOR_SELF_NODE_NAME}

timezone

optional string literal
The name of the time zone to apply to timestamp conversions that do not contain an explicit time zone. This overrides the global timezone option. The time zone name may be any name in the TZ database, or local to indicate system local time.
Examples 
"local"
"America/NewYork"
"EST5EDT"
default: local

Outputs

<component_id>

Default output stream of the component. Use this component’s ID as an input to downstream transforms and sinks.

Output Data

Logs

Line

An individual line from a Pod log file.
Fields
file required string literal
The absolute path of originating file.
Examples
/var/log/pods/pod-namespace_pod-name_pod-uid/container/1.log
kubernetes.container_image optional string literal
Container image.
Examples
busybox:1.30
kubernetes.container_name optional string literal
Container name.
Examples
coredns
kubernetes.namespace_labels optional object
Set of labels attached to the Namespace.
Examples
{
  "mylabel": "myvalue"
}
kubernetes.pod_annotations optional object
Set of annotations attached to the Pod.
Examples
{
  "myannotation": "myvalue"
}
kubernetes.pod_ip optional string literal
Pod IPv4 address.
Examples
192.168.1.1
kubernetes.pod_ips optional string literal
Pod IPv4 and IPv6 addresses.
Examples
192.168.1.1
::1
kubernetes.pod_labels optional object
Set of labels attached to the Pod.
Examples
{
  "mylabel": "myvalue"
}
kubernetes.pod_name optional string literal
Pod name.
Examples
coredns-qwertyuiop-qwert
kubernetes.pod_namespace optional string literal
Pod namespace.
Examples
kube-system
kubernetes.pod_node_name optional string literal
Pod node name.
Examples
minikube
kubernetes.pod_uid optional string literal
Pod uid.
Examples
ba46d8c9-9541-4f6b-bbf9-d23b36f2f136
message required string literal
The raw line from the Pod log file.
Examples
53.126.150.246 - - [01/Oct/2020:11:25:58 -0400] "GET /disintermediate HTTP/2.0" 401 20308
source_type required string literal
The name of the source type.
Examples
kubernetes_logs
stream required string literal
The name of the stream the log line was submitted to.
Examples
stdout
stderr
timestamp required timestamp
The exact time the event was processed by Kubernetes.
Examples
2020-10-10T17:07:36.452332Z

Telemetry

Metrics

link

component_received_events_total

counter
The number of events accepted by this component either from tagged origins like file and uri, or cumulatively from other origins.
component_id required
The Vector component ID.
component_kind required
The Vector component kind.
component_name required
Deprecated, use component_id instead. The value is the same as component_id.
component_type required
The Vector component type.
container_name optional
The name of the container from which the data originated.
file optional
The file from which the data originated.
host required
The hostname of the system Vector is running on.
mode optional
The connection mode used by the component.
peer_addr optional
The IP from which the data originated.
peer_path optional
The pathname from which the data originated.
pid required
The process ID of the Vector instance.
pod_name optional
The name of the pod from which the data originated.
uri optional
The sanitized URI from which the data originated.

component_sent_event_bytes_total

counter
The total number of event bytes emitted by this component.
component_id required
The Vector component ID.
component_kind required
The Vector component kind.
component_name required
Deprecated, use component_id instead. The value is the same as component_id.
component_type required
The Vector component type.
host required
The hostname of the system Vector is running on.
pid required
The process ID of the Vector instance.

component_sent_events_total

counter
The total number of events emitted by this component.
component_id required
The Vector component ID.
component_kind required
The Vector component kind.
component_name required
Deprecated, use component_id instead. The value is the same as component_id.
component_type required
The Vector component type.
host required
The hostname of the system Vector is running on.
pid required
The process ID of the Vector instance.

events_in_total

counter
The number of events accepted by this component either from tagged origins like file and uri, or cumulatively from other origins. This metric is deprecated and will be removed in a future version. Use component_received_events_total instead.
component_id required
The Vector component ID.
component_kind required
The Vector component kind.
component_name required
Deprecated, use component_id instead. The value is the same as component_id.
component_type required
The Vector component type.
container_name optional
The name of the container from which the data originated.
file optional
The file from which the data originated.
host required
The hostname of the system Vector is running on.
mode optional
The connection mode used by the component.
peer_addr optional
The IP from which the data originated.
peer_path optional
The pathname from which the data originated.
pid required
The process ID of the Vector instance.
pod_name optional
The name of the pod from which the data originated.
uri optional
The sanitized URI from which the data originated.

events_out_total

counter
The total number of events emitted by this component. This metric is deprecated and will be removed in a future version. Use component_sent_events_total instead.
component_id required
The Vector component ID.
component_kind required
The Vector component kind.
component_name required
Deprecated, use component_id instead. The value is the same as component_id.
component_type required
The Vector component type.
host required
The hostname of the system Vector is running on.
pid required
The process ID of the Vector instance.

k8s_docker_format_parse_failures_total

counter
The total number of failures to parse a message as a JSON object.
component_id required
The Vector component ID.
component_kind required
The Vector component kind.
component_name required
Deprecated, use component_id instead. The value is the same as component_id.
component_type required
The Vector component type.
host required
The hostname of the system Vector is running on.
pid required
The process ID of the Vector instance.

k8s_event_annotation_failures_total

counter
The total number of failures to annotate Vector events with Kubernetes Pod metadata.
component_id required
The Vector component ID.
component_kind required
The Vector component kind.
component_name required
Deprecated, use component_id instead. The value is the same as component_id.
component_type required
The Vector component type.
host required
The hostname of the system Vector is running on.
pid required
The process ID of the Vector instance.

k8s_format_picker_edge_cases_total

counter
The total number of edge cases encountered while picking format of the Kubernetes log message.
component_id required
The Vector component ID.
component_kind required
The Vector component kind.
component_name required
Deprecated, use component_id instead. The value is the same as component_id.
component_type required
The Vector component type.
host required
The hostname of the system Vector is running on.
pid required
The process ID of the Vector instance.

k8s_reflector_desyncs_total

counter
The total number of desyncs for the reflector.
component_id required
The Vector component ID.
component_kind required
The Vector component kind.
component_name required
Deprecated, use component_id instead. The value is the same as component_id.
component_type required
The Vector component type.
host required
The hostname of the system Vector is running on.
pid required
The process ID of the Vector instance.

k8s_state_ops_total

counter
The total number of state operations.
component_id required
The Vector component ID.
component_kind required
The Vector component kind.
component_name required
Deprecated, use component_id instead. The value is the same as component_id.
component_type required
The Vector component type.
host required
The hostname of the system Vector is running on.
op_kind optional
The kind of operation performed.
pid required
The process ID of the Vector instance.

k8s_stream_chunks_processed_total

counter
The total number of chunks processed from the stream of Kubernetes resources.
component_id required
The Vector component ID.
component_kind required
The Vector component kind.
component_name required
Deprecated, use component_id instead. The value is the same as component_id.
component_type required
The Vector component type.
host required
The hostname of the system Vector is running on.
pid required
The process ID of the Vector instance.

k8s_stream_processed_bytes_total

counter
The number of bytes processed from the stream of Kubernetes resources.
component_id required
The Vector component ID.
component_kind required
The Vector component kind.
component_name required
Deprecated, use component_id instead. The value is the same as component_id.
component_type required
The Vector component type.
host required
The hostname of the system Vector is running on.
pid required
The process ID of the Vector instance.

k8s_watch_requests_failed_total

counter
The total number of watch requests failed.
component_id required
The Vector component ID.
component_kind required
The Vector component kind.
component_name required
Deprecated, use component_id instead. The value is the same as component_id.
component_type required
The Vector component type.
host required
The hostname of the system Vector is running on.
pid required
The process ID of the Vector instance.

k8s_watch_requests_invoked_total

counter
The total number of watch requests invoked.
component_id required
The Vector component ID.
component_kind required
The Vector component kind.
component_name required
Deprecated, use component_id instead. The value is the same as component_id.
component_type required
The Vector component type.
host required
The hostname of the system Vector is running on.
pid required
The process ID of the Vector instance.

k8s_watch_stream_failed_total

counter
The total number of watch streams failed.
component_id required
The Vector component ID.
component_kind required
The Vector component kind.
component_name required
Deprecated, use component_id instead. The value is the same as component_id.
component_type required
The Vector component type.
host required
The hostname of the system Vector is running on.
pid required
The process ID of the Vector instance.

k8s_watch_stream_items_obtained_total

counter
The total number of items obtained from a watch stream.
component_id required
The Vector component ID.
component_kind required
The Vector component kind.
component_name required
Deprecated, use component_id instead. The value is the same as component_id.
component_type required
The Vector component type.
host required
The hostname of the system Vector is running on.
pid required
The process ID of the Vector instance.

k8s_watcher_http_error_total

counter
The total number of HTTP error responses for the Kubernetes watcher.
component_id required
The Vector component ID.
component_kind required
The Vector component kind.
component_name required
Deprecated, use component_id instead. The value is the same as component_id.
component_type required
The Vector component type.
host required
The hostname of the system Vector is running on.
pid required
The process ID of the Vector instance.

processed_bytes_total

counter
The number of bytes processed by the component.
component_id required
The Vector component ID.
component_kind required
The Vector component kind.
component_name required
Deprecated, use component_id instead. The value is the same as component_id.
component_type required
The Vector component type.
container_name optional
The name of the container from which the bytes originate.
file optional
The file from which the bytes originate.
host required
The hostname of the system Vector is running on.
mode optional
The connection mode used by the component.
peer_addr optional
The IP from which the bytes originate.
peer_path optional
The pathname from which the bytes originate.
pid required
The process ID of the Vector instance.
pod_name optional
The name of the pod from which the bytes originate.
uri optional
The sanitized URI from which the bytes originate.

processed_events_total

counter
The total number of events processed by this component. This metric is deprecated in place of using component_received_events_total and component_sent_events_total metrics.
component_id required
The Vector component ID.
component_kind required
The Vector component kind.
component_name required
Deprecated, use component_id instead. The value is the same as component_id.
component_type required
The Vector component type.
host required
The hostname of the system Vector is running on.
pid required
The process ID of the Vector instance.

Examples

Sample Output

Given this event...
F1015 11:01:46.499073       1 main.go:39] error getting server version: Get "https://10.96.0.1:443/version?timeout=32s": dial tcp 10.96.0.1:443: connect: network is unreachable
...and this configuration...
[sources.my_source_id]
type = "kubernetes_logs"

---
sources:
  my_source_id:
    type: kubernetes_logs

{
  "sources": {
    "my_source_id": {
      "type": "kubernetes_logs"
    }
  }
}
...this Vector event is produced:
{
  "file": "/var/log/pods/kube-system_storage-provisioner_93bde4d0-9731-4785-a80e-cd27ba8ad7c2/storage-provisioner/1.log",
  "kubernetes.container_image": "gcr.io/k8s-minikube/storage-provisioner:v3",
  "kubernetes.container_name": "storage-provisioner",
  "kubernetes.namespace_labels": {
    "kubernetes.io/metadata.name": "kube-system"
  },
  "kubernetes.pod_annotations": {
    "prometheus.io/scrape": "false"
  },
  "kubernetes.pod_ip": "192.168.1.1",
  "kubernetes.pod_ips": [
    "192.168.1.1",
    "::1"
  ],
  "kubernetes.pod_labels": {
    "addonmanager.kubernetes.io/mode": "Reconcile",
    "gcp-auth-skip-secret": "true",
    "integration-test": "storage-provisioner"
  },
  "kubernetes.pod_name": "storage-provisioner",
  "kubernetes.pod_namespace": "kube-system",
  "kubernetes.pod_node_name": "minikube",
  "kubernetes.pod_uid": "93bde4d0-9731-4785-a80e-cd27ba8ad7c2",
  "message": "F1015 11:01:46.499073       1 main.go:39] error getting server version: Get \"https://10.96.0.1:443/version?timeout=32s\": dial tcp 10.96.0.1:443: connect: network is unreachable",
  "source_type": "kubernetes_logs",
  "stream": "stderr",
  "timestamp": "2020-10-15T11:01:46.499555308Z"
}

How it works

Checkpointing

Vector checkpoints the current read position after each successful read. This ensures that Vector resumes where it left off if restarted, preventing data from being read twice. The checkpoint positions are stored in the data directory which is specified via the global data_dir option, but can be overridden via the data_dir option in the file source directly.

Container exclusion

The kubernetes_logs source can skip the logs from the individual containers of a particular Pod. Add an annotation vector.dev/exclude-containers to the Pod, and enumerate the names of all the containers to exclude in the value of the annotation like so:

vector.dev/exclude-containers: "container1,container2"

This annotation will make Vector skip logs originating from the container1 and container2 of the Pod marked with the annotation, while logs from other containers in the Pod will still be collected.

Context

By default, the kubernetes_logs source augments events with helpful context keys.

Enrichment

Vector will enrich data with Kubernetes context. A comprehensive list of fields can be found in the kubernetes_logs source output docs.

Filtering

Vector provides rich filtering options for Kubernetes log collection:

  • Built-in Pod and container exclusion rules.
  • The exclude_paths_glob_patterns option allows you to exclude Kubernetes log files by the file name and path.
  • The extra_field_selector option specifies the field selector to filter Pods with, to be used in addition to the built-in Node filter.
  • The extra_label_selector option specifies the label selector to filter Pods with, to be used in addition to the built-in vector.dev/exclude filter.

Globbing

By default, the kubernetes_logs source ignores compressed and temporary files. This behavior can be configured with the exclude_paths_glob_patterns option.

Globbing is used to continually discover Pods log files at a rate defined by the glob_minimum_cooldown option. In environments when files are rotated rapidly, we recommend lowering the glob_minimum_cooldown to catch files before they are compressed.

Kubernetes API access control

Vector requires access to the Kubernetes API. Specifically, the kubernetes_logs source uses the /api/v1/pods endpoint to “watch” the pods from all namespaces.

Modern Kubernetes clusters run with RBAC (role-based access control) scheme. RBAC-enabled clusters require some configuration to grant Vector the authorization to access the Kubernetes API endpoints. As RBAC is currently the standard way of controlling access to the Kubernetes API, we ship the necessary configuration out of the box: see ClusterRole, ClusterRoleBinding and a ServiceAccount in our kubectl YAML config, and the rbac configuration at the Helm chart.

If your cluster doesn’t use any access control scheme and doesn’t restrict access to the Kubernetes API, you don’t need to do any extra configuration - Vector will just work.

Clusters using legacy ABAC scheme are not officially supported (although Vector might work if you configure access properly) - we encourage switching to RBAC. If you use a custom access control scheme - make sure Vector Pod/ServiceAccount is granted access to the /api/v1/pods resource.

Kubernetes API communication

Vector communicates with the Kubernetes API to enrich the data it collects with Kubernetes context. Therefore, Vector must have access to communicate with the Kubernetes API server. If Vector is running in a Kubernetes cluster then Vector will connect to that cluster using the Kubernetes provided access information.

In addition to access, Vector implements proper desync handling to ensure communication is safe and reliable. This ensures that Vector will not overwhelm the Kubernetes API or compromise its stability.

Partial message merging

Vector, by default, will merge partial messages that are split due to the Docker size limit. For everything else, it is recommended to use the reduce transform which offers the ability to handle custom merging of things like stacktraces.

Pod exclusion

By default, the kubernetes_logs source will skip logs from the Pods that have a vector.dev/exclude: "true" label. You can configure additional exclusion rules via label or field selectors, see the available options.

Pod removal

To ensure all data is collected, Vector will continue to collect logs from the Pod for some time after its removal. This ensures that Vector obtains some of the most important data, such as crash details.

Resource limits

Vector recommends the following resource limits.

Agent resource limits

If deploy Vector as an agent (collecting data for each of your Nodes), then we recommend the following limits:

resources:
  requests:
    memory: "64Mi"
    cpu: "500m"
  limits:
    memory: "1024Mi"
    cpu: "6000m"

As with all Kubernetes resource limit recommendations, use these as a reference point and adjust as necessary. If your configured Vector pipeline is complex, you may need more resources; if you have a more straightforward pipeline, you may need less.

State

This component is stateless, meaning its behavior is consistent across each input.

State management

Agent state management

For the agent role, Vector stores its state at the host-mapped dir with a static path, so if it’s redeployed it’ll continue from where it was interrupted.

Testing & reliability

Vector is tested extensively against Kubernetes. In addition to Kubernetes being Vector’s most popular installation method, Vector implements a comprehensive end-to-end test suite for all minor Kubernetes versions starting with 1.15.
Sidebar