AWS Kinesis Firehose
Collect logs from AWS Kinesis Firehose
Requirements
tls.*
options.Configuration
Example configurations
{
"sources": {
"my_source_id": {
"type": "aws_kinesis_firehose",
"address": "0.0.0.0:443",
"access_key": "A94A8FE5CCB19BA61C4C08",
"acknowledgements": null,
"record_compression": "text"
}
}
}
[sources.my_source_id]
type = "aws_kinesis_firehose"
address = "0.0.0.0:443"
access_key = "A94A8FE5CCB19BA61C4C08"
record_compression = "text"
---
sources:
my_source_id:
type: aws_kinesis_firehose
address: 0.0.0.0:443
access_key: A94A8FE5CCB19BA61C4C08
acknowledgements: null
record_compression: text
{
"sources": {
"my_source_id": {
"type": "aws_kinesis_firehose",
"address": "0.0.0.0:443",
"access_key": "A94A8FE5CCB19BA61C4C08",
"acknowledgements": null,
"tls": null,
"record_compression": "text"
}
}
}
[sources.my_source_id]
type = "aws_kinesis_firehose"
address = "0.0.0.0:443"
access_key = "A94A8FE5CCB19BA61C4C08"
record_compression = "text"
---
sources:
my_source_id:
type: aws_kinesis_firehose
address: 0.0.0.0:443
access_key: A94A8FE5CCB19BA61C4C08
acknowledgements: null
tls: null
record_compression: text
access_key
common optional string literalaccess_key
should
be set to the same value. If not specified, vector will treat
all requests as authenticated.acknowledgements
common optional objectacknowledgement
settings. This setting is deprecated in favor of enabling acknowledgements
in the destination sink.acknowledgements.enabled
optional boolfalse
address
required string literalrecord_compression
common optional string literal enumThe compression of records within the Firehose message.
Some services, like AWS CloudWatch Logs, will compress the events with gzip, before sending them AWS Kinesis Firehose. This option can be used to automatically decompress them before forwarding them to the next component.
Note that this is different from Content encoding option of the Firehose HTTP endpoint destination. That option controls the content encoding of the entire HTTP request.
Option | Description |
---|---|
auto | Vector will try to determine the compression format of the object by looking at its file signature, also known as magic bytes. Given that determining the encoding using magic bytes is not a perfect check, if the record fails to
decompress with the discovered format, the record will be forwarded as-is. Thus, if you know the
records will always be gzip encoded (for example if they are coming from AWS CloudWatch Logs) then
you should prefer to set |
gzip | GZIP format. |
none | Uncompressed. |
text
tls
optional objecttls.ca_file
optional string literaltls.crt_file
optional string literalkey_file
must also be set. This is required if enabled
is set to true
.tls.enabled
optional boolfalse
tls.key_file
optional string literaltls.key_pass
optional string literalkey_file
is set.tls.verify_certificate
optional booltrue
, Vector will require a TLS certificate from the connecting host and terminate the connection if the certificate is not valid. If false
(the default), Vector will not request a certificate from the client.false
Outputs
<component_id>
Output Data
Logs
Line
Started GET / for 127.0.0.1 at 2012-03-10 14:28:14 +0100
X-Amz-Firehose-Request-Id
header.ed1d787c-b9e2-4631-92dc-8e7c9d26d804
X-Amz-Firehose-Source-Arn
header.arn:aws:firehose:us-east-1:111111111111:deliverystream/test
2020-10-10T17:07:36.452332Z
Telemetry
Metrics
linkcomponent_discarded_events_total
countercomponent_id
instead. The value is the same as component_id
.component_errors_total
countercomponent_id
instead. The value is the same as component_id
.component_received_bytes_total
countercomponent_id
instead. The value is the same as component_id
.component_received_event_bytes_total
countercomponent_id
instead. The value is the same as component_id
.component_received_events_total
countercomponent_id
instead. The value is the same as component_id
.component_sent_event_bytes_total
countercomponent_id
instead. The value is the same as component_id
.component_sent_events_total
countercomponent_id
instead. The value is the same as component_id
.events_in_total
countercomponent_received_events_total
instead.component_id
instead. The value is the same as component_id
.events_out_total
countercomponent_sent_events_total
instead.component_id
instead. The value is the same as component_id
.processed_bytes_total
countercomponent_id
instead. The value is the same as component_id
.request_automatic_decode_errors_total
countercomponent_id
instead. The value is the same as component_id
.request_read_errors_total
countercomponent_id
instead. The value is the same as component_id
.requests_received_total
countercomponent_id
instead. The value is the same as component_id
.Examples
AWS CloudWatch Subscription message
Given this event...{
"requestId": "ed1d787c-b9e2-4631-92dc-8e7c9d26d804",
"timestamp": 1600110760138,
"records": [
{
"data": "H4sIABk1bV8AA52TzW7bMBCE734KQ2db/JdI3QzETS8FAtg91UGgyOuEqCQq5Mqua+TdS8lu0hYNUpQHAdoZDcn9tKfJdJo0EEL5AOtjB0kxTa4W68Xdp+VqtbheJrPB4A4t+EFiv6yzVLuHa+/6blARAr5UV+ihbH4vh/4+VN52aF37wdYIPkTDlyhF8SrabFsOWhIrtz+Dlnto8dV3Gp9RstshXKhMi0xpqk3GpNJccpFRKYw0WvCM5kIbzrVWipm4VK55rrSk44HGHLTx/lg2wxVYRiljVGWGCvPiuPRn2O60Se6P8UKbpOBZrulsk2xLhCEjljYJk2QFHeGU04KxQqpCsumcSko3SfQ+uoBnn8pTJmjKWZYyI0axAXx021G++bweS5136CpXj8WP6/UNYek5ycMOPPhReETsQkHI4XBIO2/bynZlXXkXwryrS9w536TWkab0XwED6e/tU2/R9eGS9NTD5VgEvnWwtQikcu0e/AO0FYyu4HpfwR3Gf2R0Btza9qxgiUNUISiLr30AP7fbyMzu7OWA803ynIzdfJ69B1EZpoVhsWMRZ8a5UVJoRoUyUlDNspxzZWiEnOXiXYiSvQOR5TnN/xsiNalmKZcy5Yr/yfB6+RZD/gbDC0IbOx8wQrMhxGGYx4lBW5X1wJBLkpO981jWf6EXogvIrm+rYYrKOn4Hgbg4b439/s8cFeVvcNwBtHBkOdWvQIdRnTxPfgCXvyEgSQQAAA=="
}
]
}
[sources.my_source_id]
type = "aws_kinesis_firehose"
address = "0.0.0.0:443"
---
sources:
my_source_id:
type: aws_kinesis_firehose
address: 0.0.0.0:443
{
"sources": {
"my_source_id": {
"type": "aws_kinesis_firehose",
"address": "0.0.0.0:443"
}
}
}
[{"log":{"message":"{\"messageType\":\"DATA_MESSAGE\",\"owner\":\"111111111111\",\"logGroup\":\"test\",\"logStream\":\"test\",\"subscriptionFilters\":[\"Destination\"],\"logEvents\":[{\"id\":\"35683658089614582423604394983260738922885519999578275840\",\"timestamp\":1600110569039,\"message\":\"{\\\"bytes\\\":26780,\\\"datetime\\\":\\\"14/Sep/2020:11:45:41 -0400\\\",\\\"host\\\":\\\"157.130.216.193\\\",\\\"method\\\":\\\"PUT\\\",\\\"protocol\\\":\\\"HTTP/1.0\\\",\\\"referer\\\":\\\"https://www.principalcross-platform.io/markets/ubiquitous\\\",\\\"request\\\":\\\"/expedite/convergence\\\",\\\"source_type\\\":\\\"stdin\\\",\\\"status\\\":301,\\\"user-identifier\\\":\\\"-\\\"}\"},{\"id\":\"35683658089659183914001456229543810359430816722590236673\",\"timestamp\":1600110569041,\"message\":\"{\\\"bytes\\\":17707,\\\"datetime\\\":\\\"14/Sep/2020:11:45:41 -0400\\\",\\\"host\\\":\\\"109.81.244.252\\\",\\\"method\\\":\\\"GET\\\",\\\"protocol\\\":\\\"HTTP/2.0\\\",\\\"referer\\\":\\\"http://www.investormission-critical.io/24/7/vortals\\\",\\\"request\\\":\\\"/scale/functionalities/optimize\\\",\\\"source_type\\\":\\\"stdin\\\",\\\"status\\\":502,\\\"user-identifier\\\":\\\"feeney1708\\\"}\"}]}","request_id":"ed1d787c-b9e2-4631-92dc-8e7c9d26d804","source_arn":"arn:aws:firehose:us-east-1:111111111111:deliverystream/test","timestamp":"2020-09-14T19:12:40.138Z"}}]
How it works
Forwarding CloudWatch Log events
This source is the recommended way to ingest logs from AWS CloudWatch logs via AWS CloudWatch Log subscriptions. To set this up:
Deploy vector with a publicly exposed HTTP endpoint using this source. You will likely also want to use the
aws_cloudwatch_logs_subscription_parser
transform to extract the log events. Make sure to set theaccess_key
to secure this endpoint. Your configuration might look something like:[sources.firehose] # General type = "aws_kinesis_firehose" address = "127.0.0.1:9000" access_key = "secret" [transforms.cloudwatch] type = "aws_cloudwatch_logs_subscription_parser" inputs = ["firehose"] [sinks.console] type = "console" inputs = ["cloudwatch"] encoding.codec = "json"
Create a Kinesis Firewatch delivery stream in the region where the CloudWatch Logs groups exist that you want to ingest.
Set the stream to forward to your Vector instance via its HTTP Endpoint destination. Make sure to configure the same
access_key
you set earlier.Setup a CloudWatch Logs subscription to forward the events to your delivery stream
Transport Layer Security (TLS)
tls.*
options.