目录
输入和输出集成概述
Kinesis 插件使您能够从 Kinesis 数据流中读取数据,支持各种数据格式和配置。
Graylog 插件允许您将 Telegraf 指标发送到 Graylog 服务器,利用 GELF 格式进行结构化日志记录。
集成详情
Kinesis
Kinesis Telegraf 插件旨在从 Amazon Kinesis 数据流中读取数据,使用户能够实时收集指标。作为服务输入插件,它通过监听传入数据而不是定期轮询来运行。配置指定了各种选项,包括 AWS 区域、流名称、身份验证凭据和数据格式。它支持跟踪未传递的消息以防止数据丢失,用户可以利用 DynamoDB 来维护上次处理记录的检查点。此插件对于需要可靠且可扩展的流处理以及其他监控需求的应用程序特别有用。
Graylog
Graylog 插件旨在用于使用 GELF (Graylog Extended Log Format) 格式将指标发送到 Graylog 实例。GELF 有助于标准化日志记录数据,使系统更轻松地发送和分析日志。该插件遵循 GELF 规范,该规范规定了有效负载中特定字段的要求。值得注意的是,时间戳必须为 UNIX 格式,如果存在,插件会将时间戳原样发送到 Graylog,不做任何更改。如果省略,它会自动生成时间戳。此外,任何规范未明确定义的额外字段都将以一个下划线作为前缀,这有助于保持数据井井有条并符合 GELF 的要求。此功能对于实时监控应用程序和基础设施的用户尤其有价值,因为它允许跨多个系统进行无缝集成和提高可见性。
配置
Kinesis
# Configuration for the AWS Kinesis input.
[[inputs.kinesis_consumer]]
## Amazon REGION of kinesis endpoint.
region = "ap-southeast-2"
## Amazon Credentials
## Credentials are loaded in the following order
## 1) Web identity provider credentials via STS if role_arn and web_identity_token_file are specified
## 2) Assumed credentials via STS if role_arn is specified
## 3) explicit credentials from 'access_key' and 'secret_key'
## 4) shared profile from 'profile'
## 5) environment variables
## 6) shared credentials file
## 7) EC2 Instance Profile
# access_key = ""
# secret_key = ""
# token = ""
# role_arn = ""
# web_identity_token_file = ""
# role_session_name = ""
# profile = ""
# shared_credential_file = ""
## Endpoint to make request against, the correct endpoint is automatically
## determined and this option should only be set if you wish to override the
## default.
## ex: endpoint_url = "http://localhost:8000"
# endpoint_url = ""
## Kinesis StreamName must exist prior to starting telegraf.
streamname = "StreamName"
## Shard iterator type (only 'TRIM_HORIZON' and 'LATEST' currently supported)
# shard_iterator_type = "TRIM_HORIZON"
## Max undelivered messages
## This plugin uses tracking metrics, which ensure messages are read to
## outputs before acknowledging them to the original broker to ensure data
## is not lost. This option sets the maximum messages to read from the
## broker that have not been written by an output.
##
## This value needs to be picked with awareness of the agent's
## metric_batch_size value as well. Setting max undelivered messages too high
## can result in a constant stream of data batches to the output. While
## setting it too low may never flush the broker's messages.
# max_undelivered_messages = 1000
## Data format to consume.
## Each data format has its own unique set of configuration options, read
## more about them here:
## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
data_format = "influx"
##
## The content encoding of the data from kinesis
## If you are processing a cloudwatch logs kinesis stream then set this to "gzip"
## as AWS compresses cloudwatch log data before it is sent to kinesis (aws
## also base64 encodes the zip byte data before pushing to the stream. The base64 decoding
## is done automatically by the golang sdk, as data is read from kinesis)
##
# content_encoding = "identity"
## Optional
## Configuration for a dynamodb checkpoint
[inputs.kinesis_consumer.checkpoint_dynamodb]
## unique name for this consumer
app_name = "default"
table_name = "default"
Graylog
[[outputs.graylog]]
## Endpoints for your graylog instances.
servers = ["udp://127.0.0.1:12201"]
## Connection timeout.
# timeout = "5s"
## The field to use as the GELF short_message, if unset the static string
## "telegraf" will be used.
## example: short_message_field = "message"
# short_message_field = ""
## According to GELF payload specification, additional fields names must be prefixed
## with an underscore. Previous versions did not prefix custom field 'name' with underscore.
## Set to true for backward compatibility.
# name_field_no_prefix = false
## Connection retry options
## Attempt to connect to the endpoints if the initial connection fails.
## If 'false', Telegraf will give up after 3 connection attempt and will
## exit with an error. If set to 'true', the plugin will retry to connect
## to the unconnected endpoints infinitely.
# connection_retry = false
## Time to wait between connection retry attempts.
# connection_retry_wait_time = "15s"
## Optional TLS Config
# tls_ca = "/etc/telegraf/ca.pem"
# tls_cert = "/etc/telegraf/cert.pem"
# tls_key = "/etc/telegraf/key.pem"
## Use TLS but skip chain & host verification
# insecure_skip_verify = false
输入和输出集成示例
Kinesis
-
使用 Kinesis 进行实时数据处理:此用例涉及将 Kinesis 插件与监控仪表板集成,以实时分析传入的数据指标。例如,应用程序可以从多个服务使用日志并在视觉上呈现它们,使运营团队能够快速识别趋势并对发生的异常做出反应。
-
无服务器日志聚合:在无服务器架构中使用此插件,其中 Kinesis 流聚合来自各种微服务的日志。该插件可以创建指标,帮助检测系统中的问题,通过第三方集成自动化警报流程,使团队能够最大限度地减少停机时间并提高可靠性。
-
基于流指标的动态扩展:实施一种解决方案,其中 Kinesis 插件使用的流指标可用于动态调整资源。例如,如果处理的记录数量激增,则可以触发相应的扩展操作来处理增加的负载,从而确保最佳的资源分配和性能。
-
使用检查点到 S3 的数据管道:创建一个强大的数据管道,其中 Kinesis 流数据通过 Telegraf Kinesis 插件处理,检查点存储在 DynamoDB 中。这种方法可以确保数据的一致性和可靠性,因为它管理已处理数据的状态,从而实现与下游数据湖或存储解决方案的无缝集成。
Graylog
-
增强云应用程序的日志管理:使用 Graylog Telegraf 插件聚合来自跨多个服务器的云部署应用程序的日志。通过集成此插件,团队可以集中管理日志记录数据,从而更轻松地排除问题、监控应用程序性能并保持符合日志记录标准。
-
实时安全监控:利用 Graylog 插件收集安全相关的指标和日志,并将其发送到 Graylog 服务器进行实时分析。这使安全团队能够通过关联来自基础设施内各种来源的日志,快速识别异常、跟踪潜在的漏洞并及时响应事件。
-
动态警报和通知系统:实施 Graylog 插件以增强基础设施中的警报机制。通过将指标发送到 Graylog,团队可以根据日志模式或意外行为设置动态警报,从而实现主动监控和快速事件响应策略。
-
跨平台日志整合:使用 Graylog 插件促进跨平台日志整合,跨越本地、混合和云等各种环境。通过以 GELF 格式标准化日志记录,组织可以确保一致的监控和故障排除实践,无论其服务托管在何处。
反馈
感谢您成为我们社区的一份子!如果您有任何一般性反馈或在这些页面上发现了任何错误,我们欢迎并鼓励您提出意见。请在 InfluxDB 社区 Slack 中提交您的反馈。