这不是大规模实时查询的推荐配置。 为了查询和压缩优化、高速摄取和高可用性，您可能需要考虑 Kafka 和 InfluxDB。
50 亿+
Telegraf 下载量
#1
时序数据库
来源：DB Engines
10 亿+
InfluxDB 下载量
2,800+
贡献者
目录
输入和输出集成概述
此插件允许您实时从 Kafka 主题收集指标，从而增强 Telegraf 设置中的数据监控和收集功能。
此输出插件有助于将 Telegraf 收集的指标通过 HTTP 事件收集器直接流式传输到 Splunk，从而轻松集成 Splunk 强大的分析平台。
集成详情
Kafka
Kafka Telegraf 插件旨在从 Kafka 主题读取数据，并使用支持的输入数据格式创建指标。 作为服务输入插件，它持续监听传入的指标和事件，这与按固定间隔运行的标准输入插件不同。 此特定插件可以使用各种 Kafka 版本的功能，并且能够从指定主题消费消息，应用诸如使用 SASL 的安全凭证之类的配置，以及使用消息偏移量和消费者组选项管理消息处理。 此插件的灵活性使其能够处理各种消息格式和用例，使其成为依赖 Kafka 进行数据摄取的应用程序的宝贵资产。
Splunk
使用 Telegraf 可以轻松地从许多不同的来源收集和聚合指标，并将它们发送到 Splunk。 利用 HTTP 输出插件与专门的 Splunk 指标序列化器相结合，此配置可确保高效地将数据摄取到 Splunk 的指标索引中。 HEC 是 Splunk 提供的一种高级机制，旨在通过 HTTP 或 HTTPS 可靠地大规模收集数据，为安全、监控和分析工作负载提供关键功能。 Telegraf 与 Splunk HEC 的集成通过利用标准 HTTP 协议、内置身份验证和结构化数据序列化来简化操作，优化指标摄取并实现即时可操作的见解。
配置
Kafka
[[inputs.kafka_consumer]]
## Kafka brokers.
brokers = ["localhost:9092"]
## Set the minimal supported Kafka version. Should be a string contains
## 4 digits in case if it is 0 version and 3 digits for versions starting
## from 1.0.0 separated by dot. This setting enables the use of new
## Kafka features and APIs. Must be 0.10.2.0(used as default) or greater.
## Please, check the list of supported versions at
## https://pkg.go.dev/github.com/Shopify/sarama#SupportedVersions
## ex: kafka_version = "2.6.0"
## ex: kafka_version = "0.10.2.0"
# kafka_version = "0.10.2.0"
## Topics to consume.
topics = ["telegraf"]
## Topic regular expressions to consume. Matches will be added to topics.
## Example: topic_regexps = [ "*test", "metric[0-9A-z]*" ]
# topic_regexps = [ ]
## When set this tag will be added to all metrics with the topic as the value.
# topic_tag = ""
## The list of Kafka message headers that should be pass as metric tags
## works only for Kafka version 0.11+, on lower versions the message headers
## are not available
# msg_headers_as_tags = []
## The name of kafka message header which value should override the metric name.
## In case when the same header specified in current option and in msg_headers_as_tags
## option, it will be excluded from the msg_headers_as_tags list.
# msg_header_as_metric_name = ""
## Set metric(s) timestamp using the given source.
## Available options are:
## metric -- do not modify the metric timestamp
## inner -- use the inner message timestamp (Kafka v0.10+)
## outer -- use the outer (compressed) block timestamp (Kafka v0.10+)
# timestamp_source = "metric"
## Optional Client id
# client_id = "Telegraf"
## Optional TLS Config
# enable_tls = false
# tls_ca = "/etc/telegraf/ca.pem"
# tls_cert = "/etc/telegraf/cert.pem"
# tls_key = "/etc/telegraf/key.pem"
## Use TLS but skip chain & host verification
# insecure_skip_verify = false
## Period between keep alive probes.
## Defaults to the OS configuration if not specified or zero.
# keep_alive_period = "15s"
## SASL authentication credentials. These settings should typically be used
## with TLS encryption enabled
# sasl_username = "kafka"
# sasl_password = "secret"
## Optional SASL:
## one of: OAUTHBEARER, PLAIN, SCRAM-SHA-256, SCRAM-SHA-512, GSSAPI
## (defaults to PLAIN)
# sasl_mechanism = ""
## used if sasl_mechanism is GSSAPI
# sasl_gssapi_service_name = ""
# ## One of: KRB5_USER_AUTH and KRB5_KEYTAB_AUTH
# sasl_gssapi_auth_type = "KRB5_USER_AUTH"
# sasl_gssapi_kerberos_config_path = "/"
# sasl_gssapi_realm = "realm"
# sasl_gssapi_key_tab_path = ""
# sasl_gssapi_disable_pafxfast = false
## used if sasl_mechanism is OAUTHBEARER
# sasl_access_token = ""
## SASL protocol version. When connecting to Azure EventHub set to 0.
# sasl_version = 1
# Disable Kafka metadata full fetch
# metadata_full = false
## Name of the consumer group.
# consumer_group = "telegraf_metrics_consumers"
## Compression codec represents the various compression codecs recognized by
## Kafka in messages.
## 0 : None
## 1 : Gzip
## 2 : Snappy
## 3 : LZ4
## 4 : ZSTD
# compression_codec = 0
## Initial offset position; one of "oldest" or "newest".
# offset = "oldest"
## Consumer group partition assignment strategy; one of "range", "roundrobin" or "sticky".
# balance_strategy = "range"
## Maximum number of retries for metadata operations including
## connecting. Sets Sarama library's Metadata.Retry.Max config value. If 0 or
## unset, use the Sarama default of 3,
# metadata_retry_max = 0
## Type of retry backoff. Valid options: "constant", "exponential"
# metadata_retry_type = "constant"
## Amount of time to wait before retrying. When metadata_retry_type is
## "constant", each retry is delayed this amount. When "exponential", the
## first retry is delayed this amount, and subsequent delays are doubled. If 0
## or unset, use the Sarama default of 250 ms
# metadata_retry_backoff = 0
## Maximum amount of time to wait before retrying when metadata_retry_type is
## "exponential". Ignored for other retry types. If 0, there is no backoff
## limit.
# metadata_retry_max_duration = 0
## When set to true, this turns each bootstrap broker address into a set of
## IPs, then does a reverse lookup on each one to get its canonical hostname.
## This list of hostnames then replaces the original address list.
## resolve_canonical_bootstrap_servers_only = false
## Strategy for making connection to kafka brokers. Valid options: "startup",
## "defer". If set to "defer" the plugin is allowed to start before making a
## connection. This is useful if the broker may be down when telegraf is
## started, but if there are any typos in the broker setting, they will cause
## connection failures without warning at startup
# connection_strategy = "startup"
## Maximum length of a message to consume, in bytes (default 0/unlimited);
## larger messages are dropped
max_message_len = 1000000
## Max undelivered messages
## This plugin uses tracking metrics, which ensure messages are read to
## outputs before acknowledging them to the original broker to ensure data
## is not lost. This option sets the maximum messages to read from the
## broker that have not been written by an output.
##
## This value needs to be picked with awareness of the agent's
## metric_batch_size value as well. Setting max undelivered messages too high
## can result in a constant stream of data batches to the output. While
## setting it too low may never flush the broker's messages.
# max_undelivered_messages = 1000
## Maximum amount of time the consumer should take to process messages. If
## the debug log prints messages from sarama about 'abandoning subscription
## to [topic] because consuming was taking too long', increase this value to
## longer than the time taken by the output plugin(s).
##
## Note that the effective timeout could be between 'max_processing_time' and
## '2 * max_processing_time'.
# max_processing_time = "100ms"
## The default number of message bytes to fetch from the broker in each
## request (default 1MB). This should be larger than the majority of
## your messages, or else the consumer will spend a lot of time
## negotiating sizes and not actually consuming. Similar to the JVM's
## `fetch.message.max.bytes`.
# consumer_fetch_default = "1MB"
## Data format to consume.
## Each data format has its own unique set of configuration options, read
## more about them here:
## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
data_format = "influx"
Splunk
[[outputs.http]]
## Splunk HTTP Event Collector endpoint
url = "https://splunk.example.com:8088/services/collector"
## HTTP method to use
method = "POST"
## Splunk authentication token
headers = {"Authorization" = "Splunk YOUR_SPLUNK_HEC_TOKEN"}
## Serializer for formatting metrics specifically for Splunk
data_format = "splunkmetric"
## Optional parameters
# timeout = "5s"
# insecure_skip_verify = false
# tls_ca = "/path/to/ca.pem"
# tls_cert = "/path/to/cert.pem"
# tls_key = "/path/to/key.pem"输入和输出集成示例
Kafka
-
实时数据处理：使用 Kafka 插件将来自 Kafka 主题的实时数据馈送到监控系统。 这对于需要即时反馈性能指标或用户活动的应用尤其有用，使企业能够更快地对其环境中的变化条件做出反应。
-
动态指标收集：利用此插件根据 Kafka 中发生的事件动态调整正在捕获的指标。 例如，通过与其他服务集成，用户可以让插件即时重新配置自身，确保始终根据业务或应用程序的需求收集相关指标。
-
集中式日志记录和监控：使用 Kafka Consumer Plugin 实现集中式日志记录系统，以将来自多个服务的日志聚合到统一的监控仪表板中。 此设置可以帮助识别不同服务之间的问题，并提高整体系统可观察性和故障排除能力。
-
异常检测系统：将 Kafka 与机器学习算法相结合，用于实时异常检测。 通过不断分析流数据，此设置可以自动识别异常模式，触发警报并更有效地缓解潜在问题。
Splunk
-
实时安全分析：利用此插件将来自各种应用程序的安全相关指标实时流式传输到 Splunk。 组织可以通过关联跨系统的数据流立即检测到威胁，从而显着缩短检测和响应时间。
-
多云基础设施监控：集成 Telegraf 以将来自多云环境的指标直接整合到 Splunk 中，从而实现全面的可见性和运营智能。 这种统一的监控使团队能够快速检测性能问题并简化云资源管理。
-
动态容量规划：部署插件以将来自容器编排平台（如 Kubernetes）的资源指标持续推送到 Splunk。 利用 Splunk 的分析功能，团队可以自动化预测性扩展和资源分配，避免资源瓶颈并最大限度地降低成本。
-
自动化事件响应工作流程：将此插件与 Splunk 的警报系统相结合，以创建自动化事件响应工作流程。 Telegraf 收集的指标会触发实时警报和自动化修复脚本，从而确保快速解决问题并保持高系统可用性。
反馈
感谢您成为我们社区的一份子！ 如果您有任何一般性反馈或在这些页面上发现任何错误，我们欢迎并鼓励您提出您的意见。 请在 InfluxDB 社区 Slack 中提交您的反馈。
