Kafka 和 ServiceNow 集成

强大的性能和简单的集成,由 InfluxData 构建的开源数据连接器 Telegraf 提供支持。

info

对于大规模实时查询,这不是推荐的配置。为了进行查询和压缩优化、高速摄取和高可用性,您可能需要考虑 Kafka 和 InfluxDB

50 亿+

Telegraf 下载量

#1

时间序列数据库
来源:DB Engines

10 亿+

InfluxDB 的下载量

2,800+

贡献者

目录

强大的性能,无限的扩展能力

收集、组织和处理海量高速数据。当您将任何数据视为时间序列数据时,它都更有价值。借助 InfluxDB,这个排名第一的时间序列平台旨在通过 Telegraf 进行扩展。

查看入门方法

输入和输出集成概述

此插件允许您从 Kafka 主题实时收集指标,从而增强 Telegraf 设置中的数据监控和收集能力。

此输出插件通过 HTTP 将指标从 Telegraf 直接流式传输到 ServiceNow MID Server,利用 nowmetric 序列化器与 ServiceNow 的 Operational Intelligence 和 Event Management 高效集成。

集成详情

Kafka

Kafka Telegraf 插件旨在从 Kafka 主题读取数据,并使用受支持的输入数据格式创建指标。作为服务输入插件,它持续监听传入的指标和事件,这与按固定间隔运行的标准输入插件不同。此特定插件可以使用来自各种 Kafka 版本的功能,并且能够使用 SASL 等配置的安全性凭据,以及使用消息偏移量和消费者组选项管理消息处理,从而使用来自指定主题的消息。此插件的灵活性使其能够处理各种消息格式和用例,使其成为依赖 Kafka 进行数据摄取的应用程序的宝贵资产。

ServiceNow

Telegraf 可用于将指标数据直接发送到 ServiceNow MID Server REST 端点。指标使用 ServiceNow 的 Operational Intelligence (OI) 格式或 JSONv2 格式进行格式化,从而实现与 ServiceNow 的 Event Management 和 Operational Intelligence 平台的无缝集成。序列化器有效地批量处理指标,通过最大限度地减少 HTTP POST 请求的数量来降低网络开销。此集成允许用户在 ServiceNow 中快速利用指标,以增强可观测性、主动事件管理和性能监控以及 ServiceNow 的 Operational Intelligence 功能。

配置

Kafka


[[inputs.kafka_consumer]]
              ## Kafka brokers.
              brokers = ["localhost:9092"]

              ## Set the minimal supported Kafka version. Should be a string contains
              ## 4 digits in case if it is 0 version and 3 digits for versions starting
              ## from 1.0.0 separated by dot. This setting enables the use of new
              ## Kafka features and APIs.  Must be 0.10.2.0(used as default) or greater.
              ## Please, check the list of supported versions at
              ## https://pkg.go.dev/github.com/Shopify/sarama#SupportedVersions
              ##   ex: kafka_version = "2.6.0"
              ##   ex: kafka_version = "0.10.2.0"
              # kafka_version = "0.10.2.0"

              ## Topics to consume.
              topics = ["telegraf"]

              ## Topic regular expressions to consume.  Matches will be added to topics.
              ## Example: topic_regexps = [ "*test", "metric[0-9A-z]*" ]
              # topic_regexps = [ ]

              ## When set this tag will be added to all metrics with the topic as the value.
              # topic_tag = ""

              ## The list of Kafka message headers that should be pass as metric tags
              ## works only for Kafka version 0.11+, on lower versions the message headers
              ## are not available
              # msg_headers_as_tags = []

              ## The name of kafka message header which value should override the metric name.
              ## In case when the same header specified in current option and in msg_headers_as_tags
              ## option, it will be excluded from the msg_headers_as_tags list.
              # msg_header_as_metric_name = ""

              ## Set metric(s) timestamp using the given source.
              ## Available options are:
              ##   metric -- do not modify the metric timestamp
              ##   inner  -- use the inner message timestamp (Kafka v0.10+)
              ##   outer  -- use the outer (compressed) block timestamp (Kafka v0.10+)
              # timestamp_source = "metric"

              ## Optional Client id
              # client_id = "Telegraf"

              ## Optional TLS Config
              # enable_tls = false
              # tls_ca = "/etc/telegraf/ca.pem"
              # tls_cert = "/etc/telegraf/cert.pem"
              # tls_key = "/etc/telegraf/key.pem"
              ## Use TLS but skip chain & host verification
              # insecure_skip_verify = false

              ## Period between keep alive probes.
              ## Defaults to the OS configuration if not specified or zero.
              # keep_alive_period = "15s"

              ## SASL authentication credentials.  These settings should typically be used
              ## with TLS encryption enabled
              # sasl_username = "kafka"
              # sasl_password = "secret"

              ## Optional SASL:
              ## one of: OAUTHBEARER, PLAIN, SCRAM-SHA-256, SCRAM-SHA-512, GSSAPI
              ## (defaults to PLAIN)
              # sasl_mechanism = ""

              ## used if sasl_mechanism is GSSAPI
              # sasl_gssapi_service_name = ""
              # ## One of: KRB5_USER_AUTH and KRB5_KEYTAB_AUTH
              # sasl_gssapi_auth_type = "KRB5_USER_AUTH"
              # sasl_gssapi_kerberos_config_path = "/"
              # sasl_gssapi_realm = "realm"
              # sasl_gssapi_key_tab_path = ""
              # sasl_gssapi_disable_pafxfast = false

              ## used if sasl_mechanism is OAUTHBEARER
              # sasl_access_token = ""

              ## SASL protocol version.  When connecting to Azure EventHub set to 0.
              # sasl_version = 1

              # Disable Kafka metadata full fetch
              # metadata_full = false

              ## Name of the consumer group.
              # consumer_group = "telegraf_metrics_consumers"

              ## Compression codec represents the various compression codecs recognized by
              ## Kafka in messages.
              ##  0 : None
              ##  1 : Gzip
              ##  2 : Snappy
              ##  3 : LZ4
              ##  4 : ZSTD
              # compression_codec = 0
              ## Initial offset position; one of "oldest" or "newest".
              # offset = "oldest"

              ## Consumer group partition assignment strategy; one of "range", "roundrobin" or "sticky".
              # balance_strategy = "range"

              ## Maximum number of retries for metadata operations including
              ## connecting. Sets Sarama library's Metadata.Retry.Max config value. If 0 or
              ## unset, use the Sarama default of 3,
              # metadata_retry_max = 0

              ## Type of retry backoff. Valid options: "constant", "exponential"
              # metadata_retry_type = "constant"

              ## Amount of time to wait before retrying. When metadata_retry_type is
              ## "constant", each retry is delayed this amount. When "exponential", the
              ## first retry is delayed this amount, and subsequent delays are doubled. If 0
              ## or unset, use the Sarama default of 250 ms
              # metadata_retry_backoff = 0

              ## Maximum amount of time to wait before retrying when metadata_retry_type is
              ## "exponential". Ignored for other retry types. If 0, there is no backoff
              ## limit.
              # metadata_retry_max_duration = 0

              ## When set to true, this turns each bootstrap broker address into a set of
              ## IPs, then does a reverse lookup on each one to get its canonical hostname.
              ## This list of hostnames then replaces the original address list.
              ## resolve_canonical_bootstrap_servers_only = false

              ## Strategy for making connection to kafka brokers. Valid options: "startup",
              ## "defer". If set to "defer" the plugin is allowed to start before making a
              ## connection. This is useful if the broker may be down when telegraf is
              ## started, but if there are any typos in the broker setting, they will cause
              ## connection failures without warning at startup
              # connection_strategy = "startup"

              ## Maximum length of a message to consume, in bytes (default 0/unlimited);
              ## larger messages are dropped
              max_message_len = 1000000

              ## Max undelivered messages
              ## This plugin uses tracking metrics, which ensure messages are read to
              ## outputs before acknowledging them to the original broker to ensure data
              ## is not lost. This option sets the maximum messages to read from the
              ## broker that have not been written by an output.
              ##
              ## This value needs to be picked with awareness of the agent's
              ## metric_batch_size value as well. Setting max undelivered messages too high
              ## can result in a constant stream of data batches to the output. While
              ## setting it too low may never flush the broker's messages.
              # max_undelivered_messages = 1000

              ## Maximum amount of time the consumer should take to process messages. If
              ## the debug log prints messages from sarama about 'abandoning subscription
              ## to [topic] because consuming was taking too long', increase this value to
              ## longer than the time taken by the output plugin(s).
              ##
              ## Note that the effective timeout could be between 'max_processing_time' and
              ## '2 * max_processing_time'.
              # max_processing_time = "100ms"

              ## The default number of message bytes to fetch from the broker in each
              ## request (default 1MB). This should be larger than the majority of
              ## your messages, or else the consumer will spend a lot of time
              ## negotiating sizes and not actually consuming. Similar to the JVM's
              ## `fetch.message.max.bytes`.
              # consumer_fetch_default = "1MB"

              ## Data format to consume.
              ## Each data format has its own unique set of configuration options, read
              ## more about them here:
              ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
              data_format = "influx"

ServiceNow

[[outputs.http]]
  ## ServiceNow MID Server metrics endpoint
  url = "http://mid-server.example.com:9082/api/mid/sa/metrics"

  ## HTTP request method
  method = "POST"

  ## Basic Authentication credentials
  username = "evt.integration"
  password = "P@$$w0rd!"

  ## Data serialization format for ServiceNow
  data_format = "nowmetric"

  ## Metric format type: "oi" (default) or "jsonv2"
  nowmetric_format = "oi"

  ## HTTP Headers
  [outputs.http.headers]
    Content-Type = "application/json"
    Accept = "application/json"

  ## Optional timeout
  # timeout = "5s"

  ## TLS configuration options
  # tls_ca = "/etc/telegraf/ca.pem"
  # tls_cert = "/etc/telegraf/cert.pem"
  # tls_key = "/etc/telegraf/key.pem"
  # insecure_skip_verify = false

输入和输出集成示例

Kafka

  1. 实时数据处理:使用 Kafka 插件将来自 Kafka 主题的实时数据馈送到监控系统。这对于需要即时反馈性能指标或用户活动的应用尤其有用,使企业能够更快地对其环境中的变化条件做出反应。

  2. 动态指标收集:利用此插件根据 Kafka 内发生的事件动态调整正在捕获的指标。例如,通过与其他服务集成,用户可以让插件即时重新配置自身,确保始终根据业务或应用程序的需求收集相关指标。

  3. 集中式日志记录和监控:实施集中式日志记录系统,使用 Kafka Consumer Plugin 将来自多个服务的日志聚合到统一的监控仪表板中。此设置可以帮助识别不同服务之间的问题,并提高整体系统可观测性和故障排除能力。

  4. 异常检测系统:将 Kafka 与机器学习算法结合用于实时异常检测。通过不断分析流式数据,此设置可以自动识别异常模式,触发警报并更有效地缓解潜在问题。

ServiceNow

  1. 主动事件管理:利用 Telegraf 和 ServiceNow 集成将基础设施和应用程序指标实时流式传输到 ServiceNow Event Management。根据阈值自动触发事件或补救工作流程,从而显著缩短事件检测和响应时间。

  2. 端到端应用程序监控:在应用程序堆栈的多个层部署 Telegraf 代理,将性能指标直接发送到 ServiceNow。利用 ServiceNow 的 Operational Intelligence,团队可以将跨组件的指标关联起来,快速识别性能瓶颈。

  3. 动态 CI 性能跟踪:通过使用此插件推送性能数据,将 Telegraf 指标与 ServiceNow 的 CMDB 集成,从而允许根据实时指标自动更新配置项 (CI) 健康状态。这确保了 ServiceNow 中基础设施健康状态的准确和最新状态。

  4. 云资源优化:使用 Telegraf 从混合云和多云基础设施收集指标,直接流式传输到 ServiceNow。利用这些指标进行实时分析、预测性容量规划和资源优化,从而实现主动管理并降低运营成本。

反馈

感谢您成为我们社区的一份子!如果您有任何一般性反馈或在这些页面上发现任何错误,我们欢迎并鼓励您提出意见。请在 InfluxDB 社区 Slack 中提交您的反馈。

强大的性能,无限的扩展能力

收集、组织和处理海量高速数据。当您将任何数据视为时间序列数据时,它都更有价值。借助 InfluxDB,这个排名第一的时间序列平台旨在通过 Telegraf 进行扩展。

查看入门方法

相关集成

HTTP 和 InfluxDB 集成

HTTP 插件从一个或多个 HTTP(S) 端点收集指标。它支持各种身份验证方法和数据格式的配置选项。

查看集成

Kafka 和 InfluxDB 集成

此插件从 Kafka 读取消息,并允许基于这些消息创建指标。它支持各种配置,包括不同的 Kafka 设置和消息处理选项。

查看集成

Kinesis 和 InfluxDB 集成

Kinesis 插件允许从 AWS Kinesis 流中读取指标。它支持多种输入数据格式,并提供带有 DynamoDB 的检查点功能,以实现可靠的消息处理。

查看集成