Amazon Kinesis Monitoring

What is Kinesis Consumer?

Kinesis Consumer is a managed, scalable, fully cloud-based service that allows users to process massive amounts of data per second in real time. It’s designed to allow developers to take literally any amount of data from several sources, all of which can then be scaled up or down as needed in a way that can be run on EC2 instances.

It is used to process high volumes of streaming data from sources such as operating logs, financial transactions and social media feeds in order to gain insights that can trigger workflows.

Why use a Telegraf plugin for Amazon Kinesis Consumer?

One of the most important challenges that developers face when it comes to their data involves the sheer volume that they’re working with on a daily basis. On its own, the data is largely meaningless if you can’t uncover the true story hidden inside. All of that insight is necessary to make smarter and more informed decisions during the development process, but you can’t hope to get to that point if you don’t have some type of solution that will unlock that analytical power in real time.

The Amazon Kinesis Telegraf plugin is designed to be precisely that, pulling in all time series data (including metrics and events) from the various applications you’re working with, from your infrastructure and even from sensors - making it easier than ever for developers to not just create and store this data, but to actually put it to good use.

Amazon Kinesis Consumer pulls in all your time series data (metrics and events) from your applications, infrastructure, and even sensors to make it easy for your developers to use this data. This data can be used for monitoring your applications and infrastructure as well as enhancing your applications by being queried in a real-time dashboard in your application - a single source of truth for all your time series data.

In addition, InfluxDB can handle the large write and query load for not just the data collected with an Amazon Kinesis Telegraf Plugin, but combined with the other Telegraf plugins and client libraries so that all the data you need is easy to access.

How to use the Amazon Kinesis Consumer Telegraf Plugin

The Kinesis Consumer Telegraf Plugin reads from a Kinesis data stream and creates metrics using one of the supported input data formats. Those formats will vary depending on the project you’re working on.

Note that in order for all of this to happen as expected, there are a few AWS Identity and Access Management (IAM) permissions that are required. For AWS Kinesis, this includes the DescribeStream, GetRecords and GetShardIterator permissions. For Telegraf, this includes the GetItem and PutItem permissions.

If you want to leverage the functionality of storing the last processed record, you can create a table using the following command:

Partition key: namespace
Sort key: shard_id

Key use cases using the Amazon Kinesis Consumer Telegraf Plugin

With Amazon Kinesis, you can stream data between different applications, application monitoring, fraud detection and live leader-boards.

With regards to application monitoring, this functionality is a great way to catch a small problem today before it has a chance to become a much larger one down the road. By understanding what the “normal” performance of an application looks like, developers can closely monitor anything that deviates from that status. They can then investigate themselves and make changes as necessary, all in the name of offering the best possible performance to consumers at the same time.

Fraud detection would be performed in a similar way, looking for any and all events that fall outside the normal boundaries. This is a perfect opportunity to catch fraud in its early stages, all while understanding exactly what occurred and putting yourself in a better position to stop it from happening again in the future.

Project URL   Documentation