Google Cloud PubSub Push Telegraf Input Plugin

Use This InfluxDB Integration for Free

Google Cloud PubSub is a messaging queue and ingestion solution for event-driven systems and streaming analytics. It’s fully managed and lets independent applications communicate asynchronously using a publisher/subscriber architecture. An application sends messages to the Pub/Sub service and other applications can subscribe to topics to receive relevant messages. Google Cloud Pub/Sub works well in Google Compute Engine instances with large sets of data that can be distributed efficiently.

What is Pub/Sub?

Publish/subscribe (pub/sub) messaging is an asynchronous messaging pattern where publishers of messages do not send messages directly to receivers but instead send categorized messages that can be subscribed to by other devices without the publisher’s knowledge.

Pub/sub allows decoupling of senders and receivers, which can simplify the design and implementation of applications. With pub/sub neither the subscriber or publisher need to know about the other.

A pub/sub messaging architecture typically consists of three components: a publisher, a broker, and subscribers. The publisher sends messages to the broker, which in turn distributes them to all currently connected subscribers. This allows for horizontal scalability, as more subscribers are added, the broker can handle distributing more messages without affecting the publisher.

Benefits of Pub/Sub messaging

Flexibility

With a Pub/Sub messaging architecture, developers can decouple their applications by sending messages to a central message broker. This can make it easier to develop and test new applications, as well as to modify existing applications.

Pub/sub brokers generally have support for multiple different communication protocols beyond just HTTP. They also make adding new services to a system easy because those services can just subscribe to the topics they need information from, rather than needing to know about the publisher directly.

Performance and efficiency

Pub/Sub messaging can help reduce the number of messages that need to be sent between different parts of an application, which can improve performance.

A pub/sub architecture can make an application more efficient and performant in a number of ways. Pub/sub can eliminate the need for polling devices for monitoring type workloads, which will reduce load on the monitored devices and reduce bandwidth usage on unneeded polling requests. Pub/sub also allows different parts of an application to be scaled up or down independently, which makes them more cost efficient.

Reliability

Pub/Sub messaging architecture can help ensure that messages are delivered reliably by modifying message broker configuration.

Scalability

A publish-subscribe system is highly scalable as it can easily accommodate a large number of publishers and subscribers without compromising performance. The message broker can act as a load balancer or rate limiter for subscribers if message volume is high by holding onto messages until the subscribers can catch up

A pub/sub application can also scale up pieces independently. The broker can be replicated to handle more throughput of messages, some logic for filtering messages can be put into the broker rather than sending useless messages to subscribers. Subscribers can also be scaled up independently of the brokers if necessary.

Challenges of using Pub/Sub messaging

Latency

Latency can become a major problem for pub/sub applications if subscribers need to receive messages within a very small period of time. The additional network hop from publisher to broker and then to the subscriber may be too much overhead for certain use cases. This is especially true when the message broker receives a high volume of requests.

Complexity

For simple applications the additional complexity of adding a broker that needs to be managed and the potential issues related to asynchronous messaging might not be worth the benefits provided by a pub/sub architecture.

Message delivery guarantees

A common problem with pub/sub architectures can be related to how messages are delivered to subscribers. Messages can be delivered out of order in terms of when the message was created, which must be accounted for in your application code if that is an issue.

Messages may also be sent multiple times to the same subscriber and the broker could potentially fail before delivering a message.

Most pub/sub brokers have features to handle the problems described above, but they will have an impact on system performance. Generally speaking the more guarantees you want on message order and delivery, the slower your application will perform.

Pub/Sub Architecture Components

Publisher

The publisher is the component of a publish/subscribe system that produces messages. Depending on the architecture a publisher can send the messages directly to subscribers or send the message to a dedicated message broker.

Subscriber

The subscriber is the component of a pub/sub system that receives messages from publishers. In most cases subscribers must explicitly subscribe to the topics that they are interested in to receive messages. Once the message has been received the subscriber can then act on the data contained in the message and potentially act as a publisher itself by sending a message back.

Broker

A pub/sub broker is the component that mediates between publishers and subscribers by receiving and sending the messages between them.

The value of using a broker is that it can provide features like:

• redundancy via persisting messages
• filtering and routing messages
• rate limiting
• provide different levels of delivery guarantees for messages

Topic

A topic is a label used to define an available channel for messages in a publish/subscribe system that can be published to and subscribed to by publishers and subscribers.

Topics are typically organized into a hierarchy, with each level of the hierarchy representing a more specific topic.

Message

A message is the unit of information that is exchanged between publishers and subscribers. Messages are associated with a particular topic and they can contain any type of information.

Messages can have a defined data structure enforced for certain use cases to ensure the format is standardized for subscribers receiving the messages.

Pub/Sub Key Concepts

Loose coupling

A key part of the pub/sub design pattern is that publishers and subscribers are decoupled from one another. This means that a publisher does not need to know who is subscribed to its messages, and a subscriber does not need to know who the publisher is. This loose coupling allows for greater flexibility and scalability as the number of publishers and subscribers grows.

Asynchronous communication

Communication between publishers and subscribers is asynchronous. This means that a publisher can send a message without waiting for a response from the subscriber, and vice versa. Asynchronous communication allows for greater efficiency as both publishers and subscribers can continue working while messages are being sent and received.

Message filtering

With pub/sub messages can be filtered so that only those subscribers who are interested in a particular message will receive it. The broker can also read and filter messages depending on their content if configured for the task. Message filtering can help reduce clutter and ensure that subscribers only receive the information that they are interested in.

Message routing

Messages can be routed to different subscribers based on certain criteria. For example, messages could be routed based on the topic of the message or the location of the subscriber. This helps to ensure that messages are delivered to the most relevant subscribers.

Fault tolerance

Because the different parts of a pub/sub architecture are loosely coupled as described above, fault tolerance is effectively built in. The broker handling communication between publishers and subscribers can be replicated for availability and provides durability for messages. For example, if a subscribing device is down for some reason, the broker can hold a copy of the message until the device is back online and then deliver the message. The other devices don’t need to be aware of the status of each other because the broker handles things.

Pub/Sub Use Cases

Application Messaging and notifications

One of the most common use cases for a publish-subscribe architecture is messaging. With this type of system, messages can be sent to a large number of subscribers without the need for each subscriber to constantly poll for new messages. This can be useful in a variety of scenarios, such as sending out alerts or notifications to users. This type of messaging can also be used internally for communication between services in the backend of the application.

Data Streaming

Another common use case for a publish-subscribe architecture is data streaming. With this type of system, data can be streamed to a large number of subscribers. This can be useful in scenarios where it is important to get data to users as quickly as possible, such as in the case of live updates or stock prices.

Geolocation Tracking

A publish-subscribe architecture can also be used for geolocation tracking. Asset locations can be tracked in real time and subscribers can be notified when an asset enters or leaves a particular area for example.

IoT Applications

A publish-subscribe architecture can also be used for IoT applications. Using pub/sub sensors can stream data in real time to subscribers which can be useful in a variety of scenarios, such as monitoring the environment or tracking the status of machinery. If a value sent by a device is over some threshold the subscriber could then send a message back with data telling the device to shut down or take some other action.

Event-Driven Architecture

Event-driven architecture (EDA) is a messaging pattern in which services communicate with each other by producing and consuming events. This type of architecture is often used in applications that require high levels of scalability and reliability, such as financial trading systems and social media platforms.

Pub/Sub works well for Event-Driven Architectures because it enables services to produce and consume events without having to explicitly connect to each other. This helps reduce the complexity of distributed systems, making them more maintainable and scalable. It also makes it easier for new services or features to be integrated into the system, as they can simply subscribe to the relevant events without those event producers needing to know about the new service.

Microservices Architecture

Microservices architecture is a style of software development in which applications are built as a collection of small, independent services.

Pub/Sub messaging is a good fit for microservices because it allows services to communicate without having to be explicitly connected. This helps reduce the complexity of managing the system, as services can publish and subscribe to events without knowing anything about the other services in the system.

Big Data Analytics

Big data analytics is a field that deals with large volumes of data that cannot be processed using traditional methods.

Publish and subscribe messaging can help with big data analytics by allowing data to be streamed in real time from multiple sources. This can be useful for scenarios where it is important to quickly analyze large amounts of data, such as fraud detection or predictive analytics. Pub/Sub can also be used for managing multi-step data processing workflows and processing large amounts of data in parallel.

Why use the Google Cloud PubSub Push Telegraf Plugin?

The Google Cloud PubSub Push Input Plugin listens for messages sent via an HTTP POST from Google Cloud PubSub. Telegraf serves as an endpoint to the Google Pub/Sub ‘Push’ service. Google’s PubSub service sends over HTTPS/TLS, so this plugin must be behind a valid proxy or configured to use TLS.

Collecting your messages with this Telegraf plugin allows you to store your messages in InfluxDB along with all of your other metric and event data, such as message attributes and subscription topics.

How to use the Google Cloud PubSub Push Telegraf Plugin

The Google Cloud PubSub Push Telegraf Plugin is simple to use. First you need to create a PUSH subscription for a PubSub topic. Then you can choose the configuration options that fit your environment (such as service address, tokens, timeouts, and body size). In addition, you can enable TLS by specifying the file names of a service TLS certificate and key. You can also enable mutually authenticated TLS and authorize client connections by including a list of allowed CA certificate file names in tls_allowed_cacerts.</p>

It’s important to note that this plugin expects messages in Google’s Pub/Sub JSON format only.

Key Google Cloud PubSub Push metrics to monitor

With Google Cloud PubSub Push, you can stream data between different applications, application monitoring, fraud detection and live leader-boards. The data type Message defines the structure of a Google Pub/Sub message and the data type Payload is the received Google Pub/Sub data.

Project URL   Documentation

Pub/Sub Tools

Redis

Redis is an in-memory database that can be used for pub/sub use cases like real-time messaging, streaming data and geolocation tracking.

Redis is fast, reliable and scalable and has a wide range of features for managing messages such as message acknowledgements, message ordering and message delivery guarantees.

Google Cloud Pubsub

Google Cloud Pubsub is a managed service for building pub/sub messaging systems. It is focused on event streaming for tasks like moving data into data warehouses for analytics, real time alerting or notifications, and also streaming data to operational databases.

Kafka

Apache Kafka is an open source distributed messaging system that is designed for high throughput and scalability. This makes Kafka ideal for pub/sub applications and it is often used for event-driven architectures and streaming data applications.

Kafka has a wide range of features such as message acknowledgements, delivery guarantees and scalability. It also supports both push and pull messaging patterns which makes it a versatile tool for pub/sub use cases.

Pulsar

Apache Pulsar is an open-source distributed pub-sub messaging system originally created at Yahoo and now part of the Apache Software Foundation. It is designed for high performance, with a simple API, scalability, flexibility and operability.

Pulsar provides two types of messaging: queuing (message ordering) and publish-subscribe (messages broadcast to multiple consumers). It also supports data streaming, message replay and other features. The system is fault-tolerant with the ability to detect and recover from node failure.

MQTT

MQTT is a protocol designed for IoT use cases by IBM. The protocol is based around many pub/sub concepts and relies on an MQTT broker to handle passing messages between devices. MQTT places a heavy emphasis on durability and delivery guarantees because it is designed for working in edge computing environments where network connectivity isn’t guaranteed.

FAQs

Pub/Sub vs Message Queue

Although the terms “publish/subscribe” and “message queue” are often used interchangeably, they actually refer to different technologies. Publish/subscribe systems provide a way for applications to communicate through topics that can be subscribed to by multiple services, while message queues provide a way for applications to communicate through individual messages that are sent to a single service.

Pub/Sub vs event brokers

Pub/Sub is a type of messaging pattern where publishers send messages to subscribers. On the other hand, an event broker is a software application that acts as the middle-man between two or more applications by receiving, transforming and routing data from one system to another.