TECHNICAL PAPER
Time series database (TSDB) explained
In this technical paper, InfluxData CTO, Paul Dix, will walk you through what time series is (and isn’t), what makes it different from stream processing, full-text search and other solutions.
By reading this tech paper, you will:
- Learn how time series data is all around us,
- See why a purpose built TSDB is important.
- Read about how a Time Series database is optimized for time-stamped data.
- Understand the differences between metrics, events, & traces and some of the key characteristics of time series data.
Download this technical paper
What is a time series database?
A time series database (TSDB) is a database optimized for time-stamped or time series data. Time series data are simply measurements or events that are tracked, monitored, downsampled, and aggregated over time. This could be server metrics, application performance monitoring, network data, sensor data, events, clicks, trades in a market, and many other types of analytics data.
A time series database is built specifically for handling metrics and events or measurements that are time-stamped. A TSDB is optimized for measuring change over time. Properties that make time series data very different than other data workloads are data lifecycle management, summarization, and large range scans of many records.
Why is a time series database important now?
Time series databases are not new, but the first-generation time series databases were primarily focused on looking at financial data, the volatility of stock trading, and systems built to solve trading. But financial data is hardly the only application of time series data anymore — in fact, it’s only one among numerous applications across various industries. The fundamental conditions of computing have changed dramatically over the last decade. Everything has become compartmentalized. Monolithic mainframes have vanished, replaced by serverless servers, microservers, and containers.
Today, everything that can be a component is a component. In addition, we are witnessing the instrumentation of every available surface in the material world — streets, cars, factories, power grids, ice caps, satellites, clothing, phones, microwaves, milk containers, planets, human bodies. Everything has, or will have, a sensor. So now, everything inside and outside the company is emitting a relentless stream of metrics and events or time series data.
This means that the underlying platforms need to evolve to support these new workloads — more data points, more data sources, more monitoring, more controls. What we’re witnessing, and what the times demand, is a paradigmatic shift in how we approach our data infrastructure and how we approach building, monitoring, controlling, and managing systems. What we need is a performant, scalable, purpose-built time series database.
What distinguishes the time series workload?
Time series databases have key architectural design properties that make them very different from other databases. These include time-stamp data storage and compression, data lifecycle management, data summarization, ability to handle large time series dependent scans of many records, and time series aware queries.
For example: With a time series database, it is common to request a summary of data over a large time period. This requires going over a range of data points to perform some computation like a percentile increase this month of a metric over the same period in the last six months, summarized by month. This kind of workload is very difficult to optimize for with a distributed key value store. TSDB’s are optimized for exactly this use case giving millisecond level query times over months of data. Another example: With time series databases, it’s common to keep high precision data around for a short period of time. This data is aggregated and downsampled into longer term trend data. This means that for every data point that goes into the database, it will have to be deleted after its period of time is up. This kind of data lifecycle management is difficult for application developers to implement on top of regular databases. They must devise schemes for cheaply evicting large sets of data and constantly summarizing that data at scale. With a time series database, this functionality is provided out of the box.
Independent ranking of top 15 time series databases
Time series databases are the fastest growing segment in the database industry. But which time series database is the best and most popular? There are many ways of determining popularity, but an independent website, DB-Engines, ranks databases based on search engine popularity, social media mentions, job postings, and technical discussion volume. (Read their full methodology). Here are the current results:
To see trends over time, the following graphic shows the top 10 time series databases and their historical changes:
Time series – the fastest growing database category
DB-Engines also ranks time series database management systems (Time Series DBMS) according to their popularity. Time series databases are the fastest growing segment of the database industry over the past year.
What makes InfluxDB time series database unique?
InfluxDB was built from the ground up to be a purpose-built time series database; i.e., it was not repurposed to be time series. Time was built-in from the beginning. InfluxDB is part of a comprehensive platform that supports the collection, storage, monitoring, visualization and alerting of time series data. It’s much more than just a time series database.
The whole InfluxData platform is built from an open source db core. InfluxData is an active contributor to the Telegraf, InfluxDB, Chronograf and Kapacitor (TICK) projects — the “I,C,K” from the TICK Stack is being collapsed into a single binary in InfluxDB 2.0 — as well as selling InfluxDB Enterprise and InfluxDB Cloud on this open source core. The InfluxDB data model is quite different from other time series solutions like Graphite, RRD, or OpenTSDB. InfluxDB has a line protocol for sending time series data which takes the following form: measurement-name tag-set field-set timestamp. The measurement name is a string, the tag set is a collection of key/value pairs where all values are strings, and the field set is a collection of key/value pairs where the values can be int64, float64, bool, or string. The measurement name and tag sets are kept in an inverted index which make lookups for specific series very fast. For example, if we have CPU metrics:
cpu,host=serverA,region=uswest idle=23,user=42,system=12 1464623548sTimestamps in InfluxDB can be second, millisecond, microsecond, or nanosecond precision. The micro and nanosecond scales make InfluxDB a good choice for use cases in finance and scientific computing where other solutions would be excluded. Compression is variable depending on the level of precision the user needs. On disk, the data is organized in a columnar style format where contiguous blocks of time are set for the measurement, tagset, field. So, each field is organized sequentially on disk for blocks of time, which make calculating aggregates on a single field a very fast operation. There is no limit to the number of tags and fields that can be used.
Other time series solutions don’t support multiple fields, which can make their network protocols bloated when transmitting data with shared tag sets. Most other time series solutions only support float64 values, which means the user is unable to encode additional metadata along with the time series. Even OpenTSDB and KairosDB, which support tags (unlike Graphite and RRD), have limitations on the number of tags that can be used. At around 5 to 6 tags, the user will start seeing hot spots within their cluster of HBase or Cassandra machines.
InfluxDB doesn’t have this limitation because the InfluxDB data model is designed for time series specifically. It pushes the developer in the right direction to get good performance out of the database by indexing tags and keeping fields unindexed. It’s flexible in that many data types are supported, and the user can have many fields and tags. Because of all these factors, a purpose-built time series database like InfluxDB is the best solution for working with time series data.
Time series databases: FAQ
Listed below for quick reference are brief answers to frequently asked questions about time series databases:
What is a time series database?
Here’s a brief time series database definition: A time series database (TSDB) is a database optimized for time-stamped (time series) data and for measuring change over time.
What is the best time series database?
Visit this page to learn about what makes a powerful time series database and which database is best for storing large volumes of time series data.
What are time series data examples?
Visit the What is time series data page to view time series data examples.
Is InfluxDB open source?
InfluxDB is an open source time series database with a large and vibrant community.
Can I use InfluxDB with Grafana?
There are thousands of use cases utilizing InfluxDB and Grafana. Visit our Community Showcase to read about them.
How does InfluxDB compare to other databases?
View InfluxDB benchmarking tests comparing its performance to other databases (such as Cassandra, Elasticsearch, MongoDB, OpenTSDB, Graphite and Splunk) based on parameters such as write throughout, query throughput, and on-disk storage.
Is a time series database better than a relational database for handling time series data?
If you’re debating time series database vs relational database, a time series database (TSDB) is specific for sorting and querying time series data, and tends to be more efficient than a relational database, which is more generic.
Can I use a time series database for edge computing?
Transmitting data from the edge to cloud in a reliable way continues to be a challenge for many businesses. Read the Edge Computing & Data Replication with InfluxDB e-book to learn what ‘the edge’ is, edge computing use cases and benefits, and how InfluxDB time series database can be used for edge computing.
What is the difference between a time series database and a data warehouse?
While a time series database is a database optimized for time-stamped or time series data, a data warehouse stores and organizes data from multiple sources in a central location.
