Version 8 of QuestDB, an open-source time series database designed for high-performance and efficient handling of time series data, has been released. This release includes a new VARCHAR data type as a default (over STRING) that aims to provide better compression and performance, a 50% improvement for SQL query performance, and data compression via system-level ZFS that was previously "enterprise-only"
The SQL compiler has been updated for improved performance, and the data ingestion rates have been optimized to increase throughput and reduce latency. New SQL functions and improvements to existing functions aim to enhance the database’s ability to perform time series analytics.
The storage engine has also been optimized to handle larger datasets using ZFS compression, which the QuestDB team claims significantly enhances storage efficiency and performance.
ZFS, known for its advanced file system capabilities, brings robust compression algorithms to QuestDB. This allows users to store large volumes of time-series data more compactly, improving storage utilization by 6x and leading to cost savings and more efficient disk utilization. Additionally, ZFS compression improves query performance by decreasing the amount of data read from disk, accelerating data retrieval and analysis.
The adoption of the VARCHAR data type as the default over STRING marks another significant shift towards enhanced data efficiency and performance. VARCHAR, with its variable-length capability, optimizes storage by allocating space only for the actual string length, rather than a fixed size. This change contributes to reducing the memory footprint and also accelerates query execution times by minimizing the data that needs to be processed. Additionally, using VARCHAR as the default aligns QuestDB with standard SQL practices, promoting better compatibility and ease of use for users familiar with traditional relational databases.
Additional breaking changes in this release include a GLIBC upgrade, ILP and TextImport modifications, and changes to the default memory allocator, which is now jemalloc. Later updates will include disabling jemalloc as the default memory allocator and removing jemalloc from the no-jre builds.
QuestDB combines SQL and time series extensions, enabling users to perform complex queries on time-stamped data. It employs a columnar storage model, which organizes data by columns rather than rows. This model is particularly well-suited for time series data because it allows for improved data compression and faster read access. When data is stored in columns, queries that involve scanning large volumes of data can be executed more efficiently, as only the relevant columns need to be read from disk. This contrasts with traditional row-based storage, where entire rows must be read even if only a few columns are needed. The columnar model in QuestDB enhances performance, especially for analytical queries that aggregate or filter data over time.
Interested readers can learn more from the QuestDB blog post and version 8.0 release notes on GitHub.