Biweekly Report (Jul.3 - Jul.16) – Enhanced the reliability of distributed DDL operations

Summary ​

Together with all our contributors worldwide, we are glad to see GreptimeDB making remarkable progress for the better. Below are some highlights:

• Implement Create/Alter/Drop Table based on the Procedure framework
• Support building binary for CentOS 7
• Support TWCS compaction strategy
• Support Google Cloud Storage
• Add GreptimeDB metrics sink for Vector

Contributor list: (in alphabetical order) ​

For the past two weeks, our community has been super active with a total of 121 PRs merged. 11 PRs from 5 external contributors merged successfully and lots pending to be merged. Congrats on becoming our most active contributors in the past 2 weeks:

• @ccjeff (db#1864)
• @DevilExileSu (db#1932)
• @etolbakov (db#1860 db#1896 db#1931 dashboard#287)
• @Gump9 (db#1895 db#1946)
• @NiwakaDev (db#1781 db#1829 db#1911)

👏 Let's welcome @ccjeff and @Gump9 as the new contributors to join our community with their first PR merged.

A big THANK YOU for the generous and brilliant contributions! It is people like you who are making GreptimeDB a great product. Let's build an even greater community together.

Highlights of Recent PR ​

Implement Create/Alter/Drop Table based on the Procedure framework ​

We have implemented several distributed operations based on the Procedure framework, such as Create/Alter/Drop Table, to ensure the correct execution of distributed multi-step operations. We have also introduced DDL Manager and enhanced the reliability of distributed DDL operations.

Support building binary for CentOS 7 ​

This PR adds a dockerfile to support compilation to CentOS 7 machines that only have glibc 2.17.

To build CentOS 7 compatible binaries:

• docker buildx build -f ./docker/Dockerfile-centos7 -t greptime-centos7-builder . to build a builder image
• docker run --rm -v .:/greptimedb greptime-centos7-builder using that builder image to build binaries.

GreptimeDB requires Python >= 3.8. For CentOS 7 platform, we need to install extra dependencies like the following:

$ yum install -y centos-release-scl rh-python38 rh-python38-python-devel
$ source /opt/rh/rh-python38/enable

Then GreptimeDB built with PyO3 backend should work.

Support TWCS compaction strategy ​

This PR adds support for TWCS compaction strategy inspired by Cassandra.

In the previous leveled time window compaction strategy, SST files are strictly splitted and aligned to time windows. This may cause read amplification, especially when an SST file contains rows from multiple time windows, even though the out-of-order data is sparse.

TWCS computes SST file time window by their max timestamps in that out-of-order time-series data is mostly history data, it only affects min timestamps.

After assigning files into time windows, we then traverse every window to ensure that:

• For actively writing windows, we allow multiple SST files to alleviate write amplification.
• For other windows, we allow at most one file per window for better read performance.

TWCS can be enabled with table options:

CREATE TABLE IF NOT EXISTS cpu_metrics (
    hostname STRING,
    environment STRING,
    usage_user DOUBLE,
    usage_system DOUBLE,
    usage_idle DOUBLE,
    ts TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    TIME INDEX(ts),
    PRIMARY KEY(hostname, environment)

) WITH ('compaction'='twcs', 'compaction.twcs.max_active_window_files'=12, 'compaction.twcs.max_inactive_window_files'=3);

Support Google Cloud Storage ​

With OpenDAL's powerful extensibility, GreptimeDB now supports GCS (Google Cloud Storage). Previously, GreptimeDB has supported writing data to S3, OSS, Azblob, and other object storage.

Add GreptimeDB metrics sink for Vector ​

This patch adds GreptimeDB sink. For now, we use GreptimeDB's git repository for adding its client. We will eventually split it from repo and publish it to crates.io. We hope this is not a road blocker at the moment.

New Things ​

Dashboard v0.3.0 Releasing Soon

Our Dashboard is about to release version v0.3.0, this update brings some new features and improvements. Firstly, thanks to Dashboard's new contributor @etolbakov's contributions in #283 and #287, the new status page are coming.

Secondly, we've added the ability for charts to support custom configuration in #295, now users can make more in-depth customizations according to their needs. Lastly, the playground now supports gist, this improvement was proposed and implemented in #268.

This update greatly improves the user experience of the Dashboard, you are welcome to try it out and give us your valuable comments.

Greptime's First Virtual Meetup Held Successfully

Last Sunday, Greptime’s first virtual meetup has been successfully held. Our founding team shared captivating behind-the-scenes stories about Greptime, and presented a demonstration of GreptimeCloud. We have also interacted with our contributors and exchanged great ideas. Visit here for a recap of our first meetup. Thanks for your support and contributions and we'd like to meet you again!