Protocol Decoding Performance Boosted by Fivefold! Ongoing Write Performance Enhancements | Greptime Biweekly Report

Summary ​

Together with our global community of contributors, GreptimeDB continues to evolve and flourish as a growing open-source project. We are grateful to each and every one of you.

Over the past two weeks, we have released version 0.7, which fully supports cloud-native monitoring scenarios. Below are some highlights:

• Continuous optimization is underway for the write path

• Manual decoding of the Prometheus Remote Write protocol has resulted in a performance increase of approximately fivefold

• New partitioning rules and Memtable implementations are now in place

Contributor ​

For the past two weeks, our community has been super active with a total of 65 PRs merged. 6 PRs from 5 individual contributors merged successfully and lots pending to be merged.

Congrats on becoming our most active contributors in the past 2 weeks:

• @crwen （#3473)

• @etolbakov (#3445 #3390)

• @gcmutator (#3469)

• @J0HN50N133（#3379)

• @KKould（#3372)

👏 Welcome contributor @crwen @gcmutator @J0HN50N133 join to the community as the new individual contributor, and congratulations on successfully merging their first PR, more PRs are waiting to be merged.

A big THANK YOU to all our members and contributors! It is people like you who are making GreptimeDB a great product. Let's build an even greater community together.

Highlights of Recent PRs ​

Ongoing Write Path Optimization ​

Specific enhancements are detailed in PRs: #3407 and #3422, which have reduced data duplication instances; PRs #3397, #3404, #3426, and #3383, which have expanded metrics for various detailed workflows; and PRs #3415 and #3400, aimed at reducing the memory overhead of handling primary keys.

#3425 #3478 Major Reductions in Parsing Overhead for Prometheus Remote Write Format ​

Benchmark highlighted substantial CPU usage for protocol parsing, where analogous structures in Go necessitated only an eighth of the overhead. A deep dive revealed a performance bottleneck in the prost protocol library. Manual decoding for the Prometheus Remote Write protocol parsing was implemented, leading to a performance boost of approximately 5x.

#3418 Memory Usage Optimization at Restart ​

Earlier versions exhibited a spike in memory usage during restarts, a concern that has been addressed in GreptimeDB v0.7.

#3350 Introduction of New Partitioning Methodology and Syntax ​

Anticipating the need for frequent partition adjustments for features like automatic partitioning and repartitioning, we've initiated the development of a new partitioning syntax. The RFC has merged, and development progress can be followed through PR #3351.

#3430 New Memtable Implementation Now the Default Option ​

Designed to optimize for scenarios with numerous timelines, this new Memtable implementation has been fully integrated and is now the default setting, showing a potential reduction in memory overhead by up to 50%.

#3485 Fuzz Testing Enhancements Continuously ​

Our ongoing iteration of the previously developed fuzz testing suite has identified and rectified several edge cases, with an expanding scope of coverage.

Good First Issue ​

#3492 Implementing Prometheus HTTP API ​

Keywords: observability

Aiming for seamless integration with Grafana, we're enhancing GreptimeDB's compatibility with the Prometheus HTTP API.

#3435 Enforcing Strict Mode String Validation for Prometheus Remote Write Interfaces ​

Keywords: security, protocol

Difficulty: medium

While most implementations and libraries ensure that input strings are UTF-8 encoded, the possibility of malicious attacks remains. It's necessary to implement strict mode validation for relevant protocols to prevent such scenarios.