Skip to content

Prometheus 3.0: The Biggest Update in 7 Years — and How GreptimeDB Adapted

A walkthrough of every notable change in Prometheus 3.0 — UTF-8, Remote Write 2.0, native histograms, OTLP, PromQL semantics, and performance — plus how GreptimeDB has adapted to remain a drop-in Prometheus backend.
Prometheus 3.0: The Biggest Update in 7 Years — and How GreptimeDB Adapted
On this page

Prometheus 3.0 shipped in November 2024 — the first major version since 2.0 in 2017. Over seven years and 7,500 commits, Prometheus evolved from an early adopter tool into the standard cloud-native monitoring stack. During that time, several changes accumulated that could not ship as minor releases: default UTF-8 support, a new wire protocol, and a completely rewritten UI. These required a major version boundary.

This article covers every notable change in Prometheus 3.0 and explains how GreptimeDB has adapted to remain compatible as a drop-in Prometheus backend.

Prometheus 3.0 Feature Overview
Figure 1: Key features in Prometheus 3.0

Why a Major Version?

Prometheus 2.0 rewrote the storage engine (TSDB). Since then, the project shipped incremental improvements every six weeks without breaking backward compatibility.

But some changes cannot land as minor bumps — enabling UTF-8 by default, deprecating the old UI, introducing a new Remote Write protobuf message. The project also needed to clean up a pile of deprecated feature flags. That cleanup requires a major version, and 3.0 is the result.

Brand-New UI

The most visible change. The old web UI was built on an outdated Bootstrap version, cluttered with five years of ad-hoc additions. The new one uses Mantine (a modern React-based component framework) and adds features that previously required separate tools.

Prometheus 3.0 New UI with Tree View
Figure 2: The Prometheus 3.0 UI with PromLens-style tree view (source: prometheus.io)

What's new:

  • PromLens-style tree view — visualize PromQL as a tree of sub-expressions with per-node hover results. Previously this required the standalone PromLens tool.
  • Metrics explorer — browse metrics with contextual info, much richer than the old plain list.
  • Explain tab — behavioral insights about query results, directly in the UI.
  • UTF-8 display — metric names with dots, dashes, and non-ASCII characters render correctly.

The new UI still lacks exemplar display and heatmap support in the initial 3.0 release. If you need those, the old UI remains available via --enable-feature=old-ui.

UTF-8 Support: Bridging the OTel Gap

This is arguably the most impactful change for the broader observability ecosystem.

Before 3.0, Prometheus restricted metric and label names to [a-zA-Z_:][a-zA-Z0-9_:]*. OpenTelemetry uses dots extensively in its semantic conventions — http.server.duration, system.cpu.utilization, db.client.connections.usage. When OTel metrics entered Prometheus, dots were converted to underscores. You would instrument with http.server.duration but have to query http_server_duration — a confusing disconnect between what your SDK reported and what was actually queryable.

UTF-8 Support: Before and After
Figure 3: UTF-8 support eliminates the OTel–Prometheus naming mismatch

Prometheus 3.0 accepts all valid UTF-8 characters by default. PromQL adds quoting syntax for these names:

promql
# Legacy names still work
http_server_duration_seconds_bucket{job="myapp"}

# OTel metrics with original dot notation
{"http.server.duration"}{job="myapp"}

# Explicit __name__ label
{__name__="http.server.duration", job="myapp"}

For backward compatibility with older tooling, you can enforce legacy validation per scrape job:

yaml
# prometheus.yml
scrape_configs:
  - job_name: legacy-app
    metric_name_validation_scheme: legacy
  - job_name: otel-app
    metric_name_validation_scheme: utf8

Note: As of 3.0, only the Go client library (client_golang) fully supports emitting UTF-8 metric names. Other language SDKs are catching up.

Remote Write 2.0: Formalizing the Wire Protocol

Remote Write is how Prometheus pushes metrics to external storage — Thanos, Cortex, Mimir, GreptimeDB, and others. The 1.0 spec was published in April 2023, formalizing a protocol that dozens of implementations had already been using for years.

There is an important nuance here. The 1.0 specification defined a minimal proto: TimeSeries with just labels + samples. But Prometheus's actual implementation (prompb/types.proto) had already gone further — the real prometheus.WriteRequest proto included exemplars (field 3 in TimeSeries), histograms (field 4), and MetricMetadata (field 3 in WriteRequest). The 1.0 spec called these "experimental" and "out of scope," noting: "Prometheus currently has experimental support for sending metadata and exemplars."[1]

Remote Write 2.0 formalizes all of this. It introduces io.prometheus.write.v2.Request — a new protobuf message that officially specifies every data type — and adds string interning, the single biggest performance improvement.

Remote Write 1.0 vs 2.0
Figure 4: Remote Write 1.0 vs 2.0

What 2.0 changes

String interning: Each unique string is stored once in a symbols array; subsequent references use integer indices. Benchmarks from KubeCon NA 2024[2] show 60% fewer bytes on the wire, 90% fewer memory allocations, and 70% lower CPU usage during compression/decompression.

Formalized data types: Exemplars and metadata existed in the 1.0 proto implementation but lacked a formal contract. RW 2.0 specifies the full set:

  • Metadata (type, help, unit) — embedded per-TimeSeries instead of a separate array
  • Exemplars — with required timestamps for reliable deduplication
  • Created timestamps — when a counter was initialized, enabling accurate rate() across restarts
  • Native histograms — the exponential-bucket histogram type (see next section)

Partial write handling: Precise response headers tell senders exactly what was written:

X-Prometheus-Remote-Write-Samples-Written: 42
X-Prometheus-Remote-Write-Histograms-Written: 10
X-Prometheus-Remote-Write-Exemplars-Written: 5

Native Histograms

Classic Prometheus histograms require pre-defined bucket boundaries (le values). Choose them poorly and you lose resolution where it matters. Change them and you break existing dashboards.

Native histograms use exponential-growth boundaries determined by a schema parameter. You pick a resolution level; boundaries are calculated automatically. One series replaces N bucket series, and out-of-order ingestion is supported (important for OTel data and network-gap scenarios).

Classic vs Native Histograms
Figure 5: Classic histograms need pre-defined boundaries; native histograms auto-scale

Native histograms reached stable status in Prometheus v3.8 (November 2025)[3], but scraping still requires explicit opt-in:

yaml
scrape_configs:
  - job_name: my-app
    scrape_native_histograms: true

From v3.9 onward, the old --enable-feature=native-histograms flag is a no-op.

Note: The text exposition format and some accessor functions for native histograms are still being designed. Classic histograms continue to work unchanged.

Also in 3.0: holt_winters() is renamed to double_exponential_smoothing() and moved behind the experimental-promql-functions feature flag.

Native OTLP Ingestion

Prometheus 3.0 can receive OTLP metrics directly — no Collector needed as an intermediary.

bash
prometheus --web.enable-otlp-receiver

This exposes /api/v1/otlp/v1/metrics. Point any OTel SDK at it:

bash
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=http://localhost:9090/api/v1/otlp/v1/metrics
export OTEL_TRACES_EXPORTER=none
export OTEL_LOGS_EXPORTER=none

Four translation strategies control how OTLP metric names map to Prometheus:

StrategyWhat it doesWhen to use
UnderscoreEscapingWithSuffixesDots → underscores + type/unit suffixesDefault; existing Prometheus stacks
UnderscoreEscapingWithoutSuffixesDots → underscores, no suffixesRarely; risk of collisions
NoUTF8EscapingWithSuffixesKeeps dots + adds suffixesNew stacks with UTF-8 support
NoTranslationNo changesFull OTel fidelity
yaml
otlp:
  translation_strategy: NoUTF8EscapingWithSuffixes
  promote_resource_attributes:
    - service.instance.id

PromQL Behavior Changes

Left-open range selectors

Range selectors are now left-open, right-closed (previously left-closed, right-closed):

Prometheus 2.x: metric[5m] at T → samples in [T-5m, T]
  Might return 6 samples if boundary aligns perfectly

Prometheus 3.x: metric[5m] at T → samples in (T-5m, T]
  Always returns 5 samples with even spacing

This primarily affects subqueries, where evaluation timestamps are naturally aligned with resolution multiples.

Regex dot matching

. now matches all characters including \n. To preserve the old behavior, use [^\n] instead:

promql
my_metric{label=~"foo[^\n]*"}

Performance

Prometheus 3.0 delivers clear efficiency gains. On benchmark nodes (8 CPUs, 49 GB memory) running the same queries and configs via the prombench macrobenchmark[4], both memory and CPU usage dropped meaningfully compared to 2.0 and intermediate versions.

Prometheus Performance: Memory
Figure 6: Memory usage across Prometheus versions (source: prometheus.io)
Prometheus Performance: CPU
Figure 7: CPU usage across Prometheus versions (source: prometheus.io)

Other optimizations include GOMAXPROCS and GOMEMLIMIT auto-matching container limits (disable with --no-auto-gomaxprocs / --no-auto-gomemlimit), and a new Service Discovery manager that skips unchanged discoveries on reload.

Migration: What Breaks

Breaking changes are small but real.

Upgrade path: Go to v2.55 first, verify everything works, then upgrade to v3.0. Rollback from 3.0 only works back to v2.55 — the TSDB index format changed.

Key changes:

  • Seven feature flags removed (now default behavior): utf8-name, native-histograms, agent, remote-write-receiver, promql-at-modifier, promql-negative-offset, new-service-discovery-manager
  • Invalid/missing Content-Type on scrape targets fails the scrape; set fallback_scrape_protocol if needed
  • Scrape target ports no longer auto-added/removed
  • Log format switched to Go's log/slog
  • Alertmanager v0.16.0+ required (v1 API dropped)
  • Kubernetes SD dropped v1beta1 for EndpointSlice and Ingress

Full details: migration guide.

GreptimeDB and Prometheus 3.0

GreptimeDB is an open-source observability database that works as a drop-in Prometheus backend — accepting Remote Write, serving PromQL via the Prometheus HTTP API, and acting as long-term storage. With Prometheus 3.0 changing core query and protocol behaviors, GreptimeDB has adapted systematically.

GreptimeDB as Prometheus Long-Term Storage
Figure 8: GreptimeDB as Prometheus long-term storage

PromQL Compliance

GreptimeDB implements PromQL in Rust — its own parser (promql-parser) and execution engine built on Apache DataFusion, not a wrapper around Go code.

At PromCon EU 2025, GreptimeDB engineer Ruihang Xia shared[5] that on the PromLabs compliance test suite, GreptimeDB scored highest among all tested implementations outside Prometheus itself — higher even than implementations that import Prometheus's Go modules directly. The public tracking issue[6] documents the journey from 13% (February 2023) to over 82% at the time of the talk. Since then, GreptimeDB has continued closing the remaining gaps and now passes close to 100% of the official PromQL compliance test suite.

For Prometheus 3.0 specifically, GreptimeDB v1.0.0-rc.2 shipped these adaptations[7]:

  • Left-open range selectors (PR #7671) — aligned with 3.x matrix selector semantics
  • Start-exclusive instant selectors (PR #7688) — fixed lookback handling for end bounds
  • Function rename (PR #7700) — holt_wintersdouble_exponential_smoothing
  • Parser upgradepromql-parser v0.7.0 added Prometheus 3 support (string identifiers and Unicode label names)[8]; v0.7.1 declared compatibility with Prometheus v3.8 in the README[9]

Remote Write Support

GreptimeDB supports Prometheus Remote Write 1.0, including the sample, metadata, and histogram fields in the prometheus.WriteRequest proto. Exemplar ingestion is not yet supported and is on the roadmap.

yaml
# prometheus.yml
remote_write:
  - url: http://<greptimedb-host>:4000/v1/prometheus/write
remote_read:
  - url: http://<greptimedb-host>:4000/v1/prometheus/read

Remote Write 2.0 support (the new io.prometheus.write.v2.Request message with string interning) is tracked in GitHub Issue #4765[10], with the goal of adding 2.0 while keeping 1.0 fully functional.

GreptimeDB has also optimized Remote Write decode performance (PRs #7737, #7761) and supports three UTF-8 validation modes[11]: strict (default, rejects invalid UTF-8), lossy (replaces invalid chars with U+FFFD), and unchecked (no validation, maximum throughput).

Why GreptimeDB Behind Prometheus?

Prometheus's TSDB targets short-term retention (default 15 days). For longer horizons — anomaly detection training, compliance, trend analysis — external storage is needed.

Thanos and Cortex work but add operational overhead: Sidecar, StoreGateway, Compactor — multiple components to deploy and monitor.

GreptimeDB takes a different approach:

  • Single binary, one Helm chart — no multi-component orchestration
  • Object storage as primary — S3, GCS, or Azure Blob at 3–5× lower cost than block storage
  • SQL + PromQL on the same data — PromQL for dashboards, SQL for ad-hoc analysis and JOINs
  • Close to 100% PromQL compliance — existing Grafana dashboards work after an endpoint switch
  • Unified observability — metrics, logs, and traces in one database via OTLP and Loki protocols

A typical deployment:

bash
helm repo add greptime https://greptimeteam.github.io/helm-charts

helm upgrade --install prom-lts greptime/greptimedb-cluster \
  --set 'meta.backendStorage.etcd.endpoints[0]=etcd.etcd-cluster.svc.cluster.local:2379' \
  --set objectStorage.s3.bucket="prom-data" \
  --set objectStorage.s3.region="us-west-2" \
  --set objectStorage.s3.root="prom-lts" \
  --set objectStorage.credentials.accessKeyId="<your-access-key-id>" \
  --set objectStorage.credentials.secretAccessKey="<your-secret-access-key>" \
  -n observability
yaml
remote_write:
  - url: http://prom-lts-frontend:4000/v1/prometheus/write
remote_read:
  - url: http://prom-lts-frontend:4000/v1/prometheus/read

No code changes, no new query language — just scalable, durable retention.

What's Next

The community roadmap after 3.0 includes:

  • Native histogram custom buckets and continued stabilization
  • Delta temporality for counters
  • OpenMetrics 2.0, now under Prometheus governance
  • Start timestamp storage (experimental in v3.11+)
  • UTF-8 SDK support beyond Go

As of May 2026, Prometheus is at v3.12 (latest) with the LTS at v3.5.x (supported through July 2026).

What to Do Now

  1. Read the migration guide — especially the TSDB format change and the v2.55 stepping stone
  2. Audit your PromQL for . regex patterns that relied on newlines being excluded
  3. If you run GreptimeDB behind Prometheus, upgrade to v1.0+ for Prometheus 3.x selector alignment

References


  1. Prometheus Remote Write 1.0 Specification — "Prometheus currently has experimental support for sending metadata and exemplars" ↩︎

  2. Remote Write 2.0 at KubeCon NA 2024 — 60% message reduction, 90% memory reduction, 70% CPU reduction ↩︎

  3. Prometheus Native Histograms Spec — "Starting with v3.8.0, native histograms are supported as a stable feature" ↩︎

  4. Announcing Prometheus 3.0 — Performance benchmarks ↩︎

  5. Engage Prometheus in Rust — PromCon EU 2025 ↩︎

  6. PromQL Compliance Tracking Issue ↩︎

  7. GreptimeDB v1.0.0-rc.2 Release Notes ↩︎

  8. promql-parser v0.7.0 ↩︎

  9. promql-parser README — "compatible with prometheus v3.8" ↩︎

  10. GreptimeDB Issue #4765: Remote Write 2.0 ↩︎

  11. GreptimeDB Config Reference — UTF-8 validation modes ↩︎

Stay in the loop

Join our community