Streaming

Fetch Read Coalescing

This feature requires an enterprise license. To get a trial license key or extend your trial period, generate a new trial license key. To purchase a license, contact Redpanda Sales.

If Redpanda has enterprise features enabled and it cannot find a valid license, restrictions apply.

When many consumers fetch the same partition at the same offset concurrently (high read fan-out), the broker does redundant work: each fetch performs its own log read, its own serialization, and allocates its own copy of the response bytes. With a fan-out of N consumers, that is roughly N times the read CPU and N times the fetch-response memory for byte-identical output.

Fetch read coalescing removes that redundancy. The broker reads and serializes each unique read once, and shares the single result with every concurrent (and eligible back-to-back) consumer of the same data: one read, one serialization, one buffer, fanned out to all requesters. When all N consumers fetch the same partition at the same offset with the same fetch settings, read CPU and fetch-response memory drop from roughly N times to one.

Fetch read coalescing is disabled by default. Enable it when many consumers tail the same partitions with the same fetch settings, for example fan-out delivery of one stream to many downstream applications. Enable it only for high fan-out workloads: at low fan-out (for example, one or two consumers per partition), the de-duplication logic saves little to nothing and its overhead can increase reactor utilization. See Evaluate coalescing effectiveness for how to measure whether it helps your workload.

Prerequisites

A valid Redpanda Enterprise license.

How fetch read coalescing works

Redpanda coalesces fetch reads only when they request the same data in the same way: the same partition and offset, with the same isolation level and fetch size (max_bytes). When matching reads arrive concurrently, or back-to-back while a previous result is still in use, the broker performs the read once and shares the single result with every requester. Reads that differ in any of these properties run independently. If a shared read fails, all coalesced fetches receive the error.

Coalesced consumers share one response buffer instead of each holding its own copy, which is where the memory savings come from. A completed result is retained only while at least one fetch still references it, so the coalescer never pins memory on its own.

Coalescing is scoped per shard: it collapses only the fan-out that lands on the same shard. It composes with follower fetching rather than replacing it: follower fetching spreads consumers across replicas to distribute the read load, and coalescing removes the redundancy within each shard.

Limitations

Before enabling fetch read coalescing, familiarize yourself with the following limitations:

Only identical reads coalesce. Consumers reading the same data but with a different max_bytes, a different offset, or a different isolation level do not share a read. The benefit scales with how closely the concurrent fetches match: the ideal case is many consumers tailing the same partition at the same offset with the same fetch sizing.
Obligatory and strict reads are grouped apart. Redpanda distinguishes reads that must return at least one batch regardless of the byte limit (obligatory) from reads that strictly honor max_bytes (strict), and neither serves the other. A single partition and offset fetched both ways at the same time incurs one duplicate read by design.
Retention is best-effort. Completed results are only reusable by later readers while a fetch still references them. The coalescer never keeps a result alive on its own. The reliable savings are on genuinely concurrent readers of the same data, and back-to-back reuse is opportunistic.
Per-shard scope. Coalescing collapses only the fan-out that lands on the same shard. Where consumers are spread across replicas and shards, for example with follower fetching, each shard coalesces the fan-out local to it.
De-duplication adds per-read overhead. The de-duplication logic runs on every fetch read, whether or not reads coalesce. At low fan-out (for example, one or two consumers per partition), this overhead can increase reactor utilization while saving little to nothing, so only enable coalescing for high fan-out workloads.

Enable fetch read coalescing

Enable coalescing with the kafka_fetch_read_coalescing_enabled cluster property:

rpk cluster config set kafka_fetch_read_coalescing_enabled true

This is a runtime change, and no restart is required. The coalescing cache is fixed-size (about 0.5 MB per shard) and is allocated only while the feature is enabled. There is no separate sizing property. Setting the property back to false disables coalescing and clears the per-shard cache immediately.

Evaluate coalescing effectiveness

The coalescer exports four counters on the internal /metrics endpoint, one series per shard. The metrics are registered only when internal metrics are enabled (that is, when disable_metrics is set to false, the default).

Metric Description

Metric	Description
`vectorized_kafka_fetch_read_coalescer_insertions_total`	Fetch reads that missed and performed a new read.
`vectorized_kafka_fetch_read_coalescer_reinsertions_total`	Fetch reads that re-read an existing stale or expired entry.
`vectorized_kafka_fetch_read_coalescer_ready_hits_total`	Fetch reads served from a retained, already-completed read.
`vectorized_kafka_fetch_read_coalescer_inflight_hits_total`	Fetch reads served by awaiting an in-flight read.

vectorized_kafka_fetch_read_coalescer_insertions_total

Fetch reads that missed and performed a new read.

vectorized_kafka_fetch_read_coalescer_reinsertions_total

Fetch reads that re-read an existing stale or expired entry.

vectorized_kafka_fetch_read_coalescer_ready_hits_total

Fetch reads served from a retained, already-completed read.

vectorized_kafka_fetch_read_coalescer_inflight_hits_total

Fetch reads served by awaiting an in-flight read.

The effectiveness of coalescing is the ratio of hits (reads avoided) to insertions (reads actually performed):

coalescing hit ratio =
  (ready_hits_total + inflight_hits_total)
  / (insertions_total + reinsertions_total + ready_hits_total + inflight_hits_total)

A ratio near 0 means little fan-out is being collapsed: the reads aren’t aligning on the same partition, offset, and fetch settings, and the workload is unlikely to benefit from coalescing. A high ratio means the coalescer is absorbing most of the fan-out.