Fetch Read Coalescing
When many consumers fetch the same partition at the same offset concurrently (high read fan-out), the broker does redundant work: each fetch performs its own log read, its own serialization, and allocates its own copy of the response bytes. With a fan-out of N consumers, that is roughly N times the read CPU and N times the fetch-response memory for byte-identical output.
Fetch read coalescing removes that redundancy. The broker reads and serializes each unique read once, and shares the single result with every concurrent (and eligible back-to-back) consumer of the same data: one read, one serialization, one buffer, fanned out to all requesters. When all N consumers fetch the same partition at the same offset with the same fetch settings, read CPU and fetch-response memory drop from roughly N times to one.
Fetch read coalescing is available on BYOC and Dedicated clusters.
Fetch read coalescing is disabled by default, and the cluster property that controls it is managed by Redpanda. See Enable fetch read coalescing. It benefits workloads where many consumers tail the same partitions with the same fetch settings, for example fan-out delivery of one stream to many downstream applications. Request it only for high fan-out workloads: at low fan-out (for example, one or two consumers per partition), the de-duplication logic saves little to nothing and its overhead can increase reactor utilization.
How fetch read coalescing works
Redpanda coalesces fetch reads only when they request the same data in the same way: the same partition and offset, with the same isolation level and fetch size (max_bytes). When matching reads arrive concurrently, or back-to-back while a previous result is still in use, the broker performs the read once and shares the single result with every requester. Reads that differ in any of these properties run independently. If a shared read fails, all coalesced fetches receive the error.
Coalesced consumers share one response buffer instead of each holding its own copy, which is where the memory savings come from. A completed result is retained only while at least one fetch still references it, so the coalescer never pins memory on its own.
Coalescing is scoped per shard: it collapses only the fan-out that lands on the same shard. It composes with follower fetching rather than replacing it: follower fetching spreads consumers across replicas to distribute the read load, and coalescing removes the redundancy within each shard.
Limitations
Before requesting fetch read coalescing, familiarize yourself with the following limitations:
-
Only identical reads coalesce. Consumers reading the same data but with a different
max_bytes, a different offset, or a different isolation level do not share a read. The benefit scales with how closely the concurrent fetches match: the ideal case is many consumers tailing the same partition at the same offset with the same fetch sizing. -
Obligatory and strict reads are grouped apart. Redpanda distinguishes reads that must return at least one batch regardless of the byte limit (obligatory) from reads that strictly honor
max_bytes(strict), and neither serves the other. A single partition and offset fetched both ways at the same time incurs one duplicate read by design. -
Retention is best-effort. Completed results are only reusable by later readers while a fetch still references them. The coalescer never keeps a result alive on its own. The reliable savings are on genuinely concurrent readers of the same data, and back-to-back reuse is opportunistic.
-
Per-shard scope. Coalescing collapses only the fan-out that lands on the same shard. Where consumers are spread across replicas and shards, for example with follower fetching, each shard coalesces the fan-out local to it.
-
De-duplication adds per-read overhead. The de-duplication logic runs on every fetch read, whether or not reads coalesce. At low fan-out (for example, one or two consumers per partition), this overhead can increase reactor utilization while saving little to nothing, so only enable coalescing for high fan-out workloads.
Enable fetch read coalescing
The kafka_fetch_read_coalescing_enabled cluster property that controls coalescing is not user-configurable in Redpanda Cloud. To enable coalescing on a BYOC or Dedicated cluster, contact Redpanda Support.
Enabling or disabling coalescing is a runtime change: no cluster restart is required, and the coalescing cache (about 0.5 MB per shard) is allocated only while the feature is enabled.
