Consumer Group Protocol
The consumer group protocol, defined in Kafka Improvement Proposal (KIP) 848, moves partition assignment from the consumers to the broker. In the classic protocol, every member of a group stops consuming while the group rebalances. In the consumer group protocol, the broker computes each member’s assignment and hands partitions over incrementally, so members that keep their partitions don’t stop consuming while the group changes.
Use the consumer group protocol when your consumer groups scale up and down often, or when rebalance pauses cause consumer lag. Because the broker manages assignment, a single slow or misbehaving consumer can’t disrupt the whole group.
After reading this page, you will be able to:
-
Explain how the consumer group protocol differs from the classic rebalance protocol
-
Decide whether a consumer group can use the consumer group protocol
-
Enable the consumer group protocol on a cluster and opt in a consumer
Classic groups and consumer groups
A consumer group uses one of two protocols, and its type is named after the protocol:
-
A classic group uses the original rebalance protocol. All groups are classic groups unless their consumers opt in to the consumer group protocol.
-
A consumer group uses the consumer group protocol defined in KIP-848.
rpk group list and rpk group describe show this as the group’s TYPE: classic or consumer. Classic groups and consumer groups run side by side on the same cluster, and enabling the consumer group protocol doesn’t change your existing classic groups.
Most things stay the same in both protocols: offset commits, transactions, the __consumer_offsets topic, and the way you think about a group of consumers sharing partitions. Your application code doesn’t change. What changes is how the client and the broker agree on who owns each partition, and the client handles that for you.
How the consumer group protocol works
In the classic protocol, a group rebalances in rounds that every member takes part in. Each member sends JoinGroup and SyncGroup requests, one member (the group leader) computes the assignment on the client side, and members give up their partitions until the round completes. The whole group waits for the slowest member.
In the consumer group protocol, each member sends periodic ConsumerGroupHeartbeat requests to the group coordinator. Each heartbeat carries the member’s subscription and the partitions it owns, and the response carries the member’s assignment:
-
When a member joins, leaves, or changes its subscription, the coordinator computes a new target assignment for the group with a server-side partition assignor. The assignor is sticky: it moves as few partitions as possible.
-
The coordinator reconciles each member toward its target assignment. A member gives up a partition before the coordinator assigns that partition to another member, so no partition has two owners.
-
A partition moves to its new owner as soon as its previous owner stops consuming it. Members whose partitions don’t change keep consuming throughout.
The group and each member have an epoch that increases as the assignment changes. The coordinator uses the epochs to fence members that fall behind.
Example: a second consumer joins
This example follows a group that consumes a topic with four partitions, p0 to p3:
-
Consumer A sends a heartbeat to join the empty group. The coordinator assigns it all four partitions, and the group state changes to
Stable. Consumer A fetches its committed offsets (there are none yet), consumes records, and commits offsets. -
Consumer B sends a heartbeat to join. The coordinator computes a new target assignment that gives
p0andp1to consumer A andp2andp3to consumer B. The group state changes toReconciling. Consumer B has no partitions yet, because consumer A still owns them all. -
On consumer A’s next heartbeat, the coordinator tells it that its assignment is now
p0andp1. Consumer A stops consumingp2andp3and keeps consumingp0andp1without a pause. -
Consumer A’s next heartbeat reports that it owns only
p0andp1. On consumer B’s next heartbeat, the coordinator assigns itp2andp3, and the group state changes back toStable. -
Consumer B fetches the committed offsets for
p2andp3and resumes consuming from them.
In the classic protocol, consumer A would stop consuming all four partitions while the group rebalanced.
A consumer group can be in one of these states:
-
Empty: The group has no members. -
Assigning: A member joined, left, or changed its subscription, and the coordinator is computing a new target assignment. -
Reconciling: The coordinator computed a target assignment, and members are moving partitions to reach it. -
Stable: Every member has reached the target assignment.
The STATE field in rpk group list and rpk group describe shows the group’s current state.
Limitations
The consumer group protocol in Redpanda has the following limitations:
-
Static membership is not supported. The broker rejects a heartbeat that sets a group instance ID (
group.instance.id) withINVALID_REQUEST. If your consumers setgroup.instance.id, keep them on the classic protocol. -
Regex subscriptions are not supported. The broker rejects a heartbeat that subscribes with a topic regular expression with
INVALID_REQUEST. Subscribe to a list of topic names instead. -
A group with active members can’t switch protocols. A group switches protocols only when it has no members, so switching requires you to stop the group’s consumers. See Switch an existing group to the consumer group protocol.
-
Only the
uniformassignor is available. The broker responds to a heartbeat that names any other server-side assignor, such asrange, withUNSUPPORTED_ASSIGNOR. -
Group-level configuration is not supported. Session timeouts, heartbeat intervals, and assignment settings apply to every consumer group on the cluster.
Before you enable the consumer group protocol
Enabling the protocol on the cluster doesn’t move any group to it. Each consumer must opt in, and most Kafka clients still use the classic protocol by default. Before you opt in a group, check the following:
-
Test the consumer group protocol in a non-production environment first.
-
Check that the group’s clients support the protocol. See Prerequisites.
-
Keep a group on the classic protocol if its consumers need any of the following:
-
Static membership (
group.instance.id) or regex subscriptions -
An assignor other than
uniform -
Their own session timeout or heartbeat interval. In the consumer group protocol, these are cluster-wide settings. See Configure consumer groups.
-
A switch with no downtime. To switch protocols, you must stop every consumer in the group.
-
Prerequisites
-
A Redpanda cluster running version 26.3 or later, with the upgrade finalized. The consumer group protocol stays unavailable until the cluster upgrade is finalized, even when you enable it. See Defer Upgrade Finalization.
-
A Kafka client that supports the consumer group protocol:
Client Minimum version How to opt in Apache Kafka Java client
4.0.0
Set
group.protocol=consumer.librdkafka
2.12.0
Set
group.protocol=consumer.franz-go
1.22.0
Add the
kgo.ServerSideBalancer()group option. -
If authorization is enabled, the consumer’s principal needs the
READpermission on the consumer group and theDESCRIBEpermission on each topic it subscribes to. See Configure Access Control Lists.
The consumer group protocol is available in Redpanda Community Edition and doesn’t require an enterprise license.
Enable the consumer group protocol
The consumer group protocol is disabled by default. When it’s disabled, the broker responds to ConsumerGroupHeartbeat requests with UNSUPPORTED_VERSION, and consumers must use the classic protocol.
-
Enable the protocol on the cluster:
rpk cluster config set group_consumer_protocol_enabled trueThe change takes effect without a restart. If the cluster upgrade isn’t finalized, the brokers log a warning and the protocol stays unavailable until you finalize the upgrade.
-
In your consumer configuration, opt in to the consumer group protocol. For example, with the Apache Kafka Java client:
group.id=<group-id> group.protocol=consumerReplace
<group-id>with a new group ID, or with the ID of an existing group that has no active members. To move an existing classic group, see Switch an existing group to the consumer group protocol.In the consumer group protocol, the broker controls the session timeout, the heartbeat interval, and the assignor, so remove
session.timeout.ms,heartbeat.interval.ms, andpartition.assignment.strategyfrom your consumer configuration. Clients reject these settings when you use the consumer group protocol. Clients choose a server-side assignor withgroup.remote.assignorinstead. Leave it unset or set it touniform, the only assignor that Redpanda supports. -
Start the consumers, then confirm that the group uses the consumer group protocol:
rpk group list --types consumerThe output lists only consumer groups:
BROKER GROUP STATE TYPE 0 order-processors Stable consumer -
To see the group’s epochs and assignments, describe the group with
--print-members:rpk group describe <group-id> --print-membersFor a consumer group, the summary includes
TYPE consumer, the group epoch (GROUP-EPOCH), and the target assignment epoch (TARGET-ASSIGNMENT-EPOCH). The member table shows each member’s epoch (MEMBER-EPOCH), current assignment (ASSIGNMENT), and target assignment (TARGET-ASSIGNMENT):GROUP order-processors COORDINATOR-NODE 0 COORDINATOR-PARTITION __consumer_offsets/0 STATE Stable BALANCER uniform MEMBERS 3 TOTAL-LAG 0 TYPE consumer GROUP-EPOCH 4 TARGET-ASSIGNMENT-EPOCH 4 MEMBER-ID CLIENT-ID HOST MEMBER-EPOCH ASSIGNMENT TARGET-ASSIGNMENT COFm64KdBqd3VbY3ztjVWg== rpk /127.0.0.1 4 orders:2,3 orders:2,3 J8EBq40AwIV3nOyAK3306w== rpk /127.0.0.1 4 orders:4,5 orders:4,5 tntB6Ttz1Em2ObESsKjpjA== rpk /127.0.0.1 4 orders:0,1 orders:0,1When every member’s
ASSIGNMENTmatches itsTARGET-ASSIGNMENTand itsMEMBER-EPOCHmatches theTARGET-ASSIGNMENT-EPOCH, the group has finished reconciling. Without--print-members,rpk group describeshows each partition’s offsets and lag instead of the member table.In a consumer group, each client generates its own member ID, so member IDs look different from those in a classic group, where the broker assigns IDs such as
rpk-9bea3f80-7760-4392-91d0-30cce4118d41.
Switch an existing group to the consumer group protocol
An existing classic group switches to the consumer group protocol when it has no active members and a consumer that uses the consumer group protocol joins it. The group keeps its committed offsets, so the consumers resume from their last committed positions.
Redpanda doesn’t switch a group while it has active members. A rolling restart, in which some consumers use the classic protocol and some use the consumer group protocol at the same time, doesn’t work: the broker responds to the new consumers' heartbeats with GROUP_ID_NOT_FOUND until every classic consumer leaves the group.
To switch a group:
-
Stop every consumer in the group.
-
Wait until the group has no members. Run
rpk group describe <group-id>and check thatSTATEisEmpty. -
Opt in to the consumer group protocol in the consumer configuration, and remove
session.timeout.ms,heartbeat.interval.ms, andpartition.assignment.strategy. See Enable the consumer group protocol. -
Start the consumers.
-
Run
rpk group describe <group-id>and check thatTYPEisconsumer.
To switch a group back to the classic protocol, follow the same steps with group.protocol=classic. A consumer group switches back when it has no members and a classic consumer joins it, and it keeps its committed offsets.
Configure consumer groups
In the consumer group protocol, cluster properties control the timeouts and assignment behavior that the classic protocol leaves to each client. These properties apply to every consumer group on the cluster, and they take effect without a restart.
-
Session timeout:
group_consumer_session_timeout_msreplaces the client’ssession.timeout.ms. It sets how long the broker waits after a member’s last heartbeat before it removes the member from its group.group_consumer_min_session_timeout_msandgroup_consumer_max_session_timeout_msset the allowed range. -
Heartbeat interval:
group_consumer_heartbeat_interval_msreplaces the client’sheartbeat.interval.ms. It sets how long the broker tells each member to wait between heartbeats, and it must be less than the session timeout.group_consumer_min_heartbeat_interval_msandgroup_consumer_max_heartbeat_interval_msset the allowed range. -
Assignment:
group_consumer_assignorslists the server-side assignors that groups can use.group_consumer_assignment_interval_mssets the minimum time between two target assignments for a group, within the range set bygroup_consumer_min_assignment_interval_msandgroup_consumer_max_assignment_interval_ms. -
Limits:
group_consumer_max_sizecaps the number of members in a group, andgroup_consumer_write_timeout_mscaps how long the broker takes to replicate and apply a write for a group.
For the default, range, and full description of each property, see Cluster Configuration Properties.
Redpanda rejects a configuration change that puts a timeout or interval outside its allowed range. To set a value below the current minimum, lower the minimum first. For example, to set a 30-second session timeout:
rpk cluster config set group_consumer_min_session_timeout_ms 30000
rpk cluster config set group_consumer_session_timeout_ms 30000
For details on setting cluster properties, see Configure Cluster Properties.
Monitor consumer groups
Redpanda reports consumer group activity for both protocols, so you can compare them while you migrate:
-
redpanda_kafka_consumer_groupscounts the groups that each broker coordinates, withredpanda_protocolandredpanda_statelabels. -
redpanda_kafka_consumer_group_rebalances_totalcounts the rebalances that each broker coordinated, with aredpanda_protocollabel.