Consumer Group Protocol

The consumer group protocol, defined in Kafka Improvement Proposal (KIP) 848, moves partition assignment from the consumers to the broker. In the classic protocol, every member of a group stops consuming while the group rebalances. In the consumer group protocol, the broker computes each member’s assignment and hands partitions over incrementally, so members that keep their partitions don’t stop consuming while the group changes.

Use the consumer group protocol when your consumer groups scale up and down often, or when rebalance pauses cause consumer lag. Because the broker manages assignment, a single slow or misbehaving consumer can’t disrupt the whole group.

After reading this page, you will be able to:

  • Explain how the consumer group protocol differs from the classic rebalance protocol

  • Decide whether a consumer group can use the consumer group protocol

  • Enable the consumer group protocol on a cluster and opt in a consumer

Classic groups and consumer groups

A consumer group uses one of two protocols, and its type is named after the protocol:

  • A classic group uses the original rebalance protocol. All groups are classic groups unless their consumers opt in to the consumer group protocol.

  • A consumer group uses the consumer group protocol defined in KIP-848.

rpk group list and rpk group describe show this as the group’s TYPE: classic or consumer. Classic groups and consumer groups run side by side on the same cluster, and enabling the consumer group protocol doesn’t change your existing classic groups.

Most things stay the same in both protocols: offset commits, transactions, the __consumer_offsets topic, and the way you think about a group of consumers sharing partitions. Your application code doesn’t change. What changes is how the client and the broker agree on who owns each partition, and the client handles that for you.

How the consumer group protocol works

In the classic protocol, a group rebalances in rounds that every member takes part in. Each member sends JoinGroup and SyncGroup requests, one member (the group leader) computes the assignment on the client side, and members give up their partitions until the round completes. The whole group waits for the slowest member.

In the consumer group protocol, each member sends periodic ConsumerGroupHeartbeat requests to the group coordinator. Each heartbeat carries the member’s subscription and the partitions it owns, and the response carries the member’s assignment:

  1. When a member joins, leaves, or changes its subscription, the coordinator computes a new target assignment for the group with a server-side partition assignor. The assignor is sticky: it moves as few partitions as possible.

  2. The coordinator reconciles each member toward its target assignment. A member gives up a partition before the coordinator assigns that partition to another member, so no partition has two owners.

  3. A partition moves to its new owner as soon as its previous owner stops consuming it. Members whose partitions don’t change keep consuming throughout.

The group and each member have an epoch that increases as the assignment changes. The coordinator uses the epochs to fence members that fall behind.

Example: a second consumer joins

This example follows a group that consumes a topic with four partitions, p0 to p3:

  1. Consumer A sends a heartbeat to join the empty group. The coordinator assigns it all four partitions, and the group state changes to Stable. Consumer A fetches its committed offsets (there are none yet), consumes records, and commits offsets.

  2. Consumer B sends a heartbeat to join. The coordinator computes a new target assignment that gives p0 and p1 to consumer A and p2 and p3 to consumer B. The group state changes to Reconciling. Consumer B has no partitions yet, because consumer A still owns them all.

  3. On consumer A’s next heartbeat, the coordinator tells it that its assignment is now p0 and p1. Consumer A stops consuming p2 and p3 and keeps consuming p0 and p1 without a pause.

  4. Consumer A’s next heartbeat reports that it owns only p0 and p1. On consumer B’s next heartbeat, the coordinator assigns it p2 and p3, and the group state changes back to Stable.

  5. Consumer B fetches the committed offsets for p2 and p3 and resumes consuming from them.

In the classic protocol, consumer A would stop consuming all four partitions while the group rebalanced.

A consumer group can be in one of these states:

  • Empty: The group has no members.

  • Assigning: A member joined, left, or changed its subscription, and the coordinator is computing a new target assignment.

  • Reconciling: The coordinator computed a target assignment, and members are moving partitions to reach it.

  • Stable: Every member has reached the target assignment.

The STATE field in rpk group list and rpk group describe shows the group’s current state.

Limitations

The consumer group protocol in Redpanda has the following limitations:

  • Static membership is not supported. The broker rejects a heartbeat that sets a group instance ID (group.instance.id) with INVALID_REQUEST. If your consumers set group.instance.id, keep them on the classic protocol.

  • Regex subscriptions are not supported. The broker rejects a heartbeat that subscribes with a topic regular expression with INVALID_REQUEST. Subscribe to a list of topic names instead.

  • A group with active members can’t switch protocols. A group switches protocols only when it has no members, so switching requires you to stop the group’s consumers. See Switch an existing group to the consumer group protocol.

  • Only the uniform assignor is available. The broker responds to a heartbeat that names any other server-side assignor, such as range, with UNSUPPORTED_ASSIGNOR.

  • Group-level configuration is not supported. Session timeouts, heartbeat intervals, and assignment settings apply to every consumer group on the cluster.

Before you enable the consumer group protocol

Enabling the protocol on the cluster doesn’t move any group to it. Each consumer must opt in, and most Kafka clients still use the classic protocol by default. Before you opt in a group, check the following:

  • Test the consumer group protocol in a non-production environment first.

  • Check that the group’s clients support the protocol. See Prerequisites.

  • Keep a group on the classic protocol if its consumers need any of the following:

    • Static membership (group.instance.id) or regex subscriptions

    • An assignor other than uniform

    • Their own session timeout or heartbeat interval. In the consumer group protocol, these are cluster-wide settings. See Configure consumer groups.

    • A switch with no downtime. To switch protocols, you must stop every consumer in the group.

Prerequisites

  • A Redpanda cluster running version 26.3 or later, with the upgrade finalized. The consumer group protocol stays unavailable until the cluster upgrade is finalized, even when you enable it. See Defer Upgrade Finalization.

  • A Kafka client that supports the consumer group protocol:

    Client Minimum version How to opt in

    Apache Kafka Java client

    4.0.0

    Set group.protocol=consumer.

    librdkafka

    2.12.0

    Set group.protocol=consumer.

    franz-go

    1.22.0

    Add the kgo.ServerSideBalancer() group option.

  • If authorization is enabled, the consumer’s principal needs the READ permission on the consumer group and the DESCRIBE permission on each topic it subscribes to. See Configure Access Control Lists.

The consumer group protocol is available in Redpanda Community Edition and doesn’t require an enterprise license.

Enable the consumer group protocol

The consumer group protocol is disabled by default. When it’s disabled, the broker responds to ConsumerGroupHeartbeat requests with UNSUPPORTED_VERSION, and consumers must use the classic protocol.

  1. Enable the protocol on the cluster:

    rpk cluster config set group_consumer_protocol_enabled true

    The change takes effect without a restart. If the cluster upgrade isn’t finalized, the brokers log a warning and the protocol stays unavailable until you finalize the upgrade.

  2. In your consumer configuration, opt in to the consumer group protocol. For example, with the Apache Kafka Java client:

    group.id=<group-id>
    group.protocol=consumer

    Replace <group-id> with a new group ID, or with the ID of an existing group that has no active members. To move an existing classic group, see Switch an existing group to the consumer group protocol.

    In the consumer group protocol, the broker controls the session timeout, the heartbeat interval, and the assignor, so remove session.timeout.ms, heartbeat.interval.ms, and partition.assignment.strategy from your consumer configuration. Clients reject these settings when you use the consumer group protocol. Clients choose a server-side assignor with group.remote.assignor instead. Leave it unset or set it to uniform, the only assignor that Redpanda supports.

  3. Start the consumers, then confirm that the group uses the consumer group protocol:

    rpk group list --types consumer

    The output lists only consumer groups:

    BROKER  GROUP             STATE   TYPE
    0       order-processors  Stable  consumer
  4. To see the group’s epochs and assignments, describe the group with --print-members:

    rpk group describe <group-id> --print-members

    For a consumer group, the summary includes TYPE consumer, the group epoch (GROUP-EPOCH), and the target assignment epoch (TARGET-ASSIGNMENT-EPOCH). The member table shows each member’s epoch (MEMBER-EPOCH), current assignment (ASSIGNMENT), and target assignment (TARGET-ASSIGNMENT):

    GROUP                    order-processors
    COORDINATOR-NODE         0
    COORDINATOR-PARTITION    __consumer_offsets/0
    STATE                    Stable
    BALANCER                 uniform
    MEMBERS                  3
    TOTAL-LAG                0
    TYPE                     consumer
    GROUP-EPOCH              4
    TARGET-ASSIGNMENT-EPOCH  4
    
    MEMBER-ID                 CLIENT-ID  HOST        MEMBER-EPOCH  ASSIGNMENT  TARGET-ASSIGNMENT
    COFm64KdBqd3VbY3ztjVWg==  rpk        /127.0.0.1  4             orders:2,3  orders:2,3
    J8EBq40AwIV3nOyAK3306w==  rpk        /127.0.0.1  4             orders:4,5  orders:4,5
    tntB6Ttz1Em2ObESsKjpjA==  rpk        /127.0.0.1  4             orders:0,1  orders:0,1

    When every member’s ASSIGNMENT matches its TARGET-ASSIGNMENT and its MEMBER-EPOCH matches the TARGET-ASSIGNMENT-EPOCH, the group has finished reconciling. Without --print-members, rpk group describe shows each partition’s offsets and lag instead of the member table.

    In a consumer group, each client generates its own member ID, so member IDs look different from those in a classic group, where the broker assigns IDs such as rpk-9bea3f80-7760-4392-91d0-30cce4118d41.

Switch an existing group to the consumer group protocol

An existing classic group switches to the consumer group protocol when it has no active members and a consumer that uses the consumer group protocol joins it. The group keeps its committed offsets, so the consumers resume from their last committed positions.

Redpanda doesn’t switch a group while it has active members. A rolling restart, in which some consumers use the classic protocol and some use the consumer group protocol at the same time, doesn’t work: the broker responds to the new consumers' heartbeats with GROUP_ID_NOT_FOUND until every classic consumer leaves the group.

To switch a group:

  1. Stop every consumer in the group.

  2. Wait until the group has no members. Run rpk group describe <group-id> and check that STATE is Empty.

  3. Opt in to the consumer group protocol in the consumer configuration, and remove session.timeout.ms, heartbeat.interval.ms, and partition.assignment.strategy. See Enable the consumer group protocol.

  4. Start the consumers.

  5. Run rpk group describe <group-id> and check that TYPE is consumer.

To switch a group back to the classic protocol, follow the same steps with group.protocol=classic. A consumer group switches back when it has no members and a classic consumer joins it, and it keeps its committed offsets.

Configure consumer groups

In the consumer group protocol, cluster properties control the timeouts and assignment behavior that the classic protocol leaves to each client. These properties apply to every consumer group on the cluster, and they take effect without a restart.

For the default, range, and full description of each property, see Cluster Configuration Properties.

Redpanda rejects a configuration change that puts a timeout or interval outside its allowed range. To set a value below the current minimum, lower the minimum first. For example, to set a 30-second session timeout:

rpk cluster config set group_consumer_min_session_timeout_ms 30000
rpk cluster config set group_consumer_session_timeout_ms 30000

For details on setting cluster properties, see Configure Cluster Properties.

Monitor consumer groups

Redpanda reports consumer group activity for both protocols, so you can compare them while you migrate:

  • redpanda_kafka_consumer_groups counts the groups that each broker coordinates, with redpanda_protocol and redpanda_state labels.

  • redpanda_kafka_consumer_group_rebalances_total counts the rebalances that each broker coordinated, with a redpanda_protocol label.