RebalanceListener — Kafka ConsumerRebalanceListener for Partition Assignment Among Processor Tasks

RebalanceListener is a Apache Kafka ConsumerRebalanceListener callback interface that listens to changes to the partitions assigned to a single stream processor thread (aka StreamThread).

kafka streams RebalanceListener.png
Figure 1. RebalanceListener, StreamThread and KafkaStreams
The number of stream processor threads per KafkaStreams instance is controlled by configuration property with the default being 1 thread.

ConsumerRebalanceListener is a callback interface that the user can implement to trigger custom actions when the set of partitions assigned to the consumer changes.

When Kafka is managing the group membership, a partition re-assignment will be triggered any time the members of the group change or the subscription of the members changes. This can occur when processes die, new process instances are added or old instances come back to life after failure. Rebalances can also be triggered by changes affecting the subscribed topics (e.g. when the number of partitions is administratively adjusted).

It is guaranteed that all consumer processes will invoke onPartitionsRevoked prior to any process invoking onPartitionsAssigned.

RebalanceListener is created exclusively when StreamThread is created.

kafka streams RebalanceListener creating instance.png
Figure 2. Creating RebalanceListener

At the completion of a successful partition re-assignment (i.e. onPartitionsAssigned event) RebalanceListener requests TaskManager to creating processor tasks for assigned topic partitions.

At the start of a rebalance operation (i.e. onPartitionsRevoked event) RebalanceListener requests TaskManager to suspending all stream tasks and state.

RebalanceListener uses StreamThread for the following:

  1. Changing the state (of the stream processor thread) at partition assignment and partition revocation

  2. Clearing standby records at the end of partition revocation

  3. Notifying about the error caught at partition assignment and partition revocation


RebalanceListener uses the logger from the owning StreamThread for logging.

Enable DEBUG logging level for org.apache.kafka.streams.processor.internals.StreamThread logger to see what happens inside.

Add the following line to

Handling Partition Assignment — onPartitionsAssigned Handler Method

void onPartitionsAssigned(
  Collection<TopicPartition> assignment)
onPartitionsAssigned is part of ConsumerRebalanceListener Contract in Apache Kafka to…​FIXME.

Internally, onPartitionsAssigned first prints out the following DEBUG message to the logs:

at state [state]: partitions [assignment] assigned at the end of consumer rebalance.
  current suspended active tasks: [taskIds]
  current suspended standby tasks: [taskIds]

onPartitionsAssigned requests StreamThread to set the state to PARTITIONS_ASSIGNED followed by requesting TaskManager for processor tasks for assigned topic partitions.

In the end, onPartitionsAssigned prints out the following INFO message to the logs:

partition assignment took [duration] ms.
  current active tasks: [activeTaskIds]
  current standby tasks: [standbyTaskIds]
  previous active tasks: [prevActiveTaskIds]

onPartitionsAssigned does nothing (i.e. prints the messages to the logs) when the state transition was invalid.

Handling Partition Revocation — onPartitionsRevoked Handler Method

void onPartitionsRevoked(
  Collection<TopicPartition> assignment)
onPartitionsRevoked is part of ConsumerRebalanceListener Contract in Apache Kafka to…​FIXME.


Creating RebalanceListener Instance

RebalanceListener takes the following when created:

results matching ""

    No results matching ""