RebalanceListener — Kafka ConsumerRebalanceListener for Partition Assignment Among Processor Tasks
RebalanceListener is a Apache Kafka ConsumerRebalanceListener callback interface that listens to changes to the partitions assigned to a single stream processor thread (aka StreamThread).
Figure 1. RebalanceListener, StreamThread and KafkaStreams
|
Note
|
The number of stream processor threads per KafkaStreams instance is controlled by num.stream.threads configuration property with the default being 1 thread. |
From the documentation of org.apache.kafka.clients.consumer.ConsumerRebalanceListener:
ConsumerRebalanceListeneris a callback interface that the user can implement to trigger custom actions when the set of partitions assigned to the consumer changes.
When Kafka is managing the group membership, a partition re-assignment will be triggered any time the members of the group change or the subscription of the members changes. This can occur when processes die, new process instances are added or old instances come back to life after failure. Rebalances can also be triggered by changes affecting the subscribed topics (e.g. when the number of partitions is administratively adjusted).
It is guaranteed that all consumer processes will invoke
onPartitionsRevokedprior to any process invokingonPartitionsAssigned.
Figure 2. Creating RebalanceListener
At the completion of a successful partition re-assignment (i.e. onPartitionsAssigned event) RebalanceListener requests TaskManager to creating processor tasks for assigned topic partitions.
At the start of a rebalance operation (i.e. onPartitionsRevoked event) RebalanceListener requests TaskManager to suspending all stream tasks and state.
RebalanceListener uses StreamThread for the following:
-
Changing the state (of the stream processor thread) at partition assignment and partition revocation
-
Clearing standby records at the end of partition revocation
-
Notifying about the error caught at partition assignment and partition revocation
|
Tip
|
Enable Add the following line to Refer to Application Logging Using log4j. |
Handling Partition Assignment — onPartitionsAssigned Handler Method
void onPartitionsAssigned(
Collection<TopicPartition> assignment)|
Note
|
onPartitionsAssigned is part of ConsumerRebalanceListener Contract in Apache Kafka to…FIXME.
|
Internally, onPartitionsAssigned first prints out the following DEBUG message to the logs:
at state [state]: partitions [assignment] assigned at the end of consumer rebalance.
current suspended active tasks: [taskIds]
current suspended standby tasks: [taskIds]onPartitionsAssigned requests StreamThread to set the state to PARTITIONS_ASSIGNED followed by requesting TaskManager for processor tasks for assigned topic partitions.
In the end, onPartitionsAssigned prints out the following INFO message to the logs:
partition assignment took [duration] ms.
current active tasks: [activeTaskIds]
current standby tasks: [standbyTaskIds]
previous active tasks: [prevActiveTaskIds]onPartitionsAssigned does nothing (i.e. prints the messages to the logs) when the state transition was invalid.
Handling Partition Revocation — onPartitionsRevoked Handler Method
void onPartitionsRevoked(
Collection<TopicPartition> assignment)|
Note
|
onPartitionsRevoked is part of ConsumerRebalanceListener Contract in Apache Kafka to…FIXME.
|
onPartitionsRevoked…FIXME