GroupMetadataManager

GroupMetadataManager is…​FIXME

GroupMetadataManager is created exclusively when GroupCoordinator is created.

GroupMetadataManager creating instance.png
Figure 1. Creating GroupMetadataManager
Table 1. GroupMetadataManager’s Metrics
Name Description

NumOffsets

NumGroups

NumGroupsPreparingRebalance

NumGroupsCompletingRebalance

NumGroupsStable

NumGroupsDead

NumGroupsEmpty

Table 2. GroupMetadataManager’s Internal Properties (e.g. Registries and Counters)
Name Description

compressionType

CompressionType (default: NoCompressionCodec)

groupMetadataCache

Pool of GroupMetadata by group ID

groupMetadataTopicPartitionCount

Number of partitions for the __consumer_offsets consumer group metadata topic (default: 50)

scheduler

KafkaScheduler with 1 daemon thread with group-metadata-manager- prefix

Started when GroupMetadataManager is requested to start up.

Shut down when GroupMetadataManager is requested to shut down.

Used when:

enableMetadataExpiration Method

enableMetadataExpiration(): Unit

enableMetadataExpiration requests KafkaScheduler to start.

enableMetadataExpiration schedules delete-expired-group-metadata task that cleanupGroupMetadata every offsetsRetentionCheckIntervalMs milliseconds.

Note
enableMetadataExpiration is used exclusively when GroupCoordinator is started.

Creating GroupMetadataManager Instance

GroupMetadataManager takes the following when created:

GroupMetadataManager initializes the internal registries and counters.

cleanupGroupMetadata Internal Method

cleanupGroupMetadata(): Unit  (1)
cleanupGroupMetadata(deletedTopicPartitions: Option[Seq[TopicPartition]]): Unit
  1. Calls the other cleanupGroupMetadata with empty deletedTopicPartitions collection

cleanupGroupMetadata takes the current time (using time) and for every GroupMetadata in cache does the following:

  1. FIXME

In the end, cleanupGroupMetadata prints out the following INFO message to the logs:

Removed [offsetsRemoved] expired offsets in [duration] milliseconds
Note
cleanupGroupMetadata is used exclusively when GroupMetadataManager is requested to enableMetadataExpiration (as delete-expired-group-metadata task).

Getting Number of Partitions for __consumer_offsets Consumer Group Metadata Topic — getGroupMetadataTopicPartitionCount Internal Method

getGroupMetadataTopicPartitionCount: Int

getGroupMetadataTopicPartitionCount requests the KafkaZkClient for getTopicPartitionCount of __consumer_offsets consumer group metadata topic.

If not available, getGroupMetadataTopicPartitionCount uses the OffsetConfig for offsetsTopicNumPartitions (default: 50).

Note
getGroupMetadataTopicPartitionCount is used exclusively when GroupMetadataManager is requested for groupMetadataTopicPartitionCount.

shutdown Method

shutdown(): Unit

shutdown…​FIXME

Note
shutdown is used when…​FIXME

startup Method

startup(enableMetadataExpiration: Boolean): Unit

startup…​FIXME

Note
startup is used when…​FIXME

scheduleLoadGroupAndOffsets Method

scheduleLoadGroupAndOffsets(offsetsPartition: Int, onGroupLoaded: GroupMetadata => Unit): Unit

scheduleLoadGroupAndOffsets…​FIXME

Note
scheduleLoadGroupAndOffsets is used when…​FIXME

removeGroupsForPartition Method

removeGroupsForPartition(
  offsetsPartition: Int,
  onGroupUnloaded: GroupMetadata => Unit): Unit

removeGroupsForPartition…​FIXME

Note
removeGroupsForPartition is used when…​FIXME

scheduleHandleTxnCompletion Method

scheduleHandleTxnCompletion(
  producerId: Long,
  completedPartitions: Set[Int],
  isCommit: Boolean): Unit

scheduleHandleTxnCompletion…​FIXME

Note
scheduleHandleTxnCompletion is used when…​FIXME

loadGroupsAndOffsets Method

loadGroupsAndOffsets(
  topicPartition: TopicPartition,
  onGroupLoaded: GroupMetadata => Unit): Unit

loadGroupsAndOffsets…​FIXME

Note
loadGroupsAndOffsets is used when…​FIXME

removeGroupsAndOffsets Internal Method

removeGroupsAndOffsets(): Unit

removeGroupsAndOffsets…​FIXME

Note
removeGroupsAndOffsets is used when…​FIXME

handleTxnCompletion Method

handleTxnCompletion(
  producerId: Long,
  completedPartitions: Set[Int],
  isCommit: Boolean): Unit

handleTxnCompletion…​FIXME

Note
handleTxnCompletion is used when…​FIXME

partitionFor Method

partitionFor(groupId: String): Int

partitionFor…​FIXME

Note
partitionFor is used when…​FIXME

storeOffsets Method

storeOffsets(
  group: GroupMetadata,
  consumerId: String,
  offsetMetadata: immutable.Map[TopicPartition, OffsetAndMetadata],
  responseCallback: immutable.Map[TopicPartition, Errors] => Unit,
  producerId: Long = RecordBatch.NO_PRODUCER_ID,
  producerEpoch: Short = RecordBatch.NO_PRODUCER_EPOCH): Unit

storeOffsets…​FIXME

Note
storeOffsets is used exclusively when GroupCoordinator is requested to doCommitOffsets.

storeGroup Method

storeGroup(
  group: GroupMetadata,
  groupAssignment: Map[String, Array[Byte]],
  responseCallback: Errors => Unit): Unit

storeGroup…​FIXME

Note
storeGroup is used exclusively when GroupCoordinator is requested to doSyncGroup and onCompleteJoin.

appendForGroup Internal Method

appendForGroup(
  group: GroupMetadata,
  records: Map[TopicPartition, MemoryRecords],
  callback: Map[TopicPartition, PartitionResponse] => Unit): Unit

appendForGroup…​FIXME

Note
appendForGroup is used exclusively when GroupMetadataManager is requested to storeGroup and storeOffsets.

results matching ""

    No results matching ""