Partition

A Kafka topic is spread across a Kafka cluster as a virtual group of one or more partitions.

A single partition of a topic (topic partition) can be replicated across a Kafka cluster to one or more Kafka brokers.

A topic partition has one partition leader node and zero or more replicas.

Kafka producers publish messages to topic leaders as do Kafka consumers consume them from.

In-Sync Replicas (ISR) are brokers that…​FIXME

Offline Replicas are…​FIXME

Partition is an internal data structure that ReplicaManager uses to represent a TopicPartition (and manage all partitions).

Partition is created when:

Partition is a KafkaMetricsGroup and registers performance metrics.

Table 1. Partition’s Performance Metrics
Metric Name Description

InSyncReplicasCount

One of the two possible values:

LastStableOffsetLag

ReplicasCount

One of the two possible values:

UnderMinIsr

One of the two possible values:

UnderReplicated

One of the two possible values:

The performance metrics are registered in kafka.cluster:type=Partition group (only when the partition is not offline).

Partition jconsole.png
Figure 1. Partition in jconsole

Partition is given the localBrokerId that is one of the following:

Partition uses [Partition [topicPartition] broker=[localBrokerId]] as the logging prefix (aka logIdent).

Table 2. Partition’s Internal Properties (e.g. Registries, Counters and Flags)
Name Description

inSyncReplicas

In-sync Replicas

leaderReplicaIdOpt

Broker ID of the broker that manages the leader replica

Default: None / undefined

allReplicasMap

maybeExpandIsr Method

maybeExpandIsr(replicaId: Int, logReadResult: LogReadResult): Boolean

maybeExpandIsr…​FIXME

Note
maybeExpandIsr is used exclusively when Partition is requested to updateReplicaLogReadResult.

maybeShrinkIsr Method

maybeShrinkIsr(replicaMaxLagTimeMs: Long): Unit

maybeShrinkIsr…​FIXME

Note
maybeShrinkIsr is used exclusively when ReplicaManager maybeShrinkIsr.

updateReplicaLogReadResult Method

updateReplicaLogReadResult(replica: Replica, logReadResult: LogReadResult): Boolean

updateReplicaLogReadResult…​FIXME

Note
updateReplicaLogReadResult is used exclusively when ReplicaManager updateFollowerLogReadResults.

updateIsr Internal Method

updateIsr(newIsr: Set[Replica]): Unit

updateIsr…​FIXME

Note
updateIsr is used when Partition is requested to expand or shrink the ISR.

makeFollower Method

makeFollower(
  controllerId: Int,
  partitionStateInfo: LeaderAndIsrRequest.PartitionState,
  correlationId: Int): Boolean

makeFollower…​FIXME

Note
makeFollower is used exclusively when ReplicaManager is requested to makeFollowers.

leaderReplicaIfLocal Method

leaderReplicaIfLocal: Option[Replica]

leaderReplicaIfLocal returns a Replica when the leaderReplicaIdOpt is the localBrokerId. Otherwise, leaderReplicaIfLocal returns None (i.e. undefined).

Note
leaderReplicaIfLocal is used…​FIXME

maybeShrinkIsr Method

maybeShrinkIsr(replicaMaxLagTimeMs: Long): Unit

maybeShrinkIsr…​FIXME

Note
maybeShrinkIsr is used when…​FIXME

Creating Partition Instance

Partition takes the following when created:

Partition initializes the internal registries and counters.

isUnderReplicated Predicate

isUnderReplicated: Boolean

isUnderReplicated is true only if the partition isLeaderReplicaLocal and the number of in-sync replicas is below the assignedReplicas.

Note
isUnderReplicated is used when…​FIXME

isUnderMinIsr Predicate

isUnderMinIsr: Boolean

isUnderMinIsr is true only if the partition isLeaderReplicaLocal and the number of in-sync replicas is below the min.insync.replicas configuration property (as configured for the Log of the leader replica).

Note
isUnderMinIsr is used when…​FIXME

checkEnoughReplicasReachOffset Method

checkEnoughReplicasReachOffset(requiredOffset: Long): (Boolean, Errors)

checkEnoughReplicasReachOffset…​FIXME

Note
checkEnoughReplicasReachOffset is used when…​FIXME

makeLeader Method

makeLeader(
  controllerId: Int,
  partitionStateInfo: LeaderAndIsrRequest.PartitionState,
  correlationId: Int): Boolean

makeLeader…​FIXME

Note
makeLeader is used exclusively when ReplicaManager is requested to makeLeaders.

getOrCreateReplica Method

getOrCreateReplica(
  replicaId: Int = localBrokerId,
  isNew: Boolean = false): Replica

getOrCreateReplica…​FIXME

Note

getOrCreateReplica is used when:

maybeCreateFutureReplica Method

maybeCreateFutureReplica(logDir: String): Boolean

maybeCreateFutureReplica…​FIXME

Note
maybeCreateFutureReplica is used exclusively when ReplicaManager is requested to alterReplicaLogDirs.

appendRecordsToLeader Method

appendRecordsToLeader(
  records: MemoryRecords,
  isFromClient: Boolean,
  requiredAcks: Int = 0): LogAppendInfo

appendRecordsToLeader basically requests the Log (of the leader Replica) to appendAsLeader.

Internally, appendRecordsToLeader…​FIXME

Note

appendRecordsToLeader is used when:

doAppendRecordsToFollowerOrFutureReplica Internal Method

doAppendRecordsToFollowerOrFutureReplica(
  records: MemoryRecords,
  isFuture: Boolean): Option[LogAppendInfo]

doAppendRecordsToFollowerOrFutureReplica…​FIXME

Note
doAppendRecordsToFollowerOrFutureReplica is used exclusively when Partition is requested to appendRecordsToFollowerOrFutureReplica.

appendRecordsToFollowerOrFutureReplica Method

appendRecordsToFollowerOrFutureReplica(
  records: MemoryRecords,
  isFuture: Boolean): Option[LogAppendInfo]

appendRecordsToFollowerOrFutureReplica…​FIXME

Note

appendRecordsToFollowerOrFutureReplica is used when:

truncateTo Method

truncateTo(offset: Long, isFuture: Boolean): Unit

truncateTo…​FIXME

Note

truncateTo is used when:

  • ReplicaAlterLogDirsThread is requested to truncate

  • ReplicaFetcherThread is requested to truncate

truncateFullyAndStartAt Method

truncateFullyAndStartAt(newOffset: Long, isFuture: Boolean): Unit

truncateFullyAndStartAt…​FIXME

Note

truncateFullyAndStartAt is used when:

maybeReplaceCurrentWithFutureReplica Method

maybeReplaceCurrentWithFutureReplica(): Boolean

maybeReplaceCurrentWithFutureReplica…​FIXME

Note
maybeReplaceCurrentWithFutureReplica is used exclusively when ReplicaAlterLogDirsThread is requested to processPartitionData.

delete Method

delete(): Unit

delete…​FIXME

Note
delete is used exclusively when ReplicaManager is requested to stopReplica.

removeFutureLocalReplica Method

removeFutureLocalReplica(deleteFromLogDir: Boolean = true): Unit

removeFutureLocalReplica…​FIXME

Note
removeFutureLocalReplica is used when ReplicaManager is requested to alterReplicaLogDirs and handleLogDirFailure.

isLeaderReplicaLocal Internal Method

isLeaderReplicaLocal: Boolean

isLeaderReplicaLocal is positive (true) when the optional Replica is defined. Otherwise, false.

Note
isLeaderReplicaLocal is used when ReplicaManager is requested for the performance metrics (InSyncReplicasCount and ReplicasCount), isUnderReplicated, and lowWatermarkIfLeader.

Creating Partition — apply Factory Method

apply(
  topicPartition: TopicPartition,
  time: Time,
  replicaManager: ReplicaManager): Partition

apply simply creates a Partition with the given TopicPartition, Time, and ReplicaManager and the following:

Note
apply is used exclusively when ReplicaManager is requested to add a TopicPartition to allPartitions.

localReplicaOrException Method

localReplicaOrException: Replica

localReplicaOrException localReplica and returns the local replica if available. Otherwise, localReplicaOrException throws a ReplicaNotAvailableException:

Replica for partition [topicPartition] is not available on broker [localBrokerId]

localReplica Method

localReplica: Option[Replica]

localReplica simply gets the partition replica for the local broker ID.

Note

localReplica is used when:

getReplica Method

getReplica(replicaId: Int): Option[Replica]

getReplica returns the replica by the given replicaId (in the allReplicasMap registry) or None.

Note

getReplica is used when:

getLocalReplica Internal Method

getLocalReplica(
  replicaId: Int,
  currentLeaderEpoch: Optional[Integer],
  requireLeader: Boolean): Either[Replica, Errors]

getLocalReplica…​FIXME

Note
getLocalReplica is used when…​FIXME

addReplicaIfNotExists Method

addReplicaIfNotExists(replica: Replica): Replica

addReplicaIfNotExists…​FIXME

Note
addReplicaIfNotExists is used when…​FIXME

assignedReplicas Method

assignedReplicas: Set[Replica]

assignedReplicas…​FIXME

Note
assignedReplicas is used when…​FIXME

allReplicas Method

allReplicas: Set[Replica]

allReplicas…​FIXME

Note
allReplicas is used when…​FIXME

removeReplica Internal Method

removeReplica(replicaId: Int): Unit

removeReplica…​FIXME

Note
removeReplica is used when…​FIXME

removeFutureLocalReplica Method

removeFutureLocalReplica(deleteFromLogDir: Boolean = true): Unit

removeFutureLocalReplica…​FIXME

Note
removeFutureLocalReplica is used when…​FIXME

maybeReplaceCurrentWithFutureReplica Method

maybeReplaceCurrentWithFutureReplica(): Boolean

maybeReplaceCurrentWithFutureReplica…​FIXME

Note
maybeReplaceCurrentWithFutureReplica is used when…​FIXME

delete Method

delete(): Unit

delete…​FIXME

Note
delete is used when…​FIXME

toString Method

toString(): String
Note
toString is part of the java.lang.Object Contract for a string representation of the object.

toString…​FIXME

results matching ""

    No results matching ""