Partition Log

Log is a partition log that is created when LogManager is requested to loadLog (when LogManager is created) and getOrCreateLog.

Log is a KafkaMetricsGroup and registers performance metrics.

Log uses .kafka_cleanshutdown for…​FIXME

Log uses -delete suffix for…​FIXME

Log is isFuture when…​FIXME

Table 1. Log’s Performance Metrics
Metric Name Description

NumLogSegments

LogEndOffset

LogStartOffset

Size

The performance metrics are registered in kafka.log:type=Log group.

Log jconsole.png
Figure 1. Log in jconsole

Log uses a Scheduler to schedule the background tasks.

Table 2. Log’s Background Tasks
Name Period Delay Description

PeriodicProducerExpirationCheck

producerIdExpirationCheckIntervalMs

producerIdExpirationCheckIntervalMs

Requests the ProducerStateManager to removeExpiredProducers

Scheduled immediately when Log is created.

flush-log

-1 (once)

0L

flush

Scheduled when Log is requested to roll.

delete-file

-1 (once)

file.delete.delay.ms

deleteSeg

Scheduled when Log is requested to asyncDeleteSegment.

Log uses [Log partition=[topicPartition], dir=[parent]] as the logging prefix (aka logIdent).

Table 3. Log’s Internal Properties (e.g. Registries, Counters and Flags)
Name Description

nextOffsetMetadata

LogOffsetMetadata

Used when…​FIXME

segments

java.util.concurrent.ConcurrentSkipListMap of Longs and their LogSegments

Used when…​FIXME

Tip

Enable WARN, INFO, DEBUG or TRACE logging levels for kafka.log.Log logger to see what happens inside.

Add the following line to log4j.properties:

log4j.logger.kafka.log.Log=TRACE

Refer to Logging.

Creating Log Instance

Log takes the following when created:

  • Log directory

  • LogConfig

  • logStartOffset

  • recoveryPoint

  • Scheduler

  • BrokerTopicStats

  • Time

  • maxProducerIdExpirationMs

  • producerIdExpirationCheckIntervalMs

  • TopicPartition

  • ProducerStateManager

  • LogDirFailureChannel

Log initializes the internal registries and counters.

While being created, Log…​FIXME

Creating Log Instance — apply Factory Method

apply(
  dir: File,
  config: LogConfig,
  logStartOffset: Long,
  recoveryPoint: Long,
  scheduler: Scheduler,
  brokerTopicStats: BrokerTopicStats,
  time: Time = Time.SYSTEM,
  maxProducerIdExpirationMs: Int,
  producerIdExpirationCheckIntervalMs: Int,
  logDirFailureChannel: LogDirFailureChannel): Log

apply…​FIXME

Note
apply is used when LogManager is requested to loadLog and getOrCreateLog.

roll Method

roll(expectedNextOffset: Long = 0): LogSegment

roll…​FIXME

Note
roll is used when…​FIXME

asyncDeleteSegment Internal Method

asyncDeleteSegment(segment: LogSegment): Unit

asyncDeleteSegment…​FIXME

Note
asyncDeleteSegment is used when Log is requested to deleteSegment and replaceSegments.

flush Method

flush(offset: Long): Unit

flush…​FIXME

Note
flush is used when…​FIXME

deleteSeg Internal Method

deleteSeg(): Unit

deleteSeg…​FIXME

Note
deleteSeg is used when…​FIXME

appendAsLeader Method

appendAsLeader(
  records: MemoryRecords,
  leaderEpoch: Int,
  isFromClient: Boolean = true): LogAppendInfo

appendAsLeader simply append with the assignOffsets flag on.

Note
appendAsLeader is used exclusively when Partition is requested to appendRecordsToLeader.

appendAsFollower Method

appendAsFollower(records: MemoryRecords): LogAppendInfo

appendAsFollower simply append with the isFromClient and assignOffsets flags off.

Note
appendAsFollower is used exclusively when Partition is requested to doAppendRecordsToFollowerOrFutureReplica.

append Internal Method

append(
  records: MemoryRecords,
  isFromClient: Boolean,
  assignOffsets: Boolean,
  leaderEpoch: Int): LogAppendInfo

append…​FIXME

Note
append is used when Log is requested to appendAsLeader and appendAsFollower.

deleteSegment Internal Method

deleteSegment(segment: LogSegment): Unit

deleteSegment…​FIXME

Note
deleteSegment is used when Log is requested to recoverLog, deleteSegments, roll, truncateTo, and truncateFullyAndStartAt.

replaceSegments Internal Method

replaceSegments(
  newSegments: Seq[LogSegment],
  oldSegments: Seq[LogSegment],
  isRecoveredSwapFile: Boolean = false): Unit

replaceSegments…​FIXME

Note

replaceSegments is used when:

recoverLog Internal Method

recoverLog(): Long

recoverLog…​FIXME

Note
recoverLog is used exclusively when Log is requested to loadSegments.

deleteSegments Internal Method

deleteSegments(deletable: Iterable[LogSegment]): Int

deleteSegments…​FIXME

Note
deleteSegments is used exclusively when Log is requested to deleteOldSegments.

truncateTo Internal Method

truncateTo(targetOffset: Long): Boolean

truncateTo…​FIXME

Note
truncateTo is used exclusively when LogManager is requested to truncateTo.

truncateFullyAndStartAt Internal Method

truncateFullyAndStartAt(newOffset: Long): Unit

truncateFullyAndStartAt…​FIXME

Note

truncateFullyAndStartAt is used when:

loadSegments Internal Method

loadSegments(): Long

loadSegments…​FIXME

Note
loadSegments is used exclusively when Log is created (to create a LogOffsetMetadata).

deleteOldSegments Method

deleteOldSegments(): Long
// Private API
deleteOldSegments(
  predicate: (LogSegment, Option[LogSegment]) => Boolean,
  reason: String): Int

deleteOldSegments…​FIXME

Note
deleteOldSegments is used when…​FIXME

completeSwapOperations Internal Method

completeSwapOperations(swapFiles: Set[File]): Unit

completeSwapOperations…​FIXME

Note
completeSwapOperations is used when…​FIXME

splitOverflowedSegment Internal Method

splitOverflowedSegment(segment: LogSegment): List[LogSegment]

splitOverflowedSegment…​FIXME

Note
splitOverflowedSegment is used when…​FIXME

loadProducerState Internal Method

loadProducerState(lastOffset: Long, reloadFromCleanShutdown: Boolean): Unit

loadProducerState…​FIXME

Note
loadProducerState is used when Log is created and requested to truncateTo.

loadSegmentFiles Internal Method

loadSegmentFiles(): Unit

loadSegmentFiles…​FIXME

Note
loadSegmentFiles is used exclusively when Log is requested to loadSegments (when created).

onHighWatermarkIncremented Method

onHighWatermarkIncremented(highWatermark: Long): Unit

onHighWatermarkIncremented…​FIXME

Note
onHighWatermarkIncremented is used when…​FIXME

parseTopicPartitionName Object Method

parseTopicPartitionName(dir: File): TopicPartition

parseTopicPartitionName…​FIXME

Note
parseTopicPartitionName is used when…​FIXME

results matching ""

    No results matching ""