KafkaMicroBatchInputPartitionReader

KafkaMicroBatchInputPartitionReader is an InputPartitionReader (of InternalRows) that is created exclusively when KafkaMicroBatchInputPartition is requested for one (as a part of the InputPartition contract).

Creating KafkaMicroBatchInputPartitionReader Instance

KafkaMicroBatchInputPartitionReader takes the following to be created:

  • KafkaOffsetRange

  • Kafka parameters used for Kafka clients on executors (Map[String, Object])

  • Poll timeout (in ms)

  • failOnDataLoss flag

  • reuseKafkaConsumer flag

Note
All the input arguments to create a KafkaMicroBatchInputPartitionReader are exactly the input arguments used to create a KafkaMicroBatchInputPartition.

KafkaMicroBatchInputPartitionReader initializes the internal properties.

next Method

next(): Boolean
Note
next is part of the InputPartitionReader contract to proceed to next record if available (true).

next checks whether the KafkaDataConsumer should poll records or not (i.e. nextOffset is smaller than the untilOffset of the KafkaOffsetRange).

next Method — KafkaDataConsumer Polls Records

If so, next requests the KafkaDataConsumer to get (poll) records in the range of nextOffset and the untilOffset (of the KafkaOffsetRange) with the given pollTimeoutMs and failOnDataLoss.

With a new record, next requests the KafkaRecordToUnsafeRowConverter to convert (toUnsafeRow) the record to be the next UnsafeRow. next sets the nextOffset as the offset of the record incremented. next returns true.

With no new record, next simply returns false.

next Method — No Polling

If the nextOffset is equal or larger than the untilOffset (of the KafkaOffsetRange), next simply returns false.

Closing (Releasing KafkaDataConsumer) — close Method

close(): Unit
Note
close is part of the Java Closeable contract to release resources.

close simply requests the KafkaDataConsumer to release.

resolveRange Internal Method

resolveRange(
  range: KafkaOffsetRange): KafkaOffsetRange

resolveRange…​FIXME

Note
resolveRange is used exclusively when KafkaMicroBatchInputPartitionReader is created (and initializes the KafkaOffsetRange internal property).

Internal Properties

Name Description

consumer

KafkaDataConsumer for the partition (per KafkaOffsetRange)

Used in next, close, and resolveRange

converter

KafkaRecordToUnsafeRowConverter

nextOffset

Next offset

nextRow

Next UnsafeRow

rangeToRead

KafkaOffsetRange

results matching ""

    No results matching ""