KafkaMicroBatchInputPartitionReader

KafkaMicroBatchInputPartitionReader is an InputPartitionReader (of InternalRows) that is created exclusively when KafkaMicroBatchInputPartition is requested for one (as a part of the InputPartition contract).

Creating KafkaMicroBatchInputPartitionReader Instance

KafkaMicroBatchInputPartitionReader takes the following to be created:

KafkaOffsetRange
Kafka parameters used for Kafka clients on executors (Map[String, Object])
Poll timeout (in ms)
failOnDataLoss flag
reuseKafkaConsumer flag

Note	All the input arguments to create a `KafkaMicroBatchInputPartitionReader` are exactly the input arguments used to create a KafkaMicroBatchInputPartition.

KafkaMicroBatchInputPartitionReader initializes the internal properties.

`next` Method

next(): Boolean

Note	`next` is part of the `InputPartitionReader` contract to proceed to next record if available (`true`).

next checks whether the KafkaDataConsumer should poll records or not (i.e. nextOffset is smaller than the untilOffset of the KafkaOffsetRange).

`next` Method — KafkaDataConsumer Polls Records

If so, next requests the KafkaDataConsumer to get (poll) records in the range of nextOffset and the untilOffset (of the KafkaOffsetRange) with the given pollTimeoutMs and failOnDataLoss.

With a new record, next requests the KafkaRecordToUnsafeRowConverter to convert (toUnsafeRow) the record to be the next UnsafeRow. next sets the nextOffset as the offset of the record incremented. next returns true.

With no new record, next simply returns false.

`next` Method — No Polling

If the nextOffset is equal or larger than the untilOffset (of the KafkaOffsetRange), next simply returns false.

Closing (Releasing KafkaDataConsumer) — `close` Method

close(): Unit

Note	`close` is part of the Java Closeable contract to release resources.

close simply requests the KafkaDataConsumer to release.

`resolveRange` Internal Method

resolveRange(
  range: KafkaOffsetRange): KafkaOffsetRange

resolveRange…FIXME

Note	`resolveRange` is used exclusively when `KafkaMicroBatchInputPartitionReader` is created (and initializes the KafkaOffsetRange internal property).

Internal Properties

Name Description

Name	Description
`consumer`	KafkaDataConsumer for the partition (per KafkaOffsetRange) Used in next, close, and resolveRange
`converter`	`KafkaRecordToUnsafeRowConverter`
`nextOffset`	Next offset
`nextRow`	Next `UnsafeRow`
`rangeToRead`	`KafkaOffsetRange`

consumer

KafkaDataConsumer for the partition (per KafkaOffsetRange)

Used in next, close, and resolveRange

converter

KafkaRecordToUnsafeRowConverter

nextOffset

Next offset

nextRow

Next UnsafeRow

rangeToRead

KafkaOffsetRange

KafkaMicroBatchInputPartitionReader