KafkaMicroBatchInputPartition

KafkaMicroBatchInputPartition is an InputPartition (of InternalRows) that is used (created) exclusively when KafkaMicroBatchReader is requested for input partitions (when DataSourceV2ScanExec physical operator is requested for the partitions of the input RDD).

KafkaMicroBatchInputPartition takes the following to be created:

  • KafkaOffsetRange

  • Kafka parameters used for Kafka clients on executors (Map[String, Object])

  • Poll timeout (in ms)

  • failOnDataLoss flag

  • reuseKafkaConsumer flag

KafkaMicroBatchInputPartition creates a KafkaMicroBatchInputPartitionReader when requested for a InputPartitionReader[InternalRow] (as a part of the InputPartition contract).

KafkaMicroBatchInputPartition simply requests the given KafkaOffsetRange for the optional preferredLoc when requested for preferredLocations (as a part of the InputPartition contract).

results matching ""

    No results matching ""