KafkaContinuousReader — ContinuousReader for Kafka Data Source in Continuous Stream Processing

KafkaContinuousReader is created exclusively when KafkaSourceProvider is requested to create a ContinuousReader.

KafkaContinuousReader uses kafkaConsumer.pollTimeoutMs configuration parameter (default: 512) for KafkaContinuousInputPartitions when requested to planInputPartitions.


Enable INFO or WARN logging levels for org.apache.spark.sql.kafka010.KafkaContinuousReader to see what happens inside.

Add the following line to conf/log4j.properties:


Refer to Logging.

Creating KafkaContinuousReader Instance

KafkaContinuousReader takes the following to be created:

  • KafkaOffsetReader

  • Kafka parameters (as java.util.Map[String, Object])

  • Source options (as Map[String, String])

  • Metadata path

  • Initial offsets

  • failOnDataLoss flag

planInputPartitions Method

planInputPartitions(): java.util.List[InputPartition[InternalRow]]
planInputPartitions is part of the DataSourceReader contract to…​FIXME.


results matching ""

    No results matching ""