KafkaDataConsumer Contract

KafkaDataConsumer is the contract for KafkaDataConsumers that use an InternalKafkaConsumer for the following:

KafkaDataConsumer has to be released explicitly.

package org.apache.spark.sql.kafka010

sealed trait KafkaDataConsumer {
  // only required properties (vals and methods) that have no implementation
  // the others follow
  def internalConsumer: InternalKafkaConsumer
  def release(): Unit
}
Table 1. KafkaDataConsumer Contract
Property Description

internalConsumer

Used when:

release

Used when:

  • KafkaSourceRDD is requested to compute a partition

  • (Spark Structured Streaming) KafkaContinuousDataReader is requested to close
Table 2. KafkaDataConsumers
KafkaDataConsumer Description

CachedKafkaDataConsumer

NonCachedKafkaDataConsumer

Note
 KafkaDataConsumer is a Scala sealed trait which means that all the implementations are in the same compilation unit (a single file).

Getting Single Kafka ConsumerRecord — get Method

get(
  offset: Long,
  untilOffset: Long,
  pollTimeoutMs: Long,
  failOnDataLoss: Boolean): ConsumerRecord[Array[Byte], Array[Byte]]

get simply requests the InternalKafkaConsumer to get a single Kafka ConsumerRecord.

Note

get is used when:

  • KafkaSourceRDD is requested to compute a partition

  • (Spark Structured Streaming) KafkaContinuousDataReader is requested to next

Getting Single AvailableOffsetRange — getAvailableOffsetRange Method

getAvailableOffsetRange(): AvailableOffsetRange

getAvailableOffsetRange simply requests the InternalKafkaConsumer to get a single AvailableOffsetRange.

Note

getAvailableOffsetRange is used when:

results matching ""

    No results matching ""