KafkaSourceOffset

KafkaSourceOffset is a custom Offset for kafka data source.

KafkaSourceOffset is created (directly or indirectly using apply) when:

KafkaContinuousReader is requested to setStartOffset, deserializeOffset, and mergeOffsets
KafkaMicroBatchReader is requested to getStartOffset, getEndOffset, deserializeOffset, and getOrCreateInitialPartitionOffsets
KafkaOffsetReader is requested to fetchSpecificOffsets
KafkaSource is requested for the initial partition offsets (of 0th batch) and getOffset
KafkaSourceInitialOffsetWriter is requested to deserialize a KafkaSourceOffset (from an InputStream)
KafkaSourceOffset is requested for partition offsets

KafkaSourceOffset takes a collection of Kafka TopicPartitions with offsets to be created.

getPartitionOffsets(
  offset: Offset): Map[TopicPartition, Long]

getPartitionOffsets takes KafkaSourceOffset.partitionToOffsets from offset.

If offset is KafkaSourceOffset, getPartitionOffsets takes the partitions and offsets straight from it.

If however offset is SerializedOffset, getPartitionOffsets deserializes the offsets from JSON.

getPartitionOffsets reports an IllegalArgumentException when offset is neither KafkaSourceOffset or SerializedOffset.

Invalid conversion from offset of [class] to KafkaSourceOffset

Note	`getPartitionOffsets` is used when: `KafkaContinuousReader` is requested to planInputPartitions `KafkaSource` is requested to generate a streaming DataFrame with records from Kafka for a streaming micro-batch

json: String

Note	`json` is part of the Offset Contract for a JSON-encoded offset.

json…FIXME

apply(
  offsetTuples: (String, Int, Long)*): KafkaSourceOffset (1)
apply(
  offset: SerializedOffset): KafkaSourceOffset

apply…FIXME

Note	`apply` is used when: `KafkaSourceInitialOffsetWriter` is requested to deserialize a KafkaSourceOffset (from an InputStream) `KafkaSource` is requested for the initial partition offsets (of 0th batch) `KafkaSourceOffset` is requested to getPartitionOffsets

results matching ""