KafkaSourceInitialOffsetWriter

KafkaSourceInitialOffsetWriter is a Hadoop DFS-based metadata storage for KafkaSourceOffsets.

KafkaSourceInitialOffsetWriter is created exclusively when KafkaMicroBatchReader is requested to getOrCreateInitialPartitionOffsets.

KafkaSourceInitialOffsetWriter uses 1 for the version.

Creating KafkaSourceInitialOffsetWriter Instance

KafkaSourceInitialOffsetWriter takes the following to be created:

  • SparkSession

  • Path of the metadata log directory

Deserializing Metadata (Reading Metadata from Serialized Format) — deserialize Method

deserialize(
  in: InputStream): KafkaSourceOffset
Note
deserialize is part of the HDFSMetadataLog Contract to deserialize metadata (reading metadata from a serialized format)

deserialize…​FIXME

results matching ""

    No results matching ""