KafkaOffsetRangeLimit — Desired Offset Range Limits
KafkaOffsetRangeLimit
represents the desired offset range limits for starting, ending, and specific offsets in Kafka Data Source.
KafkaOffsetRangeLimit | Description |
---|---|
|
|
|
|
|
Note
|
KafkaOffsetRangeLimit is a Scala sealed trait which means that all the implementations are in the same compilation unit (a single file).
|
KafkaOffsetRangeLimit
is often used in a text-based representation and is converted to from latest, earliest or a JSON-formatted text using KafkaSourceProvider.getKafkaOffsetRangeLimit object method.
Note
|
A JSON-formatted text is of the following format {"topicName":{"partition":offset},…} , e.g. {"topicA":{"0":23,"1":-1},"topicB":{"0":-2}} .
|
KafkaOffsetRangeLimit
is used when:
-
KafkaContinuousReader is created (with the initial offsets)
-
KafkaMicroBatchReader is created (with the starting offsets)
-
KafkaRelation is created (with the starting and ending offsets)
-
KafkaSource is created (with the starting offsets)
-
KafkaSourceProvider
is requested to convert configuration options to KafkaOffsetRangeLimits