KafkaOffsetRangeLimit — Desired Offset Range Limits

KafkaOffsetRangeLimit represents the desired offset range limits for starting, ending, and specific offsets in Kafka Data Source.

Table 1. KafkaOffsetRangeLimits
KafkaOffsetRangeLimit	Description
`EarliestOffsetRangeLimit`	Intent to bind to the earliest offset
`LatestOffsetRangeLimit`	Intent to bind to the latest offset
`SpecificOffsetRangeLimit`	Intent to bind to specific offsets with the following special offset "magic" numbers: `-1` or `KafkaOffsetRangeLimit.LATEST` - the latest offset `-2` or `KafkaOffsetRangeLimit.EARLIEST` - the earliest offset

Note	`KafkaOffsetRangeLimit` is a Scala sealed trait which means that all the implementations are in the same compilation unit (a single file).

KafkaOffsetRangeLimit is often used in a text-based representation and is converted to from latest, earliest or a JSON-formatted text using KafkaSourceProvider.getKafkaOffsetRangeLimit object method.

Note	A JSON-formatted text is of the following format `{"topicName":{"partition":offset},…}`, e.g. `{"topicA":{"0":23,"1":-1},"topicB":{"0":-2}}`.

KafkaOffsetRangeLimit is used when:

KafkaContinuousReader is created (with the initial offsets)
KafkaMicroBatchReader is created (with the starting offsets)
KafkaRelation is created (with the starting and ending offsets)
KafkaSource is created (with the starting offsets)
KafkaSourceProvider is requested to convert configuration options to KafkaOffsetRangeLimits

KafkaOffsetRangeLimit

KafkaOffsetRangeLimit — Desired Offset Range Limits

results matching ""

No results matching ""