OffsetSeq

OffsetSeq is the metadata managed by Hadoop DFS-based metadata storage.

OffsetSeq is created (possibly using the fill factory methods) when:

Creating OffsetSeq Instance

OffsetSeq takes the following when created:

Converting to StreamProgress — toStreamProgress Method

toStreamProgress(
  sources: Seq[BaseStreamingSource]): StreamProgress

toStreamProgress creates a new StreamProgress and adds the streaming sources for which there are new offsets available.

Note
Offsets is a collection with holes (empty elements) for streaming sources with no new data available.

toStreamProgress throws an AssertionError if the number of the input sources does not match the offsets:

There are [[offsets.size]] sources in the checkpoint offsets and now there are [[sources.size]] sources requested by the query. Cannot continue.
Note

toStreamProgress is used when:

Textual Representation — toString Method

toString: String
Note
toString is part of the java.lang.Object contract for the string representation of the object.

toString simply converts the Offsets to JSON (if an offset is available) or - (a dash if an offset is not available for a streaming source at that position).

Creating OffsetSeq Instance — fill Factory Methods

fill(
  offsets: Offset*): OffsetSeq (1)
fill(
  metadata: Option[String],
  offsets: Offset*): OffsetSeq
  1. Uses no metadata (None)

fill simply creates an OffsetSeq for the given variable sequence of Offsets and the optional OffsetSeqMetadata (in JSON format).

Note

fill is used when:

results matching ""

    No results matching ""