OffsetSeq is the metadata managed by Hadoop DFS-based metadata storage.

OffsetSeq is created (possibly using the fill factory methods) when:

Creating OffsetSeq Instance

OffsetSeq takes the following when created:

Converting to StreamProgress — toStreamProgress Method

  sources: Seq[BaseStreamingSource]): StreamProgress

toStreamProgress creates a new StreamProgress and adds the streaming sources for which there are new offsets available.

Offsets is a collection with holes (empty elements) for streaming sources with no new data available.

toStreamProgress throws an AssertionError if the number of the input sources does not match the offsets:

There are [[offsets.size]] sources in the checkpoint offsets and now there are [[sources.size]] sources requested by the query. Cannot continue.

toStreamProgress is used when:

Textual Representation — toString Method

toString: String
toString is part of the java.lang.Object contract for the string representation of the object.

toString simply converts the Offsets to JSON (if an offset is available) or - (a dash if an offset is not available for a streaming source at that position).

Creating OffsetSeq Instance — fill Factory Methods

  offsets: Offset*): OffsetSeq (1)
  metadata: Option[String],
  offsets: Offset*): OffsetSeq
  1. Uses no metadata (None)

fill simply creates an OffsetSeq for the given variable sequence of Offsets and the optional OffsetSeqMetadata (in JSON format).


fill is used when:

results matching ""

    No results matching ""