toStreamProgress(
sources: Seq[BaseStreamingSource]): StreamProgressOffsetSeq
OffsetSeq is the metadata managed by Hadoop DFS-based metadata storage.
-
OffsetSeqLogis requested to deserialize metadata (retrieve metadata from a persistent storage) -
StreamProgressis requested to convert itself to OffsetSeq (most importantly whenMicroBatchExecutionstream execution engine is requested to construct the next streaming micro-batch to commit available offsets for a batch to the write-ahead log) -
ContinuousExecutionstream execution engine is requested to get start offsets and addOffset
Creating OffsetSeq Instance
OffsetSeq takes the following when created:
-
Collection of optional Offsets (with
Nonefor streaming sources with no new data available) -
Optional OffsetSeqMetadata (default:
None)
Converting to StreamProgress — toStreamProgress Method
toStreamProgress creates a new StreamProgress and adds the streaming sources for which there are new offsets available.
|
Note
|
Offsets is a collection with holes (empty elements) for streaming sources with no new data available. |
toStreamProgress throws an AssertionError if the number of the input sources does not match the offsets:
There are [[offsets.size]] sources in the checkpoint offsets and now there are [[sources.size]] sources requested by the query. Cannot continue.|
Note
|
|
Textual Representation — toString Method
toString: String|
Note
|
toString is part of the java.lang.Object contract for the string representation of the object.
|
toString simply converts the Offsets to JSON (if an offset is available) or - (a dash if an offset is not available for a streaming source at that position).
Creating OffsetSeq Instance — fill Factory Methods
fill(
offsets: Offset*): OffsetSeq (1)
fill(
metadata: Option[String],
offsets: Offset*): OffsetSeq-
Uses no metadata (
None)
fill simply creates an OffsetSeq for the given variable sequence of Offsets and the optional OffsetSeqMetadata (in JSON format).
|
Note
|
|