OffsetSeqMetadata

OffsetSeqMetadata holds the metadata for…​FIXME:

  • batchWatermarkMs - the current event time watermark in milliseconds (used to bound the lateness of data that will processed)

  • batchTimestampMs - the current batch processing timestamp in milliseconds

  • conf - batch configuration with spark.sql.shuffle.partitions and spark.sql.streaming.stateStore.providerClass Spark properties

Note
OffsetSeqMetadata is used mainly when IncrementalExecution is created.

OffsetSeqMetadata defines the following relevantSQLConfs:

relevantSQLConfs are used when OffsetSeqMetadata is created and is requested to setSessionConf.

Creating OffsetSeqMetadata — apply Factory Method

apply(
  batchWatermarkMs: Long,
  batchTimestampMs: Long,
  sessionConf: RuntimeConfig): OffsetSeqMetadata

apply…​FIXME

Note
apply is used when…​FIXME

setSessionConf Method

setSessionConf(metadata: OffsetSeqMetadata, sessionConf: RuntimeConfig): Unit

setSessionConf…​FIXME

Note
setSessionConf is used when…​FIXME

results matching ""

    No results matching ""