StateStoreWriter Contract — Stateful Physical Operators That Write to StateStore

StateStoreWriter is the extension of the StatefulOperator Contract for physical operators that write to a state store and collect the write metrics for execution progress reporting.

Table 1. StateStoreWriter’s Performance Metrics
Key Name (in web UI) Description

numOutputRows

number of output rows

numTotalStateRows

number of total state rows

numUpdatedStateRows

number of updated state rows

allUpdatesTimeMs

total time to update rows

allRemovalsTimeMs

total time to remove rows

commitTimeMs

time to commit changes

stateMemory

memory used by state

Table 2. StateStoreWriters
StateStoreWriter Description

FlatMapGroupsWithStateExec

StateStoreSaveExec

StreamingDeduplicateExec

StreamingGlobalLimitExec

StreamingSymmetricHashJoinExec

Setting StateStore-Specific Metrics for Stateful Physical Operator — setStoreMetrics Method

setStoreMetrics(store: StateStore): Unit

setStoreMetrics requests the specified StateStore for the metrics and records the following metrics of a physical operator:

setStoreMetrics records the custom metrics.

Note

setStoreMetrics is used when the following physical operators are executed:

getProgress Method

getProgress(): StateOperatorProgress

getProgress…​FIXME

Note
getProgress is used exclusively when ProgressReporter is requested to extractStateOperatorMetrics (when MicroBatchExecution is requested to run the activated streaming query).

Checking Out Whether Last Batch Execution Requires Another Non-Data Batch or Not — shouldRunAnotherBatch Method

shouldRunAnotherBatch(newMetadata: OffsetSeqMetadata): Boolean

shouldRunAnotherBatch is negative (false) by default (to indicate that another non-data batch is not required given the OffsetSeqMetadata with the event-time watermark and the batch timestamp).

Note
shouldRunAnotherBatch is used exclusively when IncrementalExecution is requested to check out whether the last batch execution requires another batch (when MicroBatchExecution is requested to run the activated streaming query).

stateStoreCustomMetrics Internal Method

stateStoreCustomMetrics: Map[String, SQLMetric]

stateStoreCustomMetrics…​FIXME

Note
stateStoreCustomMetrics is used when StateStoreWriter is requested for the metrics and getProgress.

timeTakenMs Method

timeTakenMs(body: => Unit): Long

timeTakenMs…​FIXME

Note
timeTakenMs is used when…​FIXME

results matching ""

    No results matching ""