StateStoreWriter Contract — Stateful Physical Operators That Write to State Store

StateStoreWriter is the extension of the StatefulOperator Contract for physical operators that write to a state store and collect the write metrics for execution progress reporting.

Table 1. StateStoreWriters
StateStoreWriter	Description
FlatMapGroupsWithStateExec
StateStoreSaveExec
StreamingDeduplicateExec
StreamingGlobalLimitExec
StreamingSymmetricHashJoinExec

Performance Metrics (SQLMetrics)

Name (in web UI)	Description
number of output rows
number of total state rows
number of updated state rows
total time to update rows
total time to remove rows
time to commit changes
memory used by state

Name (in web UI)

Description

number of output rows

number of total state rows

number of updated state rows

total time to update rows

total time to remove rows

time to commit changes

memory used by state

Setting StateStore-Specific Metrics for Stateful Physical Operator — `setStoreMetrics` Method

setStoreMetrics(store: StateStore): Unit

setStoreMetrics requests the specified StateStore for the metrics and records the following metrics of a physical operator:

numTotalStateRows as the number of keys
stateMemory as the memory used (in bytes)

setStoreMetrics records the custom metrics.

Note	`setStoreMetrics` is used when the following physical operators are executed: FlatMapGroupsWithStateExec StateStoreSaveExec StreamingDeduplicateExec StreamingGlobalLimitExec

`getProgress` Method

getProgress(): StateOperatorProgress

getProgress…FIXME

Note	`getProgress` is used exclusively when `ProgressReporter` is requested to extractStateOperatorMetrics (when `MicroBatchExecution` is requested to run the activated streaming query).

Checking Out Whether Last Batch Execution Requires Another Non-Data Batch or Not — `shouldRunAnotherBatch` Method

shouldRunAnotherBatch(newMetadata: OffsetSeqMetadata): Boolean

shouldRunAnotherBatch is negative (false) by default (to indicate that another non-data batch is not required given the OffsetSeqMetadata with the event-time watermark and the batch timestamp).

Note	`shouldRunAnotherBatch` is used exclusively when `IncrementalExecution` is requested to check out whether the last batch execution requires another batch (when `MicroBatchExecution` is requested to run the activated streaming query).

`stateStoreCustomMetrics` Internal Method

stateStoreCustomMetrics: Map[String, SQLMetric]

stateStoreCustomMetrics…FIXME

Note	`stateStoreCustomMetrics` is used when `StateStoreWriter` is requested for the metrics and getProgress.

`timeTakenMs` Method

timeTakenMs(body: => Unit): Long

timeTakenMs…FIXME

Note	`timeTakenMs` is used when…FIXME

StateStoreWriter