StateManager Contract — State Managers for Arbitrary Stateful Streaming Aggregation

Method Description

getAllState

getAllState(store: StateStore): Iterator[StateData]

Retrieves all state data (for all keys) from the StateStore

Used exclusively when InputProcessor is requested to processTimedOutState

getState

getState(
  store: StateStore,
  keyRow: UnsafeRow): StateData

Gets the state data for the key from the StateStore

Used exclusively when InputProcessor is requested to processNewData

putState

putState(
  store: StateStore,
  keyRow: UnsafeRow,
  state: Any,
  timeoutTimestamp: Long): Unit

Persists (puts) the state value for the key in the StateStore

removeState

removeState(
  store: StateStore,
  keyRow: UnsafeRow): Unit

Removes the state for the key from the StateStore

stateSchema

stateSchema: StructType

State schema

Note	It looks like (in StateManager of the FlatMapGroupsWithStateExec physical operator) `stateSchema` is used for the schema of state value objects (not state keys as they are described by the grouping attributes instead).

Used when:

FlatMapGroupsWithStateExec physical operator is requested to execute and generate a recipe for a distributed computation (as an RDD[InternalRow])
StateManagerImplBase is requested for the stateDeserializerFunc

Note	StateManagerImplBase is the one and only known direct implementation of the StateManager Contract in Spark Structured Streaming.

Note	`StateManager` is a Scala sealed trait which means that all the implementations are in the same compilation unit (a single file).

StateManager