StateManagerImplV2 — Default StateManager of FlatMapGroupsWithStateExec Physical Operator

StateManagerImplV2 is a concrete StateManager (as a StateManagerImplBase) that is used by default in FlatMapGroupsWithStateExec physical operator (per spark.sql.streaming.flatMapGroupsWithState.stateFormatVersion internal configuration property).

StateManagerImplV2 is created exclusively when FlatMapGroupsWithStateExecHelper utility is requested for a StateManager (when the stateFormatVersion is 2).

Creating StateManagerImplV2 Instance

StateManagerImplV2 takes the following to be created:

  • State encoder (ExpressionEncoder[Any])

  • shouldStoreTimestamp flag

StateManagerImplV2 initializes the internal properties.

State Schema — stateSchema Value

stateSchema: StructType
stateSchema is part of the StateManager Contract for the schema of the state.


State Serializer — stateSerializerExprs Value

stateSerializerExprs: Seq[Expression]
stateSerializerExprs is part of the StateManager Contract for the state serializer, i.e. Catalyst expressions to serialize a state object to a row (UnsafeRow).


State Deserializer — stateDeserializerExpr Value

stateDeserializerExpr: Expression
stateDeserializerExpr is part of the StateManager Contract for the state deserializer, i.e. a Catalyst expression to deserialize a state object from a row (UnsafeRow).


Internal Properties

Name Description


Position of the state in a state row (0)

Used when…​FIXME


Position of the timeout timestamp in a state row (1)

Used when…​FIXME

results matching ""

    No results matching ""