StateManagerImplV2 — Default StateManager of FlatMapGroupsWithStateExec Physical Operator

StateManagerImplV2 is a concrete StateManager (as a StateManagerImplBase) that is used by default in FlatMapGroupsWithStateExec physical operator (per spark.sql.streaming.flatMapGroupsWithState.stateFormatVersion internal configuration property).

StateManagerImplV2 is created exclusively when FlatMapGroupsWithStateExecHelper utility is requested for a StateManager (when the stateFormatVersion is 2).

Creating StateManagerImplV2 Instance

StateManagerImplV2 takes the following to be created:

  • State encoder (ExpressionEncoder[Any])

  • shouldStoreTimestamp flag

StateManagerImplV2 initializes the internal properties.

State Schema — stateSchema Value

stateSchema: StructType
Note
 stateSchema is part of the StateManager Contract for the schema of the state.

stateSchema…​FIXME

State Serializer — stateSerializerExprs Value

stateSerializerExprs: Seq[Expression]
Note
 stateSerializerExprs is part of the StateManager Contract for the state serializer, i.e. Catalyst expressions to serialize a state object to a row (UnsafeRow).

stateSerializerExprs…​FIXME

State Deserializer — stateDeserializerExpr Value

stateDeserializerExpr: Expression
Note
 stateDeserializerExpr is part of the StateManager Contract for the state deserializer, i.e. a Catalyst expression to deserialize a state object from a row (UnsafeRow).

stateDeserializerExpr…​FIXME

Internal Properties

Name Description

nestedStateOrdinal

Position of the state in a state row (0)

Used when…​FIXME

timeoutTimestampOrdinalInRow

Position of the timeout timestamp in a state row (1)

Used when…​FIXME

results matching ""

    No results matching ""