StateStoreHandler Internal Contract

StateStoreHandler is the internal base of state store handlers that manage a StateStore (i.e. commit, abortIfNeeded and metrics).

StateStoreHandler takes a single StateStoreType to be created:

Note
StateStoreHandler is a Scala private abstract class and cannot be created directly. It is created indirectly for the concrete StateStoreHandlers.
Table 1. StateStoreHandler Contract
Method Description

stateStore

stateStore: StateStore

When requested for the StateStoreMetrics, StateStoreHandler simply requests the StateStore for them.

Table 2. StateStoreHandlers
StateStoreHandler Description

KeyToNumValuesStore

KeyWithIndexToValueStore

StateStoreHandler takes a single StateStoreType when created.

Note
StateStoreHandler is a Scala abstract class and cannot be created directly. It is created indirectly when the concrete StateStoreHandlers are.
Tip

Enable DEBUG or INFO logging levels for org.apache.spark.sql.execution.streaming.state.SymmetricHashJoinStateManager.StateStoreHandler logger to see what happens inside.

Add the following line to conf/log4j.properties:

log4j.logger.org.apache.spark.sql.execution.streaming.state.SymmetricHashJoinStateManager.StateStoreHandler=DEBUG

Refer to Logging.

Committing Changes to State Store — commit Method

commit(): Unit

commit…​FIXME

Note
commit is used when…​FIXME

abortIfNeeded Method

abortIfNeeded(): Unit

abortIfNeeded…​FIXME

Note
abortIfNeeded is used when…​FIXME

getStateStore Method

getStateStore(keySchema: StructType, valueSchema: StructType): StateStore

getStateStore…​FIXME

Note
getStateStore is used when KeyToNumValuesStore and KeyWithIndexToValueStore are created.

StateStoreType Contract (Sealed Trait)

StateStoreType is required to create a StateStoreHandler.

Table 3. StateStoreTypes
StateStoreType toString Description

KeyToNumValuesType

keyToNumValues

KeyWithIndexToValueType

keyWithIndexToValue

Note
StateStoreType is a Scala private sealed trait which means that all the implementations are in the same compilation unit (a single file).

results matching ""

    No results matching ""