StateStoreProvider Contract — State Store Providers

StateStoreProvider is the abstraction of state store providers that manage state stores in Stateful Stream Processing (e.g. for persisting running aggregates in Streaming Aggregation) in stateful streaming queries.

Note
 StateStoreProvider utility uses spark.sql.streaming.stateStore.providerClass internal configuration property for the name of the class of the default StateStoreProvider implementation.
Note
 HDFSBackedStateStoreProvider is the default and only known StateStoreProvider in Spark Structured Streaming.
Table 1. StateStoreProvider Contract
Method Description

close

 
close(): Unit

Closes the state store provider

Used exclusively when StateStore helper object is requested to unload a state store provider

doMaintenance

 
doMaintenance(): Unit = {}

Optional state maintenance

Used exclusively when StateStore utility is requested to perform maintenance of registered state store providers (on a separate MaintenanceTask daemon thread)

getStore

 
getStore(
  version: Long): StateStore

Finds the StateStore for the specified version

Used exclusively when StateStore utility is requested to look up the StateStore by a given provider ID

init

 
init(
  stateStoreId: StateStoreId,
  keySchema: StructType,
  valueSchema: StructType,
  keyIndexOrdinal: Option[Int],
  storeConfs: StateStoreConf,
  hadoopConf: Configuration): Unit

Initializes the state store provider

Used exclusively when StateStoreProvider helper object is requested to create and initialize the StateStoreProvider for a given StateStoreId (when StateStore helper object is requested to retrieve a StateStore by ID and version)

stateStoreId

 
stateStoreId: StateStoreId

StateStoreId associated with the provider (at initialization)

Used when:

supportedCustomMetrics

 
supportedCustomMetrics: Seq[StateStoreCustomMetric]

StateStoreCustomMetrics of the state store provider

Used when:

Creating and Initializing StateStoreProvider — createAndInit Object Method

createAndInit(
  stateStoreId: StateStoreId,
  keySchema: StructType,
  valueSchema: StructType,
  indexOrdinal: Option[Int],
  storeConf: StateStoreConf,
  hadoopConf: Configuration): StateStoreProvider

createAndInit creates a new StateStoreProvider (per spark.sql.streaming.stateStore.providerClass internal configuration property).

createAndInit requests the StateStoreProvider to initialize.

Note
 createAndInit is used exclusively when StateStore utility is requested for the StateStore by given provider ID and version.

results matching ""

    No results matching ""