storeCheckpointLocation(): PathStateStoreId — Unique Identifier of State Store
StateStoreId is a unique identifier of a state store with the following attributes:
-
Checkpoint Root Location - the root directory for state checkpointing
-
Store Name - the name of the state store (default: default)
StateStoreId is created when:
-
StateStoreRDDis requested for the preferred locations of a partition (executed on the driver) and to compute it (later on an executor) -
StateStoreProviderIdhelper object is requested to create a StateStoreProviderId (with a StateStoreId and the run ID of a streaming query) that is then used for the preferred locations of a partition of aStateStoreAwareZipPartitionsRDD(executed on the driver) and to…FIXME
The name of the default state store (for reading state store data that was generated before store names were used, i.e. in Spark 2.2 and earlier) is default.
State Checkpoint Base Directory of Stateful Operator — storeCheckpointLocation Method
storeCheckpointLocation is Hadoop DFS’s Path of the checkpoint location (for the stateful operator by operator ID, the partition by the partition ID in the checkpoint root location).
If the default store name is used (for Spark 2.2 and earlier), the storeName is not included in the path.
|
Note
|
storeCheckpointLocation is used exclusively when HDFSBackedStateStoreProvider is requested for the state checkpoint base directory.
|