StateStoreId — Unique Identifier of State Store

StateStoreId is a unique identifier of a state store with the following attributes:

  • Checkpoint Root Location - the root directory for state checkpointing

  • Operator ID - a unique ID of the stateful operator

  • Partition ID - the index of the partition

  • Store Name - the name of the state store (default: default)

StateStoreId is created when:

The name of the default state store (for reading state store data that was generated before store names were used, i.e. in Spark 2.2 and earlier) is default.

State Checkpoint Base Directory of Stateful Operator — storeCheckpointLocation Method

storeCheckpointLocation(): Path

storeCheckpointLocation is Hadoop DFS’s Path of the checkpoint location (for the stateful operator by operator ID, the partition by the partition ID in the checkpoint root location).

If the default store name is used (for Spark 2.2 and earlier), the storeName is not included in the path.

Note
storeCheckpointLocation is used exclusively when HDFSBackedStateStoreProvider is requested for the state checkpoint base directory.

results matching ""

    No results matching ""