def stateSnapshots(): DStream[(KeyType, StateType)]
MapWithStateDStream is the result of mapWithState stateful operator.
It extends DStream Contract with the following additional method:
MapWithStateDStreamImpl is the only implementation of
MapWithStateDStreamImpl is an internal DStream with dependency on the parent
dataStream key-value dstream. It uses a custom internal dstream called
internalStream (of type InternalMapWithStateDStream).
slideDuration is exactly the slide duration of the internal stream
dependencies returns a single-element collection with the internal stream
compute method may or may not return a
getOrCompute on the internal stream and…TK
InternalMapWithStateDStream is an internal dstream to support MapWithStateDStreamImpl and uses
parent of type
DStream[(K, V)]) as well as
StateSpecImpl[K, V, S, E] (as
InternalMapWithStateDStream is a
DStream[MapWithStateRDDRecord[K, S, E]] that uses
MEMORY_ONLY storage level by default.
slideDuration is the slide duration of
dependencies is a single-element collection with the
It forces checkpointing (i.e.
mustCheckpoint flag is enabled).
When initialized, if checkpoint interval is not set, it sets it as ten times longer than the slide duration of the
parent stream (the multiplier is not configurable and always
RDD[MapWithStateRDDRecord[K, S, E]] (i.e.
compute method) first looks up a previous RDD for the last
If the RDD is found, it is returned as is given the partitioners of the RDD and the stream are equal. Otherwise, when the partitioners are different, the RDD is "repartitioned" using