def stateSnapshots(): DStream[(KeyType, StateType)]
MapWithStateDStream
MapWithStateDStream
is the result of mapWithState stateful operator.
It extends DStream Contract with the following additional method:
Note
|
MapWithStateDStream is a Scala sealed abstract class (and hence all the available implementations are in the source file).
|
Note
|
MapWithStateDStreamImpl is the only implementation of MapWithStateDStream (see below in this document for more coverage).
|
MapWithStateDStreamImpl
MapWithStateDStreamImpl
is an internal DStream with dependency on the parent dataStream
key-value dstream. It uses a custom internal dstream called internalStream
(of type InternalMapWithStateDStream).
slideDuration
is exactly the slide duration of the internal stream internalStream
.
dependencies
returns a single-element collection with the internal stream internalStream
.
The compute
method may or may not return a RDD[MappedType]
by getOrCompute
on the internal stream and…TK
Caution
|
FIXME |
InternalMapWithStateDStream
InternalMapWithStateDStream
is an internal dstream to support MapWithStateDStreamImpl and uses dataStream
(as parent
of type DStream[(K, V)]
) as well as StateSpecImpl[K, V, S, E]
(as spec
).
InternalMapWithStateDStream
is a DStream[MapWithStateRDDRecord[K, S, E]]
that uses MEMORY_ONLY
storage level by default.
InternalMapWithStateDStream
uses the StateSpec
's partitioner or HashPartitioner (with SparkContext’s defaultParallelism).
slideDuration
is the slide duration of parent
.
dependencies
is a single-element collection with the parent
stream.
It forces checkpointing (i.e. mustCheckpoint
flag is enabled).
When initialized, if checkpoint interval is not set, it sets it as ten times longer than the slide duration of the parent
stream (the multiplier is not configurable and always 10
).
Computing a RDD[MapWithStateRDDRecord[K, S, E]]
(i.e. compute
method) first looks up a previous RDD for the last slideDuration
.
If the RDD is found, it is returned as is given the partitioners of the RDD and the stream are equal. Otherwise, when the partitioners are different, the RDD is "repartitioned" using MapWithStateRDD.createFromRDD
.
Caution
|
FIXME MapWithStateRDD.createFromRDD
|