GroupStateImpl

GroupStateImpl is the default and only known GroupState in Spark Structured Streaming.

GroupStateImpl holds per-group state value of type S per group key.

GroupStateImpl is created when GroupStateImpl helper object is requested for the following:

Creating GroupStateImpl Instance

GroupStateImpl takes the following to be created:

GroupStateImpl initializes the internal properties.

Creating GroupStateImpl for Streaming Query — createForStreaming Factory Method

createForStreaming[S](
  optionalValue: Option[S],
  batchProcessingTimeMs: Long,
  eventTimeWatermarkMs: Long,
  timeoutConf: GroupStateTimeout,
  hasTimedOut: Boolean,
  watermarkPresent: Boolean): GroupStateImpl[S]

createForStreaming simply creates a new GroupStateImpl with the given input arguments.

Note
 createForStreaming is used exclusively when InputProcessor is requested to callFunctionAndUpdateState (when InputProcessor is requested to processNewData and processTimedOutState).

Creating GroupStateImpl for Batch Query — createForBatch Factory Method

createForBatch(
  timeoutConf: GroupStateTimeout,
  watermarkPresent: Boolean): GroupStateImpl[Any]

createForBatch…​FIXME

Note
 createForBatch is used when…​FIXME

Textual Representation — toString Method

toString: String
Note
 toString is part of the java.lang.Object contract for the string representation of the object.

toString…​FIXME

Specifying Timeout Duration for ProcessingTimeTimeout — setTimeoutDuration Method

setTimeoutDuration(durationMs: Long): Unit
Note
 setTimeoutDuration is part of the GroupState Contract to specify timeout duration for the state key (in millis or as a string).

setTimeoutDuration…​FIXME

Specifying Timeout Timestamp for EventTimeTimeout — setTimeoutTimestamp Method

setTimeoutTimestamp(durationMs: Long): Unit
Note
 setTimeoutTimestamp is part of the GroupState Contract to specify timeout timestamp for the state key.

setTimeoutTimestamp…​FIXME

Getting Processing Time — getCurrentProcessingTimeMs Method

getCurrentProcessingTimeMs(): Long
Note
 getCurrentProcessingTimeMs is part of the GroupState Contract to get the current processing time (as milliseconds in epoch time).

getCurrentProcessingTimeMs simply returns the batchProcessingTimeMs.

Updating State — update Method

update(newValue: S): Unit
Note
 update is part of the GroupState Contract to update the state.

update…​FIXME

Removing State — remove Method

remove(): Unit
Note
 remove is part of the GroupState Contract to remove the state.

remove…​FIXME

Internal Properties

Name Description

value

FIXME

Used when…​FIXME

defined

FIXME

Used when…​FIXME

updated

Updated flag that says whether the state has been updated or not

Default: false

Disabled (false) when GroupStateImpl is requested to remove the state

Enabled (true) when GroupStateImpl is requested to update the state

removed

Removed flag that says whether the state is marked removed or not

Default: false

Disabled (false) when GroupStateImpl is requested to update the state

Enabled (true) when GroupStateImpl is requested to remove the state

timeoutTimestamp

Current timeout timestamp (in millis) for GroupStateTimeout.EventTimeTimeout or GroupStateTimeout.ProcessingTimeTimeout

Default: -1

Defined using setTimeoutTimestamp (for EventTimeTimeout) and setTimeoutDuration (for ProcessingTimeTimeout)

results matching ""

    No results matching ""