apply[K: Encoder, V: Encoder, S: Encoder, U: Encoder](
func: (Any, Iterator[Any], LogicalGroupState[Any]) => Iterator[Any],
groupingAttributes: Seq[Attribute],
dataAttributes: Seq[Attribute],
outputMode: OutputMode,
isMapGroupsWithState: Boolean,
timeout: GroupStateTimeout,
child: LogicalPlan): LogicalPlan
FlatMapGroupsWithState Unary Logical Operator
FlatMapGroupsWithState
is a unary logical operator that is created to represent the following operators in a logical query plan of a streaming query:
Note
|
A unary logical operator ( Read up on UnaryNode (and logical operators in general) in The Internals of Spark SQL book. |
FlatMapGroupsWithState
is resolved (planned) to:
-
FlatMapGroupsWithStateExec unary physical operator for streaming datasets (in FlatMapGroupsWithStateStrategy execution planning strategy)
-
MapGroupsExec
physical operator for batch datasets (inBasicOperators
execution planning strategy)
Creating SerializeFromObject with FlatMapGroupsWithState — apply
Factory Method
apply
creates a SerializeFromObject
logical operator with a FlatMapGroupsWithState
as its child logical operator.
Internally, apply
creates SerializeFromObject
object consumer (aka unary logical operator) with FlatMapGroupsWithState
logical plan.
Internally, apply
finds ExpressionEncoder
for the type S
and creates a FlatMapGroupsWithState
with UnresolvedDeserializer
for the types K
and V
.
In the end, apply
creates a SerializeFromObject
object consumer with the FlatMapGroupsWithState
.
Note
|
apply is used in KeyValueGroupedDataset.flatMapGroupsWithState operator.
|