StreamingExecutionRelation Leaf Logical Operator for Streaming Source At Execution
StreamingExecutionRelation
is a leaf logical operator (i.e. LogicalPlan
) that represents a streaming source in the logical query plan of a streaming Dataset
.
The main use of StreamingExecutionRelation
logical operator is to be a "placeholder" in a logical query plan that will be replaced with the real relation (with new data that has arrived since the last batch) or an empty LocalRelation
when StreamExecution
is requested to transforming logical plan to include the Sources and MicroBatchReaders with new data.
StreamingExecutionRelation
is created for a StreamingRelation in analyzed logical query plan (that is the execution representation of a streaming Dataset).
Note
|
Right after |
Note
|
StreamingExecutionRelation is also resolved (aka planned) to a StreamingRelationExec physical operator in StreamingRelationStrategy execution planning strategy only when explaining a streaming Dataset .
|
Creating StreamingExecutionRelation Instance
StreamingExecutionRelation
takes the following when created:
Creating StreamingExecutionRelation (based on a Source) — apply
Object Method
apply(source: Source): StreamingExecutionRelation
apply
creates a StreamingExecutionRelation
for the input source
and with the attributes of the schema of the source
.
Note
|
apply seems to be used for tests only.
|