scala> :type sq
org.apache.spark.sql.DataFrame
import org.apache.spark.sql.execution.streaming.StreamingRelationV2
val relation = sq.queryExecution.logical.asInstanceOf[StreamingRelationV2]
assert(relation.isStreaming)
StreamingRelationV2 Leaf Logical Operator
StreamingRelationV2
is a MultiInstanceRelation
leaf logical operator that represents MicroBatchReadSupport or ContinuousReadSupport streaming data sources in a logical plan of a streaming query.
Tip
|
Read up on Leaf logical operators in The Internals of Spark SQL book. |
StreamingRelationV2
is created when:
-
DataStreamReader
is requested to "load" data as a streaming DataFrame for MicroBatchReadSupport and ContinuousReadSupport streaming data sources -
ContinuousMemoryStream is created
isStreaming
flag is always enabled (i.e. true
).
StreamingRelationV2
is resolved (replaced) to the following leaf logical operators:
-
ContinuousExecutionRelation when
ContinuousExecution
stream execution engine is requested for the analyzed logical plan -
StreamingExecutionRelation when
MicroBatchExecution
stream execution engine is requested for the analyzed logical plan
Creating StreamingRelationV2 Instance
StreamingRelationV2
takes the following to be created:
-
Optional StreamingRelation