scala> :type sq
org.apache.spark.sql.DataFrame
import org.apache.spark.sql.execution.streaming.StreamingRelationV2
val relation = sq.queryExecution.logical.asInstanceOf[StreamingRelationV2]
assert(relation.isStreaming)StreamingRelationV2 Leaf Logical Operator
StreamingRelationV2 is a MultiInstanceRelation leaf logical operator that represents MicroBatchReadSupport or ContinuousReadSupport streaming data sources in a logical plan of a streaming query.
|
Tip
|
Read up on Leaf logical operators in The Internals of Spark SQL book. |
StreamingRelationV2 is created when:
-
DataStreamReaderis requested to "load" data as a streaming DataFrame for MicroBatchReadSupport and ContinuousReadSupport streaming data sources -
ContinuousMemoryStream is created
isStreaming flag is always enabled (i.e. true).
StreamingRelationV2 is resolved (replaced) to the following leaf logical operators:
-
ContinuousExecutionRelation when
ContinuousExecutionstream execution engine is requested for the analyzed logical plan -
StreamingExecutionRelation when
MicroBatchExecutionstream execution engine is requested for the analyzed logical plan
Creating StreamingRelationV2 Instance
StreamingRelationV2 takes the following to be created:
-
Optional StreamingRelation