StreamingRelationV2 Leaf Logical Operator

StreamingRelationV2 is a MultiInstanceRelation leaf logical operator that represents MicroBatchReadSupport or ContinuousReadSupport streaming data sources in a logical plan of a streaming query.

StreamingRelationV2 is created when:

isStreaming flag is always enabled (i.e. true).

scala> :type sq
org.apache.spark.sql.DataFrame

import org.apache.spark.sql.execution.streaming.StreamingRelationV2
val relation = sq.queryExecution.logical.asInstanceOf[StreamingRelationV2]
assert(relation.isStreaming)

StreamingRelationV2 is resolved (replaced) to the following leaf logical operators:

Creating StreamingRelationV2 Instance

StreamingRelationV2 takes the following to be created:

  • DataSourceV2

  • Name of the data source

  • Options (Map[String, String])

  • Output attributes (Seq[Attribute])

  • Optional StreamingRelation

  • SparkSession

results matching ""

    No results matching ""