Continuous Stream Processing (Structured Streaming V2)

Continuous Stream Processing is a new stream processing model in Spark Structured Streaming (often referred as Structured Streaming V2) that is used for Trigger.Continuous trigger.

Continuous stream processing uses ContinuousExecution as the stream execution engine.

import org.apache.spark.sql.streaming.Trigger
import scala.concurrent.duration._
val sq = spark
  .readStream
  .format("rate")
  .load
  .writeStream
  .format("console")
  .option("truncate", false)
  .trigger(Trigger.Continuous(15.seconds)) // <-- Uses ContinuousExecution for execution
  .queryName("rate2console")
  .start

assert(sq.isActive)

scala> sq.explain
== Physical Plan ==
WriteToContinuousDataSource ConsoleWriter[numRows=20, truncate=false]
+- *(1) Project [timestamp#758, value#759L]
   +- *(1) ScanV2 rate[timestamp#758, value#759L]

// sq.stop

When DataStreamReader is requested to create a streaming query for a ContinuousReadSupport data source, it creates…​FIXME

results matching ""

    No results matching ""