import org.apache.spark.sql.streaming.Trigger import scala.concurrent.duration._ val sq = spark .readStream .format("rate") .load .writeStream .format("console") .option("truncate", false) .trigger(Trigger.Continuous(15.seconds)) // <-- Uses ContinuousExecution for execution .queryName("rate2console") .start assert(sq.isActive) scala> sq.explain == Physical Plan == WriteToContinuousDataSource ConsoleWriter[numRows=20, truncate=false] +- *(1) Project [timestamp#758, value#759L] +- *(1) ScanV2 rate[timestamp#758, value#759L] // sq.stop
Continuous Stream Processing (Structured Streaming V2)
Continuous Stream Processing is a new stream processing model in Spark Structured Streaming (often referred as Structured Streaming V2) that is used for Trigger.Continuous trigger.
Continuous stream processing uses ContinuousExecution as the stream execution engine.
DataStreamReader is requested to create a streaming query for a ContinuousReadSupport data source, it creates…FIXME