import org.apache.spark.sql.streaming.Trigger
val q = spark
.readStream
.format("rate")
.load
.writeStream
.format("console") // <-- requests ConsoleSinkProvider for a sink
.trigger(Trigger.Once)
.start
scala> println(q.lastProgress.sink)
{
"description" : "org.apache.spark.sql.execution.streaming.ConsoleSinkProvider@2392cfb1"
}
ConsoleSinkProvider
ConsoleSinkProvider
is a DataSourceV2
with StreamWriteSupport for console data source format.
Tip
|
Read up on DataSourceV2 Contract in The Internals of Spark SQL book. |
ConsoleSinkProvider
is a DataSourceRegister and registers itself as the console data source format.
When requested for a StreamWriter, ConsoleSinkProvider
simply creates a ConsoleWriter (with the given schema and options).
ConsoleSinkProvider
is a CreatableRelationProvider.
Tip
|
Read up on CreatableRelationProvider in The Internals of Spark SQL book. |
createRelation
Method
createRelation(
sqlContext: SQLContext,
mode: SaveMode,
parameters: Map[String, String],
data: DataFrame): BaseRelation
Note
|
createRelation is part of the CreatableRelationProvider Contract to support writing a structured query (a DataFrame) per save mode.
|
createRelation
…FIXME