ConsoleSinkProvider

ConsoleSinkProvider is a DataSourceV2 with StreamWriteSupport for console data source format.

Tip
 Read up on DataSourceV2 Contract in The Internals of Spark SQL book.

ConsoleSinkProvider is a DataSourceRegister and registers itself as the console data source format.

import org.apache.spark.sql.streaming.Trigger
val q = spark
  .readStream
  .format("rate")
  .load
  .writeStream
  .format("console") // <-- requests ConsoleSinkProvider for a sink
  .trigger(Trigger.Once)
  .start
scala> println(q.lastProgress.sink)
{
  "description" : "org.apache.spark.sql.execution.streaming.ConsoleSinkProvider@2392cfb1"
}

When requested for a StreamWriter, ConsoleSinkProvider simply creates a ConsoleWriter (with the given schema and options).

ConsoleSinkProvider is a CreatableRelationProvider.

Tip
 Read up on CreatableRelationProvider in The Internals of Spark SQL book.

createRelation Method

createRelation(
  sqlContext: SQLContext,
  mode: SaveMode,
  parameters: Map[String, String],
  data: DataFrame): BaseRelation
Note
 createRelation is part of the CreatableRelationProvider Contract to support writing a structured query (a DataFrame) per save mode.

createRelation…​FIXME

results matching ""

    No results matching ""