StreamSourceProvider — Streaming Data Source Provider

StreamSourceProvider is the contract of data source providers that can create a streaming data source for a format (e.g. text file) or system (e.g. Apache Kafka).

Table 1. StreamSourceProvider Contract
Method Description

createSource

createSource(
  sqlContext: SQLContext,
  metadataPath: String,
  schema: Option[StructType],
  providerName: String,
  parameters: Map[String, String]): Source

Creates a streaming source for a format or system (to continually read data)

Note
metadataPath is the value of the optional user-specified checkpointLocation option or resolved by StreamingQueryManager.

Used exclusively when DataSource is requested to create a streaming source (when MicroBatchExecution is requested to initialize the analyzed logical plan)

sourceSchema

sourceSchema(
  sqlContext: SQLContext,
  schema: Option[StructType],
  providerName: String,
  parameters: Map[String, String]): (String, StructType)

Describes a streaming source with a name and the schema

Used exclusively when DataSource is requested to describe the name and the schema of a streaming source (when MicroBatchExecution is requested to initialize the analyzed logical plan)

Note
StreamSourceProvider is an experimental contract.
Note
KafkaSourceProvider is the default implementation of the StreamSourceProvider Contract in Spark Structured Streaming.

results matching ""

    No results matching ""