ReadSupport Contract — "Readable" Data Sources

ReadSupport is the abstraction of "readable" data sources in the Data Source API V2 that can create a DataSourceReader for reading data (data scan).

ReadSupport defines a single createReader method that creates a DataSourceReader.

DataSourceReader createReader(DataSourceOptions options)
DataSourceReader createReader(StructType schema, DataSourceOptions options)

createReader is used when DataSourceV2Relation leaf logical operator is created (when DataFrameReader is requested to "load" data (as a DataFrame) from a data source with ReadSupport).

// FIXME: Demo
// spark.read.format(...) that is DataSourceV2 and ReadSupport
// DataFrameReader.load() creates a DataFrame with a DataSourceV2Relation operator

Internally, ReadSupport is accessed implicitly when DataSourceV2Relation logical operator is requested to create a DataSourceReader.

Note
 There are no production implementations of the ReadSupport Contract in Spark SQL yet.

results matching ""

    No results matching ""