DataSourceReader

DataSourceReader is the contract for data source readers with a custom DataReaderFactory.

package org.apache.spark.sql.sources.v2.reader;

interface DataSourceReader {
  StructType readSchema();
  List<DataReaderFactory<Row>> createDataReaderFactories();
}
Note

DataSourceReader is an Evolving contract that is evolving towards becoming a stable API, but is not a stable API yet and can change from one feature release to another release.

In other words, using the contract is as treading on thin ice.

Table 1. DataSourceReader Contract
Method Description

readSchema

Used when…​FIXME

createDataReaderFactories

Used when…​FIXME

Table 2. DataSourceReaders
DataSourceReader Description

ContinuousReader

Used in Spark Structured Streaming

MicroBatchReader

Used in Spark Structured Streaming

SupportsPushDownCatalystFilters

SupportsPushDownFilters

SupportsPushDownRequiredColumns

SupportsReportPartitioning

SupportsReportStatistics

SupportsScanColumnarBatch

SupportsScanUnsafeRow

results matching ""

    No results matching ""