DataSourceV2Utils Helper Object

DataSourceV2Utils is a helper object that is used exclusively to extract session configuration options (i.e. options with spark.datasource prefix for the keys in SQLConf) for DataSourceV2 data sources with SessionConfigSupport in Data Source API V2.

extractSessionConfigs(
  ds: DataSourceV2,
  conf: SQLConf): Map[String, String]

Note	`extractSessionConfigs` supports data sources with SessionConfigSupport only.

extractSessionConfigs requests the SessionConfigSupport data source for the custom key prefix for configuration options that is used to find all configuration options with the keys in the format of spark.datasource.[keyPrefix] in the given SQLConf.

extractSessionConfigs returns the matching keys with the spark.datasource.[keyPrefix] prefix removed (i.e. spark.datasource.keyPrefix.k1 becomes k1).

Note

extractSessionConfigs is used when:

DataFrameReader is requested to "load" data as a DataFrame (for a DataSourceV2 with ReadSupport)
DataFrameWriter is requested to saves a DataFrame to a data source (for a DataSourceV2 with WriteSupport)
Spark Structured Streaming’s DataStreamReader is requested to load input data stream (for data sources with MicroBatchReadSupport or ContinuousReadSupport)
Spark Structured Streaming’s DataStreamWriter is requested to start a streaming query (with data sources with StreamWriteSupport)

DataSourceV2Utils Helper Object

DataSourceV2Utils Helper Object

results matching ""

No results matching ""