RelationProvider Contract — Relation Providers With Schema Inference

RelationProvider is an abstraction of data source providers of relational data sources with schema inference (aka schema discovery).

Table 1. RelationProvider Contract
Method Description

createRelation

createRelation(
  sqlContext: SQLContext,
  parameters: Map[String, String]): BaseRelation

Creates a BaseRelation for loading data from an external data source

Used exclusively when DataSource is requested to resolve a data source for a given data source format

When resolving a data source, DataSource makes sure that a schema is not defined or matches the schema of the data source. DataSource throws an AnalysisException for a user-specified schema that does not match the data source’s schema:

[className] does not allow user-specified schemas.
Tip
Use SchemaRelationProvider for data source providers that require a user-defined schema.
Note
It is a common pattern while developing a custom data source to use RelationProvider.createRelation with CreatableRelationProvider when requested for a relation (after writing out a structured query).
Table 2. RelationProviders
RelationProvider Description

JdbcRelationProvider

Data source provider for JDBC data source

KafkaSourceProvider

Data source provider for Kafka data source

results matching ""

    No results matching ""