RelationProvider Contract — Relation Providers with Pre-Defined Schema

RelationProvider is an abstraction of providers of relational data sources with schema inference (aka schema discovery). In other words, the schema is pre-defined, and a user-specified schema is not allowed.

Table 1. RelationProvider Contract
Method Description

createRelation

 
createRelation(
  sqlContext: SQLContext,
  parameters: Map[String, String]): BaseRelation

Creates a BaseRelation for loading data from an external data source

Used exclusively when DataSource is requested to resolve a data source for a given data source format

When resolving a data source, DataSource makes sure that a schema is not defined or matches the schema of the data source. DataSource throws an AnalysisException for a user-specified schema that does not match the data source’s schema:

[className] does not allow user-specified schemas.
Tip
 Use SchemaRelationProvider for data source providers that require a user-defined schema.
Note
 It is a common pattern while developing a custom data source to use RelationProvider.createRelation with CreatableRelationProvider when requested for a relation (after writing out a structured query).
Table 2. RelationProviders
RelationProvider Description

JdbcRelationProvider

Data source provider for JDBC data source

KafkaSourceProvider

Data source provider for Kafka data source

results matching ""

    No results matching ""