RelationProvider Contract — Relation Providers with Pre-Defined Schema

RelationProvider is an abstraction of providers of relational data sources with schema inference (aka schema discovery). In other words, the schema is pre-defined, and a user-specified schema is not allowed.

Table 1. RelationProvider Contract
Method Description


  sqlContext: SQLContext,
  parameters: Map[String, String]): BaseRelation

Creates a BaseRelation for loading data from an external data source

Used exclusively when DataSource is requested to resolve a data source for a given data source format

When resolving a data source, DataSource makes sure that a schema is not defined or matches the schema of the data source. DataSource throws an AnalysisException for a user-specified schema that does not match the data source’s schema:

[className] does not allow user-specified schemas.
Use SchemaRelationProvider for data source providers that require a user-defined schema.
It is a common pattern while developing a custom data source to use RelationProvider.createRelation with CreatableRelationProvider when requested for a relation (after writing out a structured query).
Table 2. RelationProviders
RelationProvider Description


Data source provider for JDBC data source


Data source provider for Kafka data source

results matching ""

    No results matching ""