RelationProvider Contract — Relation Providers With Schema Inference

Note
Schema inference is also called schema discovery.

The requirement of not specifying a user-defined schema or having one that does not match the relation is enforced when DataSource is requested for a BaseRelation for a given data source format. If specified and does not match, DataSource throws a AnalysisException:

[className] does not allow user-specified schemas.
package org.apache.spark.sql.sources

trait RelationProvider {
  def createRelation(
    sqlContext: SQLContext,
    parameters: Map[String, String]): BaseRelation
}
Table 1. RelationProvider Contract
Method Description

createRelation

Creates a BaseRelation for loading data from an external data source

Used exclusively when DataSource is requested for a BaseRelation for a given data source format (and no user-defined schema or the user-defined schema matches schema of the BaseRelation)

Table 2. RelationProviders
RelationProvider Description

JdbcRelationProvider

KafkaSourceProvider

Tip
Use SchemaRelationProvider for relation providers that require a user-defined schema.

results matching ""

    No results matching ""