A schema needs to be specified when using [className].SchemaRelationProvider Contract — Relation Providers With Mandatory User-Defined Schema
SchemaRelationProvider is the contract of BaseRelation providers that require a user-defined schema while creating a relation.
The requirement of specifying a user-defined schema is enforced when DataSource is requested for a BaseRelation for a given data source format. If not specified, DataSource throws a AnalysisException:
package org.apache.spark.sql.sources
trait SchemaRelationProvider {
def createRelation(
sqlContext: SQLContext,
parameters: Map[String, String],
schema: StructType): BaseRelation
}| Method | Description |
|---|---|
|
Creates a BaseRelation for the user-defined schema Used exclusively when |
|
Note
|
There are no known direct implementation of PrunedFilteredScan Contract in Spark SQL. |
|
Tip
|
Use RelationProvider for data source providers with schema inference. |
|
Tip
|
Use both SchemaRelationProvider and RelationProvider if a data source should support both schema inference and user-defined schemas.
|