RelationConversions PostHoc Logical Evaluation Rule

Caution

FIXME Show example of a hive table, e.g. spark.table(…)

RelationConversions is created when the Hive-specific logical analyzer is created.

RelationConversions takes the following when created:

apply(
  plan: LogicalPlan): LogicalPlan

Note	`apply` is part of the Rule contract to execute (apply) a rule on a LogicalPlan.

apply traverses the input logical plan looking for InsertIntoTables (over a HiveTableRelation) or HiveTableRelation logical operators:

For an InsertIntoTable over a HiveTableRelation that is non-partitioned and is convertible, apply creates a new InsertIntoTable with the HiveTableRelation converted to a LogicalRelation.

isConvertible(
  relation: HiveTableRelation): Boolean

isConvertible is positive when the input HiveTableRelation is a parquet or ORC table (and corresponding SQL properties are enabled).

Internally, isConvertible takes the Hive SerDe of the table (from table metadata) if available or assumes no SerDe.

isConvertible is turned on when either condition holds:

The Hive SerDe is parquet (aka parquet table) and spark.sql.hive.convertMetastoreParquet configuration property is enabled (which is by default)
The Hive SerDe is orc (aka orc table) and spark.sql.hive.convertMetastoreOrc internal configuration property is enabled (which is by default)

Note	`isConvertible` is used when `RelationConversions` is executed.

convert(
  relation: HiveTableRelation): LogicalRelation

convert branches based on the SerDe of (the storage format of) the input HiveTableRelation logical operator.

For Hive tables in parquet format, convert creates options with one extra mergeSchema per spark.sql.hive.convertMetastoreParquet.mergeSchema configuration property and requests the HiveMetastoreCatalog to convert a HiveTableRelation to a LogicalRelation (with ParquetFileFormat).

For non-parquet Hive tables, convert assumes ORC format:

When spark.sql.orc.impl configuration property is native (default) convert requests HiveMetastoreCatalog to convert a HiveTableRelation to a LogicalRelation over a HadoopFsRelation (with org.apache.spark.sql.execution.datasources.orc.OrcFileFormat as fileFormatClass).
Otherwise, convert requests HiveMetastoreCatalog to convert a HiveTableRelation to a LogicalRelation over a HadoopFsRelation (with org.apache.spark.sql.hive.orc.OrcFileFormat as fileFormatClass).

Note	`convert` uses the HiveSessionCatalog to access the HiveMetastoreCatalog.

Note

convert is used when RelationConversions logical evaluation rule is executed and does the following transformations:

Transforms an InsertIntoTable over a HiveTableRelation with a Hive table (i.e. with hive provider) that is not partitioned and uses parquet or orc data storage format
Transforms an HiveTableRelation with a Hive table (i.e. with hive provider) that uses parquet or orc data storage format

RelationConversions