InsertIntoHiveTable Logical Command

InsertIntoHiveTable is a logical command that writes the result of executing a structured query to a Hive table.

InsertIntoHiveTable is created when:

Creating InsertIntoHiveTable Instance

InsertIntoHiveTable takes the following to be created:

  • CatalogTable

  • Partition keys with optional values (Map[String, Option[String]])

  • Structured query (as a LogicalPlan)

  • overwrite Flag

  • ifPartitionNotExists Flag

  • Names of the output columns

Executing Data-Writing Logical Command — run Method

run(
  sparkSession: SparkSession,
  child: SparkPlan): Seq[Row]
Note
 run is part of DataWritingCommand contract.

run requests the input SparkSession for SharedState that is then requested for the ExternalCatalog.

run requests the SessionState for a new Hadoop Configuration.

run converts the CatalogTable metadata to Hive’s.

run getExternalTmpPath.

run processInsert (and deleteExternalTmpPath).

run requests the input SparkSession for Catalog that is requested to uncache the table.

run un-caches the Hive table. run requests the input SparkSession for SessionState. run requests the SessionState for the SessionCatalog that is requested to invalidate the cache for the table.

In the end, run update the table statistics.

processInsert Internal Method

processInsert(
  sparkSession: SparkSession,
  externalCatalog: ExternalCatalog,
  hadoopConf: Configuration,
  tableDesc: TableDesc,
  tmpLocation: Path,
  child: SparkPlan): Unit

processInsert…​FIXME

Note
 processInsert is used when InsertIntoHiveTable logical command is executed.

results matching ""

    No results matching ""