InsertIntoHiveTable Logical Command

InsertIntoHiveTable is a logical command that writes the result of executing a structured query to a Hive table.

InsertIntoHiveTable is created when:

Creating InsertIntoHiveTable Instance

InsertIntoHiveTable takes the following to be created:

  • CatalogTable

  • Partition keys with optional values (Map[String, Option[String]])

  • Structured query (as a LogicalPlan)

  • overwrite Flag

  • ifPartitionNotExists Flag

  • Names of the output columns

Executing Data-Writing Logical Command — run Method

  sparkSession: SparkSession,
  child: SparkPlan): Seq[Row]
run is part of DataWritingCommand contract.

run requests the input SparkSession for SharedState that is then requested for the ExternalCatalog.

run requests the SessionState for a new Hadoop Configuration.

run requests the input SparkSession for Catalog that is requested to uncache the table.

run un-caches the Hive table. run requests the input SparkSession for SessionState. run requests the SessionState for the SessionCatalog that is requested to invalidate the cache for the table.

In the end, run update the table statistics.

processInsert Internal Method

  sparkSession: SparkSession,
  externalCatalog: ExternalCatalog,
  hadoopConf: Configuration,
  tableDesc: TableDesc,
  tmpLocation: Path,
  child: SparkPlan): Unit


processInsert is used when InsertIntoHiveTable logical command is executed.

results matching ""

    No results matching ""