InsertIntoHadoopFsRelationCommand Logical Command

InsertIntoHadoopFsRelationCommand is a DataWritingCommand that writes the data of the query out using the FileFormat.

InsertIntoHadoopFsRelationCommand is created when:

Executing Logical Command — run Method

run(session: SparkSession): Seq[Row]
run is part of RunnableCommand Contract to execute (run) a logical command.

run uses the spark.sql.hive.manageFilesourcePartitions configuration property to…​FIXME

FIXME When is the catalogTable defined?
FIXME When is tracksPartitionsInCatalog of CatalogTable enabled?


Creating InsertIntoHadoopFsRelationCommand Instance

InsertIntoHadoopFsRelationCommand takes the following when created:

  • Output Hadoop’s Path

  • Static table partitions (Map[String, String])

  • ifPartitionNotExists flag

  • Partition columns (Seq[Attribute])

  • BucketSpec

  • FileFormat

  • Options (Map[String, String])

  • Logical plan

  • SaveMode

  • CatalogTable

  • FileIndex

  • Output column names


staticPartitions may hold zero or more partitions as follows:

With that, staticPartitions are simply the partitions of an InsertIntoTable logical operator.

results matching ""

    No results matching ""