FileFormatWriter Helper Object

FileFormatWriter is a Scala object that allows for writing the result of a structured query.

Tip

Enable ERROR, INFO, DEBUG logging level for org.apache.spark.sql.execution.datasources.FileFormatWriter logger to see what happens inside.

Add the following line to conf/log4j.properties:

log4j.logger.org.apache.spark.sql.execution.datasources.FileFormatWriter=INFO

Refer to Logging.

Writing Query Result — write Method

write(
  sparkSession: SparkSession,
  plan: SparkPlan,
  fileFormat: FileFormat,
  committer: FileCommitProtocol,
  outputSpec: OutputSpec,
  hadoopConf: Configuration,
  partitionColumns: Seq[Attribute],
  bucketSpec: Option[BucketSpec],
  statsTrackers: Seq[WriteJobStatsTracker],
  options: Map[String, String]): Set[String]

write…​FIXME

Note

write is used when:

executeTask Internal Method

executeTask(
  description: WriteJobDescription,
  sparkStageId: Int,
  sparkPartitionId: Int,
  sparkAttemptNumber: Int,
  committer: FileCommitProtocol,
  iterator: Iterator[InternalRow]): WriteTaskResult

executeTask…​FIXME

Note
executeTask is used exclusively when FileFormatWriter is requested to write the result of a structured query.

processStats Internal Method

processStats(
  statsTrackers: Seq[WriteJobStatsTracker],
  statsPerTask: Seq[Seq[WriteTaskStats]]): Unit

processStats…​FIXME

Note
processStats is used exclusively when FileFormatWriter is requested to write the result of a structured query.

results matching ""

    No results matching ""