Powered by GitBook

FileIndex Contract

FileIndex is the abstraction of file indices that knows the root paths and partition schema of a relation.

FileIndex is associated with a HadoopFsRelation.

Table 1. FileIndex Contract
Method	Description
`inputFiles`	`inputFiles: Array[String]` File names to read when scanning this relation Used when: `CatalogFileIndex` is requested for the input files `HadoopFsRelation` is requested for the input files
`listFiles`	`listFiles( partitionFilters: Seq[Expression], dataFilters: Seq[Expression]): Seq[PartitionDirectory]` File names (grouped into partitions when the data is partitioned) Used when: `FileSourceScanExec` physical operator is requested for selectedPartitions `HiveMetastoreCatalog` is requested to inferIfNeeded OptimizeMetadataOnlyQuery logical optimization is executed `CatalogFileIndex` is requested for the files
`metadataOpsTimeNs`	`metadataOpsTimeNs: Option[Long] = None` Metadata operation time for listing files (in nanoseconds) Used when `FileSourceScanExec` leaf physical operator is requested for selectedPartitions
`partitionSchema`	`partitionSchema: StructType` Used when: `CatalogFileIndex` is requested to filterPartitions `DataSource` is requested to getOrInferFileFormatSchema and resolve a FileFormat-based relation
`refresh`	`refresh(): Unit` Refreshes cached file listings Used when: `CacheManager` is requested to lookupAndRefresh InsertIntoHadoopFsRelationCommand is executed `LogicalRelation` leaf logical operator is requested to refresh (for a HadoopFsRelation)
`rootPaths`	`rootPaths: Seq[Path]` Root paths from which the catalog gets the files (as Hadoop Paths). There could be a single root path of the entire table (with partition directories) or individual partitions. Used when: `HiveMetastoreCatalog` is requested for a LogicalRelation over a HadoopFsRelation cached (when requested to convert a HiveTableRelation) `CacheManager` is requested to lookupAndRefresh `FileSourceScanExec` physical operator is requested for the metadata `DDLUtils` utility is used to verifyNotReadPath DataSourceAnalysis logical resolution rule is executed (for a InsertIntoTable with a HadoopFsRelation)
`sizeInBytes`	`sizeInBytes: Long` Estimated size of the data of the relation (in bytes) Used when: `HadoopFsRelation` is requested for the estimated size PruneFileSourcePartitions logical optimization is executed

Table 2. FileIndexes (Direct Implementations and Extensions Only)
FileIndex	Description
CatalogFileIndex
PartitioningAwareFileIndex

results matching ""

No results matching ""