InMemoryFileIndex is a PartitioningAwareFileIndex for a partition schema and file list.
InMemoryFileIndex is created when:
CatalogFileIndexis requested for the partitions by the given predicate expressions for a non-partitioned Hive table
DataSourceis requested to createInMemoryFileIndex
Spark Structured Streaming’s
InMemoryFileIndex takes the following to be created:
InMemoryFileIndex initializes the internal properties.