InMemoryFileIndex

InMemoryFileIndex is a PartitioningAwareFileIndex for a partition schema and file list.

InMemoryFileIndex is created when:

Creating InMemoryFileIndex Instance

InMemoryFileIndex takes the following to be created:

  • SparkSession

  • Root paths (as Hadoop Paths)

  • Options for partition discovery

  • Optional user-defined schema

  • FileStatusCache (default: NoopCache)

InMemoryFileIndex initializes the internal properties.

Internal Properties

Name Description

rootPaths

The root paths with no _spark_metadata streaming metadata directories (of Spark Structured Streaming’s FileStreamSink when reading the output of a streaming query)

Note
rootPaths is part of the FileIndex contract.

results matching ""

    No results matching ""