log4j.logger.org.apache.spark.sql.execution.streaming.MetadataLogFileIndex=ALL
MetadataLogFileIndex
MetadataLogFileIndex
is a PartitioningAwareFileIndex
of metadata log files (generated by FileStreamSink).
MetadataLogFileIndex
is created when:
-
DataSource
(Spark SQL) is requested to resolve aFileFormat
relation (resolveRelation
) and creates aHadoopFsRelation
-
FileStreamSource
is requested to allFilesUsingMetadataLogFileIndex
Tip
|
Enable Add the following line to Refer to Logging. |
Creating MetadataLogFileIndex Instance
MetadataLogFileIndex
takes the following to be created:
-
Hadoop’s Path
MetadataLogFileIndex
initializes the internal properties.
While being created, MetadataLogFileIndex
prints out the following INFO message to the logs:
Reading streaming file log from [metadataDirectory]
Internal Properties
Name | Description |
---|---|
|
Metadata directory (Hadoop’s Path of the _spark_metadata directory under the path) Used when…FIXME |
|
FileStreamSinkLog (with the _spark_metadata directory) |
|