MetadataLog Contract — Metadata Storage

MetadataLog is the abstraction of metadata storage that can persist, retrieve, and remove metadata (of type T).

Table 1. MetadataLog Contract
Method	Description
`add`	`add( batchId: Long, metadata: T): Boolean` Persists (adds) metadata of a streaming batch Used when: `KafkaMicroBatchReader` is requested to getOrCreateInitialPartitionOffsets `KafkaSource` is requested for the initialPartitionOffsets `CompactibleFileStreamLog` is requested for the store metadata of a streaming batch and to compact `FileStreamSource` is requested to fetchMaxOffset `FileStreamSourceLog` is requested to store (add) metadata of a streaming batch `ManifestFileCommitProtocol` is requested to commitJob `MicroBatchExecution` stream execution engine is requested to construct a next streaming micro-batch and run a single streaming micro-batch `ContinuousExecution` stream execution engine is requested to addOffset and commit an epoch `RateStreamMicroBatchReader` is created (`creationTimeMs`)
`get`	`get( batchId: Long): Option[T] get( startId: Option[Long], endId: Option[Long]): Array[(Long, T)]` Retrieves (gets) metadata of one or more batches Used when…FIXME
`getLatest`	`getLatest(): Option[(Long, T)]` Retrieves the latest-committed metadata (if available) Used when…FIXME
`purge`	`purge(thresholdBatchId: Long): Unit` Used when…FIXME

Note	HDFSMetadataLog is the only direct implementation of the MetadataLog Contract in Spark Structured Streaming.

MetadataLog