add(
batchId: Long,
metadata: T): Boolean
MetadataLog Contract — Metadata Storage
MetadataLog
is the abstraction of metadata storage that can persist, retrieve, and remove metadata (of type T
).
Method | Description |
---|---|
|
Persists (adds) metadata of a streaming batch Used when:
|
|
Retrieves (gets) metadata of one or more batches Used when…FIXME |
|
Retrieves the latest-committed metadata (if available) Used when…FIXME |
|
Used when…FIXME |
Note
|
HDFSMetadataLog is the only direct implementation of the MetadataLog Contract in Spark Structured Streaming. |