MetadataLog Contract — Metadata Storage

MetadataLog is the abstraction of store metadata that can persist, retrieve, and remove metadata (of type T).

Table 1. MetadataLog Contract
Method Description

add

add(
  batchId: Long,
  metadata: T): Boolean

Adds metadata for a batch

get

get(batchId: Long): Option[T]
get(
  startId: Option[Long],
  endId: Option[Long]): Array[(Long, T)]

Retrieves metadata by batch id or batch range

Used when…​FIXME

getLatest

getLatest(): Option[(Long, T)]

Retrieves the latest-committed metadata (if available)

Used when…​FIXME

purge

purge(thresholdBatchId: Long): Unit

Used when…​FIXME

Note
HDFSMetadataLog is the direct implementation of the MetadataLog Contract in Spark Structured Streaming.

results matching ""

    No results matching ""