MetadataLog Contract — Metadata Storage

MetadataLog is the abstraction of metadata storage that can persist, retrieve, and remove metadata (of type T).

Table 1. MetadataLog Contract
Method Description

add

add(
  batchId: Long,
  metadata: T): Boolean

Persists (adds) metadata of a streaming batch

Used when:

get

get(
  batchId: Long): Option[T]
get(
  startId: Option[Long],
  endId: Option[Long]): Array[(Long, T)]

Retrieves (gets) metadata of one or more batches

Used when…​FIXME

getLatest

getLatest(): Option[(Long, T)]

Retrieves the latest-committed metadata (if available)

Used when…​FIXME

purge

purge(thresholdBatchId: Long): Unit

Used when…​FIXME

Note
HDFSMetadataLog is the only direct implementation of the MetadataLog Contract in Spark Structured Streaming.

results matching ""

    No results matching ""