CachedRDDBuilder

CachedRDDBuilder is created exclusively when InMemoryRelation leaf logical operator is created.

CachedRDDBuilder uses a RDD of CachedBatches that is either given or built internally.

CachedRDDBuilder uses CachedBatch data structure with the following attributes:

  • Number of rows

  • Buffers (Array[Array[Byte]])

  • Statistics (InternalRow)

CachedRDDBuilder uses isCachedColumnBuffersLoaded flag that is enabled (true) when the _cachedColumnBuffers is defined (not null). isCachedColumnBuffersLoaded is used exclusively when CacheManager is requested to recacheByCondition.

CachedRDDBuilder uses sizeInBytesStats metric (LongAccumulator) to buildBuffers and when InMemoryRelation is requested to computeStats.

Creating CachedRDDBuilder Instance

CachedRDDBuilder takes the following to be created:

  • useCompression flag

  • Batch size

  • StorageLevel

  • Physical operator

  • Table name

  • RDD[CachedBatch] (default: null)

CachedRDDBuilder initializes the internal registries and counters.

buildBuffers Internal Method

buildBuffers(): RDD[CachedBatch]

buildBuffers…​FIXME

Note
 buildBuffers is used exclusively when CachedRDDBuilder is requested to cachedColumnBuffers.

clearCache Method

clearCache(blocking: Boolean = true): Unit

clearCache…​FIXME

Note
 clearCache is used exclusively when CacheManager is requested to clearCache, uncacheQuery, and recacheByCondition.

results matching ""

    No results matching ""