ContinuousQueuedDataReader

ContinuousQueuedDataReader is created exclusively when ContinuousDataSourceRDD is requested to compute a partition.

ContinuousQueuedDataReader uses two types of continuous records:

  • EpochMarker

  • ContinuousRow (with the InternalRow at PartitionOffset)

Fetching Next Row — next Method

next(): InternalRow

next…​FIXME

Note
 next is used when…​FIXME

Closing ContinuousQueuedDataReader — close Method

close(): Unit
Note
 close is part of the java.io.Closeable to close this stream and release any system resources associated with it.

close…​FIXME

Creating ContinuousQueuedDataReader Instance

ContinuousQueuedDataReader takes the following to be created:

  • ContinuousDataSourceRDDPartition

  • TaskContext

  • Size of the data queue

  • epochPollIntervalMs

ContinuousQueuedDataReader initializes the internal properties.

Internal Properties

Name Description

coordinatorId

Epoch Coordinator Identifier

Used when…​FIXME

currentOffset

PartitionOffset

Used when…​FIXME

dataReaderThread

DataReaderThread daemon thread that is created and started immediately when ContinuousQueuedDataReader is created

Used when…​FIXME

epochCoordEndpoint

RpcEndpointRef of the EpochCoordinator per coordinatorId

Used when…​FIXME

epochMarkerExecutor

epochMarkerGenerator

EpochMarkerGenerator

Used when…​FIXME

reader

InputPartitionReader

Used when…​FIXME

queue

results matching ""

    No results matching ""