GlobalStateUpdateTask — The Default GlobalStateMaintainer

GlobalStateUpdateTask is the one and only GlobalStateMaintainer that…​FIXME

GlobalStateUpdateTask is created exclusively when GlobalStreamThread is requested to initialize (and creates a StateConsumer).

GlobalStateUpdateTask keeps track of the offsets of the records (by the topics and the partitions) that have already been processed. The offsets are then flushed to the global store when requested.

GlobalStateUpdateTask keeps track of the deserializers for…​FIXME

Initializing — initialize Method

Map<TopicPartition, Long> initialize()
initialize is part of GlobalStateMaintainer Contract to…​FIXME.


update Method

void update(final ConsumerRecord<byte[], byte[]> record)
update is part of GlobalStateMaintainer Contract to…​FIXME.

update looks up the RecordDeserializer for the topic (of the record) in the deserializers internal registry.

RecordDeserializer is the metadata of a SourceNode that knows how to deserialize a raw record in a given ProcessorContext.

update then uses the RecordDeserializer to deserialize the record (to the proper types of the key and the value).

When the record has been successfully deserialized, update does the following:

  1. Creates a ProcessorRecordContext for the deserialized record

  2. Requests InternalProcessorContext to use the record context

  3. Requests InternalProcessorContext to use the source node of the RecordDeserializer

  4. Requests the SourceNode (of the deserialized record) to process the deserialized record

In the end, update saves the topic and the partition of the record with the offset in the offsets internal registry.

Flushing State Stores — flushState Method

void flushState()
flushState is part of GlobalStateMaintainer Contract to…​FIXME.

flushState requests GlobalStateManager to flush followed by checkpointing the offsets.

Closing State Manager — close Method

void close() throws IOException
close is part of StateManager Contract to…​FIXME.

close simply requests GlobalStateManager to close (passing on the offsets).

Creating GlobalStateUpdateTask Instance

GlobalStateUpdateTask takes the following when created:

GlobalStateUpdateTask initializes the offsets and deserializers internal registries.

initTopology Internal Method

void initTopology()


initTopology is used exclusively when GlobalStateUpdateTask is requested to initialize.

results matching ""

    No results matching ""