RecordCollectorImpl is a concrete RecordCollector that is created and immediately initialized exclusively when StreamTask is created.

RecordCollectorImpl uses task [streamTaskId] for the logPrefix (that uses the specified streamTaskId).

Creating RecordCollectorImpl Instance

RecordCollectorImpl takes the following to be created:

  • StreamTask ID

  • LogContext

  • ProductionExceptionHandler

  • skippedRecords sensor

RecordCollectorImpl initializes the internal properties.

send Method

<K, V> void send(
  final String topic,
  final K key,
  final V value,
  final Long timestamp,
  final Serializer<K> keySerializer,
  final Serializer<V> valueSerializer,
  final StreamPartitioner<? super K, ? super V> partitioner)
send is part of RecordCollector Contract to…​FIXME.


Flushing Kafka Producer — flush Method

void flush()
flush is part of the RecordCollector Contract to flush the internal Kafka producer.

flush prints out the following DEBUG message to the logs:

Flushing producer

flush requests the Kafka Producer to flush the buffered (accumulated) records (via Producer.flush).

In the end, flush checkForException.

Closing RecordCollector — close Method

void close()
close is part of the RecordCollector Contract to close a RecordCollector.

close prints out the following DEBUG message to the logs:

Closing producer

close requests the internal Producer to close.

checkForException Internal Method

void checkForException()

checkForException checks the sendException internal registry and throws the exception if specified.

checkForException is used when…​FIXME

Internal Properties

Name Description




Map<TopicPartition, Long> offsets

Initialized (as empty) when RecordCollectorImpl is created

A new pair of a Kafka TopicPartition with the offset added (or updated) after RecordCollectorImpl has finished sending a record


Kafka Producer (Producer<byte[], byte[]>)

Used when…​FIXME

results matching ""

    No results matching ""