RecordCollectorImpl

RecordCollectorImpl is a concrete RecordCollector that is created and immediately initialized exclusively when StreamTask is created.

RecordCollectorImpl uses task [streamTaskId] for the logPrefix (that uses the specified streamTaskId).

Creating RecordCollectorImpl Instance

RecordCollectorImpl takes the following to be created:

StreamTask ID
LogContext
ProductionExceptionHandler
skippedRecords sensor

RecordCollectorImpl initializes the internal properties.

`send` Method

<K, V> void send(
  final String topic,
  final K key,
  final V value,
  final Long timestamp,
  final Serializer<K> keySerializer,
  final Serializer<V> valueSerializer,
  final StreamPartitioner<? super K, ? super V> partitioner)

Note	`send` is part of RecordCollector Contract to…FIXME.

send…FIXME

Flushing Kafka Producer — `flush` Method

void flush()

Note	`flush` is part of the RecordCollector Contract to flush the internal Kafka producer.

flush prints out the following DEBUG message to the logs:

Flushing producer

flush requests the Kafka Producer to flush the buffered (accumulated) records (via Producer.flush).

In the end, flush checkForException.

Closing RecordCollector — `close` Method

void close()

Note	`close` is part of the RecordCollector Contract to close a `RecordCollector`.

close prints out the following DEBUG message to the logs:

Closing producer

close requests the internal Producer to close.

In the end, close throws any KafkaException that may have been reported (but not yet thrown).

`checkForException` Internal Method

void checkForException()

checkForException checks the sendException internal registry and throws the exception if specified.

Note	`checkForException` is used when…FIXME

Internal Properties

Name Description

Name	Description
`sendException`	`KafkaException`
`offsets`	`Map<TopicPartition, Long> offsets` Initialized (as empty) when `RecordCollectorImpl` is created A new pair of a Kafka `TopicPartition` with the offset added (or updated) after `RecordCollectorImpl` has finished sending a record
`producer`	Kafka Producer (`Producer<byte[], byte[]>`) Used when…FIXME

sendException

KafkaException

offsets

Map<TopicPartition, Long> offsets

Initialized (as empty) when RecordCollectorImpl is created

A new pair of a Kafka TopicPartition with the offset added (or updated) after RecordCollectorImpl has finished sending a record

producer

Kafka Producer (Producer<byte[], byte[]>)

Used when…FIXME

RecordCollectorImpl