Fetcher

Fetcher is created exclusively when KafkaConsumer is created.

Fetcher KafkaConsumer.png
Figure 1. Fetcher and KafkaConsumer

KafkaConsumer uses the following configuration properties to create a Fetcher:

Table 1. Fetcher’s Internal Properties (e.g. Registries and Counters) (in alphabetical order)
Name Description

client

ConsumerNetworkClient that is given when Fetcher is created.

Creating Fetcher Instance

Fetcher takes the following when created:

  • ConsumerNetworkClient

  • Minimum number of bytes

  • Maximum number of bytes

  • Maximum wait time

  • Fetch size

  • How many records to poll

  • Flag to check CRC or not

  • Deserializer for keys

  • Deserializer for values

  • Metadata

  • SubscriptionState

  • Metrics

  • FetcherMetricsRegistry

  • Time

  • Retry backoff (in milliseconds)

  • Request timeout (in milliseconds)

  • IsolationLevel

Fetcher initializes the internal registries and counters.

Fetcher registers itself with SubscriptionState as a listener to receive notifications about…​FIXME

sendFetches Method

Caution
FIXME

sendMetadataRequest Internal Method

RequestFuture<ClientResponse> sendMetadataRequest(MetadataRequest.Builder request)

Internally, sendMetadataRequest requests ConsumerNetworkClient for the least loaded node.

With the node, sendMetadataRequest requests ConsumerNetworkClient to send the request to the node.

When no node was found, sendMetadataRequest returns a RequestFuture with NoAvailableBrokersException.

Note
sendMetadataRequest is used exclusively when Fetcher is requested for topic metadata.

beginningOffsets Method

Caution
FIXME

retrieveOffsetsByTimes Method

Caution
FIXME

Getting Topic Metadata — getTopicMetadata Method

Map<String, List<PartitionInfo>> getTopicMetadata(MetadataRequest.Builder request, long timeout)

Internally, getTopicMetadata sends the metadata request and requests ConsumerNetworkClient to poll until it finishes successfully or timeout expires.

After poll finishes, getTopicMetadata takes the cluster information from MetadataResponse.

When MetadataResponse is successful, getTopicMetadata takes topics (from Cluster) and requests Cluster for available partitions for every topic.

In the end, getTopicMetadata creates a collection of topic and partitions pairs.

Caution
FIXME Describe the failure path
Note

getTopicMetadata is used when:

Finding Metadata for All Topics — getAllTopicMetadata Method

Map<String, List<PartitionInfo>> getAllTopicMetadata(long timeout)

getAllTopicMetadata gets topic metadata specifying no topics (which means all topics available).

Note
getAllTopicMetadata is used exclusively when KafkaConsumer requests metadata for all topics.

results matching ""

    No results matching ""