Topics are virtual groups of one or many partitions across Kafka brokers in a Kafka cluster.

A single Kafka broker stores messages in a partition in an ordered fashion, i.e. appends them one message after another and creates a log file.

Producers write messages to the tail of these logs that consumers read at their own pace.

Kafka scales topic consumption by distributing partitions among a consumer group, which is a set of consumers sharing a common group identifier.

Due to limitations in metric names, topics with a period (.) or underscore (_) could collide. To avoid issues it is best to use either, but not both.

Use kafka-topics shell script to manage topics.


Partitions with messages — topics can be partitioned to improve read/write performance and resiliency. You can lay out a topic (as partitions) across a cluster of machines to allow data streams larger than the capability of a single machine. Partitions are log files on disk with sequential write only. Kafka guarantees message ordering in a partition.

The log end offset is the offset of the last message written to a log.

The high watermark offset is the offset of the last message that was successfully copied to all of the log’s replicas.

A consumer can only read up to the high watermark offset to prevent reading unreplicated messages.

results matching ""

    No results matching ""