Topics
Topics are virtual groups of one or many partitions across Kafka brokers in a Kafka cluster.
A single Kafka broker stores messages in a partition in an ordered fashion, i.e. appends them one message after another and creates a log file.
Kafka scales topic consumption by distributing partitions among a consumer group, which is a set of consumers sharing a common group identifier.
|
Note
|
Due to limitations in metric names, topics with a period (.) or underscore (_) could collide. To avoid issues it is best to use either, but not both.
|
Use kafka-topics shell script to manage topics.
Partitions
Partitions with messages — topics can be partitioned to improve read/write performance and resiliency. You can lay out a topic (as partitions) across a cluster of machines to allow data streams larger than the capability of a single machine. Partitions are log files on disk with sequential write only. Kafka guarantees message ordering in a partition.
The log end offset is the offset of the last message written to a log.
The high watermark offset is the offset of the last message that was successfully copied to all of the log’s replicas.
|
Note
|
A consumer can only read up to the high watermark offset to prevent reading unreplicated messages. |