Kafka Controller Election

A Kafka broker can be elected as the controller in the process known as Kafka Controller Election.

Kafka Controller Election process relies heavily on the features of Apache ZooKeeper that acts as the source of truth and guarantees that only one broker can ever be elected (due to how ephemeral nodes work).

From Nodes and ephemeral nodes in the official documentation of Apache ZooKeeper:

ZooKeeper also has the notion of ephemeral nodes. These znodes exists as long as the session that created the znode is active. When the session ends the znode is deleted.

From Leader Election in the official documentation of Apache ZooKeeper:

A simple way of doing leader election with ZooKeeper is to use the SEQUENCE|EPHEMERAL flags when creating znodes that represent "proposals" of clients. The idea is to have a znode, say "/election"

When ControllerEventThread is requested to process Startup and Reelect controller events, KafkaController (instance that runs on every Kafka broker) is requested to elect.

Tip	Consult Demo: Kafka Controller Election to learn about the process.

Given that all the state is in ZooKeeper use zookeeper-shell script to know which broker is the active controller.

$ ./bin/zookeeper-shell.sh :2181 get /controller
Connecting to :2181

{"version":1,"brokerid":0,"timestamp":"1543491973573"}
cZxid = 0x48
ctime = Thu Nov 29 12:46:13 CET 2018
mZxid = 0x48
mtime = Thu Nov 29 12:46:13 CET 2018
pZxid = 0x48
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x100073f07ba0001
dataLength = 54
numChildren = 0

If you receive Node does not exist: /controller error message, that means that no Kafka broker has been elected as the active controller yet (or that you use an incorrect ZooKeeper server to talk to).

You could also use nc to talk to ZooKeeper in a more direct way (that allows for dump command).

$ nc localhost 2181
dump
SessionTracker dump:
Session Sets (2):
0 expire at Fri Jan 02 10:57:03 CET 1970:
1 expire at Fri Jan 02 10:57:06 CET 1970:
  0x100073f07ba0001
ephemeral nodes dump:
Sessions with Ephemerals (1):
0x100073f07ba0001:
  /controller
  /brokers/ids/0

From Notable changes in 0.10.1.0 in the official documentation of Apache Kafka:

The recommended way to detect if a given broker is the controller is via the kafka.controller:type=KafkaController,name=ActiveControllerCount metric.

Figure 1. Active KafkaController in jconsole

Controller ID Registered (in ZooKeeper)

The election process stops when there is a controller ID registered in Zookeeper (using KafkaZkClient that gets the ID of the active controller and the ID is any number but -1).

No Controller ID Registered (in ZooKeeper)

If there is no controller ID registered, every KafkaController instance tries to register itself as the controller (in Zookeeper) and increment the controller epoch (using KafkaZkClient).

In the end, the active KafkaController is requested to onControllerFailover.

Kafka Controller Election