Kafka Properties

Table 1. Properties
Name Description

advertised.listeners

Comma-separated list of URIs and listener names for other Kafka brokers and Kafka clients to use

Default: listeners

In IaaS environments (e.g. Docker or Kubernetes), advertised.listeners may likely need to be different from the interface to which a Kafka broker binds. That ensures that the Kafka broker advertises an address that is accessible from both local and external hosts.

Unlike listeners it is invalid to advertise the 0.0.0.0 non-routable meta-address.

auto.commit.interval.ms

How often (in milliseconds) consumer offsets should be auto-committed when enable.auto.commit is enabled

auto.create.topics.enable

Enables auto creation of a topic

Default: true

auto.leader.rebalance.enable

Enables auto leader balancing

Default: true

A background thread checks and triggers leader balance if required.

auto.offset.reset

Reset policy — what to do when there is no initial offset in Kafka or if the current offset does not exist any more on the server (e.g. because that data has been deleted):

  • earliest — automatically reset the offset to the earliest offset

  • latest — automatically reset the offset to the latest offset

  • none — throw an exception to the consumer if no previous offset is found for the consumer’s group

  • anything else: throw an exception to a consumer

Default: latest

authorizer.class.name

broker.id

The broker id of a Kafka broker for identification purposes

If unset, a unique broker id will be generated. To avoid conflicts between zookeeper generated broker id’s and user configured broker id’s, generated broker IDs start from reserved.broker.max.id + 1.

Use KafkaConfig.brokerId to access the current value.

broker.id.generation.enable

Enables automatic broker id generation of a Kafka broker

Default: true

When enabled (true) the value configured for reserved.broker.max.id should be checked.

background.threads

The number of threads to use for various background processing tasks

As least 1 thread is required

Default: 10

bootstrap.servers

A comma-separated list of host:port pairs that are the addresses of one or more brokers in a Kafka cluster, e.g. localhost:9092 or localhost:9092,another.host:9092.

Default: (empty)

broker.rack

check.crcs

Automatically check the CRC32 of the records consumed. This ensures no on-the-wire or on-disk corruption to the messages occurred. This check adds some overhead, so it may be disabled in cases seeking extreme performance.

Use ConsumerConfig.CHECK_CRCS_CONFIG

client.id

Default: (randomly-generated)

default.replication.factor

The default replication factor that is used for auto-created topics

Default: 1

Increase the default value to at least 2

delegation.token.master.key

delete.topic.enable

Enables topic deletion

Note
Deleting topic through the admin tool has no effect with the property disabled.

Default: true

enable.auto.commit

When enabled (i.e. true) consumer offsets are committed automatically in the background (aka consumer auto commit) every auto.commit.interval.ms

Default: true

When disabled, offsets have to be committed manually (synchronously using KafkaConsumer.commitSync or asynchronously KafkaConsumer.commitAsync). On restart restore the position of a consumer using KafkaConsumer.seek.

Used when KafkaConsumer is created and creates a ConsumerCoordinator.

fetch.min.bytes

The minimum amount of data the server should return for a fetch request. If insufficient data is available the request will wait for that much data to accumulate before answering the request. The default setting of 1 byte means that fetch requests are answered as soon as a single byte of data is available or the fetch request times out waiting for data to arrive. Setting this to something greater than 1 will cause the server to wait for larger amounts of data to accumulate which can improve server throughput a bit at the cost of some additional latency.

Use ConsumerConfig.FETCH_MIN_BYTES_CONFIG

fetch.max.bytes

The maximum amount of data the server should return for a fetch request. Records are fetched in batches by the consumer, and if the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch will still be returned to ensure that the consumer can make progress. As such, this is not a absolute maximum. The maximum record batch size accepted by the broker is defined via message.max.bytes (broker config) or max.message.bytes (topic config). Note that the consumer performs multiple fetches in parallel.

Use ConsumerConfig.FETCH_MAX_BYTES_CONFIG

fetch.max.wait.ms

The maximum amount of time the server will block before answering the fetch request if there isn’t sufficient data to immediately satisfy the requirement given by fetch.min.bytes.

Use ConsumerConfig.FETCH_MAX_WAIT_MS_CONFIG

group.id

The name of the consumer group the consumer is part of.

heartbeat.interval.ms

The expected time between heartbeats to the group coordinator when using Kafka’s group management facilities.

host.name

The hostname a Kafka broker listens on

Default: (empty)

inter.broker.listener.name

Name of the listener that is used for inter-broker communication

It is not allowed to set inter.broker.listener.name and security.inter.broker.protocol properties at the same time.

inter.broker.protocol.version

interceptor.classes

Comma-separated list of ConsumerInterceptor class names.

Default: (empty)

leader.imbalance.check.interval.seconds

How often the active KafkaController schedules the partition rebalance check (aka AutoLeaderRebalance or AutoPreferredReplicaLeaderElection or auto leader balancing)

Default: 300

listeners

Comma-separated list of URIs and listener names that a Kafka broker will listen on

Default: PLAINTEXT://host.name:port

Use 0.0.0.0 to bind to all the network interfaces on a machine or leave it empty to bind to the default interface.

log.dir

The directory in which the log data is kept

Default: /tmp/kafka-logs

log.dirs

The directories in which the log data is kept

Default: log.dir

Use KafkaConfig.logDirs to access the current value.

key.deserializer

How to deserialize message keys.

max.block.ms

max.partition.fetch.bytes

The maximum amount of data per-partition the server will return. Records are fetched in batches by the consumer. If the first record batch in the first non-empty partition of the fetch is larger than this limit, the batch will still be returned to ensure that the consumer can make progress. The maximum record batch size accepted by the broker is defined via message.max.bytes (broker config) or max.message.bytes (topic config).

Use ConsumerConfig.MAX_PARTITION_FETCH_BYTES_CONFIG

Note
Use fetch.max.bytes for limiting the consumer request size.

max.poll.records

(KafkaConsumer) The maximum number of records returned from a Kafka Consumer when polling topics for records.

The default setting (-1) sets no upper bound on the number of records, i.e. Consumer.poll() will return as soon as either any data is available or the passed timeout expires.

max.poll.records was added to Kafka in 0.10.0.0 by KIP-41: KafkaConsumer Max Records.

From kafka-clients mailing list:

max.poll.records only controls the number of records returned from poll, but does not affect fetching. The consumer will try to prefetch records from all partitions it is assigned. It will then buffer those records and return them in batches of max.poll.records each (either all from the same topic partition if there are enough left to satisfy the number of records, or from multiple topic partitions if the data from the last fetch for one of the topic partitions does not cover the max.poll.records).

Use ConsumerConfig.MAX_POLL_RECORDS_CONFIG.


Internally, max.poll.records is used exclusively when KafkaConsumer is created (to create a Fetcher).

metadata.max.age.ms

metric.reporters

The list of fully-qualified classes names of the metrics reporters.

Default: JmxReporter

metrics.num.samples

Number of samples to compute metrics.

metrics.sample.window.ms

Time window (in milliseconds) a metrics sample is computed over.

min.insync.replicas

The minimum number of replicas in ISR that is needed to commit a produce request with required.acks=-1 (or all)

Default: 1

When a Kafka producer sets acks to all (or -1), this configuration specifies the minimum number of replicas that must acknowledge a write for the write to be considered successful.

If this minimum cannot be met, then the producer will raise an exception (either NotEnoughReplicas or NotEnoughReplicasAfterAppend).

When used together, min.insync.replicas and acks allow you to enforce greater durability guarantees.

A typical scenario would be to create a topic with a replication factor of 3, set min.insync.replicas to 2, and produce with acks of "all". This will ensure that the producer raises an exception if a majority of replicas do not receive a write.

num.io.threads

The number of threads that KafkaServer uses for processing requests, which may include disk I/O

Default: 8

num.network.threads

The number of threads that SocketServer uses for the number of processors per endpoint.

Default: 3

num.partitions

The number of log partitions for auto-created topics

Default: 1

Increase the default value (1) since it is better to over-partition a topic that leads to a better data balancing and aids consumer parallelism.

port

The port a Kafka broker listens on

Default: 9092

rebalance.timeout.ms

The maximum allowed time for each worker to join the group once a rebalance has begun.

receive.buffer.bytes

The hint about the size of the TCP network receive buffer (SO_RCVBUF) to use (for a socket) when reading data. If the value is -1, the OS default will be used.

replica.lag.time.max.ms

replica.socket.timeout.ms

reserved.broker.max.id

Maximum number that can be used for broker.id. Has to be at least 0.

Default: 1000

* Use KafkaConfig.MaxReservedBrokerIdProp to reference the property

* Use KafkaConfig.maxReservedBrokerId to access the current value

retry.backoff.ms

Time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

Use ConsumerConfig.RETRY_BACKOFF_MS_CONFIG

request.timeout.ms

The configuration controls the maximum amount of time the client will wait for the response of a request. If the response is not received before the timeout elapses the client will resend the request if necessary or fail the request if retries are exhausted.

Use ConsumerConfig.REQUEST_TIMEOUT_MS_CONFIG

sasl.enabled.mechanisms

send.buffer.bytes

The hint about the size of the TCP network send buffer (SO_SNDBUF) to use (for a socket) when sending data. If the value is -1, the OS default will be used.

session.timeout.ms

The timeout used to detect worker failures.

Default: 10000

unclean.leader.election.enable

Controls whether to enable replicas not in the ISR set to be elected as leader as the last resort, even though doing so may result in data loss.

Default: 1

value.deserializer

How to deserialize message values

zookeeper.connect

Comma-separated list of Zookeeper hosts (as host:port pairs) that brokers register to, e.g. localhost:2181, 127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002

Default: (empty)

Zookeeper URIs can have an optional chroot path suffix at the end, e.g. 127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002/app/a

If the optional chroot path suffix is used, all paths are relative to this path.

It is recommended to include all the hosts in a Zookeeper ensemble (cluster)

zookeeper.connection.timeout.ms

The max time that the client waits to establish a connection to zookeeper

zookeeper.max.in.flight.requests

The maximum number of unacknowledged requests the client will send to Zookeeper before blocking. Has to be at least 1

Default: 10

zookeeper.session.timeout.ms

Zookeeper session timeout

Default: 6000

zookeeper.set.acl

Enables secure ACLs

Default: false

results matching ""

    No results matching ""