KafkaConfig

KafkaConfig is the configuration of a Kafka broker and the services.

KafkaConfig is created when:

import kafka.server.KafkaConfig
assert(KafkaConfig.LogCleanupPolicyProp == "log.cleanup.policy")

import java.util.Properties
val props = new Properties()
props.put(KafkaConfig.ZkConnectProp, ":9092") // required property
val kafkaConfig = KafkaConfig.fromProps(props, doLog = false)

import scala.collection.JavaConverters._
assert(kafkaConfig.logCleanupPolicy.asScala.head == "delete")
Table 1. KafkaConfig’s Configuration Values
Value Kafka Property

advertisedListeners

advertised.listeners (see advertisedListeners)

Used when:

authorizerClassName

authorizer.class.name

Used exclusively for KafkaServer

autoCreateTopicsEnable

autoLeaderRebalanceEnable

auto.leader.rebalance.enable

Used exclusively when KafkaController is requested to onControllerFailover

backgroundThreads

brokerId

brokerIdGenerationEnable

broker.id.generation.enable

Used exclusively when KafkaServer is requested to getBrokerIdAndOfflineDirs

compressionType

compression.type

connectionsMaxIdleMs

connections.max.idle.ms

Used when:

deleteTopicEnable

delete.topic.enable

failedAuthenticationDelayMs

connection.failed.authentication.delay.ms

Used exclusively when SocketServer is requested to create a new network processor thread

hostName

host.name

interBrokerListenerName

inter.broker.listener.name (based on getInterBrokerListenerNameAndSecurityProtocol)

Used when:

interBrokerProtocolVersion

interBrokerSecurityProtocol

security.inter.broker.protocol (based on getInterBrokerListenerNameAndSecurityProtocol)

Used when:

controlPlaneListenerName

control.plane.listener.name (based on getControlPlaneListenerNameAndSecurityProtocol)

Used when:

leaderImbalanceCheckIntervalSeconds

leader.imbalance.check.interval.seconds

Used exclusively when KafkaController is requested to processAutoPreferredReplicaLeaderElection

leaderImbalancePerBrokerPercentage

leader.imbalance.per.broker.percentage

Used exclusively when KafkaController is requested to checkAndTriggerAutoLeaderRebalance

listeners

listenerSecurityProtocolMap

logCleanerThreads

logCleanerDedupeBufferSize

logCleanerDedupeBufferLoadFactor

logCleanerIoBufferSize

logCleanerIoMaxBytesPerSecond

logCleanerBackoffMs

logCleanerDeleteRetentionMs

log.cleaner.delete.retention.ms

logCleanerEnable

log.cleaner.enable

Used exclusively when LogCleaner utility is requested to build a CleanerConfig from KafkaConfig.

logCleanerMaxCompactionLagMs

log.cleaner.max.compaction.lag.ms

logCleanerMinCleanRatio

log.cleaner.min.cleanable.ratio

logCleanerMinCompactionLagMs

log.cleaner.min.compaction.lag.ms

logCleanupIntervalMs

logCleanupPolicy

log.cleanup.policy

Used exclusively when KafkaServer is requested to copyKafkaConfigToLog

logDeleteDelayMs

log.segment.delete.delay.ms

logDirs

log.dirs or log.dir

At least one log directory must be defined via log.dirs or log.dir properties.

Used when:

logFlushIntervalMessages

log.flush.interval.messages

Used exclusively when Log is requested to append records

logFlushIntervalMs

log.flush.interval.ms

Used exclusively when KafkaServer is requested for the default log properties (and sets the value of LogConfig.FlushMsProp)

logIndexIntervalBytes

log.index.interval.bytes

logIndexSizeMaxBytes

log.index.size.max.bytes

Used exclusively when KafkaServer is requested for the default log properties (and sets the value of LogConfig.SegmentIndexBytesProp)

logMessageDownConversionEnable

log.message.downconversion.enable

logMessageFormatVersion

log.message.format.version

logMessageTimestampDifferenceMaxMs

log.message.timestamp.difference.max.ms

logMessageTimestampType

log.message.timestamp.type

logPreAllocateEnable

log.preallocate

logRetentionBytes

logRetentionTimeMillis

log.retention.ms, log.retention.minutes, or log.retention.hours (from the highest to the lowest priority)

logRollTimeJitterMillis

LogRollTimeHoursProp

LogRollTimeJitterMillisProp

log.roll.jitter.ms

LogRollTimeMillisProp

log.roll.ms

Default: null

Topic-level property (TopicConfigSynonyms): segment.ms

Used when:

  • …​

logSegmentBytes

log.segment.bytes

Used exclusively when KafkaServer is requested for the default log properties (and sets the value of LogConfig.SegmentBytesProp)

maxConnectionsPerIp

max.connections.per.ip

Used exclusively when SocketServer is requested to start up

maxConnectionsPerIpOverrides

max.connections.per.ip.overrides

Used exclusively when SocketServer is requested to start up

maxReservedBrokerId

reserved.broker.max.id

Used exclusively when KafkaServer is requested to generateBrokerId

messageMaxBytes

minInSyncReplicas

numIoThreads

numNetworkThreads

numRecoveryThreadsPerDataDir

num.recovery.threads.per.data.dir

Used when LogManager is created

getNumReplicaAlterLogDirsThreads

numReplicaFetchers

port

port

PrincipalBuilderClassProp

principal.builder.class

replicaFetchMaxBytes

replicaFetchResponseMaxBytes

replicaLagTimeMaxMs

replica.lag.time.max.ms

Used when:

replicaFetchBackoffMs

replicaSocketTimeoutMs

SaslMechanismInterBrokerProtocolProp

SaslJaasConfigProp

SaslEnabledMechanismsProp

SaslKerberosServiceNameProp

SaslKerberosKinitCmdProp

SaslKerberosTicketRenewWindowFactorProp

SaslKerberosTicketRenewJitterProp

SaslKerberosMinTimeBeforeReloginProp

SaslKerberosPrincipalToLocalRulesProp

SaslLoginRefreshWindowFactorProp

SaslLoginRefreshWindowJitterProp

SaslLoginRefreshMinPeriodSecondsProp

SaslLoginRefreshBufferSecondsProp

SslProtocolProp

SslProviderProp

SslCipherSuitesProp

SslEnabledProtocolsProp

SslKeystoreTypeProp

SslKeystoreLocationProp

SslKeystorePasswordProp

SslKeyPasswordProp

SslPrincipalMappingRulesProp

SslTruststoreTypeProp

SslTruststoreLocationProp

SslTruststorePasswordProp

SslKeyManagerAlgorithmProp

SslTrustManagerAlgorithmProp

SslEndpointIdentificationAlgorithmProp

SslSecureRandomImplementationProp

SslClientAuthProp

socketRequestMaxBytes

socket.request.max.bytes

Used when SocketServer is created and requested to create a new network processor thread

tokenAuthEnabled

transactionMaxTimeoutMs

transaction.max.timeout.ms

Used exclusively when TransactionCoordinator is created

uncleanLeaderElectionEnable

unclean.leader.election.enable

Default: false

Topic-level property: unclean.leader.election.enable

Used when:

queuedMaxRequests

queued.max.requests

Used exclusively when SocketServer is created

zkConnect

zookeeper.connect

Used when:

zkConnectionTimeoutMs

zookeeper.connection.timeout.ms when set or zookeeper.session.timeout.ms

Used when:

zkEnableSecureAcls

zookeeper.set.acl

Used when:

zkMaxInFlightRequests

zookeeper.max.in.flight.requests

Used when:

zkSessionTimeoutMs

zookeeper.session.timeout.ms

Used when:

authorizer Method

authorizer: Option[Authorizer]

authorizer takes the value of KafkaConfig.AuthorizerClassNameProp property.

If defined, authorizer creates an instance and returns it as an Authorizer (or a AuthorizerWrapper for the deprecated kafka.security.auth.Authorizer API). Otherwise, authorizer returns None.

authorizer throws a ConfigException for unsupported Authorizer API:

Authorizer does not implement org.apache.kafka.server.authorizer.Authorizer or kafka.security.auth.Authorizer .
Note
 authorizer is used when KafkaServer is requested to start up.

listeners Method

listeners: Seq[EndPoint]

listeners creates the EndPoints based on listeners configuration property if defined. Otherwise, listeners assumes PLAINTEXT://[hostName]:[port] (per hostName and port).

listeners uses listenerSecurityProtocolMap for protocol mapping.

Note

listeners is used when:

advertisedListeners Method

advertisedListeners: Seq[EndPoint]

advertisedListeners…​FIXME

Note
 advertisedListeners is used when…​FIXME

listenerSecurityProtocolMap Method

listenerSecurityProtocolMap: Map[ListenerName, SecurityProtocol]

listenerSecurityProtocolMap…​FIXME

Note
 listenerSecurityProtocolMap is used when…​FIXME

getNumReplicaAlterLogDirsThreads Method

getNumReplicaAlterLogDirsThreads: Int

getNumReplicaAlterLogDirsThreads is num.replica.alter.log.dirs.threads if defined or the number of log directories.

Note
 getNumReplicaAlterLogDirsThreads is used when ReplicaAlterLogDirsManager is created.

Creating KafkaConfig Instance

KafkaConfig takes the following to be created:

KafkaConfig initializes the internal properties.

Creating KafkaConfig From Java Properties — fromProps Object Method

fromProps(props: Properties): KafkaConfig (1)
fromProps(props: Properties, doLog: Boolean): KafkaConfig
fromProps(defaults: Properties, overrides: Properties): KafkaConfig (2)
fromProps(defaults: Properties, overrides: Properties, doLog: Boolean): KafkaConfig (3)

  1. Seems to be used in tests only

  2. Seems to be used in tests only

  3. Seems to be used in tests only

fromProps…​FIXME

Note

fromProps is used when:

Creating KafkaConfig — apply Factory Method

apply(props: java.util.Map[_, _]): KafkaConfig

apply simply creates a KafkaConfig with the props and the doLog flag on.

Note
 apply seems to be used in tests only.

Adding Reconfigurable to DynamicBrokerConfig — addReconfigurable Method

addReconfigurable(
  reconfigurable: Reconfigurable): Unit

addReconfigurable simply requests the DynamicBrokerConfig to add the Reconfigurable.

Note

addReconfigurable is used when:

controlPlaneListener Method

controlPlaneListener: Option[EndPoint]

controlPlaneListener…​FIXME

Note
 controlPlaneListener is used when…​FIXME

dataPlaneListeners Method

dataPlaneListeners: Seq[EndPoint]

dataPlaneListeners…​FIXME

Note
 dataPlaneListeners is used when…​FIXME

validateValues Internal Method

validateValues(): Unit

validateValues…​FIXME

Note
 validateValues is used when KafkaConfig is created.

getInterBrokerListenerNameAndSecurityProtocol Internal Method

getInterBrokerListenerNameAndSecurityProtocol: (ListenerName, SecurityProtocol)

getInterBrokerListenerNameAndSecurityProtocol…​FIXME

Note
 getInterBrokerListenerNameAndSecurityProtocol is used when KafkaConfig is requested for interBrokerListenerName and interBrokerSecurityProtocol.

logRollTimeMillis Method

logRollTimeMillis: java.lang.Long

logRollTimeMillis is either the value of log.roll.ms or log.roll.hours (converted to millis).

Note
 logRollTimeMillis is used when KafkaServer utility is used to copyKafkaConfigToLog (and set the topic-level segment.ms).

Security Protocol for Controller-Brokers Communication — controlPlaneSecurityProtocol Method

controlPlaneSecurityProtocol: Option[SecurityProtocol]

controlPlaneSecurityProtocol is the name of the security protocol based on control.plane.listener.name and listener.security.protocol.map configuration properties (via getControlPlaneListenerNameAndSecurityProtocol).

Note
 controlPlaneSecurityProtocol is used when ControllerChannelManager is requested to add a new broker.

getControlPlaneListenerNameAndSecurityProtocol Internal Method

getControlPlaneListenerNameAndSecurityProtocol: Option[(ListenerName, SecurityProtocol)]

getControlPlaneListenerNameAndSecurityProtocol uses the control.plane.listener.name property for the listener name and looks it up in the listener.security.protocol.map for the security protocol.

getControlPlaneListenerNameAndSecurityProtocol returns None when the control.plane.listener.name property is not defined.

getControlPlaneListenerNameAndSecurityProtocol throws a ConfigException when the listener name could not be found:

Listener with [listenerName] defined in control.plane.listener.name not found in listener.security.protocol.map.
Note
 getControlPlaneListenerNameAndSecurityProtocol is used when KafkaConfig is requested for the controlPlaneListenerName and controlPlaneSecurityProtocol.

Internal Properties

Name Description

dynamicConfig

DynamicBrokerConfig (that could be provided or will be created from scratch)

Used when:

results matching ""

    No results matching ""