The Internals of Apache Kafka
Introduction
Overview of Kafka
Kafka Features
Partition Leader Election
Preferred Replica Election
Demo: Using kafka-leader-election.sh
Log Cleanup Policies
Kafka Controller Election
Topic Replication
Topic Deletion
Transactional Producer
Idempotent Producer
Log Management System
Kafka Cluster
Broker
EndPoint
Partition
Replica
PartitionStateStore
ZkPartitionStateStore
ReplicationUtils
CoreUtils
Kafka Security / Transport Layer Security (TLS) and Secure Sockets Layer (SSL)
Kafka Security / Communications Security
Demo: Securing Communication Between Clients and Brokers Using SSL
SslFactory
SslEngineBuilder
SslPrincipalMapper
SslConfigs
BrokerSecurityConfigs
Kafka Security / SSL Authentication and Authorization
SSL Authentication and Authorization
Demo: Secure Inter-Broker Communication
Demo: SSL Authentication
Demo: ACL Authorization
Authorizer
AclAuthorizer
AuthorizerWrapper
KafkaPrincipal
KafkaPrincipalBuilder
DefaultKafkaPrincipalBuilder
Kafka Networking Layer
ChannelBuilder
SaslChannelBuilder
SslChannelBuilder
SslAuthenticator
ChannelBuilders
Authenticator
KafkaChannel
Selectable
Selector
Kafka Tools (Utilities)
Kafka Tools
kafka-configs.sh
kafka-topics.sh
kafka-leader-election.sh
LeaderElectionCommand
kafka-acls.sh
AclCommand
JAuthorizerService
AclCommandService
AdminClientService
kafka-consumer-groups.sh
ConsumerGroupCommand
ConsumerGroupService
BrokerApiVersionsCommand
JmxTool
GetOffsetShell
ReplicaVerificationTool
kafka-dump-log.sh
DumpLogSegments
OffsetsMessageParser
TransactionLogMessageParser
ConfigCommand
ReassignPartitionsCommand — Partition Reassignment on Command Line
TopicCommand — Topic Management on Command Line
KafkaAdminClient
AdminClient
AdminMetadataManager
AdminClientRunnable
Admin
AdminUtils
Controller Broker
Controller Broker
KafkaController
StateChangeLogger
ControllerContext
ControllerState
PartitionStateMachine
ZkPartitionStateMachine
Election
PartitionLeaderElectionAlgorithms
ReplicaStateMachine
ZkReplicaStateMachine
ControllerEventManager
ControllerEventThread
QueuedEvent
ControllerEventProcessor
ControllerEvent
AutoPreferredReplicaLeaderElection
BrokerChange
LeaderAndIsrResponseReceived
PreferredReplicaLeaderElection
Reelect
Startup
TopicDeletion
TopicUncleanLeaderElectionEnable
UncleanLeaderElectionEnable
ControllerChannelManager
RequestSendThread
ControllerBrokerRequestBatch
AbstractControllerBrokerRequestBatch
TopicDeletionManager
AbstractControlRequest
LeaderAndIsrRequest
StopReplicaRequest
UpdateMetadataRequest
Kafka Clients / Producer API
KafkaProducer
Producer
ProducerConfig
ProducerRecord
Callback
Partitioner
ProducerInterceptor
Internals of Kafka Producer
Sender
RecordAccumulator
ProducerBatch
ProducerInterceptors
DefaultPartitioner
TransactionManager
Kafka Broker Services
KafkaServer — Kafka Broker
Kafka Server and Periodic Tasks
AdminManager
DelegationTokenManager
DynamicConfigManager
ConfigHandler
BrokerConfigHandler
ClientIdConfigHandler
TopicConfigHandler
UserConfigHandler
DynamicBrokerConfig
BrokerReconfigurable
DynamicConnectionQuota
DynamicListenerConfig
DynamicThreadPool
DynamicClientQuotaCallback
DynamicLogConfig
DynamicMetricsReporters
FetchManager
GroupCoordinator
GroupMetadataManager
GroupMetadata
Kafka
KafkaApis
KafkaHealthcheck
KafkaServerStartable
KafkaConfig
KafkaMetricsReporter
KafkaRequestHandlerPool
KafkaRequestHandler
LogManager
Log
LogSegment
LogCleanerManager
LogCleaner
CleanerConfig
CleanerThread
Cleaner
CleanedTransactionMetadata
ProducerStateManager
LogConfig
RollParams
AbstractIndex
OffsetIndex
TimeIndex
TransactionIndex
LazyIndex
MetadataCache
OffsetConfig
ReplicaManager
ReplicaFetcherManager
ReplicaFetcherThread
ReplicaAlterLogDirsManager
ReplicaAlterLogDirsThread
AbstractFetcherManager
AbstractFetcherThread
ReplicaFetcherBlockingSend
ReplicationQuotaManager
LogDirFailureHandler
ShutdownableThread
SocketServer
Network Processor Thread
RequestChannel
RequestChannel.Request
TransactionCoordinator
TransactionMarkerChannelManager
InterBrokerSendThread
TransactionStateManager
QuotaManagers
ZkUtils
ZKRebalancerListener
Kafka Performance Metrics
KafkaMetricsGroup
BrokerTopicStats
BrokerTopicMetrics
Kafka Clients / Consumer API
Consumer Contract — Kafka Clients for Consuming Records
KafkaConsumer
MockConsumer
ConsumerRecord
OffsetAndMetadata
OffsetAndTimestamp
OffsetCommitCallback
ConsumerRebalanceListener
ConsumerConfig — Configuration Properties for KafkaConsumer
CommitFailedException
InvalidOffsetException
NoOffsetForPartitionException
OffsetOutOfRangeException
RetriableCommitFailedException
ConsumerInterceptor
PartitionAssignor Contract
RangeAssignor
RoundRobinAssignor
StickyAssignor
AbstractPartitionAssignor
ConsumerCoordinator
AbstractCoordinator Contract
HeartbeatThread
GroupCoordinatorMetrics
ConsumerNetworkClient
ConsumerMetrics
Fetcher
RequestFutureListener
SubscriptionState
RequestFuture
RequestFutureAdapter Contract
CoordinatorResponseHandler Contract
FindCoordinatorResponseHandler
HeartbeatResponseHandler
JoinGroupResponseHandler
OffsetCommitResponseHandler
SyncGroupResponseHandler
Kafka Architecture
Broker Nodes — Kafka Servers
Topics
Messages
Kafka Clients
Producers
Consumers
Clusters
Kafka Monitoring (Metrics)
Metrics
Sensor
MetricsReporter
ProducerMetrics
SenderMetrics
Kafka Configuration
Properties
bootstrap.servers
client.id
enable.auto.commit
group.id
retry.backoff.ms
Logging
Tips and Tricks
Gradle Tips
Zookeeper Tips
Kafka in Scala REPL for Interactive Exploration
Running Kafka Broker in Docker
Kafka Clients
KafkaClient
NetworkClient — Non-Blocking Network KafkaClient
RequestCompletionHandler Contract
MetadataUpdater
DefaultMetadataUpdater
Metadata
Listener Contract — Intercepting Metadata Updates
ClientRequest
ClientResponse
StaleMetadataException
NetworkClientUtils
CommonClientConfigs
ClientUtils
Kafka Common
Cluster
Cluster (deprecated)
ClusterConnectionStates
ClusterResourceListener (and ClusterResourceListeners Collection)
TopicConfig
NotificationHandler Contract
ZkNodeChangeNotificationListener
Configurable Contract
Reconfigurable
MemoryRecordsBuilder
Records
FileRecords
MemoryRecords
AbstractRecords
BaseRecords
Kafka Common / Requests
AbstractRequestResponse Contract
AbstractRequest Contract
AbstractRequest.Builder Contract
AbstractResponse
DescribeLogDirsRequest
DescribeConfigsRequest
ElectPreferredLeadersRequest
FetchRequest
FindCoordinatorRequest
FindCoordinatorResponse
HeartbeatRequest
JoinGroupRequest
JoinGroupResponse
MetadataRequest
MetadataResponse
OffsetCommitRequest
ProduceRequest
SyncGroupRequest
RequestContext
Kafka Common / Serialization
Serializer Contract
Deserializer Contract
Serde Contract
Serdes Factory Object
Varia / Misc
KafkaScheduler
Scheduler
Zookeeper
ZooKeeperClient
KafkaZkClient
AdminZkClient
ZkAclChangeStore
BrokerIdZNode
Kafka Connect
WorkerGroupMember
ConnectDistributed
Kafka Demos
Demo: Kafka Controller Election
Appendix
Further reading or watching
Powered by
GitBook
Idempotent Producer
Idempotent Producer
Idempotent Producer
is a Kafka
Producer
that…FIXME
results matching "
"
No results matching "
"