The Internals of Apache Kafka
Introduction
Overview of Kafka
Kafka Features
Partition Leader Election
- Preferred Replica Election
- Demo: Using kafka-leader-election.sh
Log Cleanup Policies
Kafka Controller Election
Topic Replication
Topic Deletion
Transactional Producer
Idempotent Producer
Log Management System
Kafka Cluster
Broker
EndPoint
Partition
Replica
PartitionStateStore
- ZkPartitionStateStore
ReplicationUtils
CoreUtils
Kafka Security / Transport Layer Security (TLS) and Secure Sockets Layer (SSL)
Kafka Security / Communications Security
- Demo: Securing Communication Between Clients and Brokers Using SSL
SslFactory
SslEngineBuilder
SslPrincipalMapper
SslConfigs
BrokerSecurityConfigs
Kafka Security / SSL Authentication and Authorization
SSL Authentication and Authorization
Authorizer
- AclAuthorizer
- AuthorizerWrapper
KafkaPrincipal
KafkaPrincipalBuilder
DefaultKafkaPrincipalBuilder
Kafka Networking Layer
ChannelBuilder
- SaslChannelBuilder
- SslChannelBuilder
  - SslAuthenticator
ChannelBuilders
Authenticator
KafkaChannel
Selectable
- Selector
Kafka Tools (Utilities)
Kafka Tools
kafka-configs.sh
kafka-topics.sh
kafka-leader-election.sh
- LeaderElectionCommand
kafka-acls.sh
kafka-consumer-groups.sh
- ConsumerGroupCommand
- ConsumerGroupService
BrokerApiVersionsCommand
JmxTool
GetOffsetShell
ReplicaVerificationTool
kafka-dump-log.sh
ConfigCommand
ReassignPartitionsCommand — Partition Reassignment on Command Line
TopicCommand — Topic Management on Command Line
KafkaAdminClient
AdminUtils
Controller Broker
Controller Broker
KafkaController
StateChangeLogger
ControllerContext
ControllerState
PartitionStateMachine
- ZkPartitionStateMachine
  - Election
  - PartitionLeaderElectionAlgorithms
ReplicaStateMachine
- ZkReplicaStateMachine
ControllerEventManager
- ControllerEventThread
- QueuedEvent
ControllerEventProcessor
ControllerEvent
ControllerChannelManager
- RequestSendThread
ControllerBrokerRequestBatch
- AbstractControllerBrokerRequestBatch
TopicDeletionManager
AbstractControlRequest
Kafka Clients / Producer API
KafkaProducer
- Producer
ProducerConfig
ProducerRecord
Callback
Partitioner
ProducerInterceptor
Internals of Kafka Producer
Sender
RecordAccumulator
- ProducerBatch
ProducerInterceptors
DefaultPartitioner
TransactionManager
Kafka Broker Services
KafkaServer — Kafka Broker
Kafka Server and Periodic Tasks
AdminManager
DelegationTokenManager
DynamicConfigManager
DynamicBrokerConfig
FetchManager
GroupCoordinator
- GroupMetadataManager
- GroupMetadata
Kafka
KafkaApis
KafkaHealthcheck
KafkaServerStartable
KafkaConfig
KafkaMetricsReporter
KafkaRequestHandlerPool
- KafkaRequestHandler
LogManager
MetadataCache
OffsetConfig
ReplicaManager
ShutdownableThread
SocketServer
- Network Processor Thread
RequestChannel
- RequestChannel.Request
TransactionCoordinator
- TransactionMarkerChannelManager
- InterBrokerSendThread
TransactionStateManager
QuotaManagers
ZkUtils
ZKRebalancerListener
Kafka Performance Metrics
KafkaMetricsGroup
BrokerTopicStats
BrokerTopicMetrics
Kafka Clients / Consumer API
Consumer Contract — Kafka Clients for Consuming Records
- KafkaConsumer
- MockConsumer
ConsumerRecord
OffsetAndMetadata
OffsetAndTimestamp
OffsetCommitCallback
ConsumerRebalanceListener
ConsumerConfig — Configuration Properties for KafkaConsumer
CommitFailedException
InvalidOffsetException
NoOffsetForPartitionException
OffsetOutOfRangeException
RetriableCommitFailedException
ConsumerInterceptor
PartitionAssignor Contract
ConsumerCoordinator
ConsumerNetworkClient
ConsumerMetrics
Fetcher
RequestFutureListener
SubscriptionState
RequestFuture
RequestFutureAdapter Contract
Kafka Architecture
Broker Nodes — Kafka Servers
Topics
Messages
Kafka Clients
- Producers
- Consumers
Clusters
Kafka Monitoring (Metrics)
Metrics
Sensor
MetricsReporter
ProducerMetrics
SenderMetrics
Kafka Configuration
Properties
Logging
Tips and Tricks
Gradle Tips
Zookeeper Tips
Kafka in Scala REPL for Interactive Exploration
Running Kafka Broker in Docker
Kafka Clients
KafkaClient
- NetworkClient — Non-Blocking Network KafkaClient
RequestCompletionHandler Contract
MetadataUpdater
- DefaultMetadataUpdater
Metadata
- Listener Contract — Intercepting Metadata Updates
ClientRequest
ClientResponse
StaleMetadataException
NetworkClientUtils
CommonClientConfigs
ClientUtils
Kafka Common
Cluster
- Cluster (deprecated)
ClusterConnectionStates
ClusterResourceListener (and ClusterResourceListeners Collection)
TopicConfig
NotificationHandler Contract
ZkNodeChangeNotificationListener
Configurable Contract
Reconfigurable
MemoryRecordsBuilder
Records
Kafka Common / Requests
AbstractRequestResponse Contract
DescribeLogDirsRequest
DescribeConfigsRequest
ElectPreferredLeadersRequest
FetchRequest
FindCoordinatorRequest
FindCoordinatorResponse
HeartbeatRequest
JoinGroupRequest
JoinGroupResponse
MetadataRequest
MetadataResponse
OffsetCommitRequest
ProduceRequest
SyncGroupRequest
RequestContext
Kafka Common / Serialization
Serializer Contract
Deserializer Contract
Serde Contract
Serdes Factory Object
Varia / Misc
KafkaScheduler
Scheduler
Zookeeper
ZooKeeperClient
KafkaZkClient
AdminZkClient
ZkAclChangeStore
BrokerIdZNode
Kafka Connect
WorkerGroupMember
ConnectDistributed
Kafka Demos
Demo: Kafka Controller Election
Appendix
Further reading or watching

Metrics

Metrics is…FIXME

Metrics

Metrics

results matching ""

No results matching ""