When a Kafka broker is requested to start up, it starts LogManager.

LogManager is given the log directories that are configured using log.dirs or log.dir configuration properties (default: /tmp/kafka-logs). The log directories are immediately loaded (on a fixed thread pool with numRecoveryThreadsPerDataDir recovery threads).

LogManager uses the num.recovery.threads.per.data.dir dynamic configuration (default: 1) for the number of threads per log data directory for log recovery at startup and flushing at shutdown.

Every log directory may hold zero, one or more partition log directories. LogManager creates a Log for every partition log (that is of the format topic-partition or topic-partition.uniqueId-delete).

LogManager handles the enforcement of the flush policy and log compaction and retention.

Log is made up of log segments that are FileRecords (with the data) and an OffsetIndex (for reads by offset on the log).

Use kafka-dump-log.sh shell script for sanity check of log files and printing contents out to the console.