Log Cleanup Policies (Strategies) — Log Compaction And Retention
Kafka uses log.cleanup.policy configuration property to define cleanup strategies (policy) of logs:
Kafka uses log.retention.check.interval.ms configuration property as the interval between regular log checks.
log.cleanup.policy and cleanup.policy Configuration Properties
The cluster-wide log.cleanup.policy and the per-topic cleanup.policy configuration properties are comma-separated lists of cleanup strategies:
-
compact - enables log compaction
-
delete - enables log retention
Unless defined, cleanup.policy is exactly log.cleanup.policy.
Log Compaction
Log Compaction is a cleanup strategy in which…FIXME
Kafka brokers use LogCleaner for compact retention strategy.
Log compaction can be reconfigured dynamically at runtime.
Log Retention
Log Retention (Garbage Collection) is a cleanup strategy to discard (delete) old log segments when their retention time or size limit has been reached.
By default there is only a time limit and no size limit.
Retention time is controlled by the cluster-wide log.retention.ms, log.retention.minutes or log.retention.hours configuration properties (from the highest to the lowest priority) or their per-topic retention.ms configuration property.
Retention size is controlled by the cluster-wide log.retention.bytes or per-topic retention.bytes configuration property.
Enable ALL logging level for kafka.log.Log logger to see messages related to log retention.
Kafka brokers schedule kafka-log-retention periodic task for delete retention strategy.