Know these Kafka configurations to sleep peacefully in night
After learning the basics of Kafka one of the most important thing to is how Kafka’s configurations has power to let you sleep in night or keep you awake in night.
Following are some Kafka configurations which I have used to bring stability into systems built using Kafka and which really helped the team sleep better in night.
Configurations in Kafka are divided into multiple categories as categorized below. Kafka documentation has good explanation about all of them, so instead of copy paste the same thing here I have linked them to same documentation. I have added context more where it is relevant.
Broker configurations
- min.insync.replicas
- delete.topic.enable
- transactional.id.expiration.ms
- group.initial.rebalance.delay.ms: This is a very important configuration and we did not care about almost for 6 months, If you have less than 100 consumers in a consumer group, default value which is 3 sec is good enough. In our case one of our consumer group has around 400 consumers and 3 secs was not good enough for it. Group used to be in rebalance mode for 15 mins or sometime even an hour because consumers were going into longer cycles of errors and rebalancing specially when u restart consumers and data is also flowing into system. Setting this config t 10 secs which was good enough for all 400 consumers to be up helped in solving major production issues for us.
Topic configs
Consumer configurations
- session.timeout.ms
- auto.offset.reset
- enable.auto.commit
- max.poll.interval.ms
- max.poll.records
- partition.assignment.strategy: Can use this if we have some key with much more data load than other keys.
- request.timeout.ms
- max.partition.fetch.bytes
Producer configurations
Stream configurations
If you know any other Kafka configurations which helped you in sleeping better in night , I would love to know that so please add it into comments section.
