Consumer rebalance per topic
Hello We are starting to use Kafka in production but we found an unexpected (at least for me) behavior with the use of partitions. We have a bunch of topics with a few partitions each. We try to...
View ArticleConsumer rebalance per topic
Pablo, Currently, partition is the smallest unit that we distribute data among consumers (in the same consumer group). So, if the # of consumers is larger than the total number of partitions in a Kafka...
View ArticleConsumer rebalance per topic
Pablo, That is a good suggestion. Ideally, the partitions across all topics should be distributed evenly across consumer streams instead of a per-topic based decision. There is no particular advantage...
View ArticleConsumer rebalance per topic
Thank you Jun and Neha I was trying to avoid adding more partitions. I have enough partitions if you count all partitions in all topics. I understand the problem with different data load per topic but...
View ArticleConsumer rebalance per topic
Jira ticket https://issues.apache.org/jira/browse/KAFKA-687 2013/1/7 Pablo Barrera González < [email protected] > -- Pablo Barrera González
View ArticleConsumer rebalance per topic
(From http://kafka.apache.org/design.html) one potential benefit of the existing rebalancing logic is to reduce the number of connections to brokers per consumer instance. However, if you have a large...
View Article