Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

If a topic is split across multiple partitions and we have multiple consumers, won't messages be read out of order?


Messages within a partition are read in order. Partitions themselves may be read simultaneously or out of order. This is the trade-off for scalability.

You can compromise by using hashing or other logic to group related messages into the same partition. For example, hashing by user-id lets you process events for a particular user in order, while processing all events by users in parallel. If you can divide into logically consistent but globally isolated boundaries then this setup works very well.


Order guarantees is not part of the promise with Kafka. If your consumer drops or times out the broker will pause all active consumers for the topic, revoke assignments and issue their new assigned blocks based on the new consumer group acknowledgement count.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: