Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

I found Amazon's Builders Library to have a very insightful list of how to handle building multi tenant queue systems https://aws.amazon.com/builders-library/avoiding-insurmounta...

I think one of the big tools is Shuffle Sharding. The article talks about standard sharding by itself as not being enough to provide robustness in multitenant queues. But Shuffle Sharding I.E. assigning users to virtual groups of underlying queues and enqueueing to the queue with the smallest size gets you pretty far. It can limit throughput for individual users but implementing some simple work stealing logic on the consumer helps make sure you keep your throughput up.



David mentioned it but SQS `MessageGroupId` can get you really far on fair-ish work from the queue. Its effectively a virtual partition key for the queue. Set your customer id, resource id, etc as the MessageGroupId and use that to do the work allocation.

https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQS...


That extra layer of indirection, virtual queues, is key.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: