> what I would love is a Kafka-like "event store with MQ semantics", which is in turn backed by an object store.
I use Druid for somewhat of an off-kafka store with object store flushing. It is not an MQ log, but the ingestion flushes to object store as each segment gets finalized.
And it will directly pull off Kafka & keep the data ready to be sliced/diced easily.
I set my Druid segments to get flushed out every 24h and the Kafka queue to be kept for 7 days, which means if I lose my Druid, it will rehydrate from Kafka on startup (and I can scale the real time ingestion nodes for a faster start, because "cloud" means money is time).
Though, I've been poking about Pulsar[1] and Pravega[2], which are not exactly Kafka, but red wine/white wine in choice.
That said, there's a fair bit of ZK in it, I'm starting to tire of ZK related issues in general (like you need to declare entire ensemble before JVM startups). Also patching it is harder than usual, since the devs have gotten used to saying "No" to FB asks from ZK. It doesn't make any progress, because patches sit on an "are we sure" pile.
I use Druid for somewhat of an off-kafka store with object store flushing. It is not an MQ log, but the ingestion flushes to object store as each segment gets finalized.
And it will directly pull off Kafka & keep the data ready to be sliced/diced easily.
I set my Druid segments to get flushed out every 24h and the Kafka queue to be kept for 7 days, which means if I lose my Druid, it will rehydrate from Kafka on startup (and I can scale the real time ingestion nodes for a faster start, because "cloud" means money is time).
Though, I've been poking about Pulsar[1] and Pravega[2], which are not exactly Kafka, but red wine/white wine in choice.
That said, there's a fair bit of ZK in it, I'm starting to tire of ZK related issues in general (like you need to declare entire ensemble before JVM startups). Also patching it is harder than usual, since the devs have gotten used to saying "No" to FB asks from ZK. It doesn't make any progress, because patches sit on an "are we sure" pile.
[1] - https://pulsar.apache.org/ [2] - http://www.pravega.io/