Could be just me. I see Apache kafka more nightmarish than bureaucracy in 'The Castle' or Police/judiciary in 'The Trial' from author Franz Kafka. Mr Franz btw stole his last name from this very famous Apache Kafka project.
Unfortunately we do not have something to discuss. All these consumer/producer/broker failing randomly, system behaving erratically is me doing it wrong. Those white papers from Confluent and testimonial from customers are proof that Kafka works fine.
Individual Kafka consumers and producers tend to have simple behavior, which is good, and initially plugging them together yields simple, predictable systems, but people tend to keep going until they experience pain, at which point they have a system that is right at the limit of what they can understand. Then further feature work pushes them over the limit into darkness.
I think Kafka needs the equivalent of OpenAPI and Redoc, a simple spec and document generator, but for groups of consumers and producers rather than single applications. This would increase the tractability of complex systems, but it would also let you see the system getting more complex over time, even when you haven't reached the pain point yet.
Thanks for the tip; that's a really cool project. It looks like the documentation generator for it works on a single file at a time, and a single file defines a single service. I was thinking of something that works on the level of a system of interacting services. I want a tool that reads the API specifications for a group of services and documents the interactions between them. I'd like to look at service X and see that it emits message Y on queue Z, and then see which services read message Y on queue Z, jump to their documentation, etc. I think the AsyncAPI format is perfect to build on, though. I'll start there if I decide to take a hack at it.
I'm taking a closer look today, and it looks to me like one AsyncAPI document, which can be defined using multiple files, defines one application. If you look at the "fixed fields" section under the root object, the second field is "id"[0]:
id Identifier Identifier of the application the AsyncAPI document is defining.
This format is fine, but the tool I'm looking for is something that will read the definitions of multiple AsyncAPI documents (multiple applications) and show how their inputs and outputs connect, so I can answer a question like, "When application X publishes message Y on channel Z, which applications consume that message?"
AsyncAPI gets me 90% there by defining the service spec and providing code to parse it. It's possible somebody has already written the rest; I'll have to see.
Kafka is very configurable .. and teams struggle to optimize the production deployment configuration and tend to second guess their decisions. Getting a managed deployment from Confluent etc. mitigates these issues as you can count on proven SMEs to deliver a well optimized solution. However, if you were to DIY , you might not be so blessed. Disclosure: I never did a DIY Kafka prod deploy -- my employer was already a Confluent customer.
