Thanks for the resource! I've seen some of them and the main issue I had is I don't want to self host anything, and I want to use latest third party models as soon as they are released.

Maybe something like what I'm talking about exists already, but I think I'll still try and make my own open source version to fulfill my personal requirements.

We (disclosure: founder) do something similar at Trelent[1] but with an emphasis on security. Paid accounts can use OpenAI & Anthropic models, free ones just OpenAI. We have 3.5 sonnet live already. If you want to try it out lmk! Also totally respect building your own open-source :)

[1]: https://trelent.com

wow Trelent looks cool, how does ZDR negotiation work exactly? What do you offer to the provider that allows you ZDR?

So typically these providers only offer ZDR to "managed" customers, after a lengthy application process. For example, on Azure, "managed" means companies with >$1m, possibly more now, in annual spend. They don't want to waste their time going through this long application process with smaller companies, so we take some of that weight off their shoulders. They get the same revenue at the end of the day, so in many ways it groups smaller companies' LLM spend and sends it straight to their bottom line, and they still get to claim their rolling out AI "responsibly".

Once one provider is cracked, the others fall as well, as these AI companies are all competing viciously for customers. Et voila, ZDR across multiple providers for the small(er) companies out there :)