I think a lot of data tech has come full circle is now mostly just relational databases. Our org is invested in redshift which lets us mostly pay as we go. The DB itself is just a Postgres facade on scalable storage with some native connectors to file stores and third-parties. After rolling over our stack like three times, we're now just dumping tons of raw data into staging tables, then creating views on top of them. It's 97% raw SQL with a smattering of python for clunky extractions. And we're now true believers in ELT vs ETL.

Redshift with S3 storage is no different to Spark SQL with S3 storage.

Both are distributed compute. Except that Spark allows you to mix/match code with SQL.

I think a key driver of this is not having to use SQL. I like DynamoDB and EdgeDB because I can use a more modern and reasonable language to interact with the database.

That’s a good point, I also think that there should be some modern alternative to SQL. I really like how you can query databases with LinqPad (c#) and how it renders it into a nested table tree. All relations are clickable/expandable, so if you find something interesting in your result set, you can just expand additional rows from other tables. In the background it just creates sql via an ORM, not only once I more or less copy and pasted that generated sql into a view.

But linqpad is not useful if you don’t get the pro version, only then you get code completion. So it’s not really the answer to the problem.

its really difficult to do any kind of analysis without relational queries. The standard way you do this is to have an app datastore in DDB, and an ETL job that pipes your data into some data warehouse env.

EdgeDB works really well for relational queries, it's a graph native query language that renders into Postgresql. Check it out: https://www.edgedb.com/