for me the issue is that DuckLake's feature of flushing inlined data to parquet is still in alpha. one of the main issues with parquet is when writing small batches you end up with a lot of parquet files that are inefficient to work with using duckdb. to solve this ducklake inlines these small writes to the dbms you choose (postgres) but for a while it couldn't write them back to parquet. last I had checked this feature didn't yet exist, and now it seems to be in alpha which is nice to see, but I'd like some better support before I consider switching some personal data projects over. https://ducklake.select/docs/stable/duckdb/advanced_features...
Data inlining is also currently limited to only the DuckDB catalog (ie it doesn't work with Postgres cataglogs)[0]. It's improving very quickly though and I'm sure this will be expanded soon.
DuckLake format has an unresolved built-in chicken and egg conflict: it requires SQL database to represent its catalog. But this is what some people are running away from when they choose Parquet format in the first place. Parquet = easy, SQL = hard, adding SQL to Parquet makes the resulting format hard. I would expect a catalog to be in Parquet format as well, then it becomes something self-bootstrapping and usable.
DuckLake is more comparable to Iceberg and Delta than to raw parquet files. Iceberg requires a catalog layer too, a file system based one at its simplest. For DuckLake any RDBMS will do, including fs-based ones like DuckDB and SQLite. The difference is that DuckLake will use that database with all its ACID goodness for all metadata operations and there is no need to implement transactional semantics over a REST or object storage API.
It is not a chicken and egg problem, it is just a requirement to have an RDBMS available for systems like DuckLake and Hive to store their catalogs in. Metadata is relatively small and needs to provide ACID r/w => great RDBMS use case.
I set up something similar at work. But it was before the DuckLake format was available, so it just uses manually generated Parquet files saved to a bucket and a light DuckDB catalog that uses views to expose the parquet files. This lets us update the Parquet files using our ETL process and just refresh the catalog when there is a schema change.
We didn't find the frozen DuckLake setup useful for our use case. Mostly because the frozen catalog kind of doesn't make sense with the DuckLake philosophy and the cost-benefit wasn't there over a regular duckdb catalog. It also made making updates cumbersome because you need to pull the DuckLake catalog, commit the changes, and re-upload the catalog (instead of just directly updating the Parquet files). I get that we are missing the time travel part of the DuckLake, but that's not critical for us and if it becomes important, we would just roll out a PostgreSQL database to manage the catalog.
Yes I totally will, err..., oh my, ebook for $91.99, paperback for $127.99. What's going on with these prices? These aren't college textbooks. I'm glad to hear about the 3rd edition but the cost gives me pause.
I recall them being less expensive when first released.
Either there has been a new printing which was done overseas and was affected by tariffs or more expensive for some other reason, or the copies in the warehouse were taxed as inventory (blame Congress for that, it was a major change in the tax law and it created the current mess of remaindered books and no back-list and ever spiraling book prices).
Wait, so this object is going to pass quite close to Jupiter, Mars, and Venus? And the Juno probe, which was planned to be decommissioned soon anyway, could potentially be maneuvered to follow and get close to the object? Kind of uncanny and kind of a science fiction plot, but fun to think about.
When he claimed Oumuamua was a solar sail remnant I thought "Neat idea, let's wait for further observation." Then we discovered what it actually was and properly understood how it's shape gave us the reflection characteristics that it did.
He was simply wrong. I didn't think Avi was a crank, just that he put out an interesting guess that happened to further his solar sail work.
https://glyphsapp.com/learn/recommendation:get-started
It's a great article!