Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Good question. I actually didn't test that at all. I mostly added that as I saw a few mostly empty blocks written and compression lowered the total space usage a bit.


You might give lz4 a try as well. ZFS uses it as a compression option, and the overhead is quite small (especially compared to zlib).


A sophisticated approach to block compression would be to build a dictionary using something like zstandard's "training mode" and a random sample of data blocks, store it (versioned) in FDB as well and reference the dictionary version from compressed blocks. That would get better compression with the relatively small blocks.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: