This is so much better than automatically doing dedup and the RAM overhead that entails.
Doing offline/RAM+in memory dedup size optimizations seem like a really good optimization path. In the spirit of also paying only what you use and not the rest.
Edit: What's the RAM overhead of this? Is it ~64B per 128kB deduped block or what's the magnitude of things?
> Edit: What's the RAM overhead of this? Is it ~64B per 128kB deduped block or what's the magnitude of things?
No real memory impact. There's a regions table that uses 128k of memory per terabyte of total storage (and may be a bit more in the future). So for your 10 petabyte pool using deduping, you'd better have an extra gigabyte of RAM.
But erasing files can potentially be twice as expensive in IOPS, even if not deduped. They try to prevent this.
If I manually deduplicate/copy a 1GB file, how many such offset/refcount tuples would be created in RAM? Just one for the whole file, or one per underlying 128kB block?
n.b. I am not pjd, this is from memory and may be wrong.
The answer to that is messy, but basically, there's a table that should be kept in memory for "is there anything reflinked at all in this logical range on disk", and that covers large spans, so how many entries would depend on how contiguous your data logically was on disk; the actual precise mapping list per-vdev doesn't need to be kept continuously in memory, just the more coarse table, so that saves you a fair bit on memory requirements.
They don't go in ram. You have one 64 bit refcount number for every 64MB of storage irrespective of how much is deduplicated; this is a reduction factor of 8M, or 128k per terabyte stored.
This largely exists so you can avoid doing extra work on free.
I still believe TunSafe's interoperability and standing security issues are significant, and that TunSafe's existence and confusion-potential are generally damaging to the WireGuard project, as has been the adversarial position of their developer. That project is going in a number of really undesirable directions from a security perspective. I would encourage folks to stay away from this software.
For those who are after Windows clients, the WireGuard project will hopefully have one quite soon, and of course we're happy to work with interested Windows developers who are working on similar projects with a security-minded attitude.
---
I'm sure the subsequent replies to this message will have plenty of outcry, demands for details, misinformation, and accusations, to bait this into a long sprawling thread. I'd like to preemptively step out of that kind of mudslinging. But I do think it's important to warn users, hence the note above.
> If you decide you'd like to open source it at some point, rather than putting ads on it or selling it like you've done in the past with software, we can talk. But insofar as you're putting users in harm's way and fragmenting the project, I ask that you stay away from these parts. Nobody is interested in insecure software.
> Yet in spite of your to-date brazenness, I'm still willing to work with you if you'd like to turn things around. Shoot me an email if you'd like to talk about open sourcing this work and integrating with the community.
So far, I have not heard anyone who has found any security holes and I'm active in the #WireGuard IRC channel with 300+ users, where many have looked at the code. There may be some unscrupulous hacker who has reviewed the code and found something but choose not to publish it, but it may also apply to WireGuard's source code.
A security hole in WireGuard's wg-quick that many use to establish the connection is that it allows the .conf file to download and execute programs without asking the user, and this feature is enabled by default.
This is basically a good feature and allows admins to run custom software as soon as the connection has been established.
However, it allows an evil (or NSA-hooked) VPN provider to issue .conf files to infect the user's computer with malicious code because users of VPN services rarely review the .conf files.
TunSafe has the same feature but it is disabled by default and requires Admin privileges to enable it.
I like that TunSafe seems to have more restrictive security settings as default, though it may not be appreciated by hardcore users.
I have no dog in this fight whatsoever and I'm grateful for the work you are doing with WireGuard, but I would like to remark that at the moment the only mudslinging and accusations in this thread seem to come from yourself.
I know you pre-emptively opted out of backing up your accusations but I'm going to ask anyway because otherwise it just seems like you are spreading FUD. What are the standing security issues and interoperability issues? Also, how has the developer been adversarial in his position? I'd genuinely like to know.
> If I open source it then anyone could just take it and rebrand it and pretend it's theirs.
Not necessarily. Depends on what license you use for the code, but if it's not a copyleft license they can't even create a (legal) copy if you maintain copyright and don't give anyone a license to copy the code.
It's probably less about legal forks and more about bad actors forking it, inserting spyware, and buying Google ads against "vpn server" and "tunsafe". It seems like it's more of a problem for open source software targeting nontechnical users, and I imagine uTorrent had a lot of issues with it.
I can see that angle. We're all used to curated packages from distro maintainers, whereas Windows software is like the wild West, especially for those not so technically competent.
The only 'supported' WireGuard implementation that exists is written by you in C, yet you complain that my C++ implementation lacks memory safety. Where is the logic?
Have been looking forward to this for years!
This is so much better than automatically doing dedup and the RAM overhead that entails.
Doing offline/RAM+in memory dedup size optimizations seem like a really good optimization path. In the spirit of also paying only what you use and not the rest.
Edit: What's the RAM overhead of this? Is it ~64B per 128kB deduped block or what's the magnitude of things?