Yes; that's the whole point. There is even a computer verified proof that (for some version of std) if you only use the standard library and don't use unsafe, then you can't write a program that exhibits unsound behavior.
Of course, if there's one bug in unsafe code, then all bets are off, but that's still a bug in the unsafe code.
I think the "It’s not unsound; it’s not even incorrect." is wrong... Clearly, it's unsound -- the problem in this post was documented[0] as a safety precondition of the one function they call in that block!
In code I'd consider high-quality, there'd be a comment like this [1]:
// SAFETY: inner is only mutated by foo, bar, and baz, all of which
// ensure the pointer is valid.
[0] has an unsafe annotation, so in order to invoke it, you have to write the token "unsafe", which means all bets are off unless some sort of external proof of soundness exists.
Of course, if there's one bug in unsafe code, then all bets are off, but that's still a bug in the unsafe code.