You don't lose the optimizations because of UB and aliasing rules letting them stay in, but the people who want to make C safer by simply defining all UB would lose you all these optimizations.
ARM already includes a small part of CHERI (pointer signing) and the rest is coming.