Flipside is that with hardware virtualization, a lot of that behavior is protected in hardware which, for whatever reason, seems to be extremely secure in practice. You don't see a lot of erratum-based exploits... the recent SYSRET bug was severe but only counts somewhat ("instruction does something different than what it does on another vendor's processors, and is technically documented to do so" is bad, but it's not like there was some sequence of instructions that would just get you arbitrary memory access without interacting with the hypervisor).
The attack surface of incorrect use of the admittedly complicated x86 privilege transition and protection mechanisms is shared in its entirety by all x86 operating systems (except, to a limited extent, by those that turn off some of these mechanisms, which AFAIK none do).
Um, citation needed. The exact behaviour of the entire x86 supervisor mode instruction set is not remotely a "small attack surface".