No OS. These games were hand written in assembly direct to the metal.

The source code of the SNES Doom port was open sourced if you wanna take a look: https://github.com/RandalLinden/DOOM-FX