The molecules in a crystal still vibrate kinetically. A solid with no molecular motion would be at absolute zero temperature.

Wouldn't a fair amount of the thermal energy be hanging out in the bonds themselves? The stress in the system has to count for something.

I'm imagining a bunch of potential energy being stored in the fields at any given moment.

Absolutely, but that’s not heat. Things like heat produced from burning wood is an example of potential energy of binds getting released as kinetic energy + photons.