Great idea! There was a system in the late 1800, early 1900 in Sweden. There would be a ventilation whole from the outside leading to the cold storage area. But these seems to have been forgotten since the refrigerator was invented.

We would need todo the same thing but more modern take with a heat exchanger/heat pumps. That would save gadzallions amount of energy used for cooling refrigerators. Depending on climate of course.