Physics dictates that the energy required to cool an object by 1C increases linearly, with difference to ambient temps.

It's like trying to swim against the current - the faster the river flows, the harder it is to move forward at the same rate.