>usually, passphrases are only used to decrypt the actual encryption keys, so overwriting those keys should be super fast

I'm not sure if it's really that simple with modern flash storage. There might be no guarantee that attempting to overwrite some data will actually affect the particular memory cells where it is stored. You would probably have to trigger a secure erase to reset all memory cells and hope that it is correctly implemented by the storage device's firmware.