> With that context of course they chose to place this key in a hardware security module controlled with an m-of-n quorum of engineers to ensure no single human can steal it, expose it, or leak it. Right? ... right?
This is unfortunately not how SSH works. It needs to be unlocked for every incoming connection.
You raise valid hypotheticals about the security of the service... but fixing it involves removing SSH host key verification from Github; better OpsSec would not fully resolve this issue.
I am well aware how ssh works. I have written ssh servers and design secure enclave key management solutions for a living.
Even if they wanted the most quick and dirty lazy option with no policy management, they could stick the key in a PKCS#11 supporting enclave every cloud provider supports these days. OpenSSH natively supports them today.
At a minimum they could have placed their whole ssh connection termination engine in a immutable read only and remotely attestable system like a Nitro enclave or other TEE. You do not need to invent anything new for this.
There are just no excuses here for a company with their size and resources, because I do this stuff all the time as just one guy.
Would these secure storage methods for the key be able to scale up to the ssh connection volume an outfit like GH sees? Genuinely asking; I don't know the answer. I just feel like having to hit a HSM or something similar every time a new ssh connection comes in wouldn't work. Or at the very least I would not see the sub-100ms connection time I can get right now.
Easily. You could have only the CA key in the enclave and have it sign throw-away session keys on stateless autoscaling machines, and/or you can have your actual ssh terminations happen in cloud enclaves like nitro enclaves which have all the same specs and cost of a regular cloud server.
Hardware security modules can perform key operations without allowing anyone to access the key data. Key material being (accidentally or deliberately) leaked has been a solved problem for a long, long time.
Used in such a way, the entirety of Github's SSH connections would be bottlenecked by this HSM. It wouldn't scale and it would be a single point of failure. As lrvick points out, you'd have to use a certificate-based host key scheme like PKCS#11 to make this scalable. That's fine, but it is a different scheme than RSA host key identification.
PKCS#11 is a protocol for talking to hardware security modules for generic cryptographic operations to keep a key out of system memory.
You can totally take a single host key scheme and use HSMs or TEEs at scale.
Openssh supports using openssl as a backend which in turn can use the PKCS#11 protocol to delegate private key operations to remote enclaves or TEEs. Nothing stops you from loading your key into a fleet of them and scaling horizontally just like you scale anything else.
> Nothing stops you from loading your key into a fleet of them and scaling horizontally just like you scale anything else.
Isn't having one key the root of the problem? If you've got one key, it has to be transferred to each HSM module, which means it's going to be in their deploy code. My understanding is that the safe way to scale PKI is to have an HSM generate a key internally, and have that key be signed by a CA in a different HSM, so private key material never leaves any HSM.
I guess you're saying that SSH RSA host keys support this, but I'm only familiar with doing it using looks up correct terminology X.509 certificates like for HTTPS.
> If you've got one key, it has to be transferred to each HSM module, which means it's going to be in their deploy code
There's products out there that allow for keys to be securely shared amongst multiple HSMs, without the key ever existing in clear text outside the HSM.
This is unfortunately not how SSH works. It needs to be unlocked for every incoming connection.
You raise valid hypotheticals about the security of the service... but fixing it involves removing SSH host key verification from Github; better OpsSec would not fully resolve this issue.