In an ideal world this could be avoided by using SSH certificates - an assertion by a trusted party that the key presented by an SSH server is legitimate. The signer of those certificates could be in an HSM, ensuring that the private keys can never be leaked, and the certificates could be sufficiently short-lived that there'd be no need to deal with revocation.
Unfortunately there's no way to make this easy. To associate a CA key with a specific domain you need to add something like:
@cert-authority github.com (key)
to known_hosts, and this (as far as I know) can't currently be managed in the same way that keys are generally added to known_hosts on first use when you ssh to something. So instead we're left with every github.com SSH frontend using the same private keys, and those can't be in an HSM because it simply wouldn't scale to the required load. Which means the private key exists in a form where this kind of thing can occur.
> every github.com SSH frontend using the same private keys
HTTPS has the same problem. Compromise any frontend, and you can MITM traffic to any other front-end till the expiry of the certificate (usually 90 days).
I would really like to see some kind of almost realtime delegation - so that for example, every second a new sub certificate could be generated and signed by a HSM, valid for only 2 seconds.
Lots of bad things happen if your front-end is compromised, that's different and it's a high bar. They can persist access with a backdoor. They can exfil historical data, password hashes. They can corrupt and modify data.
With HTTPS certs you usually have at most 90 days of impact when a key is leaked (less if you revoke and software is checking CRL). GitHub used the same RSA key for over a decade, they may have continued using this key for quite some time more had they not noticed the leak this week.
Clock skew unfortunately makes that scale a little impractical, but yeah it would be great to have mechanisms to delegate endpoint certificate issuance to much shorter timescales
Unfortunately there's no way to make this easy. To associate a CA key with a specific domain you need to add something like:
@cert-authority github.com (key)
to known_hosts, and this (as far as I know) can't currently be managed in the same way that keys are generally added to known_hosts on first use when you ssh to something. So instead we're left with every github.com SSH frontend using the same private keys, and those can't be in an HSM because it simply wouldn't scale to the required load. Which means the private key exists in a form where this kind of thing can occur.