There isn't necessarily an OAuth signing key. The OAuth tokens might not be signed. They might be random values, which act like a password, with a hash of them stored in a database so they can't even be stolen from the database.
Even if they are signed, it doesn't need to be as bad as Golden SAML, because OAuth tokens have a short expiration, so the signing key can have frequent automatic rotation, so any stolen signing key will quickly be useless. For the refresh tokens, they don't have fast expiration, so frequent rotation won't work, but you could have a hybrid system where the OAuth tokens use a frequently rotated signing key, but the refresh tokens are random values with hashes stored in a database.
Even if they are signed, it doesn't need to be as bad as Golden SAML, because OAuth tokens have a short expiration, so the signing key can have frequent automatic rotation, so any stolen signing key will quickly be useless. For the refresh tokens, they don't have fast expiration, so frequent rotation won't work, but you could have a hybrid system where the OAuth tokens use a frequently rotated signing key, but the refresh tokens are random values with hashes stored in a database.