The problem is actually WAY more subtle, and pretty hard to understand unless you really get in the weeds of Certificate Transparency and certificate policy, but I'll give a shot at providing a concise explanation.
Let's Encrypt has produced two signed artifacts with the same serial number:
A precertificate is not a certificate, but it implies the existence of a corresponding certificate which can be constructed by applying an algorithm to the precertificate.
Let's Encrypt intended to create a precertificate which would result in (2) when applying the algorithm to (1). Unfortunately, applying the algorithm to (1) results in a different certificate, (3), presumably because of some bug in Let's Encrypt. Since (2) and (3) have the same serial number, it's a violation of the prohibition against duplicate serial numbers.
An easier-to-understand description of the problem is that Let's Encrypt was producing precertificates that didn't match the final certificate, but the compliance violation is duplicate serial numbers, which is why I worded my compliance bug the way I did.
If applying the algorithm to (1) produced (3), what produced (2)?
How can "no duplicate serial numbers" be enforced by any browser without having a store of all certificates? Is it simply a best-effort? Will the browser have a mapping from <serial number> to <certificate>, and whenever it sees a certificate, it will check this map to see if it has seen that serial number on a separate certificate?
> If applying the algorithm to (1) produced (3), what produced (2)?
I believe the root of the problem is that Let's Encrypt is creating certificates and precertificates independently, instead of creating a precertificate and then applying the algorithm to create the corresponding certificate. Since their processes for certificates and precertificates got out-of-sync, they ended up producing (2) instead of (3).
> How can "no duplicate serial numbers" be enforced by any browser without having a store of all certificates?
Browser software doesn't enforce this. It can only be enforced by scanning Certificate Transparency logs looking for violations.
Indeed, although the de jure requirement in the policy isn't actually important per se, the only practical way to obey the policy is to do the thing we want you to do, so that's what you'll actually do, but the way the policy is phrased makes enforcement practical.
This is different from a "Brown M&M" policy where the purpose of the policy is to easily check that you are actually reading and obeying the policy document. Here the policy is worded in a way that doesn't directly achieve what we want, but is measurable, whereas what we want isn't, but the only practical way to achieve policy compliance is to do what we wanted anyway.
Firefox does incidentally error if it sees duplicate serial numbers from the same issuer, though it wouldn't detect this case since the browser won't see precertificates in a TLS handshake.
I don't think this is intended to be a security feature, but simply an error from the depths of NSS where some code uses (issuer, serial) as a unique index.
Requiring Certificate Transparency in the browser doesn’t directly prevent this, but (as in this instance) it ensures there is public data anyone can check to see if this situation has occurred.
They will indeed need to revoke the affected certificates within 5 days, per the Baseline Requirements.
Also, the affected certificates won't be accepted by Certificate Transparency-enforcing browsers (Chrome and Safari) because of the precertificate mismatch.
Interestingly, Chrome does seem to be accepting the affected certificates based on some limited testing. I'm not sure why that would be, though. Chrome certainly requires SCTs to be present, but perhaps isn't (always?) checking the SCTs signatures over the certificates. Once we've wrapped this incident up, I'll have to follow up on that.
Safari does seem to be rejecting them as expected.
I picked 10 affected sites at random, and Chrome correctly rejected every one with ERR_CERTIFICATE_TRANSPARENCY_REQUIRED.
If Chrome isn't validating SCT signatures, that's a serious bug that would enable bypass of CT enforcement, so I'm sure the Chrome team would want to hear more about what you've seen.
Firefox, the browser, only cares if the certificate is valid (not expired, not revoked, ultimately signed by a root CA it trusts). It does not keep tabs on every certificate ever issued. You wouldn't like it if Firefox did an online check with a central authority for every website you visited, nor would you like it to bundle every single certificate ever issued (or even just serial numbers).
Mozilla, the authors of the browser, are part of the CA/Browser Forum, which holds the threat of complete distrust in all web browsers against CAs, which compels CAs to be open and provide logs of all the certificates they've issued and prove they're not mis-issuing certificates. All those extra checks happen here.
> You wouldn't like it if Firefox did an online check with a central authority for every website you visited
Enforcing Certificate Transparency does not require doing an online check for every website you visit.
> Mozilla, the authors of the browser, are part of the CA/Browser Forum, which holds the threat of complete distrust in all web browsers against CAs, which compels CAs to be open and provide logs of all the certificates they've issued and prove they're not mis-issuing certificates.
The CA/Browser Forum does not require CAs to log the certificates that they issue. CT is enforced entirely within the certificate validator code, and it is a major shortcoming that Firefox does not do it.
You can embed CT attestations (SCTs) in the certificate itself, so yes, provided the CA is in cooperation with CT log operators, and deliberately does the pre-certificate -> SCTs -> real certificate dance, it is possible for a browser to validate embedded SCTs without an online check.
However, that assumes that the CA actively does that, they don't have to. Neither does the server. What's compelling them to is _policy_, set by Google and Apple, that their respective browsers won't accept certificates _without_ CT attestations. Google's policy specifically requires that one of the SCTs on a certificate must be a CT log run by Google. Google also controls the list of CT logs that Chrome will consider as valid CT logs, as part of deciding if an SCT is valid. Antitrust, anyone?
I was trying to make a similar point about Firefox - policy vs code. And rather than saying that it's specifically the CA/Browser Forum setting policy (which it does, but only baseline policy, which does not include CT), each org in the CA/Browser Forum has their own root cert inclusion program with their own policies, that all draw from baseline policy then add to it. You are right, _baseline_ policy does not require CT....
... and neither does _Mozilla's_ policy, now I've scanned through it. It actively acknowledges that CT exists (in that it mandates that if you issue a precertificate for CT, you _must_ issue the completed certificate), but it does _not_ require CAs to use CT. In stark contrast to Google and Apple.
Perhaps this is why they also don't implement CT checking in Firefox?
> It actively acknowledges that CT exists (in that it mandates that if you issue a precertificate for CT, you _must_ issue the completed certificate)
I don't think that's what the document says. I don't see a requirement to issue the final certificate. This portion is putting pre-certificates into scope of the agreement in that a mis-issued pre-certificate is evidence of intent to mis-issue a final certificate. So, before issuing a pre-certificate, a CA has to be prepared to revoke the final certificate, even if they never actually issue the final certificate; as well as prepared to defend the issuing of the final certificate.
Presumably, this is to cover from CAs claiming a pre-certificate was issued for testing only, and wasn't going to be issued as a final certificate. Also, I'd presume that a CA issuing pre-certificates so they could embed SCTs would abort issuance if they were unable to get a response from the certificate log, but there's always the chance that the submission went fine and the pre-certificate is logged, but the response didn't make it, so the CA would abort.
There is a major distinction between root store policy and CT policy which you are missing.
Root store policy contains requirements which are enforced by audits, and if a CA violates the root store policy it is considered misissuance requiring them to revoke the offending certificates and file an incident report. Neither Chrome nor Apple root store policies require CT.
CT policy describes what CAs must do for their certificates to be accepted by the certificate validation code. CT policy is enforced entirely by code. It is not an incident if a CA doesn't comply with CT policy; it just means their certificates won't be accepted.
Certificate Transparency verification only requires the server provide stapled proof (either to the certificate itself or a OCSP response) that the certificate was submitted to the public logs. It does not involve any extra requests from the browser to a third party; at most it involves a periodic request from the server to the CA with no client-specific data.
I wonder how much this has to do with OCSP stapling being so badly implemented in Apache 2 and nginx (don't know about the other servers). This article from 2009 [1] still seems current, at least I can attest that I still have issues with nginx. Also this super user Q&A [2] which suggest priming OCSP with a cron job because nginx does not do its job by itself.
