In my last couple of posts about CAs and Root Certificates I've talked about something called Alternate Trust Paths. As a result, many people have asked me questions about how a client can use a different intermediate and/or root to the one that issued the certificate and how/why this works. In this post I'm going to look at how that works and as a result we need to touch on a few other subjects along the way including Authority Information Access (AIA) Fetching too.

First things first

To understand how alternate trust paths work we first need to understand how a certificate is signed and issued by a CA. This is a somewhat technical process that I'm going to (over) simplify here but it should be enough for us to get a good idea.

The Subscriber (the person asking a CA to issue them a certificate) sends a Certificate Signing Request (CSR) to the CA which contains all of their details like name, domain name, etc... The CA will then take a blank certificate and begins to fill in the details of the Subscriber, validating ownership of the domain along the way, until they have completed all of the fields in the certificate. At this stage we have what we call the Pre-Certificate, it's basically a finished certificate ready to issue but the only thing that's missing is the CAs signature on the bottom of the certificate to make it official. The CA has two steps to complete to sign the certificate.

The first step is the CA taking the Pre-Certificate and running it through a hash function, currently SHA256, to obtain the digest. The second step is the CA then taking that digest and encrypting it with their private key. This encrypted digest is the signature and once appended to the end of the file we now have a signed certificate that can be issued to the Subscriber. The thing to note here is the complete lack of any mention about a CA certificate. There is no intermediate certificate mentioned, no root certificate mentioned, nothing. Your certificate is not issued 'by' an intermediate certificate, it's issued 'by' a key, or more specifically it's issued by being signed by a key.

To demonstrate how this works in practise we can look at how the client would validate the signature on your certificate to understand the whole process.

The client, which is a browser for ease of explanation here, has two processes that it must complete to validate the signature. The first process is to take the signature on the bottom of the certificate and decrypt it with the CA's public key. This public key comes from the CA's intermediate certificate which should be delivered by the server during the connection at the start of the TLS handshake. This tells us that if the CA's public key can decrypt it, the CA's private key must have encrypted it. This means it definitely came from the CA in question because only they posses the private key. The second process is for the browser to calculate its own hash of the Pre-Certificate to compare to the hash stored in the signature and determine if they are identical. If they match, the browser now knows two things:

  1. The signature definitely came from the CA. (Authenticity)
  2. The certificate has not been tampered with. (Integrity)

That's an overly simplified explanation of how this all works, but hopefully it's enough to understand the basics of what's going on. The key point to take away from this is that your certificate, as the Subscriber, is issued by a CA key rather than a CA certificate. This is contrary to how most people in the industry, myself included, would explain this in casual conversation. You might hear me say things like "my certificate was issued by the Let's Encrypted X3 Intermediate", which isn't strictly true and doesn't fully convey what's happening on a technical level, but it's easy to say and most of the time it's sufficient.

Cross-Signing

Now that we understand the basics of signatures and how they work, we can look at Cross-Signing and then, eventually, how the idea of Alternate Trust Paths can exist. I'm very familiar with Let's Encrypt and their CA infrastructure so I'm going to use them to explain this as they have a cross-signed intermediate certificate. This means I can give you some real examples to help understand what's going on which I hope will make this a little easier! As I mentioned in my previous post, Let's Encrypt currently have two intermediate certificates called Let's Encrypt Authority X3, one is issued by their own ISRG Root X1 and one is issued by the IdenTrust DST Root CA X3. Here they both are.

As I'm flicking between the tabs there, note what changes and what doesn't. The issuer changes because of course they are issued by different Root CAs, the serial number changes as a result too. The fingerprint also changes because they are separate files so of course they also have unique fingerprints. Now note what doesn't change which is the information about Let's Encrypt and their public key. As I flip between the tabs the Subject and Public Key sections remain identical. Let's Encrypt basically made an exact copy of their intermediate certificate and took it to IdenTrust who signed if for them. The X3 Intermediate issued by IdenTrust is what we call a cross-signed intermediate, it's signed by a different CA. In technical terms this really doesn't mean anything, it doesn't really change anything, but we do it for the reason I outlined in my previous post. By having current certificates issued by this cross-signed intermediate (see my use of the wrong terminology there?) Let's Encrypt have better backwards compatibility for their issued certificates because the client is more likely to have the IdenTrust DST 3 Root and not the ISRG X1 Root, meaning it will trust the certificate.

Note 1: You might notice the Microsoft Trust warning when flicking between tabs but Let's Encrypt is now trusted by Microsoft.

Note 2: By getting a Cross-Signed Intermediate, Let's Encrypt could start issuing certificates immediately without waiting years for root distribution. Without this, they still wouldn't be issuing any certificates today.

Alternate Trust Paths

Now that we know we have two intermediates, both owned and operated by Let's Encrypt and both with the same public key, but issued by different Root CAs, we can talk about Alternate Trust Paths.

Imagine I have a certificate for my website and it was issued by the current intermediate that Let's Encrypt are using, their cross-signed intermediate, my chain would look like this:

scotthelme.co.uk (Leaf)

Let's Encrypt Authority X3 (Cross-Signed Intermediate)

DST Root CA X3 (Root)

The leaf and the intermediate are sent by the server during the TLS handshake and the root is embedded in the client. With these 3 certificates we have a valid certificate chain and everything is good. Let's say I then change the configuration on my server to return a different intermediate certificate and instead of serving the cross-signed intermediate I serve the 'wrong' one and serve the one issued by the ISRG Root X1.

scotthelme.co.uk (Leaf)

Let's Encrypt Authority X3 (ISRG Signed Intermediate)

Most people would expect this to fail now because this is the 'wrong' intermediate but actually, everything could work just fine and the client could build this chain.

scotthelme.co.uk (Leaf)

Let's Encrypt Authority X3 (Intermediate)

ISRG Root X1 (Root)

This is what we call an alternate trust path and you might be confused as to why this would work, so let me explain. Let's go back to this diagram right at the start of the blog post. This is how a browser would verify the signature on a certificate.

There are two steps that I described, use the public key from the intermediate certificate to decrypt the signature and then calculate our own hash to compare. In the certificate chain I served above with the 'wrong' intermediate think how this process would be different and then you realise, it's not different. It doesn't matter which intermediate out of the two I serve because they both contain the same public key! If they both contain the same public key then we can validate the signature with either of these intermediate certificates and it will work just fine. This is how and why we can have alternate trust paths.

I think a lot of the confusion here might come from thinking that our certificates are issued by the CAs intermediate certificate rather than being issued by the CAs intermediate private key. If we think that our certificate is issued by a private key, then it becomes more clear that it's possible for the public key to be present in more than one certificate, and this is how alternate trust paths are made.

Caveat Emptor

Buyer beware indeed. There are some very sharp edges here and I feel that I must point them out to avoid making this all sound simpler than it is in reality. Hopefully we now understand that alternate trust paths can be useful to expand support for legacy clients and how that is technically achieved but clients don't always behave how we expect. Assuming that you can serve either one of the Let's Encrypt intermediates mentioned above and control which Root CA the client would anchor on will leave you surprised in a lot of scenarios. Clients build the path that they want, not the path that you want.

If we're talking about legacy or somewhat simple clients then yes, they will most likely build the chain you expect, for better or for worse. If we're talking about more modern clients, especially browsers, then they will do things in slightly different ways. Imagine the chain I mentioned above with the cross-signed intermediate from Let's Encrypt:

scotthelme.co.uk (Leaf)

Let's Encrypt Authority X3 (Cross-Signed Intermediate)

DST Root CA X3 (Root)

The IdentTrust Root in that chain expires in Sep 2021 so this chain would stop working at that time. Maybe you've statically configured the intermediate in your config, which you shouldn't do but it does happen, and you can see a situation when your Leaf certificate was renewed and replaced, but using the old intermediate is causing you to push clients to anchor on an expired root. Should the client just fail there and say 'oh well, broken' or should it be smarter and look for ways to solve the problem?

Authority Information Access

Within all certificates is a field called the AIA field that contains a URL to download the certificate that signed the certificate you're looking at. Taking a look at one of my certificates, we can see the following:

openssl x509 -in aia.scotthelme.co.uk.cer -noout -text
...
            Authority Information Access:
                OCSP - URI:http://ocsp.int-x3.letsencrypt.org
                CA Issuers - URI:http://cert.int-x3.letsencrypt.org/
...

So this is my certificate for aia.scotthelme.co.uk (yes I issued a certificate just for this demo because Let's Encrypt is so damn easy and free!) and inside the AIA field we can see CA Issuers. This is a URL straight to the certificate that issued my certificate and you can go fetch it, which we call AIA Fetching.

wget http://cert.int-x3.letsencrypt.org/ -O int-x3.cert

We can then inspect that cert and see it is exactly what we expected.

openssl x509 -in int-x3.cert -inform der -noout -text

Note the addition of the -inform der flag as the cert is downloaded in DER not PEM, you can convert it to PEM easily if you need though.

openssl x509 -in int-x3.cert -inform der -out int-x3.cer -outform pem

The point is though that this information resides inside the certificate. If the client is served the leaf certificate without an intermediate, or with the wrong intermediate, why doesn't the client do an AIA Fetch to fix it? Well, the client can, and most do, but there are some that don't so be careful. There were also concerns that AIA Fetching would cause people to become lazy with their config because the client would fix it rather than the site operator serving the correct chain which could be a valid concern or not, either way, this is where we are. The final thing to note is that AIA Fetching will only fetch the certificate at the URL provided in the AIA field, it isn't ever going to be able to fetch anything other than that and you might need something else if you're looking to build an alternate chain.

In the Let's Encrypt example that I gave above, where you might be serving the old intermediate by mistake, AIA would help here because Let's Encrypt would have changed the certificate they serve at the AIA URL so clients would pull down the new one instead.  

Intermediate CA Preloading

Much in the way that CRLite tries to fix the problem of revocation by building knowledge into the client, we can try to do a similar thing here with certificates. If you aren't familiar with the issues of CRLite/revocation, think instead of how we solved the HSTS Trust On First Use (TOFU) problem by introducing the HSTS Preload list. If we have a problem we can sometimes solve it by just giving the client a little bit more information, like bundling in a list of all intermediate CAs in existence. Welcome to Intermediate CA Preloading!

This is a different approach taken by Firefox that you can read about here but it does exactly what it sounds like. This is a list of all (2,727 at present) intermediate certificates that are bundled into Firefox via the Common CA Database (CCADB). This means that Firefox shouldn't have to do any AIA Fetching and instead can have all of the possible intermediates already bundled in for a faster solution. This works great for the Internet PKI, but you need to be careful of that if you're using Private PKI as Firefox will never AIA fetch to save you there.

Demonstration

Hold on to your hats, I'm going to try and demo some things that I've come across in the past and thought that they were weird or the client wasn't behaving how I would have expected. I'm also going to try and explain how or why the client is doing what it's doing and throughout all of this you must remember that any other client could do something different.

I'm using Sandbox to give me a completely fresh environment to work with and if you aren't familiar with Sandbox on Windows then you can check out my blog post here. I have the latest (stable) Chrome and Firefox installed and shown here side by side.

To get started I'm going to visit a local test site using a Let's Encrypt issued certificate for certageddon.scotthelme.co.uk (yes, another certificate just for this demo because Let's Encrypt is so damn easy and free!) and the site is not going to serve any intermediate certificate, it's only going to serve the leaf certificate. By default the Windows Trust Store is pretty sparse and it will down roots as needed with Firefox already being preloaded with all roots in NSS.

Here's the page load in each browser and the result is that they both load fine.

Firefox already has the necessary intermediate preloaded and Chrome (or more specifically Windows) has done an AIA fetch to retrieve the missing intermediate certificate which we can now see listed in Intermediate Certification Authorities.

Next is the same scenario on the server, only the leaf certificate being served, but I'm going to remove trust in the DST X3 Root CA in Windows and Firefox.

Here, as you might expect, Chrome won't be able to AIA Fetch to save the connection as it's going to bring back the intermediate that chains to the untrusted root. I did expect Firefox to have a good chance of working though as between the preloaded intermediates and the IRSG X1 Root that's present, there is a valid chain to be built here. Next step was to serve my leaf certificate with the X3 Intermediate signed by the ISRG X1 Root instead of the 'correct' X3 Intermediate signed by the DST X3 Root.

As you can see that works fine and both clients are able to chain down to the ISRG X1 Root now, with Windows having downloaded it to update the Root Store.

Now I can demonstrate something working that maybe should or shouldn't work depending on your view. I'm going to go back one step to where I was serving just the leaf certificate from the server and the DST X3 Root is still not trusted and previously, both browsers failed to connect.

Now Firefox can connect, and I'm using the exact same configuration on the server. My guess, and I must stress this is a guess, is that Firefox has now cached the X3 Intermediate signed by ISRG X1 locally and now has it available to build a chain with, which would also explain why it didn't work before. Looking further at the Intermediate Preloading docs you can see that Firefox will pull down, by default, only 100 intermediate certificates per day. This could explain why this didn't work earlier, because the certs aren't actually preloaded at installation, they are downloaded after. Another interesting thing is that Windows pulled down the ISRG X1 Root as expected, and it cached the DST X3 signed X3 Intermediate from earlier, but it doesn't appear to have cached the ISRG X1 signed X3 Intermediate...

Even if I visit the Let's Encrypt ISRG X1 test site, Windows still doesn't cache the intermediate but the sites work and the chains build as you'd expect.

Hopefully that at least shows my point which is that different clients can behave in different ways and as the client becomes less complex than a modern browser/OS, you can expect it to do less of the helpful things above!

More to come

I know right?! This has turned into a pretty lengthy series of blog posts but sometimes you get an itch and you just have to scratch it. My concern with talking about PKI is that you have to go so deep to really understand this stuff and my blog posts are barely scratching the surface. Hopefully this got you started and gave some clarity into the 'magic' that might happen around certificates but if you enjoyed this and you really want to dive into it, consider coming along an joining me on our 2-day training course. For now though, I'm off to work on the next part and I'm tidying up a tool I've just written to show how chain building works 🤣