In my previous blog post I looked at the problem of expiring Root CA Certificates and why it exists and you should definitely read that post first. Now though, I'm going to look at the data available and gauge how big an issue this might be in the years to come.
That was the plan
When I started writing the original blog post it didn't have multiple parts and it was never meant to be that long. Likewise, when I started writing this post after breaking it out, it was meant to be a look at the largest CAs out there and then gathering some insight into when we might see similar problems to those mentioned in the first post. After many hours of laborious work I realised just how difficult that was going to be to complete. I still intend to do that but I also want to go deeper into the issue of what happened in the AddTrust expiry incident and hopefully give organisations more insight into how they may foresee and prevent similar incidents in the future.
The AddTrust External CA Root Expiration
Very similar to the problem the BBC had that I mentioned in Part 1, the issue could be solved on either the server with some certificate chain magic or on the client with an update. The problem certificate was this one:
AddTrust External CA Root (Root) Validity 30 May 2000 to 30 May 2020
This was a widely trusted and distributed Root CA that expired just a few days ago at the time of writing and that expiration is what caused issues. Very similar again to the BBC incident, sites and services were serving a specially crafted certificate chain to anchor on this older Root CA for backwards compatibility in old clients. The chains that sites could build would be this one, which is a normal chain without legacy client considerations:
Server Chain
example.com (Leaf)
Sectigo RSA Domain Validation Secure Server CA (Intermediate) Validity 2 Nov 2018 to 31 Dec 2030
Client Anchor
USERTrust RSA Certification Authority (Root) Validity 1 Feb 2010 to 18 Jan 2038
Or this one, which could be delivered for legacy client support:
Server Chain
example.com (Leaf)
Sectigo RSA Domain Validation Secure Server CA (Intermediate) Validity 2 Nov 2018 to 31 Dec 2030
USERTrust RSA Certification Authority (Intermediate) Validity 30 May 2000 to 30 May 2020
Client Anchor
AddTrust External CA Root (Root) Validity 30 May 2000 to 30 May 2020
The reason the second chain is more friendly to legacy clients is that the ultimate Root CA it anchors on is 10 years older and as a result has been trusted and distributed for much longer. This means there is a much better chance that those legacy clients have it and installed and as a result it will work. But none of this should matter...
Modern clients aren't as lame
If you're reading this blog post on a relatively modern browser, let's say anything mainstream from the last few years, your client would likely have never choked on this particular issue. A modern client like that will not only look at the chain that was served, it would look at alternate trust paths if necessary too. In the case above if the chain that was served would fail on your client then it wouldn't just give up, it'd try alternate chains itself to see if it could solve the problem (more on that in Part 3!). That's great and will help us some time into the future, but it won't help us now or in the coming years for all of those legacy clients that aren't going to save us.
Who are the biggest CAs out there?
A few years ago that was probably a really hard question to answer. You might have to ask every CA what their issuance volume was and then compare them all to each other, hoping they weren't presenting figures in creative ways. Today though, we have a very easy way of determining this, Certificate Transparency. You can read the blog for more details but in short CT is a series of public, auditable logs that contain all certificates issued by all CAs. These logs are operated by independent companies all over the world and then further companies aggregate all of these feeds together and even make cool dashboards and advanced search features available.
Merkle Town
The first such tool I'm going to look at is Merkle Town, a CT dashboard created and hosted by Cloudflare.
Using their dashboard we can quickly and easily see that Let's Encrypt, Sectigo and DigiCert are listed as the biggest issuing CAs.
Censys
Another awesome resource I will point to is Censys, much more than a CT log search tool, but that's specifically what I'm going to be using today.
You can see here that Censys provide a list of the most common issuers of certificates along with the number of certificates they've issued.
Crawler.Ninja
Yep, my Crawler.Ninja project is in there too just to add a slightly different perspective to the data. My crawler analyses the Top 1 Million ranked websites every day and amongst all of the data it logs is who issued the certificate.
Certificates can be used across an enormous range of different purposes, not just HTTPS on website. Given that, I want to see if the CT data on the largest CAs lines up with my data on the largest CAs in the Top 1 Million sites on the Web.
The Contenders
I was going to look at the 10 biggest CAs out there but I guess first of all we'd need to define what we consider to be a 'CA'. Do they have to have their own Root CA or can we consider a reseller/subordinate a CA for these purposes? For me, I'm going to grab the top 7 from this list on Crawler.Ninja, caList.txt, as it stands today as those CAs account for more than 10,000 sites (1%) each and the vast majority of the certificates in the Top 1 Million sites.
C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3 146,000
C = US, ST = CA, L = San Francisco, O = "CloudFlare, Inc.", CN = CloudFlare Inc ECC CA-2 75,204
C = GB, ST = Greater Manchester, L = Salford, O = Sectigo Limited, CN = Sectigo RSA Domain Validation Secure Server CA 28,136
C = US, ST = Arizona, L = Scottsdale, O = "GoDaddy.com, Inc.", OU = http://certs.godaddy.com/repository/, CN = Go Daddy Secure Certificate Authority - G2 22,596
C = US, O = Amazon, OU = Server CA 1B, CN = Amazon 16,823
C = US, ST = TX, L = Houston, O = "cPanel, Inc.", CN = "cPanel, Inc. Certification Authority" 12,729
C = US, O = DigiCert Inc, CN = DigiCert SHA2 Secure Server CA 11,025Taking a look at the CT data from the above sources it seems like we mostly agree too. Working with the data I'm most familiar with, and looking at things on the WWW for now, should give us a quick idea of if, or how soon, we might see similar issues.
Let's Encrypt
Let's start with the biggest CA on the list first and look at what's on the horizon for Let's Encrypt.
Their setup is probably the simplest out there and they only have 2 pretty basic and straightforward trust paths. This one:
Let's Encrypt Authority X3 (Intermediate) 17 Mar 2016 to 17 Mar 2021
DST Root CA X3 (Root) 30 Sep 2000 to 30 Sep 2021
Or this one:
Let's Encrypt Authority X3 (Intermediate) 16 Oct 2016 to 16 Oct 2021
ISRG Root X1 (Root) 4 Jun 2015 to 4 Jun 2035
At present, Let's Encrypt are currently still providing their cross-signed Intermediate when issuing certificates to chain back to the IdenTrust DST 3 Root. I talked about Let's Encrypt's plan to transition over to the ISRG Root back in April 2019 and how they had to put that on hold because of insufficient root propagation on Android devices. That bought them time, but the clock is ticking on the IdenTrust DST 3 Root expiring and at some point there won't be any choice. Let's Encrypt will be forced to transition away from depending on the IdenTrust Root and instead depending on their ISRG Root. Given their ISRG Root was only created in 2015 and started full distribution in 2018, that could leave a lot of legacy devices in a bad spot. There's no alternate option at present and this is why I mentioned in Part 1 that the BBC couldn't consider using Let's Encrypt at this time. For those wanting to use Let's Encrypt and support legacy devices, you can serve the cross-signed X3 intermediate all the way up until the expiry of the IdenTrust Root CA, at which point there's nothing more you can do. When this happens I expect a lot of legacy devices, particularly Android or Android based devices, to start encountering issues.
Cloudflare
Taking the next biggest slice of the pie is Cloudflare who, along with Let's Encrypt, dominate as the most popular CAs out there.
Looking at the current chain being served in some of the highest ranked sites in the world we see the following:
CloudFlare Inc ECC CA-2 (Intermediate) 14 Oct 2015 to 9 Oct 2020
Baltimore CyberTrust Root (Root) 12 May 2000 to 12 May 2025
The DigiCert Root in this chain is good for another 5 years but the same problem will be encountered then, which Root will they anchor on at that point? You may also notice that the Cloudflare Intermediate expires soon but it seems like they already have the Cloudflare Inc ECC CA-3 Intermediate ready to go and it's simply a case of switching in the new intermediate on the server, there shouldn't be any issues with that and it has a validity 27 Jan 2020 to 31 Dec 2024. Given the expiry of that new Intermediate seems to be 'the end of 2024', I'd bet there's already a plan in place for the Root to be transitioned out before then.
Sectigo
Next up is Sectigo and they were related to the AddTrust CA expiry I talked through above. Looking at their place in the highest ranked sites out there the certificate chain served is as follows:
Sectigo RSA Domain Validation Secure Server CA (Intermediate) Valid 2 Nov 2018 to 31 Dec 2030
USERTrust RSA Certification Authority (Intermediate) Valid 12 Mar 2019 to 31 Dec 2028
AAA Certificate Services (Root) Valid 1 Jan 2004 to 31 Dec 2028
It's that Intermediate that was the problem earlier as there are different intermediates that can be used to build different chains here. You'll notice that the first intermediate expires after the second one, which seems counter intuitive but is a result of being able to build those different chains. The first intermediate anchor on the USERTrust RSA Certification Authority (Root) certificate which is valid 1 Feb 2010 to 18 Jan 2038, giving it a much later expiry. As it's a newer Root though, the one in the chain above is 6 years older and more likely to be installed on a client device so it makes sense for now.
GoDaddy
Another big player, let's take a look at the chains served by the biggest sites using GoDaddy as their CA.
Go Daddy Secure Certificate Authority - G2 (Intermediate) Valid 3 May 2011 to 3 May 2031
Go Daddy Root Certificate Authority - G2 (Intermediate) Valid 1 Jan 2014 to 30 May 2031
Go Daddy Class 2 Certification Authority (Root) Valid 29 Jun 2004 to 29 Jun 2034
This is interesting because we have a "Root Certificate Authority" being served as an intermediate certificate here and it appears to be related to the age of the root again. There exists a Go Daddy Root Certificate Authority - G2 that is an actual Root CA with a validity period 1 Sep 2009 to 31 Dec 2037 but serving the above intermediate certificate allows the client to anchor on a Root CA that is 5 years older instead. Both of these CAs still have a long time to live so they're looking good in that regard.
Amazon
Another modern cloud hosting provider serving a chain to work around the issue of time, Amazon is no different. Here is the typical Amazon chain:
Amazon (Intermediate) Validity 22 Oct 2015 to 19 Oct 2025
Amazon Root CA 1 (Intermediate) Validity 25 May 2015 to 31 Dec 2037
Starfield Services Root Certificate Authority - G2 (Intermediate) Validity 2 Sep 2009 to 28 Jun 2034
Starfield Class 2 Certification Authority (Root) Validity 29 Jun 2004 to 29 Jun 2034
Here see more 'Root CA' certificates being served in the chain as intermediates which allows the client to anchor on and older Root CA. The Amazon Root CA 1 is available as a Root CA issued in 2015 and the Starfield Services Root Certificate Authority - G2 is also available as a Root CA issued in 2009. By serving them as intermediates in the chain the client is able to build a chain back to the root shown above issued in 2004 and much more likely to be installed. There shouldn't be any issues here until 2034 when that root expires.
cPanel
With a much shorter chain, cPanel have a less complex analysis. Here's their chain:
cPanel, Inc. Certification Authority (Intermediate) Validity 18 May 2015 to 17 May 2025
Sectigo (formerly Comodo CA) (Root) Validity 19 Jan 2010 to 18 Jan 2038
This allows clients to anchor on a chain with a Root CA issued in 2010, which could be considered quite new. Given that cPanel are probably mostly dealing with certificates for websites though, chain building in the browser and the browser being on a somewhat up-to-date system probably isn't a concern. If you do have that concern though you can use the same cPanel Intermediate but instead serve COMODO RSA Certification Authority as your intermediate which would allow the client to anchor on AAA Certificate Services Root instead, which has a validity 1 Jan 2004 to 31 Dec 2028.
DigiCert
Last on the list of the big players and we have DigiCert with another, fortunately, simple chain!
DigiCert SHA2 Secure Server CA (Intermediate) Validity 8 Mar 2013 to 8 Mar 2023
DigiCert (Root) Validity 10 Nov 2006 to 10 Nov 2031
Again, 2006 is a somewhat new Root CA but depending on the usage, and given that all of the usages I'm looking at here are websites, it might not be too bad. There is an alternate chain through DigiCert Global Root CA Intermediate to Baltimore CyberTrust Root which has a Validity 12 May 2000 to 12 May 2025 if you wanted to reach further back in client support.
Analysis
It might not seem like it but there are certainly some interesting things to understand after looking at the data. The more 'traditional' or 'legacy' CA companies of course have a lot more options here, having been in the Root CA game for a long time. If you look at newer companies coming to the party like Cloudflare or Amazon, they have their own intermediates and roots but are still chaining back through the legacy CA Roots, most likely for backwards compatibility. It's a similar problem that Let's Encrypt has, you're new to the task of being a CA but it requires many years of distribution to have a widely trusted Root CA. In the Amazon Trust Services documentation they list the Starfield Root CA as one of their own and operated by them, so one has to wonder if Amazon simply purchased the Root CA to give themselves the capability they needed.
Having looked at the chains above so closely even I wasn't aware of the diversity of available trust paths that exist. For many, getting and using a certificate is probably a simple task and ultimately none of what I've mentioned above will matter. If you have a large enough client base though, and some of those clients are not modern, updated browser installations, then perhaps it can be of some use to someone!
It's worth pointing out that I've only touched the tip of the iceberg here too. There are hundreds of (Root) CAs out there and hundreds more when you include Intermediates! If you depend on certificates for secure communications, and most of us do these days, take the time to look at your current chain and really understand it. Could you build an alternate path if needed and might you need to at some point in the future?
Coming Up!
Yep, there's going to be a 3rd part to this 'series' I guess you could call it. I want to look more closely at the upcoming Let's Encrypt Intermediate and DST 3 Root expiry, how modern clients work around all of these issues, AIA Fetching, Intermediate caching on the client and even more! Stay tuned and consider subscribing using the link up at the top if you want to get notified when I post that!