Trust Issues: How We Cured Our Android 13 Biometric Tablets of Certificate Renewal Dread (Without Shipping a Single Cert File)

TL;DR: Our biometric tablets are stuck on Android 13, and Android 13 doesn't trust SSL.com's newer 2022 root certificate. For a long time we papered over that by bundling a certificate file inside the app, which meant shipping an app release every single time the server certificate renewed. The permanent fix was on the server all along: serve a cross-signed copy of the 2022 root in the HAProxy chain, and Android 13 happily builds a path to an older SSL.com root it already trusts. Zero certificate files in the APK. Zero app releases per renewal. Bonus: even old app builds still carrying an outdated certificate file started working again.

Renewal Day: The Most Dreaded Date on the DevOps Calendar

Some dates on the calendar make you smile: payday, public holidays, the day someone finally descales the office coffee machine. And then there's certificate renewal day.

For most teams, renewing an SSL certificate is a ten-minute chore. Download the new files, drop them on the server, reload, done, back to your coffee. For us, it was a full-blown release cycle with a side of anxiety.

The Ritual Nobody Asked For: Renew, Rebuild, Redeploy, Repeat

Here's what renewal day used to look like on our team:

First, renew the certificate with SSL.com and install it on HAProxy. So far, so normal. Then the fun part begins. Export the new certificate, drop it into the Android project's res/raw folder, bump the versionCode, build, run QA, sign the release, push it to every tablet in the field, and then refresh the dashboard nervously until the whole fleet has updated.

Every. Single. Renewal.

When a Routine Certificate Renewal Becomes a Critical App Release

This wasn't a "nice to have" update we could batch with next month's features. Without the new certificate file, the app couldn't establish a secure connection to our backend at all. No enrollments syncing, no data uploads, no version checks. Biometric data captured in the field would sit on the device with nowhere to go.

When an SSL renewal silently becomes a P1 incident, something in the architecture is wrong.

The Question That Started It All: Is There a Permanent Fix?

After our latest renewal, we stopped and asked the obvious question: can the app stop carrying certificates entirely? Can we renew certificates as often as we like without touching the mobile app?

Spoiler: yes. And the fix turned out to be a one-certificate change in a file on the server. But getting there taught us more about TLS trust chains than we ever expected to learn, so buckle up.

Meet the Hardware: Biometric Tablets That Refuse to Grow Up

Darma, Morpho and the World of Purpose-Built Biometric Devices

Our apps don't run on the latest flagship phones. They run on purpose-built biometric tablets from vendors like Darma and Morpho: rugged Android devices with integrated fingerprint sensors, cameras tuned for face capture, and vendor SDKs that talk directly to that hardware. They're built for enrollment and verification work in the field, where reliability matters more than a shiny new OS.

These devices are fantastic at what they do. Keeping up with Android releases is just not one of those things.

Frozen in Time: Certifications, Vendor Firmware and the "No Updates for You" Reality

Biometric hardware lives in a world of certifications. The sensors, drivers and SDKs are tested and certified against a specific platform build, so a major OS upgrade can mean re-validating drivers, re-testing SDKs and sometimes re-certifying the whole device. That's expensive and slow, so vendors pick a stable Android version and stick with it.

It doesn't help that updates on this kind of hardware don't come from Google on a predictable schedule. They come from the device vendor, when (and if) the vendor decides to ship them. Major version upgrades may never arrive, and firmware updates are rare events that need their own testing and rollout plan.

For our fleet, that stable version is Android 13. Realistically, the Android 13 image on these tablets is the image they'll run for their entire working life.

Why These Tablets Can't Just Download New Trust

Here's the part that matters for this story. Up to and including Android 13, the list of trusted root certificates (the system trust store) lives in the system image, under /system/etc/security/cacerts. It only changes when the firmware changes.

Android 14 changed the game. System CA certificates are now primarily loaded from /apex/com.android.conscrypt/cacerts, which belongs to the Conscrypt module, and Google can update that module through Google Play system updates. So newer devices can learn about new root CAs without a full OS update. HTTP Toolkit's deep dive explains the change in detail.

A quick note to avoid a wrong impression: on an Android 14+ device, /system/etc/security/cacerts is still sitting right where it always was. It's kept as a legacy location and a fallback, and Android only reads it when the updatable Conscrypt directory isn't available. On a normal Android 14+ device that directory exists, so it's the one that counts. The old folder is still there, but it's no longer the guest list, so it isn't a reliable way for us to judge what a newer device trusts.

Our Android 13 tablets have no updatable Conscrypt store at all. They trust exactly the roots in their /system folder, the ones they shipped with, and not one root more.

Keep that in mind, because it's the whole plot.

The Error That Launched a Thousand Stack Overflow Tabs

Here's the log entry that kicked off this adventure, with our hostname swapped for a placeholder:

URL: https://api.example.com/api/mobile-app-versioning/verify-version?versionCode=10401
Method: GET
Protocol: null
Status: Failed
Response: javax.net.ssl.SSLHandshakeException:
  java.security.cert.CertPathValidatorException:
  Trust anchor for certification path not found.
SSL: Yes
Request size: 0 B
Response size: null

If you've built Android apps long enough, you've met this error. Probably late at night. Probably right before a release.

"Trust Anchor for Certification Path Not Found": Decoding the Scary Message

Let's translate it into plain English.

A certification path is the chain of certificates linking the server's certificate to a root authority. The CertPathValidator is the component that walks that chain, link by link. A trust anchor is a root certificate the device already trusts, one that lives in its trust store.

So the error says: "I followed the chain the server gave me as far as I could, and I never reached a certificate I trust." It's not saying the certificate is expired, or for the wrong domain, or broken. It's saying the device and the server can't agree on who to trust.

SSLHandshakeException and Friends: Where Exactly Things Fall Apart

The TLS handshake happens before a single byte of HTTP is exchanged. The client says hello, the server presents its certificate chain, and the client validates that chain. If validation fails, the connection is torn down on the spot, and our HTTP client threw an SSLHandshakeException.

That's why this error has nothing to do with our API code, our endpoints or our backend logic. The request never made it that far.

It Works in the Browser! (And Why That Means Absolutely Nothing)

The first thing everyone does is open the URL in Chrome on their laptop. Padlock icon, green vibes, everything looks perfect. So the server must be fine, right?

Not so fast. Desktop browsers are very forgiving. They ship with large, frequently updated trust stores. Many of them can fetch missing intermediate certificates on their own, and they cache intermediates they've seen on other sites. Our laptops' OS is also far more up to date than a tablet frozen on Android 13.

An Android app using the default networking stack has none of those tricks. It gets the chain the server sends, checks it against the device's system trust store, and that's it. "Works in my browser" is a statement about the browser, not about the server.

Reading the Clues: HTTP Logs, Null Responses and Empty Bodies

Look at that log entry again. Protocol: null, Response size: null, no status code, no response body, a request size of 0 bytes.

That's a fingerprint worth remembering. A real HTTP status, even a 500, means TLS worked fine and the problem lives somewhere else. When everything is null and the request never got a status at all, the connection died during the handshake. That was our cue to look at certificates, not code.

And yes, the irony: the endpoint failing here is the version check, the very call designed to tell the app "hey, please update." When TLS breaks, even our update prompt can't get through.

TLS Chain of Trust 101: A Family Tree With Trust Issues

Before we fix anything, let's understand what the device is actually checking. Certificates work like a family tree, and trust flows from the top down. GlobalSign has a great primer on the chain of trust if you want the long version. Here's the short one.

Leaf, Intermediate, Root: The Three Generations of Every Certificate

Every public TLS certificate setup has three generations.

The leaf certificate (also called the server or end-entity certificate) is the one issued for our domain, a wildcard like *.example.com. It contains our server's public key.

The intermediate certificate belongs to the certificate authority's issuing CA. It's the one that actually signed our leaf certificate.

The root certificate sits at the top. It signed the intermediate, and it's the one that operating systems and browsers ship in their trust stores. Roots are self-signed: nobody vouches for them, the trust store simply decides to believe them.

The Mystery of the Three "Public Keys" With Short, Medium and Long Lives

When SSL.com delivered our certificate, we got three files that looked like three public keys with very different expiry dates. Confusing at first, obvious once you know the family tree. They were three certificates, each containing a public key:

The short-lived one (around 200 days) is our leaf certificate. The medium-lived one (several years) is SSL.com's intermediate, "SSL.com TLS Issuing RSA CA R1". The long-lived one (valid until 2046) is the root, "SSL.com TLS RSA Root CA 2022".

The lifetimes tell the story. Leaf certificates rotate often to limit risk. Intermediates live longer. Roots live for decades, because replacing a root means updating trust stores on billions of devices.

Why Root CAs Hide Offline and Let Intermediates Do the Dirty Work

The private key of a root CA is the crown jewel of the whole system. If it leaks, every certificate below it becomes untrustworthy. So CAs keep root keys offline, locked away in hardware security modules, and only bring them out for rare ceremonies like signing a new intermediate.

Day-to-day issuing is done by intermediates. If an intermediate is ever compromised, it can be revoked and replaced without touching the root that every device trusts. Good security design, and also the reason our server needs to send more than one certificate.

How a Client Walks the Chain From Our Domain to a Trusted Root

During the handshake, the server sends its leaf certificate plus any intermediates. The client then climbs the tree:

*.example.com
  issued by → SSL.com TLS Issuing RSA CA R1
    issued by → SSL.com TLS RSA Root CA 2022
      is this in my trust store? → yes: connect   no: "Trust anchor not found"

Two details are crucial here. First, the client only trusts roots from its own trust store. If the server sends a root certificate, the client doesn't trust it just because it showed up. It's essentially decoration. Second, the client can stop climbing as soon as it reaches any certificate in its trust store. Remember that one, because it's the secret behind the fix.

Trust Stores Explained: The Guest List a Device Actually Checks

Think of the trust store as the guest list at a very exclusive club. A chain can look as fancy as it wants, but if the name at the top isn't on the list, it's not getting in.

On Android 13, the system trust store lives at /system/etc/security/cacerts. Each file is named after the certificate's subject hash, so we could check for a specific root with a couple of commands:

# Get the Android-style hash of the root you care about
openssl x509 -in SSLcom-TLS-Root-2022-RSA.pem -noout -subject_hash_old
 
# Look for it on the device (replace HASH with the output above)
adb shell ls /system/etc/security/cacerts/ | grep HASH

I wasn't very familiar with openssl, so these commands were mostly AI-generated. They served the purpose well, though!

No match? That root is not on the guest list, and no amount of server-side politeness will change that.

The Root of the Problem (Pun Absolutely Intended)

SSL.com's 2016 Roots vs. the Shiny New TLS RSA Root CA 2022

SSL.com has more than one generation of roots. The older family dates from around 2016 and 2017 and includes "SSL.com Root Certification Authority RSA" and "SSL.com EV Root Certification Authority RSA R2". These have been in Android's trust store for years.

The newer generation is "SSL.com TLS RSA Root CA 2022" (plus an ECC sibling). Our renewed certificate is now issued by "SSL.com TLS Issuing RSA CA R1", which chains up to the 2022 root, not to the older 2016 family.

Why Android 13 Has Never Met the 2022 Root

Our tablets shipped with a trust store that contains the older SSL.com roots and not the 2022 one. Since Android 13 can only update its roots through a firmware update, and firmware updates on these devices are rare, the 2022 root simply isn't on the guest list.

So the chain the server sent ended at a root the tablet had never heard of. The tablet did exactly what it should: it refused to trust it.

Android 14's Updatable Root Certificates and the Devices Left Behind

Android 14 introduced updatable root certificates through the Conscrypt module, so newer devices can receive new roots via Google Play system updates. It's a great change, and it's exactly why this problem is invisible on modern phones.

It's also why it's so painful on specialized hardware. Any fleet stuck on Android 13 or older is locked out of that mechanism. As CAs keep rolling out newer roots, the gap between what the server presents and what old devices trust will only grow.

Checking the Guest List: Inspecting Trusted Credentials on the Tablet

We confirmed all of this on the device itself, no adb required. On our Android 13 builds, the list lives under Settings → Security → Encryption & credentials → Trusted credentials, in the System tab (some vendor builds tuck it under "More security settings").

Scroll to the SSL Corporation entries. On our tablets, "SSL.com EV Root Certification Authority RSA R2" was right there. "SSL.com TLS RSA Root CA 2022" was nowhere to be found. Diagnosis confirmed.

The Old Workaround: Smuggling Certificates Inside the APK

Bundling Certificates in res/raw: A Quick Fix With a Long Tail

The quickest way to make an Android app trust something the OS doesn't is to bring its own certificate. We dropped ours into res/raw and pointed Android's network security config at it:

<!-- res/xml/network_security_config.xml (the old way) -->
<network-security-config>
    <domain-config>
        <domain includeSubdomains="true">example.com</domain>
        <trust-anchors>
            <certificates src="@raw/our_server_cert" />
            <certificates src="system" />
        </trust-anchors>
    </domain-config>
</network-security-config>

It works. The app connects. Everyone goes home happy. For about 200 days.

Network Security Config, Trust Anchors and Custom TrustManagers

There are several flavors of this workaround. Some teams use network security config trust anchors like the one above. Some write a custom TrustManager. Some use OkHttp's CertificatePinner to pin a certificate or public key hash.

They all share one fatal property: the app now depends on a specific certificate that lives on the server. When the server's certificate changes, the app's copy goes stale, and the connection breaks until the app gets updated.

The Maintenance Treadmill: Every Renewal Becomes an App Release

Because our bundled file was tied to the server certificate, every renewal meant a new app build. Every new build meant QA, signing, distribution and waiting for the fleet to update. The app's release schedule was effectively being dictated by a certificate authority's expiry dates.

Worse, the knowledge of "don't forget the app when we renew" lived in people's heads. The kind of thing that works perfectly right up until someone is on vacation.

The Rollout Risk: What Happens When Tablets in the Field Miss an Update

Field tablets aren't always online. Some sit in remote locations, some sync once a day, some get turned on only when there's work to do. If a tablet misses the update before the old certificate is replaced on the server, it can't talk to the backend anymore.

And remember the irony from earlier: the endpoint that tells the app to update sits behind the same TLS connection. A tablet that misses the window can't even be told that it missed the window. Someone has to physically get it updated. With a fleet of devices, that's a very bad day.

Certificate Lifetimes Are Shrinking and Our Workaround Wouldn't Survive It

From Multi-Year Certificates to the 200-Day Era

Not long ago, you could buy a certificate valid for two or even three years. Those days are gone. Under the CA/Browser Forum's Ballot SC-081, the maximum lifetime of a publicly trusted TLS certificate dropped to 200 days on March 15, 2026. It drops again to 100 days on March 15, 2027, and to just 47 days on March 15, 2029.

The industry's reasoning is sound: shorter lifetimes limit the damage of compromised keys and push everyone toward automation. But for anyone baking certificates into client apps, it's a countdown clock.

Why a "1-Year" Certificate Now Means Two Reissues

We bought a "1-year" certificate. The certificate we actually received was issued on July 22, 2026 and expires on February 5, 2027. That's roughly 200 days.

The "1 year" is now a subscription. Within it, we get reissues whenever the current certificate approaches its expiry. Which is great for the server, and terrible for an app that carries a copy of that certificate.

Doing the Math: How Many App Releases Was Our Workaround Costing Us?

Let's count. With 200-day certificates, that's about two forced app releases per year. With 100-day certificates from 2027, it's closer to four. At 47 days in 2029, we'd be looking at around eight releases a year that exist purely because a certificate rotated.

Each one needs a build, QA, signing, distribution and a fleet-wide rollout that can't miss a single device. That's not a workaround anymore. That's a part-time job.

Hunting for a Permanent Fix: Every Option on the Table

We set a simple goal: the app should never need to change when the server certificate changes. Here's everything we considered on the way there.

Certificate Pinning: Security Blanket or Ticking Time Bomb?

Pinning means telling the app "only accept this specific certificate or public key." It protects against a rogue or compromised CA issuing a certificate for a domain, which is genuinely valuable for high-risk apps.

But pinning also means the app now has opinions about the server's certificates, and those opinions only change when the app changes. Pin the leaf certificate and it's right back on the renewal treadmill. Pinning is a security blanket that can quietly turn into a ticking time bomb.

Pinning the Root Instead of the Leaf: Better, But Not Forever

Pinning the root CA instead of the leaf sounds smarter. The root lives for decades, so renewals won't break anything, right?

Mostly. Until the CA reorganizes its hierarchy. Which is exactly what happened to us: our renewed certificate moved from the older SSL.com root family to the 2022 root. A root pin would have broken on that renewal just as badly as a leaf pin. Switching CAs would break it too.

Backup Pins and Expiration Dates: Building a Safety Net

If we do pin, Android offers two important safety features. We can pin multiple keys (the connection succeeds if any of them matches), and we can set an expiration date after which pinning stops being enforced:

<network-security-config>
    <domain-config>
        <domain includeSubdomains="true">example.com</domain>
        <pin-set expiration="2028-01-01">
            <pin digest="SHA-256">BASE64_HASH_OF_PRIMARY_KEY=</pin>
            <pin digest="SHA-256">BASE64_HASH_OF_BACKUP_KEY=</pin>
        </pin-set>
    </domain-config>
</network-security-config>

With an expiration date, the worst case becomes "pinning quietly switches off" instead of "every old install is bricked." Better. But still not the permanent, zero-maintenance fix we were after.

Reusing the Same Key Pair Across Renewals: Clever But Risky

Another trick: generating the CSR from the same private key at every renewal. The certificate changes, but the public key doesn't, so a public key pin keeps working forever.

It's clever, and it puts us in control instead of the CA. The downside is that the key never rotates. If it's ever compromised, it forces an emergency app update anyway. Key rotation exists for a reason.

Running a Private CA for App-Only APIs

For APIs that only the app talks to, one option is a private CA: create a long-lived root, issue our own server certificates, and have the app trust only that root. We'd control every renewal, and no public CA could change the rules on us.

The trade-offs are real, though. Browsers won't trust those certificates, so separate app-only endpoints are needed. And we'd become responsible for protecting a root CA key, which is a serious operational commitment.

The Radical Option: Letting the OS Trust Store Do Its Job

The option that sounded almost too simple: remove all custom trust from the app and let Android's system trust store decide, like every normal app does.

No pins, no bundled certificates, no custom TrustManager. The app would never care what the server certificate looks like, as long as it chains to a root the device trusts. The catch was obvious: Android 13 doesn't trust our root. So the real question became: can we make the server present a chain that Android 13 already trusts?

That's the question that cracked it.

Detective Mode: Diagnosing the Chain Like a Pro

openssl s_client: The Stethoscope Every Sysadmin Needs

If TLS is a patient, openssl s_client was our stethoscope. It connects to a server exactly like a TLS client would and shows everything the server presents:

openssl s_client -connect api.example.com:443 \
  -servername api.example.com -showcerts </dev/null

I wasn't very familiar with openssl, so these commands were mostly AI-generated. They served the purpose well, though!

The -servername flag matters. It sends SNI, so the server presents the certificate for that hostname. Without it, we might have been looking at a default certificate and diagnosing the wrong thing entirely.

Counting Certificates in the Handshake

Near the top of the output is the certificate chain section. To see just the essentials, we used:

openssl s_client -connect api.example.com:443 -servername api.example.com \
  -showcerts </dev/null 2>/dev/null | grep -E "^ *[0-9]+ s:|i:"

I wasn't very familiar with openssl, so these commands were mostly AI-generated. They served the purpose well, though!

Each numbered entry is a certificate the server sent, with its subject (s:) and issuer (i:). Certificate 0 is always the leaf. If there's only a 0, something's wrong.

Suspect Number One: The Missing Intermediate

The single most common cause of "Trust anchor for certification path not found" is a server that only sends its leaf certificate and forgets the intermediate. Browsers often paper over it by fetching or caching the missing intermediate. Android doesn't, and fails with exactly our error.

HAProxy is especially prone to this, because it reads the certificate, the chain and the private key from one combined PEM file. If that file only contains the leaf and the key, HAProxy only sends the leaf. So that was our first suspect.

Plot Twist: The Chain Was Complete All Along

We opened our HAProxy PEM file, fully expecting to find a lonely leaf certificate. Instead, we found everything: the private key, the leaf certificate, the intermediate, and the root. A textbook complete chain.

So the chain wasn't missing anything. It just ended at the wrong place: the self-signed "SSL.com TLS RSA Root CA 2022", a root our Android 13 tablets had never met. The problem wasn't what was missing from the chain. It was where the chain led.

Dissecting the PEM File: Who Signed Whom?

We didn't need to decode base64 by hand to see what's inside a PEM bundle. This one-liner lists every certificate with its subject and issuer:

openssl crl2pkcs7 -nocrl -certfile star.example.com.pem \
  | openssl pkcs7 -print_certs -noout

I wasn't very familiar with openssl, so these commands were mostly AI-generated. They served the purpose well, though!

Our original bundle looked like this:

subject=CN = *.example.com
issuer=C = US, O = SSL Corporation, CN = SSL.com TLS Issuing RSA CA R1
 
subject=C = US, O = SSL Corporation, CN = SSL.com TLS Issuing RSA CA R1
issuer=C = US, O = SSL Corporation, CN = SSL.com TLS RSA Root CA 2022
 
subject=C = US, O = SSL Corporation, CN = SSL.com TLS RSA Root CA 2022
issuer=C = US, O = SSL Corporation, CN = SSL.com TLS RSA Root CA 2022

Look at the last entry: subject and issuer are identical. That's a self-signed root, and it's the one Android 13 doesn't have.

Verify Return Codes and What They're Trying to Tell You

At the bottom of the s_client output, there's a line like Verify return code: 0 (ok). It's useful, but with one big caveat: it reflects the trust store of the machine it runs on, not the tablet. Our laptops trust the 2022 root, so they happily reported 0 (ok) while the tablets failed.

A few codes worth knowing: 0 (ok) means the chain validated against the local trust store. 21 (unable to verify the first certificate) usually means a missing intermediate. 20 (unable to get local issuer certificate) and 19 (self-signed certificate in certificate chain) mean the chain leads to a root the machine doesn't trust.

To really see what the tablet sees, we simulated its trust store. We downloaded the old root the tablet does trust ("SSL.com EV Root Certification Authority RSA R2", also available from the SSL.com repository) and verified against that root only:

# Split the bundle: leaf in one file, everything else as "untrusted" helpers
openssl verify -no-CApath -no-CAstore \
  -CAfile SSLcom-EV-Root-RSA-R2.pem \
  -untrusted chain-without-leaf.pem \
  leaf.pem

I wasn't very familiar with openssl, so these commands were mostly AI-generated. They served the purpose well, though!

With our original bundle, this fails, exactly like the tablet. After the fix, it prints leaf.pem: OK. It's a fantastic way to test for old devices without leaving the terminal.

The Hero of This Story: Cross-Signed Certificates

What Cross-Signing Actually Means (Same Root, Two Signatures)

A certificate is basically a statement: "this public key belongs to this name," signed by an issuer. Nothing stops two different issuers from signing the same statement.

That's cross-signing. The 2022 root exists in two forms. One is self-signed: the 2022 root vouching for itself. The other is a cross-certificate: the exact same name and the exact same public key, but signed by SSL.com's older EV Root Certification Authority RSA R2.

Same identity, two signatures. Anything the 2022 root signed can now be traced up either path.

Lessons From Let's Encrypt and the Great DST Root Expiry

This isn't a niche trick. It's how new CAs have bootstrapped trust for years. Let's Encrypt launched with its ISRG Root X1 cross-signed by IdenTrust's DST Root CA X3, a root that devices already trusted, so certificates worked everywhere from day one.

When DST Root CA X3 expired in September 2021, Let's Encrypt kept older Android devices working with a longer-lived cross-sign, taking advantage of the fact that Android doesn't enforce the expiry date of roots in its trust store. That arrangement kept ancient Android phones connected until 2024. Cross-signing has been quietly rescuing old devices for a long time.

How Older Devices Build a Path Through a Root They Already Trust

Once we swapped the self-signed 2022 root in our chain for the cross-signed one, here's what Android 13 sees:

*.example.com                                         (leaf)
  └─ issued by SSL.com TLS Issuing RSA CA R1           (intermediate)
       └─ issued by SSL.com TLS RSA Root CA 2022       (cross-signed copy)
            └─ issued by SSL.com EV Root Certification Authority RSA R2
                                                        already in the tablet's trust store

The tablet climbs one extra step, lands on a root it has trusted all along, and the handshake succeeds. No app changes. No certificate files. No firmware update.

Why Newer Devices Don't Even Notice the Extra Hop

Remember the rule from earlier: a client stops climbing as soon as it reaches a certificate in its trust store. Newer devices and browsers already trust "SSL.com TLS RSA Root CA 2022", so they recognize it as a trust anchor the moment they reach it and never look at who cross-signed it.

Old devices get an extra hop that saves them. New devices don't even notice. Everyone wins.

Shopping the SSL.com Repository Without Picking the Wrong Certificate

SSL.com publishes all its root, intermediate and cross-certificates in its CA repository. It's a long page full of very similar names, so it pays to shop carefully.

Self-Signed vs. Cross-Certificate: Spotting the Difference

"SSL.com TLS RSA Root CA 2022" is listed more than once. The difference is in the details. The self-signed version shows Issuer: Self-signed, and it's the one that was already in our bundle. The cross-certificate shows a different issuer and is explicitly labeled Type: Cross-certificate.

If the issuer says "Self-signed," we keep scrolling.

RSA or ECC? Matching the Cross-Cert to Our Chain

The repository offers cross-certificates for both the RSA and ECC 2022 roots. The right one matches the chain. Our intermediate is "SSL.com TLS Issuing RSA CA R1", so we needed the RSA one. To double-check, we asked the certificate itself:

openssl x509 -in star.example.com.crt -noout -text | grep "Public Key Algorithm"

I wasn't very familiar with openssl, so these commands were mostly AI-generated. They served the purpose well, though!

rsaEncryption means RSA. id-ecPublicKey means ECC.

The Winner: TLS RSA Root CA 2022 Cross-Signed by EV Root CA RSA R2

The certificate that solved our problem:

Subject: CN = SSL.com TLS RSA Root CA 2022, O = SSL Corporation, C = US
Issuer:  CN = SSL.com EV Root Certification Authority RSA R2, O = SSL Corporation,
         L = Houston, ST = Texas, C = US
Type:    Cross-certificate
Valid:   2023-11-07 to 2038-11-03
SHA-256: 6B:CB:14:7D:41:E9:B5:77:F4:00:02:A5:C4:BA:08:84:
         67:37:80:5A:AF:E8:55:4D:D0:E7:22:04:F6:00:7B:A7
File:    SSLcom-TLS-Root-2022-RSA_cross.pem

It's signed by a root our tablets already trust, and it's valid until 2038. That's a lot of renewals without a single app release.

The Backup Plan: The SecureTrust CA Cross-Signed Alternative

Before committing, we checked that our tablets actually trust the cross-signer by looking for "SSL.com EV Root Certification Authority RSA R2" under Trusted credentials. It was there, so we were good to go.

If it isn't, the same repository page has another option in a section called "SecureTrust CA Cross-signed SSL.com Root Certificates": the same 2022 root, cross-signed by SecureTrust CA, a long-established root, valid until the end of 2029. What matters is picking the cross-signer the devices already trust.

Double-Checking Subject, Issuer and Expiry Before Committing

Once downloaded, we verified it before it went anywhere near production:

openssl x509 -in SSLcom-TLS-Root-2022-RSA_cross.pem -noout \
  -subject -issuer -enddate -fingerprint -sha256

I wasn't very familiar with openssl, so these commands were mostly AI-generated. They served the purpose well, though!

The subject should be the 2022 root, the issuer should be EV Root Certification Authority RSA R2 (not the 2022 root itself), the end date should be November 2038, and the fingerprint should match the one published in the repository. Four checks, thirty seconds, zero regrets.

HAProxy Surgery: Swapping the Root Without Dropping a Single Request

Time to operate. The good news: the whole fix is replacing one certificate block in one file, and HAProxy can reload without dropping connections.

Anatomy of an HAProxy PEM: Key, Leaf, Intermediate, Root

HAProxy expects everything for a TLS endpoint in a single PEM file: the private key, the leaf certificate and the chain. Our bundle was ordered like this:

-----BEGIN PRIVATE KEY-----          ← private key
-----BEGIN CERTIFICATE-----          ← leaf: *.example.com
-----BEGIN CERTIFICATE-----          ← intermediate: SSL.com TLS Issuing RSA CA R1
-----BEGIN CERTIFICATE-----          ← root: SSL.com TLS RSA Root CA 2022 (self-signed)

The plan: keep the first three blocks exactly as they are, and swap the last one for the cross-signed version.

Finding the Right File From the crt Directive

First, we confirmed which file HAProxy was really serving. The answer was in the crt option of our HTTPS bind line:

grep -n "crt" /etc/haproxy/haproxy.cfg
# bind *:443 ssl crt /etc/haproxy/certs/star.example.com.pem

In setups where crt points to a directory, HAProxy loads every PEM in it and picks by SNI, and with crt-list, the file list lives there instead. Editing the wrong file is a classic way to "fix" nothing for an hour.

Backups First, Heroics Later

Before touching anything:

cd /etc/haproxy/certs
sudo cp star.example.com.pem star.example.com.pem.bak

If anything goes sideways, restoring is one cp and one reload away. Future us would be grateful.

One awk Command to Drop the Self-Signed Root

This awk command counts certificate blocks as it reads the file and keeps everything before the third certificate, which means the key, the leaf and the intermediate:

sudo awk '/-----BEGIN CERTIFICATE-----/{n++} n<3' star.example.com.pem > /tmp/new.pem

It assumes the order shown above (key first, then leaf, intermediate, root). A bundle in a different order would need to be rebuilt from the individual files instead.

Appending the Cross-Signed Root in the Right Place

Next, we added the cross-signed root at the end, where the self-signed one used to be:

cat SSLcom-TLS-Root-2022-RSA_cross.pem >> /tmp/new.pem

Order matters in a certificate chain: leaf first, then each issuer after the certificate it signed. The cross-signed root signed the intermediate, so it goes after the intermediate.

Listing Every Certificate in the Bundle to Confirm the Order

Trust, but verify (pun still intended):

openssl crl2pkcs7 -nocrl -certfile /tmp/new.pem | openssl pkcs7 -print_certs -noout

I wasn't very familiar with openssl, so these commands were mostly AI-generated. They served the purpose well, though!

We got exactly three certificates:

subject=CN = *.example.com
issuer=... CN = SSL.com TLS Issuing RSA CA R1
 
subject=... CN = SSL.com TLS Issuing RSA CA R1
issuer=... CN = SSL.com TLS RSA Root CA 2022
 
subject=... CN = SSL.com TLS RSA Root CA 2022
issuer=... CN = SSL.com EV Root Certification Authority RSA R2

The last issuer is the one that matters. If it still says "SSL.com TLS RSA Root CA 2022", the self-signed version was appended by mistake.

Validating the Config and Reloading Without Downtime

We installed the new bundle, locked down its permissions, validated, and reloaded:

sudo cp /tmp/new.pem /etc/haproxy/certs/star.example.com.pem
sudo chmod 600 /etc/haproxy/certs/star.example.com.pem
sudo haproxy -c -f /etc/haproxy/haproxy.cfg
sudo systemctl reload haproxy

We only reloaded after haproxy -c reported that the configuration was valid. A reload starts fresh workers with the new certificate while the old ones finish serving their existing connections, so nobody gets kicked out mid-request.

The Moment of Truth: Proving the Fix Works

Confirming the New Chain From the Outside World

First, we confirmed that the live server was actually sending the new chain:

openssl s_client -connect api.example.com:443 -servername api.example.com \
  -showcerts </dev/null 2>/dev/null | grep -E "^ *[0-9]+ s:|i:"

I wasn't very familiar with openssl, so these commands were mostly AI-generated. They served the purpose well, though!

The last i: line should now read "SSL.com EV Root Certification Authority RSA R2". Then we ran the old-device simulation from the detective section again. When openssl verify against only the old EV root printed OK, we'd proven Android 13 could build the path, before touching a single tablet.

Testing on a Real Android 13 Biometric Tablet

Simulations are great. Real hardware is better. We built a test version of the app with the bundled certificate removed, installed it on an Android 13 biometric tablet, force-stopped it, and hit the version check endpoint.

Status 200. Real response body. Real duration. After years of shipping certificates around, it felt almost suspicious how boring it was.

Deleting the Bundled Certificate From the App (Goodbye, Old Friend)

With the server fixed, the app no longer needs its own certificate. We deleted the file from res/raw and removed the code that loaded it (teams with a custom TrustManager or CertificatePinner built around a bundled certificate would remove those too).

This is the most satisfying git rm we ran all year.

Cleaning Up network_security_config So the System Store Takes Over

If we still want a network security config (to block cleartext traffic, for example), the key is making sure it trusts the system store and nothing else:

<!-- res/xml/network_security_config.xml (the new way) -->
<network-security-config>
    <base-config cleartextTrafficPermitted="false">
        <trust-anchors>
            <certificates src="system" />
        </trust-anchors>
    </base-config>
</network-security-config>

The other option is removing the config and its manifest reference entirely. Either way, the app now trusts exactly what the OS trusts, like every well-behaved Android app.

Plot Twist Number Two: Old App Versions Work Again Too

Here's a bonus we didn't see coming: the older builds of the app, the ones still carrying an outdated certificate file, started working again too.

The reason is that Android's trust anchors are additive. The old network security config listed two sources of trust: the bundled file and system. During validation, Android only needs one valid path from the server's certificate to any trust anchor. The outdated bundled file no longer matches anything the server sends, so Android simply never uses it. But the chain now leads through the cross-signed root to SSL.com EV Root Certification Authority RSA R2, which sits in the system store, and the old config still trusts the system store.

A stale trust anchor that's never used is harmless. The system store does the work, and the old file just sits there like a forgotten gym membership card.

In practice, that means tablets that missed an update (the ones sitting in a drawer in a remote office) reconnect the moment they're switched on. No emergency update campaign, no field trips to reinstall the app.

One important caveat: this only works because the old config included <certificates src="system" />. A build that trusted only its bundled file, or pinned a certificate or key that has since changed, ignores the system store and still needs an update. It's worth checking our old configs before promising anyone a miracle.

Regression Check: Making Sure Newer Devices and Browsers Still Smile

Fixing old devices must not break new ones. We tested on newer Android versions, desktop browsers, curl and our API tools. All of them still stop at the 2022 root they already trust and never notice the cross-signature.

For a thorough external check, the Qualys SSL Labs server test is a great option. It shows every trust path a chain supports and flags chain problems, so it's a great way to double-check that nothing got weird.

Gotchas, Traps and Things That Almost Went Wrong

Browsers Quietly Fixing Broken Chains Behind Our Backs

Worth repeating, because it fools everyone: browsers are not a reliable test for TLS chains. They fetch missing intermediates, cache certificates from other sites and ship newer trust stores than our devices. What actually told us the truth: openssl s_client, a simulated old trust store, and the actual app on the actual hardware.

Load Balancers and Gateways: Where TLS Actually Terminates

It's worth knowing where TLS actually terminates. In our case, it was HAProxy. In setups with a load balancer, CDN or API gateway in front of the server, that's the component presenting certificates to clients, and that's where the chain needs to be fixed. Managed load balancers often have a separate "certificate chain" field that's very easy to leave empty. When our fixes seem to have no effect, s_client quickly shows whether we're looking at the right box.

PEM Files With Missing Newlines and Glued-Together Blocks

Some certificate files don't end with a newline. Concatenate a few of those with cat and the result is lines like -----END CERTIFICATE----------BEGIN CERTIFICATE-----, which quietly corrupt the bundle. A quick check:

grep -n "END.*-----.*BEGIN" star.example.com.pem

No output is good output. If it prints a line, we split it so each -----BEGIN starts on its own line.

Grabbing the Self-Signed Root Instead of the Cross-Cert

The self-signed root and the cross-certificate have the same name and live on the same repository page. Grabbing the wrong one changes nothing and can easily cost us an afternoon of head-scratching. The issuer line never lies, which is why we always check it before installing.

Leftover Trust Anchor Configs That Ignore the System Store

On the app side, a leftover network_security_config.xml can sabotage the whole fix. If its trust anchors list only a bundled certificate, without <certificates src="system" />, the app ignores the system store entirely. Deleting the bundled certificate while keeping that config leads to the very same "Trust anchor not found" error with a perfectly good server.

The New Renewal Playbook: Boring, Predictable and Beautiful

Renewal day should be boring. Here's how we keep it that way.

Step-by-Step Checklist for Every Future Reissue

1. Reissue the certificate in the SSL.com dashboard and download the new leaf certificate and intermediate.
2. Build the HAProxy bundle: private key, leaf, intermediate, cross-signed root.
3. List the certificates in the bundle and confirm the last issuer is EV Root Certification Authority RSA R2.
4. Verify the chain against the old EV RSA R2 root only.
5. Validate the HAProxy config and reload.
6. Confirm the live chain with openssl s_client.
7. Touch the mobile app: nope, not anymore.

Keeping the Cross-Signed Root in the Chain (Every Single Time)

The one thing that must never be forgotten: the cross-signed root stays in the bundle at every renewal. SSL.com's download will typically include the self-signed root, and if that slips back in, the Android 13 error comes right back. Keep the cross-certificate file somewhere obvious, right next to our renewal notes, so nobody has to go hunting through the repository again.

Calendar Reminders for 2038 and Other Distant Deadlines

The cross-certificate is valid until November 3, 2038. That feels like forever, which is exactly how deadlines sneak up on people. A reminder in a shared calendar a year or so before it expires is cheap insurance (before the end of 2029 for anyone using the SecureTrust variant). When the time comes, we'll check the repository for whatever SSL.com offers next.

It's also worth reviewing whether our fleet has changed by then. If the Android 13 tablets have been retired, the cross-certificate may quietly stop mattering.

Monitoring Certificate Expiry Before Users Notice

With certificate lifetimes heading toward 47 days, expiry monitoring isn't optional. Most uptime monitoring tools can alert us when a certificate is about to expire, and in Prometheus setups, the blackbox exporter's probe_ssl_earliest_cert_expiry metric does the same job. The earlier the alert, the better. Thirty days ahead is a comfortable buffer.

Just as important: after every renewal, the chain check from the "Moment of Truth" section is worth re-running to confirm the last issuer is still EV Root Certification Authority RSA R2. That thirty-second look is what saves us from a sneaky regression after a hasty renewal.

Trust, Finally Earned: Wrapping Up

From App Release to Server Tweak: What Actually Changed

Before: every certificate renewal meant a new certificate file in the app, a new build, QA, signing and a nerve-wracking rollout to a fleet of field tablets.

After: the app carries no certificates at all and trusts the Android system store like any normal app. Renewals happen entirely on the server. The Android 13 tablets connect through a cross-signed certificate that stays valid until 2038. Even old app builds still carrying an outdated certificate file reconnect, because they trust the system store too. The whole fix was swapping one certificate block in one PEM file on HAProxy.

Why Fixing Trust at the Server Beats Patching Every Client

When a trust problem shows up on clients, it's tempting to fix it on the clients. But we have one server and many clients, and the clients are the part we control least. Every certificate we baked into the app was a promise that the server would never change, and with certificate lifetimes shrinking every year, that promise gets harder to keep.

The server decides which chain it presents. When we present a chain that every client can validate, the clients don't need any special treatment at all.

Final Thoughts for Teams Running Devices That Can't Update

For those of us running hardware that's frozen on an older Android version (biometric tablets, POS terminals, kiosks, rugged scanners), our devices' trust stores are frozen too. As CAs roll out new roots, we'll keep running into this problem in new forms.

What helped us most: knowing which roots our devices actually trust, being able to read a certificate chain, keeping an eye out for cross-signed certificates from our CA, testing with a simulated old trust store, and never trusting "it works in my browser."

Most importantly, before we ship another certificate file inside an app, it's worth asking whether the problem can be fixed on the server instead. For us, it could. And our future selves, calmly renewing certificates in ten minutes with a coffee in hand, will be grateful.

More Relevant Resources

• The Chain of Trust: What it is, Key Concepts and Applications (GlobalSign): a friendly primer on root, intermediate and end-entity certificates.
• Network security configuration (Android Developers): the official docs for trust anchors, pinning and cleartext settings.
• Android 14 blocks modification of system certificates, even as root (HTTP Toolkit): how Android 14 moved system certificates into the updatable Conscrypt module.
• Extending Android Device Compatibility for Let's Encrypt Certificates (Let's Encrypt): the classic story of a cross-sign rescuing old Android devices.
• SSL.com CA Repository: SSL.com's roots, intermediates and cross-certificates, with fingerprints.
• Industry Leaders Approve the Move to a 47-Day SSL Certificate Validity Period (SectigoStore): the SC-081 timeline and what it means for renewals.
• Qualys SSL Labs Server Test: a free, thorough external check of your TLS setup and certificate chains.
• openssl s_client manual (OpenSSL): every flag of the stethoscope, explained.