Your Let’sEncrypt Certificates Could Be Leaking Your (Company) Secrets

Photo by chris panas on Unsplash

Let’s Encrypt is a fantastic service that allows anyone to generate an SSL certificate for their domain for FREE. It’s backed and funded by the biggest players in the industry, with the goal of creating a more secure Internet by enabling “SSL Everywhere”.

Many developers use it to generate certificates that encrypt their web traffic using HTTPS / TLS / SSL, but the way that they’re doing so may inadvertently leak their entire client list to anyone on the web. From a business perspective this can be catastrophic — especially when Non-Disclosure Agreements are in place to prevent such disclosure.

In this article, I’ll demonstrate the issue and provide some solutions that will keep your client list secure from this type of accidental disclosure.

The Problem

Background

First, I must say: I LOVE Let’sEncrypt. Coming from the old days of the web when you had to pay the Thawte / Verisign monopoly hundreds of dollars a just to secure your website traffic, I see Let’sEncrypt as a necessary step in the evolution of the free (as in freedom) secure web.

However, it does have some requirements and pitfalls — one in particular that leads to this issue. Certificates generated by LetsEncrypt are only valid for 90 days (after which point they need to be renewed). This is actually a great security feature, as it limits the scope of key compromise and ensures that certificates for domains are always being refreshed. Much like rotating your passwords every 90 days, it is a cumbersome but critical task from a security perspective.

Fortunately, auto-renewal is a relatively straight-forward task. The developer needs to set up a CRON job that refreshes the certificate (which involves re-validating the domain) and then install that refreshed certificate onto their web or application server.

For developers using Kubernetes, there’s an application called cert-manager that connects with your Ingress controller to automatically renew the certificates on a regular schedule and install those new certificates in the ingress controller.

Unfortunately — this process also creates an opening for developers to make a mistake. And, it turns out, a lot of them are making it.

The Fatal Flaw: Issuers

I’ll use Cert-Manager as the example, but this flaw can be found in any script that performs the same or similar task of auto-renewing Let’sEncrypt certificates.

Cert-Manager uses a Custom Resource Definition (CRD) in Kubernetes called an “Issuer” (cluster-wide issuers are called a “ClusterIssuer”). These issuers tell cert-manager (and by extension LetsEncrypt) how to validate that a domain is owned by the person or process requesting the certificate. This validation (also called a “challenge”) happens either by adding a piece of content at a given URL on the domain (called HTTP01 Challenge) or via DNS by adding and then validating a txt DNS record on that domain (called a DNS01 Challenge)

An issuer can resolve that challenge, and therefore issue certificates, on multiple domains at the same time and can generate a certificate that’s valid for any of the domains that the issuer has validated. And this, dear reader, is where we encounter a problem.

FIRST, AN EXAMPLE

To find an example of this flaw in action, I needed only to look as far as my own website.

I run a Platform as a Service incubator called Propl Labs, and I run a Kubernetes cluster for ProplLabs that hosts services for several of the startup experiments that the lab fosters. These services all run in the same Kubernetes namespace, and because they are part of the same organization and hosted in the same Amazon AWS account, I use the same ClusterIssuer to resolve the DNS challenges for those domains. This means that the certificate for my main site is valid for certificates on the rest of the sites.

Want to see which domains I’m hosting?

First, navigate to the URL (www.propl.us). Then, click on the lock icon next to my URL to view the certificate.

Click the “Certificate (valid)” button.

Finally, open the details toggle below the domain-specific certificate:

and scroll down to theSubject Alternative Namesection.

Boom — there it is, for the world to see. All of the URL’s that the certificate is valid for.

I initially discovered this unintentional disclosure on the website of a very prominent open source project and found a HUGE list of domains in the Subject Alternative Name section that were likely clients of the maintainers of that project. Given the nature of these sites, I find it highly likely that these DNS names are actually clients of the projects creators and that this disclosure was unintentional.

The Fix

So how do you prevent the accidental disclosure of a large part of your client list?

The answer lies in the Issuers. In Cert-Manager, the best way to fix this problem is to create a new issuer for every client or customer domain, rather than adding new domains to the existing Issuer (or ClusterIssuer).

By creating a different issuer for each will ensure that the Issuer generates different certificates for each of your clients. That way, when the certificates are generated and auto-renewed, the Subject Alternative Name list only contains the names of URL’s valid for that site.

Security Ingress in Kubernetes and Amazon Route 53

You can create a new cluster issuer for each client using a DNS challenge on Amazon AWS, you can use the following YAML.

apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: <clientname>-issuer
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: youremail@yourdomain.tld
    privateKeySecretRef:
      name: <clientname>-certificate
    # ACME DNS-01 provider configurations
    solvers:
    - selector:
        dnsZones:
          - "*.<clientdomain>.tld"
      dns01:
        route53:
          region: us-west-2
          accessKeyID: <your_AWS_access_key>
          secretAccessKeySecretRef:
            name: aws-creds
            key: AWS_SECRET_KEY

NOTE: This also expects a Kubernetes SECRET with the name aws-creds (exactly this) and a single key of AWS_SECRET_KEY (exactly this). You can use the YAML below as an example:

apiVersion: v1
kind: Secret
metadata:
  name: aws-creds
type: Opaque
stringData:
  AWS_SECRET_KEY: "<your_AWS_secret_here>"

Then, for the ingress of your client, just make sure to have the following annotation:

cert-manager.io/cluster-issuer: <clientname>-issuer

And a “TLS” section in your ingress that determines which certificate to use and with which domain for that ingress

tls: # < placing a host in the TLS config will indicate a certificate should be created
  - hosts:
    - *.<clientdomain>.tld
    secretName: <clientname>-certificate

Combined together, the ingress might look something this:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: <clientname>-issuer
  name: myIngress
  namespace: myIngress
spec:
  rules:
  - host: <clientdomain>.tld
    http:
      paths:
      - backend:
          serviceName: myservice
          servicePort: 80
        path: /
  tls:
  - hosts:
    - clientdomain.tld
    secretName: <clientname>-certificate

Conclusion

In this article I demonstrated a subtle but very serious issue with Certificates issued by Let’sEncrypt could potentially expose your client list. I also demonstrated a technique for securing ingresses using Kubernetes and cert-manager. While this article discussed how to mitigate this risk on Kubernetes, it’s an issue that can occur for anyone that uses Let’s Encrypt Certificates. Keep an eye out for more, and if you or your organization need help with anything Kubernetes, Rancher, Server Ops, or Software, drop me a line at john@propl.us and ProplLabs can help.