The fake website that kept our mail server certificate alive

A website nobody visited

While cleaning up an old server, we found a web virtual host serving a single index.html from 2018. No links pointed to it, no one used it, and it had its own TLS certificate.

It turned out to be the most important website on the machine. It existed so that the mail server could have a certificate.

Let's Encrypt's most common validation method, HTTP-01, proves you control a hostname by serving a token over HTTP on that hostname. Our mail server — Postfix for SMTP, Dovecot for IMAP and POP3 — does not speak HTTP. So at some point someone set up a web host for mail.example.net, let certbot's Apache plugin answer the challenges there, and pointed Postfix and Dovecot at the resulting certificate files. It worked, and it kept working for years.

When we looked closely, it had three problems. One of them had a timer on it.

Problem one: the mail certificate depended on a web server

The obvious one. We were in the middle of moving websites off that machine. Once the last site was gone, Apache would have had exactly one job left: answering certificate challenges for a service that is not a website. Switch it off, and the mail certificate stops renewing roughly ninety days later — long after anyone remembers why.

Problem two: nobody told Dovecot

Renewal replaces the certificate files on disk. The services using them have to pick up the new files, and the setup had nothing to make that happen. The only renewal hook on the machine belonged to an unrelated service.

Postfix mostly copes: its worker processes are recycled regularly and read the files again when they start. Dovecot reads its certificate when it starts and keeps it in memory. After a renewal, it carries on presenting the old certificate — valid for a while longer, which is exactly why nobody notices — until the day that old certificate expires and every IMAP client starts showing a security warning.

At the moment we looked, it was fine, and only by luck: Dovecot had happened to be restarted for an unrelated reason a couple of weeks after the last renewal. The next renewal would not have been so lucky.

So we checked what the services were actually presenting, rather than what the files on disk said:

openssl s_client -connect 127.0.0.1:993 </dev/null | openssl x509 -noout -subject -enddate
openssl s_client -starttls smtp -connect 127.0.0.1:587 </dev/null | openssl x509 -noout -subject -enddate

That is the only check that matters. A certificate on disk is a promise; the one in the handshake is the fact.

Problem three: one name out of six

The server handles mail for several of our customers' domains, and users configure their mail clients with the name that matches their domain: mail.example.com, mail.example.org and so on. All of those names pointed at the same machine. The certificate covered exactly one of them.

Anyone who had set up their client with their own domain's mail name was either looking at a certificate warning or had clicked "trust" long ago and forgotten about it. Neither is a good state for mail.

Server-to-server SMTP was unaffected — mail servers use opportunistic TLS and generally do not check names — which is why this could stay hidden: mail kept flowing.

The fix: prove control of DNS, not of a web page

Let's Encrypt has a second validation method, DNS-01: instead of serving a token over HTTP, you publish it as a TXT record at _acme-challenge.<hostname>. No web server involved, no port 80 required.

It is especially convenient when you run your own authoritative DNS. The same machine was the primary name server for every one of those domains, and it already had a TSIG key used by certbot on another host. certbot's dns-rfc2136 plugin sends signed dynamic updates straight to BIND:

# /etc/letsencrypt/rfc2136.ini   (chmod 600)
dns_rfc2136_server    = 127.0.0.1
dns_rfc2136_port      = 53
dns_rfc2136_name      = acme-key
dns_rfc2136_secret    = ...
dns_rfc2136_algorithm = HMAC-SHA512

Each zone then gets a grant — and it is worth keeping it as narrow as possible. The key may change exactly one TXT record per zone, the challenge name for the mail host, and nothing else:

zone "example.com" {
    type master;
    file "example.com";
    update-policy {
        grant acme-key name _acme-challenge.mail.example.com. TXT;
    };
};

If that key ever leaks, the worst anyone can do with it is issue a certificate for a mail hostname — not rewrite the zone.

Test the plumbing before involving Let's Encrypt

Dynamic DNS updates fail in unhelpful ways: a wrong key name, a grant with a missing trailing dot, a zone file BIND is not allowed to write. Rather than finding that out through a failed certificate request, we tested each zone directly:

nsupdate -k /etc/bind/acme.key <<EOF
server 127.0.0.1
update add _acme-challenge.mail.example.com. 60 TXT "test"
send
EOF

dig +short TXT _acme-challenge.mail.example.com @127.0.0.1   # expect "test"
# ...then the same with "update delete" to clean up

That caught the one real obstacle: several zone files were owned by root, so BIND could not write the changes back. Making them writable by BIND solved it — with a consequence covered below.

Only then did we ask for a certificate, first as a dry run against Let's Encrypt's staging environment, then for real, with every mail hostname on it:

certbot certonly --dns-rfc2136 \
    --dns-rfc2136-credentials /etc/letsencrypt/rfc2136.ini \
    --dns-rfc2136-propagation-seconds 30 \
    --cert-name mail.example.net \
    -d mail.example.net -d mail.example.com -d mail.example.org

The propagation wait matters when your zones have secondary name servers: Let's Encrypt may query any of them, and the secondary needs a moment to receive the update.

And tell the services

A deploy hook runs after every successful renewal. Ours reloads the mail services — but only when the renewed certificate is the mail one, so an unrelated renewal does not bounce the mail server:

#!/bin/sh
# /etc/letsencrypt/renewal-hooks/deploy/reload-mail.sh
case "$RENEWED_LINEAGE" in
  */mail.example.net) ;;
  *) exit 0 ;;
esac
systemctl is-active --quiet postfix && systemctl reload postfix
systemctl is-active --quiet dovecot && systemctl reload dovecot
exit 0

A reload, not a restart: open IMAP sessions survive, new connections get the new certificate. The hook ran on its own during the first real issuance, which was the most convincing test it could have had.

With the certificate renewing over DNS and the services reloading themselves, the web host was switched off and its directory archived. A final renewal dry run confirmed the certificate no longer needs a web server at all.

The price: dynamic zones

One trade-off is worth knowing before you copy this. Once BIND accepts dynamic updates for a zone, it owns that zone file. It keeps changes in a journal and periodically rewrites the file in its own canonical format. Editing the file by hand while BIND is running risks losing either your edit or its changes.

Manual edits now go through a freeze and thaw:

rndc freeze example.com
# edit the zone file, bump the serial
rndc thaw example.com

It is a small change in habit, but it is a change for everyone who touches those zones, and it is better announced than discovered.

What we took away

• A certificate for a non-web service should not depend on a web server. If you control DNS, DNS-01 removes the dependency entirely.
• Renewal is two steps. Getting a new certificate is the first; getting every service to present it is the second, and the one that gets forgotten.
• Check the handshake, not the file. openssl s_client against each port tells you what clients actually see.
• List every name users connect with. For mail that is rarely just one.
• Keep DNS update grants narrow. One record type, one name, per zone.