Moving websites off the mail server without anyone noticing

One server, too many jobs

For years one of our machines did everything: it was the mail server, the authoritative DNS server for our domains and our customers' domains, and the web server for every site anyone had ever put on it. Apache, a decade of virtual hosts, certificates for all of them.

Mail and DNS are the two services you least want to disturb. Every website upgrade, every new PHP module, every certificate problem on a site nobody visits happened on the same machine that delivered everyone's mail. So we decided to leave the old server with mail and DNS only, and move the websites to a newer machine that already ran nginx.

The move itself was uneventful, which was the goal. This article is about what made it uneventful, and about the few things that still surprised us.

Step one: find out what is actually alive

The Apache configuration listed far more sites than were really in use. Before moving anything, we checked every enabled virtual host against the outside world: does the domain's DNS still point at this machine, and does the site answer with something meaningful?

dig +short www.example.com A        # does it still point here?
curl -sI https://www.example.com/   # and what does it answer?

That sorted the list into three groups:

• Live sites — DNS pointed here and the site worked. These were the ones to move.
• Zombies — the virtual host was enabled, but the domain had long since moved to another provider, or was no longer registered at all. One "live" site was only a redirect to another site that had been dead for years.
• Directories without a site — old content under the web root with no virtual host pointing at it.

The zombies were switched off and their directories archived, not deleted. Nothing was lost, and the list of things to migrate shrank to a handful of sites.

One small trap here: a couple of the zombie hosts were enabled through symlinks from before the configuration files had to end in .conf. a2dissite did not recognise them, so they had to be removed by hand. If a2dissite says a site does not exist, check what is actually in sites-enabled.

Step two: build the new home in parallel

Each site was set up on the new server while the old one kept serving it. Static sites were copied as they were. Our own React site was cloned from git and built there. The certificates were copied from the old server too, so the new server could answer HTTPS correctly from the first second, before any DNS change.

Everything was then tested against the new server without touching DNS. curl can be told to send a hostname to a specific address:

curl -sI --resolve www.example.com:443:203.0.113.20 https://www.example.com/

For the static sites we went one step further and compared checksums of every page on both servers. When the old and the new server return byte-identical content, there is nothing left to argue about.

Translating Apache to nginx

Most of the Apache configuration translated one to one. The part that needed thought was our single-page application, and it had already bitten us once on the old server.

We had added an .htaccess file to the site's build to set cache headers. Rules in .htaccess do not add to the rewrite rules in the virtual host's <Directory> block — they replace them. The virtual host held the fallback that sends every application route to index.html. The moment the new .htaccess was deployed, that fallback disappeared, and every page except the homepage returned 404. It lasted about two minutes, because we checked the routes right after the deploy.

nginx has no per-directory files, so on the new server everything lives in the server block and the whole behaviour is visible in one place:

location / {
    try_files $uri $uri/index.html =404;
}

nginx has its own version of the same trap. add_header is inherited from the server block only by locations that do not define any add_header of their own. Add one Cache-Control header to a location, and every security header from the server level silently disappears for those URLs. The fix is dull but reliable: repeat the security headers in every location that sets its own.

Step three: lower the TTL a day before

Our zones used a TTL of twelve hours. Changing an A record with a twelve-hour TTL means that some visitors keep going to the old address for up to twelve hours, and if anything goes wrong, rolling back takes just as long.

So the day before the move, we lowered the TTL of only the records that would change to five minutes. Then we waited for the old TTL to expire. Lowering the TTL does nothing for resolvers that already cached the record under the old value — they keep it until the old twelve hours are up.

After that, the switch itself was changing a few A records, and a rollback would have taken five minutes. It was not needed.

Two details are worth knowing:

• Some of our zones accept dynamic updates (certbot uses them for DNS validation). BIND owns those zone files, so every manual change went through rndc freeze, the edit, and rndc thaw.
• For a few hours after the switch, some resolvers still returned the old address. That was harmless, because the old server was still serving exactly the same content. It does mean that on the day of the switch you either deploy to both servers or to neither.

Step four: make the certificates renew on the new server

Copied certificates are valid, but a copied certificate is not a renewing one. Each certificate's renewal configuration still referred to the old server's Let's Encrypt account and to the Apache plugin. Neither existed on the new server.

We changed every renewal configuration to the new server's account and to the nginx plugin, then checked each one with a dry run:

certbot renew --cert-name www.example.com --dry-run

A dry run talks to Let's Encrypt's staging environment, and that environment is sometimes overloaded. Twice, a dry run failed with rateLimited: Service busy; retry later, and once it simply hung. That message is about their capacity, not about your configuration. Try again later before you start changing things.

The dry run we repeated a few days later found something the first ones had not. Two of the copied renewal files still carried hooks from the old server:

post_hook  = systemctl reload apache2
renew_hook = systemctl reload postfix dovecot

The new server has no Apache and no mail server. The certificate itself renewed, but every renewal would have ended with an error, and an error in the renewal log is exactly what makes people start ignoring the renewal log. The nginx plugin reloads nginx on its own, so both lines were simply removed.

When you copy /etc/letsencrypt between servers, read every file in renewal/ as if it were new. It describes the old server, not the new one.

Step five: switch off the old sites

A few days later, with no real visitors left on the old server for these domains, it was time to remove the old virtual hosts and certificates.

This produced the only real scare of the whole move. After disabling the sites and deleting their certificates, apache2ctl configtest failed. One of the HTTPS virtual hosts had been created years earlier by certbot as a regular file in sites-enabled, not as a symlink. a2dissite only removes symlinks, so it left that file in place, still pointing at a certificate that no longer existed.

Nothing went down. The test ran before the reload, and Apache keeps running with the configuration it already has in memory. Moving the stray file out fixed the test. Had we reloaded without testing, Apache would have refused to start again — and it was still serving the sites that had not moved yet.

Before switching sites off, it is worth a look at what is really in there:

find /etc/apache2/sites-enabled -type f    # regular files, not symlinks

The part everyone forgets

A migration leaves temporary things behind. Ours were:

• the five-minute TTLs, which stayed on the moved records for several days before we put them back to twelve hours;
• the old server's deploy key, still able to read the repository;
• temporary administrative access set up on the new server to make the move easier.

None of these are dangerous on the day of the move. All of them are easy to forget, because nothing breaks when you forget them. Write them down while you create them.

What we took away

• Take an inventory before you migrate. Half of what was configured was no longer in use, and did not need to be moved at all.
• Build in parallel and test without DNS. curl --resolve and checksums let you prove the new server is ready before anyone can reach it.
• Lower the TTL a day ahead, and wait for the old one to expire. The switch and the rollback then take minutes.
• Copied certificates are not renewing certificates. Rewrite the renewal configuration for the new server and read every hook in it.
• Run configtest before every reload, especially when you are removing things.
• Keep a list of temporary changes, and undo them when the move is done.