Managing secrets for FDE is tricky. Do it manually and you’ll need to be ready to enter the passphrase whenever your server reboots (not fun during a short outage at night). Since I didn’t want to take that availability trade-off, I use Mandos (https://www.recompile.se/mandos/man/intro.8mandos).

With Mandos, servers can boot unattended, but there’s a bit of a security and complexity trade-off (the Mandos documentation has a deep-dive on this, good read) 6/n

CoreDNS is used as the caching DNS resolver, forwarding to Cloudflare via DNS over TLS.

Main reasons for using CoreDNS: it’s written in a memory-safe language, offers great Prometheus integration and detailed query logging.

DNS logs offer great insights in incident response scenarios (“This server was hacked at 12:15 UTC, which DNS queries were sent around that time?”).

Cloudflare was chosen as the upstream because of performance and DoT support. 7/n

Finally, networking. All servers run ufw (an iptables frontend) and block incoming traffic except for SSH and HTTP/HTTPS (on the application servers).

There’s no blocking on outgoing traffic (would be good as additional defense-in-depth, but tricky to get right with federation, so it’s in my backlog).

When servers need to talk to each other (e.g. app servers to redis or postgres), that happens on a private, firewalled network encrypted via Wireguard. 8/n

Let’s dive into the server roles, starting with Postgres, the database server.

I run Postgres 11, installed via the Postgres apt repository (https://wiki.postgresql.org/wiki/Apt). The server also runs an instance of pgbouncer, a connection pooler for Postgres.

I do some minimal tuning of Postgres, mostly in terms of buffer size. All of these depend highly on RAM, CPU and I/O constraints.

I also enable data checksums to prevent silent database corruption. 9/n

Postgres is Mastodon’s main data store, so backing up that data regularly is crucial.

I use three approaches to cover different data loss scenarios.

First, a disk snapshot of the server is created at least once a month as well as before and after any maintenance work on the server. The main purpose of this backup is to reduce recovery time by providing a fast way to spin up a new instance of Postgres with the right configuration. 10/n

I also create a logical backup of the database using pg_dump once a day. The backup is encrypted and stored on a separate volume on the server. It’s also pulled by my home server once a day, where backup rotation keeps daily backups for a week as well as weekly backups for a month.

Completion and pulling of backups is monitored using something like a dead man’s switch – if no backup completes or is pulled in a 25-hour period, monitoring screams at me. 11/n

Finally, I use wal-g (https://github.com/wal-g/wal-g) to ship encrypted WAL files to Amazon’s S3 every 5 minutes.

WAL (Write-Ahead Logging) can be thought of as files containing changes the database made to its permanent records.

By continuously archiving WAL files, you get point-in-time recovery for your data. In the configuration I use, the maximum window for data loss is roughly 5 minutes (unless the failure scenario involves WAL archive failure). 12/n

Being paranoid about backups, monitoring success and not putting all your eggs in one basket is an important lesson often learned the hard way.

Things you should think about:

Can I access my backups if my hosting provider throws me out?

Are my backups safe if my server is compromised?

Would I notice if my backups are silently failing?

Do I know restoration works?

Do I have a fallback if one of my backup tools has a bug that leads to corruption? 13/n