arlofox

a fox's internet corner

← Back to Home

privacy-systems

The Red Fox Server

2026.08.17

I wrote about my personal server, Red Fox, a while back. It has grown since then more services, but more to the point, a lot of scaffolding around them that didn't exist before. Backups, monitoring, locked-down containers. The boring half. The half that decides whether the interesting half survives contact with a bad week.

Here's what's on it now and, more usefully, what I got wrong the first time. The repo is on Codeberg.

What it runs

The blog you're reading is Ghost, containerized behind a Caddy reverse proxy that handles TLS. That much hasn't changed. What has: Ghost 5 stored its data in SQLite, which upstream treats as development-only, and Ghost 6 assumes MySQL. So there's now a MySQL container underneath it. The admin interface sits behind HTTP basic auth in addition to Ghost's own login, because /ghost is the one door on this box that faces the internet with a password behind it, and I'd rather an attacker have to get through two.

ntfy is still the notification backbone. You post a message to a topic, anything subscribed to that topic gets it. Everything on this box that can fail talks to it: backups, upgrades, monitoring. Self-hosted, auth locked down, no anonymous writes except the one narrow case UnifiedPush needs.

MollySocket is still the piece I'm most pleased with. Molly is a hardened Signal fork; standard Signal notifications route through Google's Firebase, so Google learns something is happening on your phone even without reading it. MollySocket keeps its own connection to Signal and pings my ntfy instance instead, which wakes Molly directly. No Google in the loop. It runs airgapped no public web endpoint, registration done once over SSH.

Then the things I added because I wanted them, not because the architecture demanded them:

Miniflux for RSS, running against the host's PostgreSQL. Navidrome for music, streaming from a Storage Box mounted over CIFS, so the library isn't eating the server's own disk. LibreSpeed for checking bandwidth without loading someone else's ad-laden speed test. And Endurain, which took over fitness tracking after I moved off Garmin Connect my watch talks to Gadgetbridge on my phone, which syncs here. That one brought its own PostgreSQL and Redis with it.

Uptime Kuma watches the rest of it and shouts through ntfy when something stops answering.

Keeping the data

The rule I settled on: a file-level copy of a running database is not a backup. It's a snapshot of a file that was being written to, which restores into something that might be fine and might be subtly broken, and you find out at the worst possible time.

So the nightly job dumps every database properly first mysqldump for Ghost, pg_dump for Endurain, sqlite3 .backup for Navidrome, Kuma, and the ntfy auth database and then restic picks up the dumps. The live database directories are explicitly excluded. The dump is the artifact.

Each dump writes to a temp file and only moves into place if it succeeded. Get that wrong and a failed dump truncates the good copy from yesterday, which converts a small problem into a total one.

The backups go to Scaleway, deliberately a different provider from the server. Backups living on the same provider as the machine they protect share a failure domain: an account suspension, a billing dispute, a region-wide incident takes both at once, and no amount of redundancy inside that provider helps. This isn't a knock on my host I'd do the same anywhere. The whole point of a backup is surviving the scenario you didn't plan for, and "I lost access to my account" is on that list.

Restic checks the repository integrity monthly. Anything failing anywhere in that chain pushes to ntfy.

I'll be honest that the piece I haven't finished is rehearsing a restore. Verifying the repository proves the bytes are readable. It doesn't prove I can rebuild the box from them, and those are different claims.

Keeping it shut

Every container runs with all Linux capabilities dropped and only the specific ones it needs added back, with no-new-privileges set so nothing inside can escalate. Each application gets its own Docker network, so a compromise in one doesn't get to talk to the others. Nothing binds to anything but loopback; Caddy is the only thing listening publicly, and the firewall allows three ports.

Caddy writes access logs per virtual host, and fail2ban reads them including a filter for repeated auth failures against the Ghost admin gate. That one is doing double duty. The obvious job is stopping password guessing. The less obvious one is that verifying a password costs real CPU by design, so an unthrottled guesser can pin a core and take the whole box down without ever getting in.

SSH is key-only. Secrets live outside the git repo entirely; the config references paths, never values, which is the only reason it's safe to publish at all.

Knowing when it breaks

This is the part I got wrong, and it took an outage to see it.

An automatic image update pulled a new PostgreSQL build for the fitness tracker. That build had changed how it starts where the old one ran as root and fixed up its own file permissions, the new one doesn't, and the security capabilities I'd granted to allow that were suddenly doing nothing. The database couldn't write to its own directory. It died. Systemd restarted it. It died again. That went on for seventeen hours at full CPU, and I found it because I happened to open the hosting dashboard for an unrelated reason.

Two things had failed, and neither was the database.

The first: my monitoring watched endpoints. Uptime Kuma asks "does this URL respond," and the app did respond it answered HTTP perfectly well while being completely unable to reach its database. Nothing was checking the machine itself. No alert on CPU, load, memory, or disk.

The second is subtler and I liked finding it. A service stuck in a restart loop never enters a failed state. It's perpetually almost starting, so any check looking for failed units sees a clean system. It was invisible by construction.

So there's now a job every five minutes that checks the host rather than the services: load average, disk usage, available memory, failed units, and specifically units whose restart count has jumped since the last check. It alerts through the same ntfy path as everything else, and reports in to Kuma so that if the check itself stops running, that's an alert too.

The fix took about five minutes once I knew what was wrong. The seventeen hours were entirely a detection problem.

About that reproducibility claim

Last time I wrote that everything was pinned and a rebuild would bring back the exact same stack. That was more aspiration than description, and worth correcting.

The system config is one version-controlled file, and that part is true and genuinely valuable if this box dies I rebuild from a repo rather than from memory. But I'm tracking a release channel, with auto-upgrades on, and the containers pull moving tags. So the config is reproducible and the stack drifts underneath it. The outage above is exactly that drift arriving unannounced on a Sunday morning.

Which is fine for a blog, where I'd rather have security patches land unattended, and much less fine for anything holding state. That distinction is the actual lesson, and I'd rather write it down than keep claiming a property I don't have.

The config is on Codeberg if you want to read it. It's commented more heavily than it probably needs to be, mostly with notes to myself about why something is the way it is which is the only documentation I've ever consistently gone back and read.

2026.07.07

I've been building out a small personal server which I call it Red Fox. Here's a quick rundown of what's on it and why. The repo is on Codeberg.

Ghost

The blog itself runs on Ghost, a lightweight publishing platform. It's containerized, sitting behind a Caddy reverse proxy that handles TLS automatically. Nothing exotic here I wanted something I control end to end, rather than handing my writing over to a platform I don't own.

ntfy

ntfy is a simple push notification service you send a message to a topic, and anything subscribed to that topic gets notified. I self-host my own instance on rather than using the public one, mainly so notifications about my own infrastructure (upgrades, backups, uptime) stay on infrastructure I control, with auth locked down rather than open to anyone who finds the URL.

MollySocket

This is the one I'm most pleased with. I run Molly — a hardened fork of Signal — on my phone, but standard Signal notifications route through Google's Firebase Cloud Messaging, which means Google knows something's happening on your phone even if it can't read the message. MollySocket sidesteps that: it maintains its own connection to Signal's servers and, when a message arrives, pings my self-hosted ntfy instance instead, which wakes up Molly directly. No Google in the loop at all.

I run it in airgapped mode, meaning there's no public web server exposed for it registration happens once, manually, over SSH, rather than through an internet-facing endpoint. Fewer open doors on the box.

NixOS

The whole thing runs on NixOS. I like it for the same reason I like most of my setup choices: everything is declared in one config file, version-controlled, and reproducible. The flake pins Ghost, Caddy, ntfy, and MollySocket together, so a rebuild brings back the exact same stack, not whatever's current when I get around to it. If Red Fox ever dies, I rebuild it from a repo, not from memory.

Mastodon