The last post ended on a number I don't like: 97. A break-in sat on a production server for 97 days, and the only reason it was found at all is that I happened to be checking something else. Found by accident. Accident is not a security strategy, and I knew that while I was writing the story down.
So the week after the cleanup went into a different kind of work. No drama, no forensics, nothing that makes a good movie scene. Just making sure that nothing on that server can fail silently again. This is the part where the house gets locks that fit how the burglar actually operated.
Watch the door they used
The burglar had a parking spot: the folder where Linux keeps every user's scheduled tasks. That is where the miner's restart job lived for three months, and it is the first place any future intruder with the same playbook would go.
So now something watches it. Every ten minutes, the server checks that folder against a short list of what belongs there. Anything else, and it flips a status file from OK to TRIPPED. Every fifteen minutes, a machine on my desk reads that status over a secure connection and reacts.
The detail I care most about is not the check. It's what happens when the check stops. If that status file goes stale, the alarm fires anyway, because an attacker smart enough to install a miner is smart enough to kill the thing watching for miners. A dead tripwire is a tripped tripwire. Monitoring that can fail without anyone noticing is just the old problem wearing a new badge.
An alarm has to ring where somebody lives
Here is the thing about the four silent failures from the last post. None of them was truly silent. The dependency errors were being written sixty times an hour to a place that discards everything sent to it. The firewall's config file said it was switched on, and it wasn't filtering a thing. The information existed. It was just written to places nobody reads.
An alert nobody sees is a diary entry.
So this alarm doesn't email me, because email is where alerts go to die politely. It turns a screen red. There is a display on my desk that spends its day glowing along with the systems it watches, and when the tripwire trips, or goes silent, or the server stops answering for an hour, the whole thing goes red and stays red for as long as the problem lasts. It is impossible to ignore, which is the entire point. The fanciest monitoring in the world is worth less than a light in the room where you actually sit.
Backups that assume the worst
Every morning, the database leaves the building.
The direction matters more than the schedule, and it's the one genuinely counterintuitive idea in this post. The server does not send its backup anywhere. A machine at my office reaches out, takes a copy, and brings it home. The server has no key to the place the backups live. It doesn't know where they are.
Why? Because this particular server has already been burgled once. A machine you have seen compromised does not get the keys to its own insurance policy. If an intruder gets back in, they get the server. They do not get the month of backups that would let me rebuild it, because from where they're standing, those backups do not exist. It's the difference between keeping copies of your documents at a relative's house and keeping them in a fireproof box the burglar can also find.
And the copy is tested the moment it arrives. Is it a valid file, is it a plausible size, does it end the way a complete backup ends. Fail any of those and the copy is deleted rather than kept, and the screen on the desk goes red. A corrupt backup filed away as a good one is worse than no backup, because it buys a month of false confidence.
A backup you haven't restored is a rumor
The first thing I did with the first backup was try to destroy its reputation. Restored it into a throwaway database on a different machine, then counted everything. Forty-two tables. Every blog post. Every user. All of it matched production exactly, to the row.
This is the step everybody skips. I understand why: restoring a backup feels redundant, like test-driving your own parked car. But businesses discover their backups were decorative at exactly one moment, and it is the worst possible moment. A restore test moves that discovery to a Tuesday morning when nothing is on fire. So it goes back on the calendar every month, because a backup system only proven once is a backup system slowly becoming a rumor again.
The morning after I built it, the machine on my desk came up from a night powered off, took the day's copy on its own before I'd asked it to, verified it, and logged one line to say so. Boring. On purpose. Boring is what this entire week of work was for.
What this doesn't fix
I want to be straight about the limits, because the last post was and that's the only reason it's worth believing.
None of this stops a break-in. There is no tool I could install that guarantees nobody ever gets into that server again, and anyone who sells you one is lying. What this fixes is the number. Last time, the gap between break-in and discovery was 97 days. With the tripwire watching the door they used, the same move now buys an intruder about twenty minutes at worst before a screen goes red in a room where I sit every day.
The honest gap: the watcher runs on a machine in my office, so a stretch with that machine off is a stretch with nobody looking. The server keeps its own record either way, and the screen goes red when I come back to it.
Detection is the unglamorous half of security, and it's the half nobody sells because it isn't shiny. Prevention fails eventually. Detection decides whether that failure is a Tuesday or a season.
The same three questions, with teeth
The last post ended by asking when somebody last looked at your server, how you'd find out if something broke silently, and what you would restore from. I can now answer all three for this machine with numbers instead of intentions. Something looks every ten minutes. I find out inside the half hour, from a red screen I cannot miss. And I would restore from this morning's copy, which is off the server and verified, from a system I have actually restored from once already.
Getting there took about a day of real work. It has run by itself ever since, unattended, on mornings I never thought about it.
That day of work is what "somebody looking" costs. If nobody is doing it for your business, that is exactly what a care plan is.
