Your application can pass every test and still fail because the host runs out of inodes, a service restarts in a loop, or a restore cannot complete. For a backend developer taking a first on-call rotation, the costly mistake is treating system administration as a checklist to finish before an incident. I would start with evidence, ownership, and recovery instead, because those determine whether a fault stays small.
A green deployment can hide a broken host
Backend developers often use the deployment result as the boundary of their responsibility. That boundary fails in production because a successful deploy says nothing about disk space, time synchronization, file descriptors, or whether the service stayed healthy after the deployment job exited. System Administration Best Practices for Dev Teams is a broad starting point; I would put failure evidence ahead of a broader checklist because the first restart can erase the clearest account of what went wrong.
Consider an illustrative incident: an API returns HTTP 200 from its health endpoint for 6 minutes while requests that need PostgreSQL fail. The process is alive, but its connection pool cannot open new connections. Restarting the API may briefly clear its pool without fixing the database limit, so the apparent recovery can cost another interruption and a harder investigation. A readiness check should exercise a dependency the service actually needs; a liveness check can remain narrower because killing a healthy process during a database outage makes recovery noisier.
Before changing a failed Linux host, capture its state. The following script runs on a systemd-based machine and takes a unit name, such as api.service, as its argument:
#!/usr/bin/env bash
set -euo pipefail
unit="${1:?usage: inspect-unit UNIT}"
systemctl status "$unit" --no-pager -l || true
journalctl -u "$unit" -n 100 --no-pager || true
systemctl show "$unit" -p Restart -p NRestarts -p MemoryCurrent
df -h /var /var/log
ss -ltn
Run it before the restart and attach its output to the incident record. journalctl access depends on host permissions, so an empty or denied result is a reason to fix diagnostic access, not to assume there were no errors. NRestarts can expose a restart loop, while df and ss answer different questions about storage and listening sockets. Add df -i when free bytes look adequate, because a filesystem with no free inodes cannot create files even when its capacity graph looks comfortable.
For ongoing detection, Prometheus and node_exporter can show node_filesystem_avail_bytes and node_filesystem_files_free; Grafana can display the trend, and Alertmanager can route the alert. As an initial threshold to tune, warn when available space falls below 10% for 15 minutes, then compare that window with the host’s observed rate of consumption. A fixed threshold without that comparison can page too late on a rapidly filling log volume or too often on a small, stable filesystem. The cost of missing either condition is time spent debugging application symptoms after the host has already made the real decision.
Shared administrator access makes small mistakes expensive to explain
A developer moving into operations may ask for a shared root login because it removes friction during an outage. I would not make that the normal access path, because a shared identity makes it difficult to establish who changed a firewall rule, rotated a credential, or stopped a service. The immediate cost is investigative time; the longer cost is that a team cannot confidently distinguish an authorized emergency action from an unrelated change.
Compare individual SSH identities with sudo against a shared root account. Individual access wins during normal operations because OpenSSH records a user identity and sudo can log elevated commands; it costs account provisioning, key revocation, and review of privileges. A shared root credential can win as an offline, sealed break-glass path when the usual identity system is unavailable, but it costs attribution and requires rotation after use because every holder knows the same secret. These are different failure paths, not interchangeable convenience settings.
On a Linux host, give each operator a named account, use OpenSSH public-key authentication, and scope sudo rules to the commands the role needs. Set and verify PermitRootLogin no in sshd_config for the normal SSH path, then test a second session before closing the first; otherwise an access-control improvement can lock out the people needed to repair it. Treat sudoers edits the same way: use visudo, because its syntax check reduces the chance that a typo disables elevation during an incident. Keep the break-glass procedure separate and test it under supervision, because an untested emergency account is only a theory of access.
The more subtle mistake is failing to define who may make which change. A backend developer may know why PostgreSQL needs more connections, but increasing max_connections can also increase memory use and move contention elsewhere. The database owner should review the capacity effect while the application owner explains pool behavior; without both, a quick fix can turn a bounded error into host-wide memory pressure. Record the requester, approver, command or configuration diff, and rollback condition in the change record. That record costs minutes to write but can save an entire shift of reconstructing intent from shell history.
Access policy should also account for service identities. A systemd service should run under its own User= where practical, because a compromised process running as root has a larger set of files and commands available to it. This does not eliminate application vulnerabilities; it limits what one vulnerability can reach. A developer who inherits a root-run service should first identify the files, ports, and device access it actually uses, because changing its identity without that inventory may create an outage while leaving the underlying privilege assumption undocumented.
A backup schedule does not establish a recovery time
Teams frequently report that backups are “working” because a scheduled job exits successfully. That result proves only that the job completed its own steps; it does not prove the files can be read, the database can start, or the application can use the restored data. I would put a restore rehearsal ahead of Essential System Administration Best Practices for IT Teams because a backup schedule cannot tell an on-call developer how long a customer-visible recovery will take.
For PostgreSQL, compare a filesystem snapshot with pg_basebackup plus archived WAL. A filesystem snapshot wins for quick recovery of a whole volume when the storage system can take a consistent snapshot and the database’s requirements are met; it costs dependence on that storage platform and can preserve the same operator error or corruption you hoped to escape. pg_basebackup with WAL archiving wins when you need to restore to a selected point in time or move recovery to another host; it costs archive storage, retention management, and a more involved restore procedure. A snapshot taken without checking database consistency is not a shortcut, because fast access to unusable bytes has no recovery value.
Write down two targets before selecting either method. The recovery point objective is how much committed data the team can afford to lose; the recovery time objective is how long the service can remain unavailable. As a planning assumption, not a measured result, suppose the service must recover within 45 minutes and lose no more than 5 minutes of committed transactions. A nightly pg_dump alone cannot meet that recovery-point target because it does not capture the transactions between dumps. Likewise, a theoretically suitable WAL archive does not meet the time target if nobody has timed the restore and application checks.
Perform the drill on an isolated host or database instance. Restore the files, start PostgreSQL, check that the expected tables and recent records exist, and run a representative application read and write against the restored instance. Record when each step starts and ends; those observations turn “we have backups” into a measured recovery path. PostgreSQL’s pg_verifybackup can check a base backup against its manifest, but it cannot substitute for starting the database and testing application behavior, because file verification does not prove that the service can resume useful work.
Make retention an explicit decision rather than accepting a tool default. For example, 30 days could be a policy value to review against storage cost and the time it takes to discover a bad deletion; it is not automatically the right answer for every dataset. Keep a copy outside the failure domain of the primary host, because a backup on the same lost volume disappears with the data it was meant to protect. The cost of getting this wrong is not merely a longer incident: it can be permanent loss of data that the team believed was recoverable.
An undocumented fix becomes the next outage’s first mistake
Application developers are used to putting behavior in version control, yet host changes often arrive as one-off shell commands. That mismatch creates configuration drift because the next deployment or replacement host may not contain the change that repaired production. If someone edits a systemd unit in place, capture the resulting unit with systemctl cat and put the intended change in the repository that owns host configuration; otherwise the live machine becomes the only reliable specification.
Use a small change loop: state the observed fault, save the evidence, make one change, check the service and its dependent requests, and define the condition that triggers rollback. Ansible can apply a reviewed host configuration, while Terraform can describe infrastructure resources, but neither tool makes an unreviewed assumption safe. For example, applying a new firewall rule before confirming the management path can sever SSH access; testing the rule on a disposable host and retaining an out-of-band route costs preparation time but reduces the chance of a repair creating a second incident.
Time synchronization deserves a line in that loop. Check timedatectl and the host’s chrony status when logs disagree about event order, because skewed clocks can make a correct sequence of events appear contradictory. Record the old and new configuration and the exact check used to accept the change. “The service started” is too weak an acceptance test when the failure affected database writes, because a process can start successfully and still reject the requests customers need.
Do not turn every incident into an automation project. A short, executable runbook wins before a complex remediation system when the failure has happened only once, because the team does not yet know which observations distinguish the cause from its symptoms. Update that runbook after a real drill or incident, including the command that failed and the permission it required. The cost of leaving those details in one person’s memory is paid by the next on-call developer, usually at the least convenient hour.
Your first improvement should survive a failed restart
Pick one service you may have to recover and arrange a short rehearsal with its current owner. On a non-production host, cause it to fail, capture its systemd state and logs before restarting it, and time the return of a real application request. Put the commands, required access, and rollback condition in a shared runbook. That single exercise will expose a concrete gap you can fix before the pager does.



