Why a Simple Cron Job Feels Safe—At First Glance

Many operations teams adopt a “run‑every‑night” cron script to replace SSH host keys or user authorized keys. The premise is straightforward: generate a fresh key pair, push the public part to the remote authorized_keys file, discard the old private key, and repeat. On paper this reduces the attack surface of long‑lived credentials and satisfies compliance check‑boxes that demand periodic key renewal.

However, this convenience hides a cluster of subtle dangers that become apparent only when the rotation process collides with real‑world operational constraints—file‑system race conditions, audit‑trail gaps, and inadvertent denial‑of‑service for legitimate users. The following sections unpack those hidden internals.

A Minimalist Cron‑Based Rotator (What You Might Find Online)

Below is a common example found in internal wikis. It runs at 02:00 AM on every day, creates a new RSA key, updates authorized_keys, and overwrites the old private key in /root/.ssh/id_rsa. 

#!/bin/bash
# /etc/cron.d/ssh-key-rotate
0 2 * * * root /usr/local/bin/rotate_ssh_key.sh > /var/log/ssh-key-rotate.log 2>&1

The accompanying script is equally terse: 

#!/usr/bin/env bash
set -euo pipefail

KEY_DIR="/root/.ssh"
TMP_DIR="/tmp/ssh-key-rotate-$(date +%s)"

mkdir -p "$TMP_DIR"
chmod 700 "$TMP_DIR"

# Generate a fresh 4096‑bit RSA key without a passphrase
ssh-keygen -t rsa -b 4096 -N "" -f "$TMP_DIR/id_rsa"

# Replace the authorized_keys entry for the “deploy” user
sed -i '/^deploy:/d' "$KEY_DIR/authorized_keys"
cat "$TMP_DIR/id_rsa.pub" >> "$KEY_DIR/authorized_keys"

# Atomically swap the private key
mv "$TMP_DIR/id_rsa" "$KEY_DIR/id_rsa"
chmod 600 "$KEY_DIR/id_rsa"

# Clean up
rm -rf "$TMP_DIR"

At first sight the script appears solid: it uses a temporary directory, respects permissions, and updates the public key atomically. Yet each line carries hidden assumptions that, when violated, open a security back‑door.

Hidden Pitfalls That Turn a Routine Job into a Liability

1. Uncontrolled Pass‑phrase Removal
By generating a key without a passphrase you eliminate the need for an interactive unlock step, but you also place an unprotected private key on disk. If an attacker gains root access—even briefly—they can copy the key before it is rotated again. The script’s chmod 600 mitigates exposure to other users, yet it does nothing against a compromised root account.

2. Race Conditions Between Update and Connection
The script updates authorized_keys while existing SSH sessions may still be authenticating. A client that began a handshake before the update may complete successfully using the now‑revoked key, creating a window where an old credential remains valid. Conversely, a client that starts after the update but before the remote side reloads the file can be rejected, causing an unexpected outage.

3. Loss of Auditable History
The script overwrites authorized_keys in place, erasing the previous entry. Without a versioned log, compliance audits cannot demonstrate which key was valid at a given timestamp. A simple git‑backed repository or immutable append‑only log would be required to retain that evidence.

4. Incomplete Host‑Key Rotation
The example focuses on user keys but ignores host keys stored in /etc/ssh/ssh_host_*_key. Clients that cache the host fingerprint will see a mismatch after the next rotation, triggering man‑in‑the‑middle warnings that may be ignored by users, thereby weakening trust.

5. Unchecked Exit Paths
The script uses set -euo pipefail, which is good practice, yet it does not trap signals such as SIGINT or SIGTERM. An abrupt termination (for instance, during a system reboot) could leave a partially written authorized_keys, breaking all SSH access.

Deep Dive: What Happens Inside the SSH Daemon

When sshd receives a connection request, it reads the authorized_keys file once per authentication attempt. The daemon does not cache the file across connections, which means any change is visible immediately to new sessions but not to sessions that have already read the file. This behavior explains the race condition described earlier.

Moreover, sshd checks the file’s owner and mode (600 for the file, 700 for the directory). If a rotation script momentarily creates the file with broader permissions, a brief window exists where another process could read the private key. The temporary directory in the example mitigates this, but the mv operation itself is not atomic on all filesystems (e.g., NFS). On such mounts, the rename can be implemented as a copy‑and‑delete, leaving the original key exposed for longer than expected. 

# Example of a non‑atomic rename on NFS
cp "$TMP_DIR/id_rsa" "$KEY_DIR/id_rsa"
rm -f "$TMP_DIR/id_rsa"
# At this point both copies exist simultaneously

Understanding these low‑level mechanics is essential before trusting a black‑box cron job with credential lifecycles.

Safer Alternatives and Mitigations

Rather than a bare cron script, consider the following hardened pattern: 

#!/usr/bin/env bash
set -euo pipefail
trap 'rm -rf "$TMP_DIR"' EXIT

KEY_DIR="/root/.ssh"
TMP_DIR="$(mktemp -d /tmp/ssh-rotate.XXXXXX)"
KEY_FILE="$KEY_DIR/id_rsa"
AUTH_KEYS="$KEY_DIR/authorized_keys"
GIT_REPO="/opt/ssh-key-audit"

# 1. Generate a passphrase‑protected key
PASSPHRASE=$(openssl rand -base64 32)
ssh-keygen -t ed25519 -a 100 -N "$PASSPHRASE" -f "$TMP_DIR/id_ed25519"

# 2. Append new public key to a version‑controlled file
git -C "$GIT_REPO" pull
cat "$TMP_DIR/id_ed25519.pub" >> "$GIT_REPO/authorized_keys"
git -C "$GIT_REPO" add authorized_keys
git -C "$GIT_REPO" commit -m "Rotate deploy key $(date -u +%Y-%m-%dT%H:%M:%SZ)"
git -C "$GIT_REPO" push

# 3. Deploy atomically using a lockfile
exec 200>"$KEY_DIR/rotate.lock"
flock -n 200 || { echo "Another rotation in progress"; exit 1; }

# Replace the private key only after the public key is safely stored
mv "$TMP_DIR/id_ed25519" "$KEY_FILE"
chmod 600 "$KEY_FILE"

# 4. Signal sshd to reload authorized keys without restarting
kill -HUP "$(cat /var/run/sshd.pid)"

Key improvements:

  • Pass‑phrase protection reduces the impact of a stolen key file.
  • Version‑controlled authorized_keys retains a full audit trail.
  • File‑system lock (flock) prevents concurrent rotations.
  • Explicit SIGHUP forces sshd to re‑read the file, eliminating stale sessions that might otherwise accept revoked keys.

Security and Best Practices

Limit the rotation window. Schedule the job during a low‑traffic maintenance window and monitor for failed logins immediately after rotation. Use a centralized logging platform to correlate authentication attempts with rotation timestamps.

Enforce key type and size. Modern best practice favors Ed25519 or ECDSA over RSA, providing stronger security with smaller key material and faster cryptographic operations.

Separate duties. The process that generates the key