Why Automated SSH Key Rotation Can Undermine Security in Production

Introduction – the temptation of “set it and forget it”

Many operations teams adopt scripts that generate a new key pair every few days, push the public key to every host, and delete the old key automatically. The idea sounds attractive: a constantly refreshed credential should reduce the window of exposure if a key is compromised. In practice, however, the automation introduces race conditions, audit gaps, and a false sense of safety that can be more dangerous than a static key managed with discipline.

What the automation typically does

A typical automated rotation workflow follows these steps:

# pseudo‑code for a naive rotation script
for host in $(cat inventory.txt); do
  ssh-keygen -t ed25519 -f /tmp/new_key -N "" > /dev/null
  ssh-copy-id -i /tmp/new_key.pub user@$host
  ssh user@$host "sed -i '/old_key/d' ~/.ssh/authorized_keys"
  rm /tmp/new_key*
done

The script runs from a central CI runner every 24 hours. It appears to keep the environment clean, but several hidden problems emerge once the script is used at scale.

Hidden failure modes

1. Transient connectivity loss – If the network hiccups while the script is pushing the new key, the old key may be removed before the new one is installed, leaving the host inaccessible. Automated recovery is rarely built in, and the result is a production outage that requires manual console access.

2. Inconsistent state across hosts – When the inventory list is out of date, the script may skip a host that still trusts the old key, creating a “ghost key” that an attacker can exploit indefinitely.

3. Auditing blind spots – Every rotation overwrites the authorized_keys file without logging who performed the change or why. Security auditors lose the ability to trace which key was active at a given moment, breaking chain‑of‑custody requirements for many compliance regimes.

Manual rotation – a safer baseline

Instead of a fully automated loop, use a controlled, staged process that lets you verify each step before moving on. Below is a Bash function that performs a single‑host rotation with explicit checks and logging.

# rotate_ssh_key.sh – rotate a key on a single host safely
rotate_key() {
  local HOST=$1
  local USER=$2
  local KEY_NAME="deploy_$(date +%Y%m%d%H%M%S)"
  local NEW_PRIV="/tmp/${KEY_NAME}"
  local NEW_PUB="${NEW_PRIV}.pub"

  # 1. Generate a new ed25519 key pair
  ssh-keygen -t ed25519 -f "${NEW_PRIV}" -N "" -C "${USER}@${HOST}" > /dev/null

  # 2. Verify we can log in with the existing key before adding the new one
  if ! ssh -o BatchMode=yes -o ConnectTimeout=5 ${USER}@${HOST} true; then
    echo "❗ Cannot reach ${HOST} with current credentials"
    return 1
  fi

  # 3. Append the new public key, keep a backup of the old authorized_keys
  ssh ${USER}@${HOST} "cp ~/.ssh/authorized_keys ~/.ssh/authorized_keys.bak_$(date +%s)"
  scp "${NEW_PUB}" ${USER}@${HOST}:/tmp/new_pub.key
  ssh ${USER}@${HOST} "cat /tmp/new_pub.key >> ~/.ssh/authorized_keys && rm /tmp/new_pub.key"

  # 4. Test the new key before removing the old one
  if ssh -i "${NEW_PRIV}" ${USER}@${HOST} true; then
    # 5. Remove the old key (identified by comment)
    ssh ${USER}@${HOST} "sed -i '/${HOST}_old_key/d' ~/.ssh/authorized_keys"
    echo "✅ Rotation successful on ${HOST}"
  else
    echo "⚠️ New key failed on ${HOST}; restoring backup"
    ssh ${USER}@${HOST} "mv ~/.ssh/authorized_keys.bak_* ~/.ssh/authorized_keys"
    return 1
  fi

  # 6. Securely delete the temporary private key locally
  shred -u "${NEW_PRIV}" "${NEW_PUB}"
}

The function performs three critical safety checks: (a) it confirms the existing connection works, (b) it validates the new key before any removal, and (c) it keeps a timestamped backup of the original authorized_keys file. By running this function host‑by‑host, you maintain full visibility and can halt the process immediately if anything goes wrong.

Integrating the manual step into a CI pipeline

Automation is still valuable, but it should orchestrate rather than execute the rotation. A CI job can iterate over the inventory, call the rotate_key function on each host, and capture the exit status. The job should also produce a signed JSON report that auditors can verify later.

# ci_rotate.yml – GitHub Actions workflow fragment
jobs:
  rotate-ssh-keys:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout scripts
        uses: actions/checkout@v3

      - name: Install SSH client
        run: sudo apt-get update && sudo apt-get install -y openssh-client

      - name: Execute rotation
        env:
          INVENTORY: inventory.txt
        run: |
          while read -r line; do
            HOST=$(echo $line | cut -d',' -f1)
            USER=$(echo $line | cut -d',' -f2)
            bash rotate_ssh_key.sh $HOST $USER || echo "Rotation failed on $HOST" >> failures.log
          done < $INVENTORY

      - name: Generate signed report
        run: |
          jq -n --arg date "$(date -u +"%Y-%m-%dT%H:%M:%SZ")" \
                --argfile failures failures.log \
                '{date:$date, failures:$failures}' > report.json
          gpg --clearsign --output report.json.asc report.json
      - name: Upload artifact
        uses: actions/upload-artifact@v3
        with:
          name: rotation-report
          path: report.json.asc

The workflow still runs automatically, but the critical rotation logic lives in a script that can be inspected, version‑controlled, and reviewed by security engineers before each deployment. The signed report provides immutable evidence for compliance audits.

Security and Best Practices

Never store private keys in plain text on CI runners. Use secret‑management tools (e.g., HashiCorp Vault, AWS Secrets Manager) to inject the temporary private key into the runner’s memory only.

Prefer ed25519 over RSA. The shorter key size reduces the attack surface and improves performance without sacrificing security.

Enforce key comment conventions. Include host, purpose, and creation date in the comment field; this makes the sed removal step reliable and audit‑friendly.

Rotate only when you have a documented change window. Align key rotation with other scheduled maintenance to minimise the impact of unexpected lock‑outs.

“Automation without verification is a shortcut to failure; the most reliable pipelines are those that pause for human confirmation at the highest risk points.” – Senior DevSecOps Engineer

Conclusion – balance automation with control

Fully automated SSH key rotation promises perpetual freshness, yet it often hides connectivity glitches, audit blind spots, and inadvertent lock‑outs. By extracting the risky portion of the workflow into a reviewed, logged script and letting CI orchestrate the process, you keep the benefits of regular rotation while preserving operational safety and regulatory compliance.

Treat key rotation as a security control, not a convenience feature. A disciplined, observable process will protect your production environment far better than a blind “set it and forget it” script.