Cybersecurity

Why Automatic Certificate Renewal for Internal Services Is a Hidden Liability

Understanding the Appeal of Automation

Many organizations treat TLS certificates as a routine operational detail. The rise of ACME‑compatible tools makes it tempting to let a daemon handle renewal for every endpoint, even those that never face the public internet. While automation reduces manual effort, it also introduces subtle failure modes that can cripple internal traffic, expose services to man‑in‑the‑middle attacks, or break compliance checks.

The Core Problem: Blind Trust in Renewal Daemons

Renewal services typically run with elevated privileges, have access to private keys, and assume that a successful HTTP challenge equals a trustworthy certificate. In internal environments, the following hidden issues arise:

Certificates may be issued by an external CA that does not meet internal policy (key length, algorithm, or audit requirements).
Renewal can happen at unexpected times, causing services to reload with new certificates without proper coordination.
Expired certificates are sometimes cached by load balancers, leading to intermittent failures that are hard to trace.
Automatic renewal masks configuration drift; teams may forget to rotate root trust stores.

The result is a false sense of security. When a renewal fails, the service may continue presenting an expired certificate, and operators often discover the outage only after users report errors.

Reproducing the Failure: A Minimal Lab

The following steps set up a small internal service that uses a self‑signed CA. We will then attach certbot to auto‑renew the leaf certificate and observe the hidden breakage.

# Install dependencies (Ubuntu/Debian)
sudo apt-get update
sudo apt-get install -y nginx openssl certbot python3-certbot-nginx

# Create a private CA
mkdir -p ~/internal-ca && cd ~/internal-ca
openssl genrsa -out ca.key 4096
openssl req -x509 -new -nodes -key ca.key -sha256 -days 3650 -out ca.crt -subj "/CN=Internal CA"

# Generate a server key and CSR
mkdir -p ~/service && cd ~/service
openssl genrsa -out server.key 2048
openssl req -new -key server.key -out server.csr -subj "/CN=internal.example.local"

# Sign the CSR with the internal CA (valid for 30 days)
openssl x509 -req -in server.csr -CA ~/internal-ca/ca.crt -CAkey ~/internal-ca/ca.key -CAcreateserial -out server.crt -days 30 -sha256

# Configure nginx to use the certificate
sudo tee /etc/nginx/sites-available/internal.conf > /dev/null <<'EOF'
server {
    listen 443 ssl;
    server_name internal.example.local;

    ssl_certificate     /home/$(whoami)/service/server.crt;
    ssl_certificate_key /home/$(whoami)/service/server.key;

    location / {
        return 200 'OK';
    }
}
EOF

sudo ln -s /etc/nginx/sites-available/internal.conf /etc/nginx/sites-enabled/
sudo nginx -t && sudo systemctl restart nginx

At this point, a client can verify the connection using the internal CA:

# Verify with curl
curl -k --cacert ~/internal-ca/ca.crt https://internal.example.local
# Expected output: OK

Hooking Up Automatic Renewal

We now configure certbot to treat the internal service as a regular domain and renew the leaf certificate automatically. Note that we intentionally do not change the CA; certbot will request a public Let’s Encrypt certificate, which violates the internal policy.

# Simulate a public domain (replace with a real reachable domain for a real test)
sudo certbot certonly --nginx -d internal.example.local --register-unsafely-without-email --agree-tos --non-interactive

# Link the renewed cert to our nginx config (dangerous!)
sudo ln -sf /etc/letsencrypt/live/internal.example.local/fullchain.pem /home/$(whoami)/service/server.crt
sudo ln -sf /etc/letsencrypt/live/internal.example.local/privkey.pem /home/$(whoami)/service/server.key

# Verify renewal schedule
sudo systemctl list-timers | grep certbot

The daemon now renews every 60 days, silently replacing the internal CA‑signed certificate with a public one. This violates compliance, and if the public CA revokes the certificate, the internal service will lose trust instantly.

Designing a Safer Renewal Workflow

Instead of blind automation, we propose a “semi‑automated” approach:

Run a scheduled check that only detects impending expiration.
Emit a structured alert (JSON, Slack, or PagerDuty) for a human operator.
When approved, invoke a controlled signing script that uses the internal CA.
Reload the service in a coordinated fashion (e.g., via systemctl reload).

Implementation: Expiration Checker

The following Bash script scans a list of PEM files, reports those expiring within a configurable window, and exits with a non‑zero status if any are found. It can be run from cron or a CI job.

#!/usr/bin/env bash
# check_cert_expiry.sh
# Usage: ./check_cert_expiry.sh /path/to/cert1.crt /path/to/cert2.crt ...

THRESHOLD_DAYS=7
ALERTS=()

for CERT in "$@"; do
    if [[ ! -f "$CERT" ]]; then
        echo "File not found: $CERT"
        continue
    fi

    # Extract expiry in seconds since epoch
    EXPIRY=$(openssl x509 -enddate -noout -in "$CERT" | cut -d= -f2)
    EXPIRY_EPOCH=$(date -d "$EXPIRY" +%s)
    NOW_EPOCH=$(date +%s)
    DIFF_DAYS=$(( (EXPIRY_EPOCH - NOW_EPOCH) / 86400 ))

    if (( DIFF_DAYS <= THRESHOLD_DAYS )); then
        ALERTS+=("$CERT expires in $DIFF_DAYS days (on $EXPIRY)")
    fi
done

if (( ${#ALERTS[@]} )); then
    echo "Certificates nearing expiration:"
    for A in "${ALERTS[@]}"; do echo "  - $A"; done
    # Send to Slack (example, replace webhook URL)
    PAYLOAD=$(jq -n --arg text "$(printf '%s\n' "${ALERTS[@]}")" '{text: $text}')
    curl -X POST -H 'Content-Type: application/json' -d "$PAYLOAD" https://hooks.slack.com/services/XXXX/XXXX/XXXX
    exit 1
else
    echo "All certificates are healthy."
    exit 0
fi

Save the script as check_cert_expiry.sh, make it executable, and add a cron entry:

# Run daily at 02:00
0 2 * * * /home/$(whoami)/scripts/check_cert_expiry.sh /home/$(whoami)/service/server.crt >> /var/log/cert_check.log 2>&1

Implementation: Controlled Signing Script

When the alert is acknowledged, an operator runs the following script to create a fresh certificate using the internal CA. The script also backs up the previous key/cert pair.

#!/usr/bin/env bash
# sign_internal_cert.sh
# Arguments: domain, key_path, csr_path, cert_path, days_valid

set -euo pipefail

DOMAIN=$1
KEY_PATH=$2
CSR_PATH=$3
CERT_PATH=$4
VALID_DAYS=${5:-30}
CA_DIR=~/internal-ca

# Backup old files
timestamp=$(date +%Y%m%d%H%M%S)
cp "$KEY_PATH" "${KEY_PATH}.bak.${timestamp}"
cp "$CERT_PATH" "${CERT_PATH}.bak.${timestamp}"

# Generate a new key if needed
if [[ ! -f "$KEY_PATH" ]]; then
    openssl genrsa -out "$KEY_PATH" 2048
fi

# Create a new CSR
openssl req -new -key "$KEY_PATH" -out "$CSR_PATH" -subj "/CN=$DOMAIN"

# Sign with internal CA
openssl x509 -req -in "$CSR_PATH" -CA "$CA_DIR/ca.crt" -CAkey "$CA_DIR/ca.key" \
    -CAcreateserial -out "$CERT_PATH" -days "$VALID_DAYS" -sha256

# Reload nginx (or any service)
sudo systemctl reload nginx

echo "Certificate for $DOMAIN renewed and service reloaded."

The script can be wrapped in a small CI job that requires a manual approval step, ensuring that a human reviews the renewal request before the private key is touched.

Security and Best Practices

The semi‑automated model mitigates the hidden risks discussed earlier:

♥ Enjoyed this article? Use the like banner at the top of the page to let us know — you can like it more than once! Each additional like from the same reader carries a little less weight than the first, so our appreciation scores reflect genuine enthusiasm rather than accidental clicks. Your feedback helps us understand which topics resonate most.