Background: Spot Instances and Self‑Hosted Runners

Cloud providers offer spot (or preemptible) VMs at a fraction of the regular price. The cost savings are attractive, especially for workloads that can tolerate interruption. GitHub Actions allows you to register self‑hosted runners on any machine you control, which has led many teams to experiment with spot‑based runners for CI/CD jobs that appear “non‑critical”. While the idea looks appealing, the reality is far more complex. This article explains why using spot instances for production‑grade pipelines is a hidden liability, and walks through the internal mechanisms that cause unexpected failures.

Hidden Internals: How Spot Preemption Works

When a spot VM receives a termination notice, the cloud platform sends a short‑lived signal (usually a two‑minute warning) via metadata service or an event stream. The VM’s operating system does not automatically halt running processes; instead, it must query the metadata endpoint and react accordingly. If your runner does not poll this endpoint, the shutdown proceeds abruptly, killing any active jobs. 

# Bash snippet that polls the metadata service for termination notice
while true; do
  if curl -s -m 1 http://169.254.169.254/spot/termination-time | grep -q '202'; then
    echo "Spot termination detected – stopping runner"
    # Gracefully stop the GitHub Actions runner
    ./svc.sh stop
    exit 0
  fi
  sleep 5
done &

The code above is a minimal watchdog that can be bundled with the runner’s startup script. Without it, a running job may be killed mid‑execution, leaving artifacts in an inconsistent state and causing downstream pipelines to fail.

Why Production Pipelines Can’t Afford Sudden Termination

Production CI/CD pipelines typically involve:

  • Building container images that are later pushed to a registry.
  • Running integration tests that provision temporary cloud resources.
  • Deploying to staging or production environments.

Any interruption during these steps can:

  1. Leave dangling images that waste storage.
  2. Leak credentials if temporary secrets are not revoked.
  3. Corrupt database migrations that have already been partially applied.

Spot‑based runners also inherit the “capacity‑drain” behavior of the underlying cloud zone. When a zone experiences high demand, spot capacity can disappear entirely, causing multiple runners to disappear simultaneously. The cascade effect is difficult to debug because the GitHub UI only reports “runner unavailable” without details.

Step‑by‑Step Tutorial: Building a Safer Runner Setup

Below is a practical example that shows how to:

  1. Create an EC2 spot instance with a persistent root volume.
  2. Install the GitHub Actions self‑hosted runner.
  3. Wrap the runner with a termination‑aware supervisor.
  4. Configure a fallback on‑demand runner for critical jobs.

1. Launch a Spot Instance with a Dedicated EBS Volume

# Terraform snippet – spot instance with a 30 GB gp3 volume
resource "aws_instance" "ci_spot" {
  ami                         = data.aws_ami.ubuntu.id
  instance_type               = "t3.medium"
  spot_price                  = "0.015"
  availability_zone           = "us-east-1a"
  user_data                   = file("install_runner.sh")
  root_block_device {
    volume_size = 30
    volume_type = "gp3"
  }
  tags = {
    Name = "ci-spot-runner"
  }
}

The persistent volume ensures that the runner’s state survives a stop/start cycle, allowing you to resume work after a preemption if you choose to re‑launch the instance automatically.

2. Install the Runner and the Watchdog

# install_runner.sh – executed on instance boot
#!/bin/bash
set -e

# Install dependencies
apt-get update && apt-get install -y curl jq

# Create a dedicated user
useradd -m -s /bin/bash runner
su - runner -c "
  mkdir -p ~/actions-runner && cd ~/actions-runner
  curl -O -L https://github.com/actions/runner/releases/download/v2.311.0/actions-runner-linux-x64-2.311.0.tar.gz
  tar xzf actions-runner-linux-x64-2.311.0.tar.gz
  ./config.sh --url https://github.com/yourorg/yourrepo --token $RUNNER_TOKEN --name ci-spot-$(hostname)
  ./svc.sh install
  ./svc.sh start
"

# Start termination watchdog in background
cat <<'EOF' > /usr/local/bin/spot-watcher.sh
#!/bin/bash
while true; do
  if curl -s -m 1 http://169.254.169.254/spot/termination-time | grep -q '202'; then
    echo "$(date) Spot termination detected – stopping runner"
    su - runner -c "~/actions-runner/svc.sh stop"
    # Optionally trigger a Lambda to spin a new spot instance
    exit 0
  fi
  sleep 5
done
EOF
chmod +x /usr/local/bin/spot-watcher.sh
nohup /usr/local/bin/spot-watcher.sh &

The script installs the runner, registers it, and launches a background watchdog that reacts to termination notices.

3. Define a Fallback On‑Demand Runner

# Terraform for a small on‑demand runner (always available)
resource "aws_instance" "ci_fallback" {
  ami           = data.aws_ami.ubuntu.id
  instance_type = "t3.small"
  user_data     = file("install_fallback.sh")
  tags = {
    Name = "ci-fallback-runner"
  }
}

The fallback runner should be configured with a label (e.g., fallback) and used only for jobs marked as runs-on: [fallback]. This segregation prevents critical deployments from being scheduled on volatile spot nodes.

4. Adjust Workflow Files

# .github/workflows/build.yml
name: Build & Push
on:
  push:
    branches: [main]

jobs:
  build:
    runs-on: [self-hosted, linux, spot]   # spot label for non‑critical jobs
    steps:
      - uses: actions/checkout@v3
      - name: Build Docker image
        run: |
          docker build -t myapp:${{ github.sha }} .
          docker push myapp:${{ github.sha }}

  deploy:
    runs-on: [self-hosted, linux, fallback]   # fallback label for critical deploy
    needs: build
    steps:
      - uses: actions/checkout@v3
      - name: Deploy to prod
        run: ./deploy.sh

By explicitly labeling jobs, you keep the risk surface limited to non‑essential builds, while protecting production deployments.

Security and Best Practices

Even with the safeguards above, there are additional concerns:

  • Credential Leakage: Spot instances may be reclaimed without a graceful shutdown if the watchdog fails. Always use short‑lived, scoped tokens (e.g., GitHub’s RUNNER_TOKEN) and rotate them frequently.
  • Network Policies: Ensure that spot runners cannot reach production databases directly; use VPC security groups that only permit read‑only access for build stages.
  • Logging: Forward runner logs to a central Log Analytics workspace before termination. A simple CloudWatch agent configuration can capture the watchdog output.
  • Cost Monitoring: Spot prices fluctuate. Set a maximum hourly budget in your automation to prevent runaway costs if spot capacity spikes.
“Spot instances are a powerful tool, but treating them as a drop‑in replacement for reliable CI/CD workers is a recipe for hidden outages.” – Senior DevOps Engineer, CloudOps Corp.

Conclusion

Spot‑based self‑hosted runners can reduce CI costs, yet the hidden mechanics of preemption introduce reliability, security, and debugging challenges that are unsuitable for production pipelines. By understanding the termination workflow, adding a watchdog, and separating critical jobs onto a stable on‑demand runner, you gain the cost benefits without sacrificing pipeline integrity.

The key takeaway is to treat spot runners as “best‑effort” workers, not as the backbone of your release process. When you align job labels, enforce short‑lived credentials, and monitor termination events, you turn a risky shortcut into a controlled, cost‑effective layer of your DevOps stack.