Building a Zero-Downtime Application

Deploying software to production is easy. Deploying it without interrupting users is where things get hard.

My application is a Spring Boot microservices platform: a React frontend, an API gateway, a backend service, PostgreSQL, and Keycloak for authentication. Early on, deployments were simple build Docker images, swap the running containers, and hope everything came back online.

As the application matured, the cracks became obvious. Users experienced downtime whenever containers restarted. If a deployment failed halfway through, rolling back meant manually rebuilding the previous environment. Infrastructure services like PostgreSQL and Keycloak were entangled with application deployments changing one line in the backend could trigger a Keycloak restart. Every release carried more operational risk than it needed to.

I wanted a deployment strategy that could:

• deploy a new release without affecting active users
• verify health before switching traffic
• switch traffic instantly once a release was validated
• automatically recover if the new release became unhealthy
• keep shared infrastructure stable across deployments
• support canary rollouts later without redesigning everything.

No Kubernetes. I’ll explain why.

---

Why Not Kubernetes

Kubernetes is usually the first recommendation for blue-green deployments. For this workload, it was the wrong choice.

Parameters	Docker Compose	Kubernetes
Environment	Single VM	Cluster of nodes
Operational overhead	Low — one config file per stack	High — controllers, operators, CRDs
Debugging	docker logs, docker inspect	kubectl, multiple namespaces, contexts
Blue-green support	Manual but fully controllable	Native via Deployments + Services
Fit for this workload	Good — predictable single-VM traffic	Overkill
When to reconsider	When you outgrow one VM	When you need multi-node resilience

---

The Core Architecture

The fundamental insight that made everything else possible: separate infrastructure from application services, and run two application releases simultaneously.

Two Compose Projects

Instead of one large docker-compose.yml containing everything, I split the environment into two independent Compose projects.

Platform stack long-lived infrastructure that rarely changes:

• PostgreSQL
• Keycloak database
• Keycloak

Application stack services that change with every release:

• Backend
• API Gateway
• Frontend

Deploying a new backend version should never restart the authentication server. Keeping them in separate projects enforces this cleanly.

/infra/
  platform/
    docker-compose.yml        # PostgreSQL + Keycloak
  app/
    docker-compose.mainline.yml
    docker-compose.beta.yml

Two Application Releases at Once

Both mainline and beta application stacks run simultaneously, sharing platform infrastructure via a common Docker network.

# docker-compose.mainline.yml
networks:
  platform-net:
    external: true

services:
  backend-mainline:
    image: ${REGISTRY}/backend:${MAINLINE_TAG}
    container_name: backend-mainline
    networks: [platform-net]
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/actuator/health"]
      interval: 10s
      timeout: 5s
      retries: 12
    environment:
      SPRING_DATASOURCE_URL: jdbc:postgresql://postgres:5432/appdb

  gateway-mainline:
    image: ${REGISTRY}/gateway:${MAINLINE_TAG}
    container_name: gateway-mainline
    ports:
      - "8103:8080"
    networks: [platform-net]
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/actuator/health"]
      interval: 10s
      timeout: 5s
      retries: 12

  frontend-mainline:
    image: ${REGISTRY}/frontend:${MAINLINE_TAG}
    container_name: frontend-mainline
    ports:
      - "8101:80"
    networks: [platform-net]
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:80/health"]
      interval: 10s
      timeout: 5s
      retries: 6

# docker-compose.beta.yml -- same structure, different ports and container names
services:
  backend-beta:
    image: ${REGISTRY}/backend:${BETA_TAG}
    container_name: backend-beta
    networks: [platform-net]
    # no port published -- only reachable through gateway

  gateway-beta:
    image: ${REGISTRY}/gateway:${BETA_TAG}
    container_name: gateway-beta
    ports:
      - "28103:8080"
    networks: [platform-net]

  frontend-beta:
    image: ${REGISTRY}/frontend:${BETA_TAG}
    container_name: frontend-beta
    ports:
      - "28101:80"
    networks: [platform-net]

The active release is determined entirely by Nginx. Docker just keeps both stacks alive.

                  Internet
                      │
                      ▼
                   Nginx
                      │
         ┌────────────┴────────────┐
         │                         │
   Mainline Stack             Beta Stack
   :8101  :8103             :28101  :28103
         └────────────┬────────────┘
                      │
        Shared Platform Services
        PostgreSQL + Keycloak

---

Traffic Routing with Nginx

Running two stacks is the easy half. The harder problem: routing traffic between them without restarting Nginx or changing DNS.

Permanent Upstreams

Both release environments are permanently defined. Deployments never touch upstream configuration.

# /etc/nginx/conf.d/upstreams.conf

upstream app_frontend_mainline {
    server 127.0.0.1:8101;
}
upstream app_frontend_beta {
    server 127.0.0.1:28101;
}

upstream app_gateway_mainline {
    server 127.0.0.1:8103;
}
upstream app_gateway_beta {
    server 127.0.0.1:28103;
}

Release Selection — Priority Order

Every request resolves to one value: mainline or beta. The decision follows a strict priority chain:

Incoming Request
       │
       ▼
 X-Release header?  ──yes──▶  Use header value
       │ no
       ▼
 ?release= param?   ──yes──▶  Use query value
       │ no
       ▼
 Release cookie?    ──yes──▶  Use cookie value
       │ no
       ▼
 Traffic bucket     ──────▶   Default (mainline)

# /etc/nginx/conf.d/release-routing.conf

# 1. Explicit header override (highest priority)
map $http_x_release $release_from_header {
    "beta"     "beta";
    "mainline" "mainline";
    default    "";
}

# 2. Query parameter override
map $arg_release $release_from_query {
    "beta"     "beta";
    "mainline" "mainline";
    default    "";
}

# 3. Sticky cookie
map $cookie_app_release $release_from_cookie {
    "beta"     "beta";
    "mainline" "mainline";
    default    "";
}

# 4. Traffic bucket (default: 100% mainline)
split_clients "${remote_addr}${http_user_agent}" $app_release_bucket {
    # 20% beta;   # uncomment for canary rollout
    * mainline;
}

# Resolve final release -- first non-empty value wins
map "$release_from_header:$release_from_query:$release_from_cookie" $app_release {
    ~^beta      "beta";
    ~^:beta     "beta";
    ~^::beta    "beta";
    default     $app_release_bucket;
}

Header override — QA sends X-Release: beta to access the candidate release. Normal users are unaffected. No separate URL needed.

Query override — https://app.example.com?release=beta. Useful for mobile testing and external tools where header injection is awkward.

Sticky cookie — Once a release is selected, Nginx writes a cookie so subsequent requests in the same session stay on the same environment. Without this, a user could bounce between releases across page loads.

Traffic bucket — The fallback. Designed with canary deployment in mind from day one. Enabling 20% beta traffic later is a single config change — no application code, no Docker changes, no workflow modifications.

Server Block

server {
    listen 443 ssl;
    server_name app.example.com;

    # Frontend routing
    location / {
        set $upstream_frontend "app_frontend_${app_release}";
        proxy_pass http://$upstream_frontend;

        # Write sticky cookie
        add_header Set-Cookie "app_release=${app_release}; Path=/; SameSite=Lax";

        # Automatic failover
        proxy_next_upstream error timeout http_502 http_503 http_504;
        proxy_next_upstream_tries 2;
    }

    # API routing
    location /api/ {
        set $upstream_gateway "app_gateway_${app_release}";
        proxy_pass http://$upstream_gateway;

        add_header Set-Cookie "app_release=${app_release}; Path=/; SameSite=Lax";

        proxy_next_upstream error timeout http_502 http_503 http_504;
        proxy_next_upstream_tries 2;
    }
}

Automatic Failover

proxy_next_upstream handles the case where the selected release becomes unhealthy mid-request. If beta returns a 502/503/504, Nginx transparently retries using mainline. The user gets a response; the release cookie updates so future requests avoid the broken environment.

This matters most in the first few minutes after promotion, when you’re watching logs and hoping nothing breaks.

---

GitHub Actions Deployment Pipeline

Repository Layout

.github/
  workflows/
    platform-deploy.yml    # PostgreSQL + Keycloak -- runs rarely
    app-deploy.yml         # Build + deploy beta
    app-promote.yml        # Promote beta → mainline (manual trigger)

Deployment Timeline

Every release follows the same sequence. Nothing is skipped.

Code Push / Tag
      │
      ▼
Build Docker Images
      │
      ▼
Push to Artifact Registry
      │
      ▼
Deploy Beta Stack
(mainline untouched)
      │
      ▼
Health Checks
(all 3 services must pass)
      │
      ├── FAIL → Pipeline stops, mainline stays active
      │
      ▼
Beta running, validated
      │
   [manual]
      │
      ▼
Promote Beta → Mainline
      │
      ▼
Users receive new release

Application Deploy Workflow

# .github/workflows/app-deploy.yml
name: Deploy Application

on:
  push:
    tags: ['v*']

env:
  REGISTRY: us-central1-docker.pkg.dev/${{ vars.GCP_PROJECT }}/app

jobs:
  build:
    runs-on: ubuntu-latest
    outputs:
      image_tag: ${{ steps.meta.outputs.version }}
    steps:
      - uses: actions/checkout@v4

      - name: Authenticate to Google Cloud
        uses: google-github-actions/auth@v2
        with:
          credentials_json: ${{ secrets.GCP_SA_KEY }}

      - name: Configure Docker for Artifact Registry
        run: gcloud auth configure-docker us-central1-docker.pkg.dev

      - name: Extract tag
        id: meta
        run: echo "version=${GITHUB_REF#refs/tags/}" >> $GITHUB_OUTPUT

      - name: Build and push backend
        run: |
          docker build -t $REGISTRY/backend:${{ steps.meta.outputs.version }} ./backend
          docker push $REGISTRY/backend:${{ steps.meta.outputs.version }}

      - name: Build and push gateway
        run: |
          docker build -t $REGISTRY/gateway:${{ steps.meta.outputs.version }} ./gateway
          docker push $REGISTRY/gateway:${{ steps.meta.outputs.version }}

      - name: Build and push frontend
        run: |
          docker build -t $REGISTRY/frontend:${{ steps.meta.outputs.version }} ./frontend
          docker push $REGISTRY/frontend:${{ steps.meta.outputs.version }}

  deploy-beta:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - name: Deploy beta to production VM
        uses: appleboy/ssh-action@v1
        with:
          host: ${{ secrets.VM_HOST }}
          username: deploy
          key: ${{ secrets.VM_SSH_KEY }}
          script: |
            cd /opt/app

            export BETA_TAG=${{ needs.build.outputs.image_tag }}
            export REGISTRY=${{ env.REGISTRY }}
            docker compose -f docker-compose.beta.yml pull

            # Recreate beta containers only - mainline untouched
            docker compose -f docker-compose.beta.yml up -d --force-recreate

            echo "$BETA_TAG" > .beta-tag

  health-check:
    needs: deploy-beta
    runs-on: ubuntu-latest
    steps:
      - name: Wait for beta to become healthy
        uses: appleboy/ssh-action@v1
        with:
          host: ${{ secrets.VM_HOST }}
          username: deploy
          key: ${{ secrets.VM_SSH_KEY }}
          script: |
            check_health() {
              local service=$1
              local url=$2
              local retries=24
              local wait=5

              echo "Waiting for $service..."
              for i in $(seq 1 $retries); do
                if curl -sf "$url" > /dev/null 2>&1; then
                  echo "$service is healthy"
                  return 0
                fi
                echo "  attempt $i/$retries - sleeping ${wait}s"
                sleep $wait
              done

              echo "$service failed health check after $((retries * wait))s"
              return 1
            }

            check_health "backend-beta"  "http://localhost:8080/actuator/health"
            check_health "gateway-beta"  "http://localhost:28103/actuator/health"
            check_health "frontend-beta" "http://localhost:28101/health"

            echo "All beta services healthy. Ready for promotion."

At this point, production traffic is still 100% on mainline. Beta is running, healthy, and waiting.

Promotion Workflow – Separate, Manual

# .github/workflows/app-promote.yml
name: Promote Beta to Mainline

on:
  workflow_dispatch:   # manual trigger only
    inputs:
      confirm:
        description: 'Type PROMOTE to confirm'
        required: true

jobs:
  promote:
    runs-on: ubuntu-latest
    if: ${{ github.event.inputs.confirm == 'PROMOTE' }}
    steps:
      - name: Promote beta → mainline
        uses: appleboy/ssh-action@v1
        with:
          host: ${{ secrets.VM_HOST }}
          username: deploy
          key: ${{ secrets.VM_SSH_KEY }}
          script: |
            cd /opt/app

            export MAINLINE_TAG=$(cat .beta-tag)
            envsubst < docker-compose.mainline.yml.tpl > docker-compose.mainline.yml

            docker compose -f docker-compose.mainline.yml up -d --force-recreate

            for i in $(seq 1 24); do
              if curl -sf http://localhost:8103/actuator/health > /dev/null 2>&1; then
                echo "Mainline promoted successfully to $MAINLINE_TAG"
                exit 0
              fi
              sleep 5
            done

            echo "Mainline failed to start - traffic still on previous release"
            exit 1

The promotion workflow is triggered manually from GitHub Actions after I’ve spent time validating beta – checking API responses, running smoke tests, verifying logs.

---

What Happened the First Time It Saved Me

Three months into running this setup, I promoted a release that had a bug in the JWT token validation logic. The gateway was rejecting tokens with a non-standard aud claim that worked fine in the previous version but broke after a library upgrade I hadn’t caught in testing.

Here’s exactly what happened:

14:30 – Promotion workflow ran. Mainline updated to v1.4.2.

14:34 – First user report: login returning 401. I checked gateway-mainline logs:

WARN  o.s.s.web.DefaultSecurityFilterChain - JWT validation failed:
  Invalid audience claim. Expected [api], got [api, mobile]

Immediately clear – the new gateway was rejecting tokens the old one accepted.

14:35 – Checked that gateway-beta (now the old version, still running) was still healthy:

curl -sf http://localhost:28103/actuator/health | jq .status
# "UP"

14:36 – Updated .mainline-tag back to v1.4.1, ran promotion in reverse:

export MAINLINE_TAG=v1.4.1
docker compose -f docker-compose.mainline.yml up -d --force-recreate

14:38 – Mainline healthy again. Login working. Nginx continued routing to mainline throughout no config changes needed.

Eight minutes from first report to resolution. No database restore. No emergency image rebuild. The previous release was sitting right there.

Before this setup, the same incident would have been: identify bug, find the commit, rebuild image, push to registry, redeploy, wait for Spring Boot startup. Probably 25–30 minutes during which users couldn’t log in.

---

Observing the Deployment

Blue-green deployment without visibility is just hope with extra steps. Three sources cover the operational picture without adding infrastructure.

Docker Health Status

# See health state of all app containers at a glance
docker ps --format "table {{.Names}}\t{{.Status}}" | grep -E "mainline|beta"

# NAMES                  STATUS
# frontend-mainline      Up 3 hours (healthy)
# gateway-mainline       Up 3 hours (healthy)
# backend-mainline       Up 3 hours (healthy)
# frontend-beta          Up 12 minutes (healthy)
# gateway-beta           Up 12 minutes (healthy)
# backend-beta           Up 12 minutes (starting)

The (starting) status tells you a container passed its first check but hasn’t reached the full retry threshold yet. This is why the health-check script waits for all services explicitly rather than trusting Docker’s “running” status.

Nginx Access Logs

Once beta is promoted, the access log is the fastest way to confirm traffic switched:

tail -f /var/log/nginx/access.log | grep --line-buffered "app_release"

# 203.0.113.42 - [14:38:01] "GET /api/user" 200 - cookie: app_release=mainline
# 198.51.100.7 - [14:38:02] "GET /api/orders" 200 - cookie: app_release=mainline

Adding $cookie_app_release to the Nginx log format makes it trivial to see which release is serving each request and spot any unexpected routing.

Spring Boot Actuator

Each service exposes /actuator/health with component-level detail:

curl -s http://localhost:8080/actuator/health | jq .

# {
#   "status": "UP",
#   "components": {
#     "db": { "status": "UP" },
#     "keycloak": { "status": "UP" },
#     "diskSpace": { "status": "UP" }
#   }
# }

A gateway that is “UP” overall but has a downstream component in “DOWN” state is a problem that Docker’s container-level health check would miss. The actuator endpoint catches it.

---

Secrets and Configuration

Production config is injected through GitHub Secrets and Variables – never stored in the repository.

# Injected at deploy time, not committed
environment:
  SPRING_DATASOURCE_URL: ${{ secrets.DB_URL }}
  SPRING_DATASOURCE_PASSWORD: ${{ secrets.DB_PASSWORD }}
  KEYCLOAK_CLIENT_SECRET: ${{ secrets.KC_CLIENT_SECRET }}
  JWT_SECRET: ${{ secrets.JWT_SECRET }}
  STORAGE_BUCKET: ${{ vars.GCP_STORAGE_BUCKET }}
  SMTP_HOST: ${{ vars.SMTP_HOST }}
  SMTP_PASSWORD: ${{ secrets.SMTP_PASSWORD }}

The repository contains deployment logic. Not production credentials.

---

Nginx Config – Transactional Updates

For the rare infrastructure changes that do require Nginx config updates, I wrote a script that treats the update transactionally:

#!/bin/bash
# nginx-update.sh

NGINX_CONF="/etc/nginx/conf.d/release-routing.conf"
NGINX_CONF_BACKUP="${NGINX_CONF}.bak"

# Backup current working config
cp "$NGINX_CONF" "$NGINX_CONF_BACKUP"

# Apply new config
cp "$1" "$NGINX_CONF"

# Validate before touching the live server
if ! nginx -t 2>/dev/null; then
  echo "Nginx config validation failed - restoring backup"
  cp "$NGINX_CONF_BACKUP" "$NGINX_CONF"
  nginx -t && echo "Backup restored successfully"
  exit 1
fi

# Safe to reload
nginx -s reload
echo "Nginx updated successfully"

If validation fails, the previous working config is restored automatically. This has already caught a broken upstream block during an infrastructure change before it reached the live server.

---

Lessons That Actually Mattered

Separate infrastructure from applications. PostgreSQL and Keycloak should never restart because you changed a Spring Boot endpoint. Once I split the Compose projects, deployments became far less risky.

Deployment and promotion are different operations. Deploy prepares a release. Promotion exposes it. The gap between those two steps is where you validate. Collapsing them into one step is where most deployment anxiety comes from.

Health checks, not container status. Docker reports a container as “running” long before the Spring Boot application is ready. Use healthcheck in Compose and poll the actuator endpoint explicitly before considering a deployment complete.

Keep the previous release alive. The entire value of rollback depends on not destroying mainline when you deploy beta. If you tear it down before validating the new release, you’ve eliminated your escape route.

Nginx doesn’t need to change during deployments. Both upstreams are permanently configured. Deploying a new version changes containers, not routing config. This made the system significantly simpler to reason about under pressure.

Log which release is serving each request. Adding $cookie_app_release to the Nginx log format was a 5-minute change that saved hours during the JWT incident. Know which environment your traffic is hitting before something breaks, not after.