Chapter 04

CI/CD and the cloud

A commit on main should produce a tested, scanned, tagged image and put it in production without anyone opening a terminal — and let you undo it just as easily.

⏱ 24 min readLevel: intermediate → advancedHands-on

The pipeline

Commit, build and test, build image, scan, push, deploy to staging, smoke test, deploy to production
Fast checks first. Nothing is pushed until the tests pass.
.github/workflows/deploy.yml
name: build-and-deploy

on:
  push:
    branches: [main]
  pull_request:

env:
  REGISTRY: ghcr.io
  IMAGE_API: ghcr.io/${{ github.repository }}/api
  IMAGE_WEB: ghcr.io/${{ github.repository }}/web

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-java@v4
        with: { java-version: '21', distribution: temurin, cache: maven }
      - run: mvn -B verify            # Testcontainers spins up PostgreSQL here
        working-directory: api

      - uses: actions/setup-node@v4
        with: { node-version: '20', cache: npm, cache-dependency-path: web/package-lock.json }
      - run: npm ci && npm run lint && npm run test -- --watch=false --browsers=ChromeHeadless
        working-directory: web

  images:
    needs: test
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
      id-token: write               # for keyless signing
    steps:
      - uses: actions/checkout@v4
      - uses: docker/setup-buildx-action@v3
      - uses: docker/login-action@v3
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - id: meta
        uses: docker/metadata-action@v5
        with:
          images: ${{ env.IMAGE_API }}
          tags: |
            type=sha,format=long
            type=raw,value=latest

      - uses: docker/build-push-action@v6
        with:
          context: ./api
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

      - name: Scan for vulnerabilities
        uses: aquasecurity/trivy-action@0.24.0
        with:
          image-ref: ${{ env.IMAGE_API }}:sha-${{ github.sha }}
          severity: CRITICAL,HIGH
          exit-code: '1'            # fail the build, do not just warn

  deploy:
    needs: images
    runs-on: ubuntu-latest
    environment: production          # requires an approval, records the deploy
    steps:
      - name: Roll out
        run: |
          kubectl set image deployment/oms-api \
            api=${{ env.IMAGE_API }}:sha-${{ github.sha }}
          kubectl rollout status deployment/oms-api --timeout=180s
Three details that make this pipeline good

Tests run on pull requests too, so main is protected rather than repaired. Images are tagged by commit SHA, so any past state is one command away. The scan uses exit-code: 1 — a warning nobody reads is not a control.

Secrets

WhereMechanism
Local.env, gitignored
CIGitHub encrypted secrets / OIDC federation to the cloud
RuntimeCloud secret manager, injected as env vars or mounted files
Prefer OIDC over long-lived cloud keys

Storing a permanent access key in CI means a compromised workflow is a compromised cloud account, forever. GitHub's OIDC federation exchanges a short-lived token for a scoped role per run — nothing durable to steal. Every major cloud supports it.

Choosing a target

Single VM, managed container platform and Kubernetes compared
Single VMManaged containersKubernetes
ExamplesEC2, Droplet + ComposeCloud Run, App Runner, Container AppsEKS, GKE, AKS
SetupAn afternoonAn hourDays, then ongoing
ScalingManualAutomaticAutomatic, configurable
Zero-downtime deployDIYBuilt inBuilt in
Ops burdenYou patch the OSNear zeroHigh
Good forSide projectsMost teamsMany services, a platform team
Use managed PostgreSQL regardless

Running the database yourself means owning backups, point-in-time recovery, failover, minor-version upgrades and disk growth. RDS, Cloud SQL or Azure Database do all of it for a predictable monthly cost — reliably cheaper than the incident you avoid. See PostgreSQL chapter 6.

Kubernetes, if you need it

Ingress routing to web and api services, each backed by a deployment with replicas, plus secrets and a managed database
k8s/api-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: oms-api
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate: { maxSurge: 1, maxUnavailable: 0 }   # never dip below capacity
  selector:
    matchLabels: { app: oms-api }
  template:
    metadata:
      labels: { app: oms-api }
    spec:
      securityContext:
        runAsNonRoot: true
        seccompProfile: { type: RuntimeDefault }
      containers:
        - name: api
          image: ghcr.io/acme/oms/api:sha-9f2c1ab
          ports: [{ containerPort: 8080 }]
          env:
            - name: SPRING_PROFILES_ACTIVE
              value: prod
            - name: SPRING_DATASOURCE_PASSWORD
              valueFrom:
                secretKeyRef: { name: oms-db, key: password }
          resources:
            requests: { cpu: 250m, memory: 512Mi }
            limits:   { memory: 1Gi }        # memory limit yes, CPU limit no
          startupProbe:                       # JVMs are slow to boot
            httpGet: { path: /actuator/health/liveness, port: 8080 }
            failureThreshold: 30
            periodSeconds: 5
          readinessProbe:                     # gates traffic
            httpGet: { path: /actuator/health/readiness, port: 8080 }
            periodSeconds: 10
          livenessProbe:                      # restarts a wedged pod
            httpGet: { path: /actuator/health/liveness, port: 8080 }
            periodSeconds: 20
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities: { drop: ["ALL"] }
          volumeMounts:
            - { name: tmp, mountPath: /tmp }
      volumes:
        - name: tmp
          emptyDir: {}
Readiness and liveness are not the same probe

Readiness answers "should I get traffic?" — a pod that fails it is removed from the load balancer but left alone. Liveness answers "am I broken?" — failing it gets the pod killed. Pointing liveness at an endpoint that checks the database means a brief database blip restarts your entire fleet and turns a small outage into a large one.

Migrations during a rolling deploy

For a few seconds, version N and N+1 run simultaneously against one database. Every migration must therefore be compatible with both.

ChangeSafe?How to do it
Add a nullable columnYesDeploy directly
Add a NOT NULL columnNoAdd nullable → backfill → add constraint later
Rename a columnNoAdd new → write both → migrate reads → drop old
Drop a columnNoStop using it, deploy, drop in the next release
Add an indexCarefulCREATE INDEX CONCURRENTLY, outside a transaction
Expand, migrate, contract

Every breaking schema change becomes three safe releases. It feels slow until the first time a rename would have taken production down mid-deploy. And always set lock_timeout in migrations — a blocked ALTER TABLE queues every other query behind it.

Observability

api/src/main/resources/application-prod.yml
management:
  endpoints:
    web:
      exposure:
        include: health,info,metrics,prometheus
  endpoint:
    health:
      probes:
        enabled: true            # gives /health/liveness and /health/readiness
      show-details: when-authorized
  metrics:
    tags:
      application: oms-api
      version: ${APP_VERSION:unknown}

server:
  shutdown: graceful             # finish in-flight requests on SIGTERM
spring:
  lifecycle:
    timeout-per-shutdown-phase: 25s

logging:
  structured:
    format:
      console: ecs               # JSON logs the platform can index
Alert on four things first

Error rate, p99 latency, saturation (CPU, memory, connection pool) and database replication lag. Almost every real incident shows up in one of them, and a short alert list is one people still read after six months.

Rollback

the thing to practise before you need it
kubectl rollout undo deployment/oms-api            # previous revision
kubectl rollout history deployment/oms-api
kubectl set image deployment/oms-api api=ghcr.io/acme/oms/api:sha-7a1b3cd

# managed platforms: shift traffic back to the previous revision
gcloud run services update-traffic oms-api --to-revisions=oms-api-00042=100
Code rolls back. Data does not.

Redeploying the old image is seconds. Undoing a migration that dropped a column is a restore. This is the real reason for expand-migrate-contract: it keeps rollback a one-line operation.

Production checklist

  • ☐ Images tagged by commit SHA; latest never deployed.
  • ☐ Tests and an image scan block the pipeline on failure.
  • ☐ Secrets from a secret manager; CI authenticates via OIDC.
  • ☐ Containers non-root, read-only root filesystem, all capabilities dropped.
  • ☐ Startup, readiness and liveness probes configured separately.
  • ☐ Graceful shutdown enabled and the termination grace period longer than it.
  • ☐ Managed PostgreSQL with PITR; a restore tested.
  • ☐ Migrations backwards compatible with a lock_timeout.
  • ☐ TLS terminated at the ingress; HSTS on.
  • ☐ Structured logs, metrics and four alerts that page a human.
  • ☐ A rollback performed at least once, on purpose.

Why should a liveness probe avoid checking the database?

Liveness means "this process is unrecoverable, kill it". A dependency being briefly unavailable is not that. Put dependency checks in readiness, where the pod is simply taken out of rotation until it recovers.

You have shipped it

Four courses, a full application and a deployment pipeline. If you want the argument for why these four technologies belong together, read The Stack; if you want to go deeper on any one of them, every course is on the tutorials page.