Ship, Securely, Ch. 4: Ratcheting and Supply-Chain Hardening

September 23, 2025 · 13 min read · #devsecops, #ship-securely, #security

Part of Ship, Securely, a hands-on DevSecOps book.

Chapter 4. Trivy Image Workflow

1) Build → Scan → Decide (quarantine first)

Tag every fresh build as quarantine until it passes policy.

Makefile additions

QUAR := $(IMAGE_REPO):quarantine-$(SHA)
GOOD := $(IMAGE_REPO):good-$(TAG)

build:
	docker build -t $(IMAGE) app
	docker tag $(IMAGE) $(QUAR)   # mark as quarantine by default

sca:
	@echo "== Trivy (image scan) =="
	docker run --rm -v $(PWD):/workspace \
	  -v $(PWD)/.trivy-db:/root/.cache/trivy \
	  aquasec/trivy:latest image --ignore-unfixed \
	  --format json -o /workspace/artifacts/trivy-image.json $(QUAR) || true

We scan $(QUAR) specifically so it never accidentally ships before review.

2) Policy: when to promote

Day-1 default (tune later):

PASS if no CRITICAL vulnerabilities and no HIGH with a fix available in direct dependencies.
Else: stay in quarantine, open a ticket, or accept temporary risk with expiry.

Automate with a tiny shell gate:

sca-gate:
	@./scripts/sca_gate.sh artifacts/trivy-image.json

scripts/sca_gate.sh (example)

#!/usr/bin/env bash
set -euo pipefail
f="${1:-artifacts/trivy-image.json}"
[ -s "$f" ] || { echo "no trivy report"; exit 1; }
crit=$(jq '[.Results[].Vulnerabilities[]? | select(.Severity=="CRITICAL")] | length' "$f")
high_fix=$(jq '[.Results[].Vulnerabilities[]? | select(.Severity=="HIGH" and (.FixedVersion != null and .FixedVersion != ""))] | length' "$f")
if [ "$crit" -gt 0 ] || [ "$high_fix" -gt 0 ]; then
  echo "❌ Gate fail: CRITICAL=$crit HIGH(w/fix)=$high_fix"
  exit 2
else
  echo "✅ Gate pass"
fi

Make it executable:

chmod +x scripts/sca_gate.sh

3) Promote (or don’t)

If sca-gate passes, promote the image tag we scan:

promote:
	@echo "Promoting $(QUAR) -> $(GOOD)"
	@docker tag $(QUAR) $(GOOD)
	@docker tag $(GOOD) $(IMAGE_LATEST)

If it fails, do not promote. Patch deps, rebuild, and rescan.

4) Deploy to Minikube (from promoted image)

Even without a registry, Minikube can pull whatever you load into the node.

cd: mk-up build sca sca-gate promote mk-image-load k8s-ns k8s-apply k8s-verify
	@echo "✅ Deployed $(GOOD) to Minikube ($(PROFILE))"

Note: k8s-apply should reference $(IMAGE_REPO) and a tag variable; or use kubectl set image … $(GOOD) post-apply.

5) Evidence pack

Every run leaves:

artifacts/trivy-image.json (scan result)
artifacts/sbom.json (inventory)
artifacts/cosign-verify.txt (if signing enabled)
Optional: make evidence zips everything for sharing.

6) Optional: sign & verify before promotion

sign:
	@if [ -f cosign.key ]; then cosign sign --key cosign.key $(QUAR); else echo "cosign.key missing; skipping"; fi

verify:
	@if [ -f cosign.pub ]; then cosign verify --key cosign.pub $(QUAR) > artifacts/cosign-verify.txt || true; else echo "cosign.pub missing; skipping"; fi

# Promote only after verify (optional stronger gate)
promote: verify

End-to-end demo (copy/paste)

make build                  # builds + tags :quarantine-<sha>
make sca                    # writes artifacts/trivy-image.json
make sca-gate               # enforces CRITICAL/HIGH(w/fix) policy
make promote                # only if gate passes (tags :good-<tag> + :latest)
make cd                     # loads/promotes to Minikube & deploys
make k8s-port               # get the URL; then run make zap if desired
make evidence               # bundle artifacts into a dated zip

Checklist

Quarantine tag exists and is scanned by default
Gate script enforces a simple, explainable policy
Promote target only runs on gate pass
Deployed image equals the promoted tag
Evidence saved under ./artifacts/ (scan + SBOM + verify)

Maturity ladder

Bronze: scanning + quarantine pattern in place
Silver: promotion gated; deploys only promoted images
Gold: signing enabled; verify required before promotion
Platinum: monthly thresholds tighten (e.g., block HIGH regardless of fix), metrics tracked

Should Fail

Short answer: yes—this is expected with the policy we put in. Your zap-k8s depends on cd, and cd depends on sca-gate. Trivy found 1 HIGH with a fix available, so the gate failed by design and blocked deploy—therefore ZAP didn’t run.

What the log means

Built image → tagged :quarantine-dev
Trivy scanned :quarantine-dev and wrote artifacts/trivy-image.json
Gate rule: fail if any CRITICAL, or any HIGH that has a fix → ❌ Gate fail
Because zap-k8s calls cd first, deploy was blocked and ZAP never started

See which vuln tripped the gate

jq -r '
  .Results[]
  | .Vulnerabilities[]?
  | select(.Severity=="HIGH" and (.FixedVersion != null and .FixedVersion != ""))
  | "\(.VulnerabilityID) | \(.PkgName) \(.InstalledVersion) -> fix: \(.FixedVersion) | \(.Title)"
' artifacts/trivy-image.json

Your choices (in class)

A) Fix it (preferred)

Most likely it’s an Alpine base package. Try bumping the final stage base image (and rebuild):

If your final image is alpine:3.20@sha256:…, bump to the latest patch (or 3.21):

# In app/Dockerfile final stage:
FROM alpine:3.21

Then:

make build sca
make sca-gate     # should pass if the fix is in the newer base
make cd           # deploys; now zap-k8s will work too

Even better long term: use a minimal base (e.g., distroless nonroot) to shrink CVE surface.

B) Temporarily relax for the demo (skip gate just this run)

If you just need ZAP to run right now, you can bypass the gate once:

# Manually do what cd would do, but without the gate:
make build sca
make promote                       # promote quarantine tag anyway
make mk-image-load k8s-ns k8s-apply k8s-verify
# Now run ZAP without re-running cd (since zap-k8s calls cd)
# Option 1: port-forward + docker ZAP manually:
kubectl -n ship port-forward deploy/app 8080:3000 &
docker run --rm -t --add-host=host.docker.internal:host-gateway \
  -v "$PWD/tools/zap":/zap/wrk -w /zap/wrk \
  zaproxy/zap-stable:latest \
  zap-baseline.py -t http://host.docker.internal:8080 -m 1 -T 3 -s \
  -r report.html -J report.json

C) Add a controlled bypass switch (nice for teaching)

If you want a clean toggle:

Patch your Makefile:

SKIP_GATE ?= 0

sca-gate:
	@if [ "$(SKIP_GATE)" = "1" ]; then \
	  echo "⚠️  Skipping SCA gate (SKIP_GATE=1)"; \
	else \
	  ./scripts/sca_gate.sh artifacts/trivy-image.json; \
	fi

cd: mk-up build sca sca-gate promote mk-image-load k8s-ns k8s-apply k8s-verify

Use it once:

make cd SKIP_GATE=1
make zap-k8s SKIP_GATE=1

(Still produces the Trivy report so you can discuss it, but doesn’t block.)

Quick recap for students

The block is intentional: “quarantine → scan → gate → promote.”
Today it failed on HIGH (fix available). Either patch (best) or use SKIP_GATE=1 for the exercise, then come back and remediate.

If you paste the jq output for the HIGH finding (ID + package), I’ll suggest the exact Dockerfile or go.mod change to clear it.

Yep—this is exactly the kind of thing the gate is supposed to catch.

You’ve got a HIGH in the Go stdlib bundled into your binary:

CVE-2025-47907 | stdlib v1.22.12 -> fix: 1.23.12, 1.24.6 | database/sql: Postgres Scan Race Condition

What it means (plain English)

Trivy inspects your gobinary and sees it was built with Go 1.22.12.
That Go toolchain version carries a HIGH vuln affecting database/sql (Postgres scanning race).
The fixed toolchains are Go 1.23.12 and Go 1.24.6 (no fixed 1.22 per your report).
Our gate fails on “HIGH with a fix available” → so it blocked deploy. Correct behavior.

Fastest fix (recommended)

Edit your builder image in app/Dockerfile to a patched Go:

- FROM golang:1.22-alpine AS build
+ FROM golang:1.24.6-alpine AS build

(If you prefer to be conservative: golang:1.23.12-alpine is also listed as fixed.)

Then rebuild/scan:

make build sca
make sca-gate      # should pass now
make cd            # deploys (then you can run make zap-k8s)

Optional but tidy: update your go.mod to align with the toolchain:

go mod tidy -go=1.24   # or -go=1.23

If you can’t upgrade right now (teaching/demo fallback)

Add the toggle we discussed so you can continue the flow while acknowledging risk:

# in Makefile
SKIP_GATE ?= 0

sca-gate:
	@if [ "$(SKIP_GATE)" = "1" ]; then \
	  echo "⚠️  Skipping SCA gate (SKIP_GATE=1)"; \
	else \
	  ./scripts/sca_gate.sh artifacts/trivy-image.json; \
	fi

Use it once:

make cd SKIP_GATE=1
make zap-k8s SKIP_GATE=1

(You’ll still have artifacts/trivy-image.json to show the class why you skipped.)

Want to be extra precise? (Reachability signal)

This CVE is in database/sql for Postgres. If your demo app doesn’t use a DB at all, the vuln is likely not reachable. For a more nuanced gate:

Add a govulncheck step on source (reachable vulns) and gate primarily on that, keeping Trivy as a secondary “FYI” signal.
Keep today’s simple gate for class, then upgrade policy after students see it in action.

TL;DR for the class

The gate blocked deploy because your Go toolchain had a HIGH with a fix.
Upgrade the builder image to Go 1.24.6 (or 1.23.12), rebuild, rescan → pass.
Or temporarily bypass with SKIP_GATE=1 to continue the exercise, then circle back and remediate.

Chapter 4. SBOM Diff & License Policy

Why this

Vulns rise when new deps quietly appear. A small SBOM diff catches supply-chain drift. A license allowlist avoids last-minute release blockers.

1) Produce today’s SBOM

You already get ./artifacts/sbom.json from make sbom or make ci. If not:

make sbom   # ensures artifacts/sbom.json exists

2) Keep a baseline and diff

Create or update a dated copy after each “release”:

make sbom-snapshot

Compare current SBOM to the previous release (the script auto-detects sensible pairs):

make sbom-diff

Artifact: artifacts/sbom-diff.txt shows ADDED/REMOVED packages (name@version) and a short summary.

3) License allowlist (policy you can enforce)

Create an allowlist at repo root:

# licenses-allowlist.txt
MIT
BSD-2-Clause
BSD-3-Clause
Apache-2.0
ISC
MPL-2.0

Scan licenses found in the SBOM:

make license-scan

Artifact: artifacts/license-scan.txt includes:

counts per license
any UNKNOWN licenses
any DISALLOWED licenses (not in your allowlist)

4) (Optional) Gate on licenses (WARN→BLOCK later)

Keep WARN mode for the MVP. When ready to enforce:

PROMOTE_BLOCK=1 make license-gate

Checklist

artifacts/sbom.json present (today)
A dated SBOM baseline kept (at least one)
artifacts/sbom-diff.txt reviewed (ADDED/REMOVED)
licenses-allowlist.txt committed
artifacts/license-scan.txt shows 0 DISALLOWED (or exception recorded)

Chapter 4. Golden Container Images

Pre-reqs

app/Dockerfile.base with placeholder digests
Cosign keypair (cosign generate-key-pair)
Kyverno installed (optional enforcement)

Steps

Pin base image digests (once)

make pin-digests
# paste the printed digests into app/Dockerfile.base ARGs

Build + secure the golden base
```
make golden-image
```
This runs:
- base-build → builds $(IMAGE_REPO):base-$(TAG) and :base-latest
- base-scan → Trivy → artifacts/trivy-image-base.json
- base-sbom → Syft → artifacts/sbom-base.json
- base-sign / base-verify → Cosign bundle → artifacts/sbom-base.bundle, artifacts/cosign-verify-base.txt
(Optional) Enforce org-only images at admission
```
make kyverno-apply-allow-org
make kyverno-test-allow-org
```
Evidence: artifacts/policy-allow-org-test.txt

Verify

Check artifacts exist:

artifacts/trivy-image-base.json
artifacts/sbom-base.json
artifacts/sbom-base.bundle
artifacts/cosign-verify-base.txt

Run make evidence (your ZIP now includes the golden artifacts).

Makefile commands

One-shot: make golden-image
Policy apply/test: make kyverno-apply-allow-org / make kyverno-test-allow-org
Optional CI: make ci INCLUDE_GOLDEN=1 KYVERNO_ENFORCE_ORG=1

Checklist

Base Dockerfile pinned by digest
Golden image built and scanned
SBOM generated and signed
(Optional) Kyverno org-allow policy applied & tested
Evidence ZIP contains base scan/SBOM/signature + policy test

Chapter 4. Golden Node Images

Steps (simulate outcomes on Minikube)

Enforce Pod Security (Restricted) on namespace
```
make pss-enforce
```

Harden the workload spec (already compatible with your app): In infra/k8s/deployment.yaml under spec.template.spec:

securityContext:
  runAsNonRoot: true
  seccompProfile: { type: RuntimeDefault }
containers:
- name: app
  securityContext:
    readOnlyRootFilesystem: true
    allowPrivilegeEscalation: false
    capabilities: { drop: ["ALL"] }

Apply:

make k8s-apply && make k8s-verify

(Optional) Runtime detection

make falco-install
sleep 10
make falco-export   # artifacts/falco-last10m.log

Verify

Try deploying an intentionally “bad” pod (privileged/root) to see enforcement fail (you can reuse your kyverno-bad-apply flow if you prefer a policy-based test).
Confirm k8s-verify passes for the hardened deployment.

Makefile commands

Namespace PSA labels: make pss-enforce
Re-apply manifests: make k8s-apply && make k8s-verify
Falco: make falco-install → make falco-export

Checklist

Namespace labeled pod-security: restricted
Deployment uses non-root, RO rootfs, no privilege escalation, drop ALL caps, seccomp RuntimeDefault
(Optional) Falco installed and logs captured
Evidence ZIP contains falco-last10m.log (if used)

Chapter 4. Dependency Mirrors

Pre-reqs

Docker Desktop (macOS/Windows) or Docker Engine (Linux)
Your repo root with infra/docker/dep-mirror.yaml

Steps

Start the mirrors
```
make dep-mirror-up
```
- Go proxy at http://localhost:8081
- (Optional) Verdaccio (npm) at http://localhost:4873

Point the project at the mirror

Go:

export GOPROXY=http://localhost:8081,direct
go clean -modcache && (cd app && go mod download)

npm (optional): add a project .npmrc
```
registry=http://localhost:4873/
```

Build as usual (downloads now pass via the mirror)
```
make build
```

Verify

For Go:

curl -fsS http://localhost:8081/healthz

Confirm a fresh dependency fetch is fast (served from cache) and your app builds.

Makefile commands

Start/stop: make dep-mirror-up / make dep-mirror-down
CI tip: set GOPROXY in your pipeline env so builds are deterministic.

Checklist

Mirror started (dep-mirror-up)
Project points to mirror (GOPROXY=... or .npmrc)
Build succeeds with mirror in place
Evidence captured (document GOPROXY in your runbook)

Chapter 4. Artifact Hygiene at Home

1) Tagging: predictable, unique, effortless

Use a semver + git metadata tag on every build. Example:

<app>:<semver>-<YYYYMMDD>-<shortsha> # e.g., app:0.1.0-20251009-a1b2c3d

Makefile snippet

APP_NAME ?= app
IMAGE_REG ?=
IMAGE_REPO ?= $(IMAGE_REG)ship-securely/$(APP_NAME)
DATE := $(shell date +%Y%m%d)
SHA  := $(shell git rev-parse --short HEAD 2>/dev/null || echo "dev")
TAG  ?= 0.1.0-$(DATE)-$(SHA)

IMAGE := $(IMAGE_REPO):$(TAG)
IMAGE_LATEST := $(IMAGE_REPO):latest

print-image:
	@echo "IMAGE=$(IMAGE)"

2) Immutability: don’t mutate tags

Treat every content tag (:0.1.0-…) as immutable.
You may move :latest for convenience, but never overwrite a content tag.

Makefile helper

tag-latest:
	@docker tag $(IMAGE) $(IMAGE_LATEST)

3) Retention: keep what matters, prune the rest

Keep last N content tags locally (e.g., 20).
Keep all evidence (./artifacts/*) for releases.
Periodically prune unused images:

docker image prune -f

Optionally log built images:

log-image:
	@mkdir -p artifacts
	@echo "$(IMAGE)" >> artifacts/images-$(DATE).txt

4) Registry or no registry?

Option A — No registry (simplest, default)

Use Minikube’s in-node cache:

mk-image-load:
	minikube -p $(PROFILE) image load $(IMAGE)

Pros: fewer moving parts, faster in class. Cons: not shareable with teammates out of the box.

Option B — Local registry (`registry:2`)

Spin up a registry on localhost:5000 to push/pull locally.

docker compose file (optional): infra/docker/registry.yaml

version: "3.9"
services:
  registry:
    image: registry:2
    ports: [ "5000:5000" ]
    restart: unless-stopped
    environment:
      REGISTRY_STORAGE_DELETE_ENABLED: "true"

Makefile helpers

reg-up:
	docker compose -f infra/docker/registry.yaml up -d

reg-down:
	docker compose -f infra/docker/registry.yaml down

push:
	@docker tag $(IMAGE) localhost:5000/$(APP_NAME):$(TAG)
	@docker push localhost:5000/$(APP_NAME):$(TAG)

If you use a local registry with Minikube, tell K8s to pull from localhost:5000 by either (a) enabling the Docker driver so node can reach host, or (b) port-forwarding/hostAliases. For this book, Option A is simpler.

5) Store SBOMs & signature proofs under `./artifacts`

You already generate:

artifacts/sbom.json (Syft)
artifacts/cosign-verify.txt (cosign verify)
artifacts/trivy-image.json (Trivy)

Keep them per build (zip with date) using your evidence target.

Checklist

Content tag format selected and implemented in Makefile
Immutable content tags (no overwrites)
Decision made: minikube image load (default) vs local registry
SBOM & signature evidence present under ./artifacts/

Chapter 4. Checklist

Gates & Ratcheting (policy-first)

Quarantine tag: New images are tagged quarantine-<sha> until they pass gates.
Pass criteria: No CRITICAL, and no HIGH with fixes in direct deps (tune per team).
Gate implementation: CI step sca-gate (Trivy JSON → pass/fail) wired after build/scan.
Promotion tag: On pass, retag to good-<tag> and/or :release-<semver>.
Ratcheting plan: 6-week path from WARN → ENFORCE documented (which checks flip each week).
Override path: Temporary risk acceptance file with owner + expiry required for bypass.

Golden Node Outcomes (simulated on Minikube)

Namespace PSS=Restricted enforced (e.g., make pss-enforce).
Pod hardening: drop caps, read-only FS, seccomp/apparmor profiles applied to sample app.
Runtime detection (optional): basic signals enabled; evidence captured in artifacts.
Verification: kubectl/audit evidence attached to the runbook.

Golden Container Images (digest-pinned, scanned, SBOM’d, signed)

Minimal runtime base (distroless/alpine), non-root USER, fixed UID/GID.
Digest pins: base and runtime pinned; pins recorded in repo.
Build base image via Make: base-build.
Scan: base-scan (Trivy) → artifacts/trivy-image-base.json.
SBOM: base-sbom (Syft) → artifacts/sbom-base.json.
Sign & verify: base-sign / base-verify (Cosign) → bundle + verify log.
(Admission) Enforce org-only image prefixes with Kyverno; tests included.

SBOM Diff & License Policy

SBOM produced for each build (make sbom) → artifacts/sbom.json.
Diff rule: Define when to run diff (every merge to main or at RC).
License allowlist: committed list (e.g., MIT/Apache-2.0/BSD/ISC/MPL-2.0); reviewers named.
License scan: make license-scan → artifacts/license-scan.txt (flags UNKNOWN/DISALLOWED).
Ticketing: violations create issues with context (package, version, license, rationale).

Dependency Mirrors (local-first supply chain)

Mirror up: make dep-mirror-up (Go proxy + optional npm cache).
Route deps: set GOPROXY=http://localhost:8081,direct and/or .npmrc registry to mirror.
Build via mirror: make build succeeds with caching in place.
Evidence: health checks + config captured in artifacts and runbook.

Artifact Hygiene at Home

Tag schema: <semver>-<YYYYMMDD>-<shortsha> used consistently.
Repro build: pinned versions; immutable artifacts with checksums.
Registry choice: documented (Minikube image cache vs local registry) with pros/cons.
SBOM + signatures stored alongside artifacts; retention policy defined.

Promotion & Evidence

Promotion rule: “promote on pass” criteria captured (which gates must be green).
Bundle: evidence ZIP contains Trivy JSON, SBOM, license scan, Cosign bundle, logs.
Runbook: links to artifacts, rollback steps, smoke endpoints.
Audit trail: CI summary lists artifacts & commit; exceptions tracked with expiry.

← Back to the blog

Ship, Securely, Ch. 4: Ratcheting and Supply-Chain Hardening

Chapter 4. Trivy Image Workflow

1) Build → Scan → Decide (quarantine first)

2) Policy: when to promote

3) Promote (or don’t)

4) Deploy to Minikube (from promoted image)

5) Evidence pack

6) Optional: sign & verify before promotion

End-to-end demo (copy/paste)

Checklist

Maturity ladder

Should Fail

What the log means

See which vuln tripped the gate

Your choices (in class)

A) Fix it (preferred)

B) Temporarily relax for the demo (skip gate just this run)

C) Add a controlled bypass switch (nice for teaching)

Quick recap for students

What it means (plain English)

Fastest fix (recommended)

If you can’t upgrade right now (teaching/demo fallback)

Want to be extra precise? (Reachability signal)

TL;DR for the class

Chapter 4. SBOM Diff & License Policy

Why this

1) Produce today’s SBOM

2) Keep a baseline and diff

3) License allowlist (policy you can enforce)

4) (Optional) Gate on licenses (WARN→BLOCK later)

Checklist

Chapter 4. Golden Container Images

Pre-reqs

Steps

Verify

Makefile commands

Checklist

Chapter 4. Golden Node Images

Steps (simulate outcomes on Minikube)

Verify

Makefile commands

Checklist

Chapter 4. Dependency Mirrors

Pre-reqs

Steps

Verify

Makefile commands

Checklist

Chapter 4. Artifact Hygiene at Home

1) Tagging: predictable, unique, effortless

2) Immutability: don’t mutate tags

3) Retention: keep what matters, prune the rest

4) Registry or no registry?

Option A — No registry (simplest, default)

Option B — Local registry (registry:2)

5) Store SBOMs & signature proofs under ./artifacts

Checklist

Chapter 4. Checklist

Gates & Ratcheting (policy-first)

Golden Node Outcomes (simulated on Minikube)

Golden Container Images (digest-pinned, scanned, SBOM’d, signed)

SBOM Diff & License Policy

Dependency Mirrors (local-first supply chain)

Artifact Hygiene at Home

Promotion & Evidence

Option B — Local registry (`registry:2`)

5) Store SBOMs & signature proofs under `./artifacts`