t0ul

Ship, Securely: Appendix

· 7 min read · #devsecops, #ship-securely, #security

Part of Ship, Securely, a hands-on DevSecOps book.

Appendix: Image Signing — Offline Digest vs Registry Signatures

Two ways to prove “what we built is what we run”:

A) Offline (no registry): sign the image digest as a blob

  • What it proves: You signed the SHA256 digest of the promoted image (GOOD).
  • Where the signature lives: In your repo as artifacts/image-digest.bundle (not attached to the image).
  • When to use: Air-gapped demos, workshops without a registry, or when you only need an evidence pack (auditors/customers).
  • Limitations: Admission controllers (Kyverno) can’t verify this at deploy time because the signature isn’t stored alongside the image.

B) Registry signatures (recommended for class): sign the image reference in a registry

  • What it proves: Your registry holds an OCI-native signature attached to the image (subject to the image’s digest).
  • Where the signature lives: In the registry (e.g., localhost:5000) as an OCI artifact.
  • When to use: Any environment with a registry (local or remote). Enables cluster-side verification via Kyverno/Gatekeeper.
  • Bonus: Lets you block unsigned images at admission using a Kyverno verifyImages policy.

Class guidance: favor Registry signatures so students can see Kyverno reject unsigned images and allow signed ones.

Kyverno: Verify Image Signatures (keyed, registry mode)

1) Makefile helpers (generate policy from your cosign.pub and apply)

Add these targets:

# Create a verify policy by embedding your cosign.pub (no guessing about ConfigMaps)
KYVERNO_VERIFY_OUT ?= policies/verify-image-signatures.yaml
VERIFY_IMAGE_PATTERN ?= "localhost:5000/ship-securely/*"  # <-- adjust for your class registry/repo

kyverno-gen-verify-policy:
	@test -s cosign.pub || (echo "cosign.pub missing. Run: cosign generate-key-pair"; exit 1)
	@echo "== Generating Kyverno verifyImages policy =="
	@echo "apiVersion: kyverno.io/v1"                                         >  $(KYVERNO_VERIFY_OUT)
	@echo "kind: ClusterPolicy"                                             >> $(KYVERNO_VERIFY_OUT)
	@echo "metadata:"                                                       >> $(KYVERNO_VERIFY_OUT)
	@echo "  name: verify-image-signatures"                                 >> $(KYVERNO_VERIFY_OUT)
	@echo "spec:"                                                           >> $(KYVERNO_VERIFY_OUT)
	@echo "  rules:"                                                        >> $(KYVERNO_VERIFY_OUT)
	@echo "  - name: require-cosign"                                        >> $(KYVERNO_VERIFY_OUT)
	@echo "    match:"                                                      >> $(KYVERNO_VERIFY_OUT)
	@echo "      resources:"                                                >> $(KYVERNO_VERIFY_OUT)
	@echo "        kinds: [\"Pod\",\"Deployment\",\"StatefulSet\",\"DaemonSet\",\"Job\",\"CronJob\"]" >> $(KYVERNO_VERIFY_OUT)
	@echo "    verifyImages:"                                               >> $(KYVERNO_VERIFY_OUT)
	@echo "    - imageReferences: [$(VERIFY_IMAGE_PATTERN)]"                >> $(KYVERNO_VERIFY_OUT)
	@echo "      attestors:"                                                >> $(KYVERNO_VERIFY_OUT)
	@echo "      - entries:"                                                >> $(KYVERNO_VERIFY_OUT)
	@echo "        - keys:"                                                 >> $(KYVERNO_VERIFY_OUT)
	@echo "            publicKeys: |"                                       >> $(KYVERNO_VERIFY_OUT)
	@sed 's/^/              /' cosign.pub                                   >> $(KYVERNO_VERIFY_OUT)
	@echo "✅ wrote $(KYVERNO_VERIFY_OUT)"

kyverno-apply-verify: kyverno-gen-verify-policy
	@$(MAKE) kyverno-apply

VERIFY_IMAGE_PATTERN controls which images must be signed. For the class, set it to the registry/repo you’ll use (examples below).

2) Class flow — Registry signing + Kyverno verification

Prereqs

# Once
cosign generate-key-pair                 # creates cosign.key / cosign.pub
make kyverno-install                     # install Kyverno via helm (already in Makefile)

A. Start / choose a registry

  • If you already have a registry URL for class, use that as IMAGE_REG.
  • For a local demo registry:
make reg-up                               # starts localhost:5000 via docker compose

B. Build, promote, push

# Build and deploy locally (optional), but we need a PUSHED image to sign in the registry
IMAGE_REG=localhost:5000/ make build promote
docker push localhost:5000/ship-securely/app:good-$(date +%Y%m%d)-<sha>

C. Sign the image in the registry

IMAGE_REG=localhost:5000/ make sign_image_registry
# optional: prove verification from CLI (also saved into artifacts/cosign-image-verify.txt)
IMAGE_REG=localhost:5000/ make verify_image_registry

D. Enforce at admission (Kyverno)

# Generate policy that embeds cosign.pub and targets your registry/images:
VERIFY_IMAGE_PATTERN='"localhost:5000/ship-securely/*"' make kyverno-apply-verify

E. See it block an unsigned image

  • Create a quick “unsigned” tag and try to apply a workload using it:
# Make an unsigned tag by pushing without signing:
docker tag localhost:5000/ship-securely/app:good-$(TAG) localhost:5000/ship-securely/app:unsigned-$(TAG)
docker push localhost:5000/ship-securely/app:unsigned-$(TAG)

# Patch your Deployment to use :unsigned-$(TAG) temporarily
kubectl -n ship set image deploy/app app=localhost:5000/ship-securely/app:unsigned-$(TAG) || true
# Expect: Kyverno admission rejection. Capture the output as evidence:
kubectl -n ship apply -f infra/k8s/deployment.yaml 2>&1 | tee artifacts/policy-tests.txt

F. Switch back to the signed image (allowed)

kubectl -n ship set image deploy/app app=$(IMAGE_REPO):good-$(TAG)
# or re-apply your standard deployment pointing at the GOOD tag

Tips for class:

  • Put your registry URL and repo pattern on a slide (VERIFY_IMAGE_PATTERN students must set).
  • Have students run cosign verify --key cosign.pub <their-image> to reassure themselves before Kyverno enforces it.
  • If anyone hits “refused by policy”, the error will name verify-image-signatures/require-cosign and show the image ref that failed.

3) Example configurations per environment

  • Local registry demo (Docker compose registry):

    • IMAGE_REG=localhost:5000/
    • VERIFY_IMAGE_PATTERN='"localhost:5000/ship-securely/*"'

  • Shared class registry (e.g., registry.class.tld:5000):

    • IMAGE_REG=registry.class.tld:5000/
    • VERIFY_IMAGE_PATTERN='"registry.class.tld:5000/ship-securely/*"'

  • Cloud registry (if later):

    • Use the real host (e.g., ghcr.io/org/*, gcr.io/project/*, 123456.dkr.ecr.region.amazonaws.com/*)

4) Evidence to expect (already wired in your Makefile’s evidence)

  • artifacts/cosign-image-verify.txt (CLI proof for registry mode)
  • artifacts/image-digest.txt + .bundle (offline mode)
  • policies/verify-image-signatures.yaml (the exact policy applied)
  • artifacts/policy-tests.txt (admission rejection from an unsigned tag)

Need me to generate a one-page “Instructor Notes” (slide-ready) that lists the exact 6 commands your students will run, with the two variables they must set (IMAGE_REG, VERIFY_IMAGE_PATTERN)?

Appendix: Instructor Notes — Registry Signing + Admission Verification (Kyverno)

What students will do (6 steps)

  1. Start/choose a registry
  2. Build → promote → push image
  3. Sign the pushed image with cosign
  4. Install Kyverno and apply a verifyImages policy (using your cosign.pub)
  5. Prove enforcement (unsigned image rejected)
  6. Deploy signed image, collect evidence

Two variables to set (put these on a slide)

export IMAGE_REG=localhost:5000/                 # or your class registry, e.g., registry.class.tld:5000/
export VERIFY_IMAGE_PATTERN='"localhost:5000/ship-securely/*"'   # pattern Kyverno must enforce

If you’re using a shared registry, replace localhost:5000 with the class registry host and keep the trailing slash.

One-time prep (everyone)

cosign generate-key-pair            # creates cosign.key / cosign.pub
make kyverno-install                # installs Kyverno (Helm)

6-Command student runbook (copy/paste)

1) Start a local registry (skip if you provide a class registry)

make reg-up                         # starts localhost:5000 via docker compose

2) Build → promote → push to the registry

IMAGE_REG=$IMAGE_REG make build promote
docker push ${IMAGE_REG}ship-securely/app:good-$(date +%Y%m%d)-$(git rev-parse --short HEAD)

3) Sign the image in the registry (OCI signature)

IMAGE_REG=$IMAGE_REG make sign_image_registry
IMAGE_REG=$IMAGE_REG make verify_image_registry     # writes artifacts/cosign-image-verify.txt

4) Enforce at admission (Kyverno verifyImages with your public key)

VERIFY_IMAGE_PATTERN=$VERIFY_IMAGE_PATTERN make kyverno-apply-verify
# (this generates policies/verify-image-signatures.yaml from cosign.pub and applies all policies/)

5) Prove it blocks an unsigned image

# create/push an unsigned tag
docker tag ${IMAGE_REG}ship-securely/app:good-$(date +%Y%m%d)-$(git rev-parse --short HEAD) \
           ${IMAGE_REG}ship-securely/app:unsigned-$(date +%Y%m%d)-$(git rev-parse --short HEAD)
docker push ${IMAGE_REG}ship-securely/app:unsigned-$(date +%Y%m%d)-$(git rev-parse --short HEAD)

# try to deploy the unsigned tag (expect admission rejection)
kubectl -n ship set image deploy/app app=${IMAGE_REG}ship-securely/app:unsigned-$(date +%Y%m%d)-$(git rev-parse --short HEAD) || true
kubectl -n ship apply -f infra/k8s/deployment.yaml 2>&1 | tee artifacts/policy-tests.txt

6) Deploy the signed image + collect evidence

# point back to the signed GOOD tag
kubectl -n ship set image deploy/app app=${IMAGE_REG}ship-securely/app:good-$(date +%Y%m%d)-$(git rev-parse --short HEAD)

# optional: re-run ZAP and metrics, and zip the pack
make zap-k8s
make metrics
make evidence

Evidence students should see (already wired in Makefile)

  • artifacts/cosign-image-verify.txt — proof the registry signature verifies
  • policies/verify-image-signatures.yaml — the exact Kyverno verify policy used
  • artifacts/policy-tests.txt — admission rejection for the unsigned image
  • artifacts/sbom.json, artifacts/sbom-image.json, artifacts/sbom.bundle, artifacts/cosign-verify.txt
  • artifacts/trivy-image.json, artifacts/semgrep.sarif, artifacts/checkov.json, artifacts/hadolint.txt
  • (if run) artifacts/zap/report.html, artifacts/falco-last10m.log, metrics snapshot

Troubleshooting slide (quick hitters)

  • Policy didn’t enforce?

    • Confirm kyverno-admission-controller is Ready: kubectl -n kyverno get deploy/kyverno-admission-controller
    • Verify policy target pattern matches your registry: grep imageReferences policies/verify-image-signatures.yaml
    • Re-apply policies: make kyverno-apply

  • Verify fails (cosign verify …):

    • Ensure you pushed the GOOD tag to the registry before signing.
    • Make sure cosign.pub matches the key used to sign.

  • Students using Windows: run in Git Bash / WSL; ensure Docker Desktop is running.

Optional: Offline mode (no registry)

If someone can’t reach a registry, they can still produce cryptographic proof:

make cd                    # builds, scans, promotes, deploys locally
make sign_image            # signs the image digest as a blob (offline)
make verify_image          # writes artifacts/cosign-image-verify.txt (offline)
make evidence

This doesn’t enable admission enforcement (no registry signature present), but it’s valid for the evidence pack.

Instructor checklist (before class)

  • Choose and announce IMAGE_REG (e.g., registry.class.tld:5000/ or localhost:5000/)
  • Put VERIFY_IMAGE_PATTERN on a slide (e.g., "registry.class.tld:5000/ship-securely/*")
  • Validate Kyverno install on your Minikube: make kyverno-install && make kyverno-apply
  • Dry run the 6 commands end-to-end; confirm policy-tests.txt shows rejection text
  • Ensure students have Docker, Minikube, kubectl, helm, cosign available (your Part I installs)

Want me to generate a single “Student Handout” PDF page from this (markdown source) you can print or share?

← Back to the blog