Ship, Securely, Ch. 3: Reviewing Results

September 16, 2025 · 18 min read · #devsecops, #ship-securely, #security

Part of Ship, Securely, a hands-on DevSecOps book.

Chapter 3. Reviewing Results

SAST — Static Application Security Testing

What it is
SAST analyzes source code without executing it to find risky patterns (e.g., unsanitized inputs, unsafe APIs). In this project we use Semgrep (via the sast target) to scan the Go app and write results to artifacts/semgrep.json.

Why we need it
Catches code-level flaws early, while they’re cheap to fix, and shifts security left. Typical hits: injection, XSS, insecure crypto, unsafe deserialization, etc.

What it shows (how to read it)
Each finding includes a rule/check ID, severity, a message, and a location (file + line). Prioritize by severity and by “tainted data flow” (user input reaching a dangerous sink).

Use:

jq -r '.results[]? | "\(.extra.severity)\t\(.check_id)\t\(.path):\(.start.line)\t\(.extra.message)"' artifacts/semgrep.json | column -t | head

Signal vs noise

• Signal: “High/Critical” issues; user-controlled data in sinks (e.g., HTML output, DB queries).
• Noise: Style or low-confidence patterns that aren’t reachable or relevant. Triaging is normal.

Limits May miss runtime-only bugs and can raise false positives. Tune rules over time.

Makefile link make sast (and optionally a gate like sast_block if you enable it).

Quick checklist

• Triage High/Critical first.
• Validate the code path (is the finding reachable?).
• Fix or suppress with rationale (minimize ignores).
• Re-run and confirm the artifact is clean(er).

SiC — Secrets in Code

What it is Search for credentials in code (API keys, tokens, passwords). We use TruffleHog via secrets and write to artifacts/trufflehog.json. In this book we favor offline, permissive scanning for training.

Why we need it Leaked secrets are a fast path to compromise (cloud keys, DB creds). Catching them before commit/merge is critical.

What it shows (how to read it) Look for the detector (e.g., AWS, GitHub PAT), the file/line, and whether it’s verified (if verification is enabled) or redacted. Example summaries:

jq -r 'select(.Source=="filesystem") | [.Detector, .Path, .RawV2] | @tsv' artifacts/trufflehog.json | head

Signal vs noise

• Signal: Real-looking keys/tokens, especially with known formats.
• Noise: Dummy/test strings, IDs that look random but aren’t secrets.

Limits Pattern/entropy-based: may miss weak-looking secrets or raise false positives. History scans can be noisy (we keep it local and focused for speed).

Makefile link make secrets (scans app/ only in this starter).

Quick checklist

• If real: remove from code and rotate the secret immediately.
• If false positive: document and suppress carefully.
• Add guardrails (env vars, .gitignore, pre-commit hooks).
• Keep the scanner in CI.

IaC — Infrastructure as Code (Kubernetes, Terraform, etc.)

What it is Static analysis of infra config for misconfigurations. We use Checkov to scan Kubernetes manifests and output artifacts/checkov.json.

Why we need it A secure app can be undone by an insecure deployment (e.g., privileged pods, public buckets, no encryption).

What it shows (how to read it) Look for FAILED checks, their resource (e.g., Deployment), file/line, and recommended fix/guideline. Start with high-severity misconfigs: privileged containers, missing runAsNonRoot, lack of readOnlyRootFilesystem, wide network policies, etc.

jq -r '.results.failed_checks[]? | [.check_id, .check_name, .file_path, .resource] | @tsv' artifacts/checkov.json | column -t

Signal vs noise

• Signal: Privilege escalation, root containers, public exposure, missing auth/enc.
• Noise: Pure best-practices with low risk in your context (document if skipping).

Limits Template-only view (lacks runtime context), may need exceptions for valid deviations.

Makefile link make iac

Quick checklist

• Fix high-risk misconfigs first.
• Apply secure defaults in base templates.
• Document justified exceptions (with reasons).
• Re-scan to confirm.

SBOM — Software Bill of Materials

What it is An inventory of components (OS packages, libraries, modules). We use Syft to generate artifacts/sbom.json.

Why we need it Visibility: answer “What’s in our software?” It’s essential for rapid CVE impact checks, license review, and supply-chain transparency.

What it shows (how to read it) Base image OS and version, language/runtime versions, third-party packages. Use it to answer “Do we include X?” and to feed SCA.

jq -r '.artifacts[]? | [.name, .version, .type] | @tsv' artifacts/sbom.json | column -t | head

Signal vs noise

• Signal: Presence and versions of major components (OS, language runtime, key libs).
• Noise: Long tail of transitive packages (query as needed; don’t read linearly).

Limits Snapshot in time; may miss unusual binaries; large but queryable.

Makefile link make sbom

Quick checklist

• Verify key components (OS, runtime) match expectations.
• Store SBOM as a release artifact.
• Use SBOM to drive SCA and incident response.
• Consider SBOM diffs per release.

SCA — Software Composition Analysis (Vulnerability Scanning)

What it is Scan dependencies/OS packages for known CVEs. We use Trivy to scan the built image into artifacts/trivy-image.json.

Why we need it Most risk comes from third-party code. SCA flags known-bad versions and points to fixed versions.

What it shows (how to read it) For each vulnerable package: CVE ID, severity, installed vs fixed version, and references. Prioritize Critical/High. Example:

jq -r '.Results[]?.Vulnerabilities[]? | "\(.Severity)\t\(.PkgName)@\(.InstalledVersion)\tfix:\(.FixedVersion // "none")\t\(.VulnerabilityID)"' artifacts/trivy-image.json | column -t | head

Signal vs noise

• Signal: Critical/High vulns in reachable components.
• Noise: Low/Medium in unused tools; duplicates; “no fix” items (track & mitigate).

Limits Depends on vuln DB freshness; zero-days won’t appear; remediation can require upgrades.

Makefile link make sca

Action plan

• Triage Critical/High immediately; note fixes.
• Upgrade base image / libs; re-scan.
• Mitigate or accept risk temporarily with rationale if no fix.
• Automate regular updates (e.g., Dependabot) and nightly scans.

DAST — Dynamic Application Security Testing

What it is Probe the running app like an external attacker. We use OWASP ZAP (baseline) via zap and save reports under artifacts/zap/.

Why we need it Finds runtime issues: reflected XSS, missing headers, auth/logic exposure, misconfig in real responses.

What it shows (how to read it) Alerts grouped by type with risk level, description, solution, and example URLs. Expect common findings like missing security headers (CSP, X-Frame-Options). Treat High (e.g., exploitable injection) as urgent; fix Mediums for hardening.

Signal vs noise

• Signal: Confirmed vulns (XSS/SQLi), exposed admin endpoints, auth bypass.
• Noise: Info-level headers, duplicate URLs, occasional false positives—verify manually.

Limits Coverage depends on crawl/auth; can be slow or noisy; logic flaws may need manual testing.

Makefile link make zap (runs after deploy in make cd)

Quick checklist

• Prioritize High/Medium alerts; reproduce locally.
• Add/adjust headers (CSP, X-Frame-Options, CORP, HSTS) or sanitize inputs.
• Re-scan after fixes.
• Schedule deeper scans periodically; script auth for protected areas if needed.

Evidence & Index (Optional but Recommended)

Bundle current artifacts and generate an index for quick review and sharing.

• Make targets (optional additions):

  • make evidence → artifacts/evidence-YYYY-MM-DD.zip
  • make artifacts-index → artifacts/ARTIFACTS.md

These help you hand a reviewer/manager a portable proof-pack.

Putting It All Together

Each artifact covers a different angle:

• SAST → your code
• SiC → your secrets
• IaC → your infra configs
• SBOM → your ingredients
• SCA → known CVEs in those ingredients
• DAST → your live surface

Run them, read them, fix the high-signal items, and iterate. Over time you’ll tune rules, reduce noise, and ship with a clean, auditable security posture—locally, on your laptop.

Chapter 3. Vulnerability Management

Approaches to Handling Scanner Results

Teams typically choose one of three models. Pick the one that matches your risk tolerance and delivery needs.

Recommendation: Start with B (Tuned Blocking). Block only on issues you’re confident about; track everything else. Revisit thresholds as the team’s signal/noise improves.

When (and How) to Block — By Tool

Use these practical policies to decide what should actually fail the pipeline.

Secrets (TruffleHog)

• Block if: A finding is clearly a real credential (e.g., verified by the tool or matches a high-confidence detector you trust).
• Don’t block if: Ambiguous/entropy-only strings. Log and review.
• Tuning tips:
  • Restrict scope (we scan app/ only).
  • Add allowlists for known fake/test keys.
• Make target: make secrets → artifacts/trufflehog.json.

SAST (Semgrep)

• Block if: Rules tagged High/Critical or OWASP Top 10 patterns that are obviously exploitable (e.g., reflected XSS sinks, SQLi).
• Don’t block if: Style-only or low-confidence rules.
• Tuning tips:
  • Keep a small curated ruleset in tools/semgrep/.
  • Use // nosemgrep sparingly with a comment explaining why.
• Make target: make sast → artifacts/semgrep.json.

IaC (Checkov)

• Block if: Severe misconfigs (privileged pods, runAsRoot, missing readOnlyRootFilesystem, overly broad RBAC, public exposure).
• Don’t block if: Cosmetic tags/labels or org-specific exceptions with justification.
• Tuning tips:
  • Inline suppressions with reason (e.g., # checkov:skip=CKV_K8S_...: reason).
  • Maintain a small policy file of allowed skips.
• Make target: make iac → artifacts/checkov.json.

SCA / Container Vulns (Trivy)

• Block if: Critical (and optionally High) vulns with fixes present—especially in the base image or critical runtime libs.
• Don’t block if: “Unfixed” items (track them), dev-only deps, or non-runtime paths.
• Tuning tips:
  • Use flags like --ignore-unfixed and a .trivyignore for accepted CVEs.
  • Pin and regularly update base images.
• Make target: make sca → artifacts/trivy-image.json.

DAST (ZAP)

• Block if: For gated environments only (staging/prod) and confirmed exploitable Highs (rare for baseline).
• Default here: Don’t block; treat as actionable feedback for hardening (headers, input handling).
• Make target: make zap → artifacts/zap/.

False Positives & Tuning (Quick Guide)

Tip: After your first run, invest 30–60 minutes tuning; you’ll earn that time back every day.

Vulnerability Management Lifecycle

1. Triage
   Classify new findings: severity, exploitability, exposure (internet-facing?), and environment (prod vs. dev). Decide: Critical/High (now), Medium/Low (backlog), False Positive, Info.

2. Assign / Ownership
   Create a ticket (or use DefectDojo assignment) and give each finding an owner. Add a target due date based on severity (e.g., Critical ≤ 7 days, High ≤ 30 days).

3. Validate
   Reproduce or verify the issue. Mark False Positive if it’s not real; otherwise confirm scope. Add repro steps or links for the assignee.

4. Remediate / Mitigate / Accept

   • Fix (preferred): Code change, dependency upgrade, or config hardening.
   • Mitigate: Temporary control (WAF rule, feature flag).
   • Risk accept: Document rationale + an expiry/review date.

5. Retest
   Re-run the relevant scan; confirm the finding disappears. Keep evidence (screenshot/report snippet or commit SHA).

6. Close & Document
   Close the item; note the fix and any preventative measures (e.g., new rule, pre-commit hook, pinned base image). Review accepted risks periodically.

What “Good” Looks Like (Checklist)

• Scans automated in CI (sast, secrets, iac, sbom, sca, zap as needed).
• Clear blocking policy documented and implemented (only high-signal issues fail the build).
• Exception flow exists (inline suppressions with reasons; reviewable allows).
• Single source of truth for findings (DefectDojo, issues, or a shared tracker).
• Regular triage cadence; every finding has an owner and due date.
• SLAs by severity (e.g., Critical ≤ 7d, High ≤ 30d) and reported on.
• Retest required before closure; evidence stored in ./artifacts (or tracker).
• False positives managed (marked, suppressed, tuned at the tool).
• Risk acceptances documented with expiry/review.
• Metrics (open by severity, age, SLA compliance) reviewed; continuous tuning.

Practical Next Steps in This Repo

1. Start with Tuned Blocking:

   • Secrets: block only high-confidence.
   • SAST/IaC: block critical rules; log others.
   • SCA: block Critical (and optionally High) with fixes.

2. Add a Findings Triage readme page for your team (who triages, when, SLAs).

3. (Optional) Bring up DefectDojo via Docker Compose and import today’s artifacts/*.json.

4. Track & close 2–3 issues end-to-end to practice the lifecycle.

You now have a clear, practical path from “scans produced findings” to “we handled them, and can prove it.”

Chapter 3. Triage

Plan (short & actionable)

• Scope parsed: artifacts/semgrep.json, trufflehog.json, checkov.json, trivy-image.json, sbom.json Code & k8s: app/main.go, app/Dockerfile, infra/k8s/deployment.yaml, service.yaml, kustomization.yaml

• Top issues to fix now (P0/P1):

  1. SCA (Trivy): 1× HIGH CVE in Go stdlib (Go 1.22.12) with fixes available.
  2. IaC (Checkov): missing readiness/liveness probes, service account token hardening, seccomp, caps drop, default namespace, image digest, imagePullPolicy, network policy.
  3. Secrets (TruffleHog): AWS key pattern in app/main.go, unverified canary (training string) — clean up or allowlist.
  4. SAST (Semgrep): “HTTP server without TLS” warning (local dev) + we know a reflected XSS in /echo (not flagged by the current ruleset) — we’ll remediate.

• Retest targets: make sca iac sast secrets cd zap then capture deltas under ./artifacts.

Triage (actual findings)

Validate (quick proofs)

• CVE-2025-47907 present: Trivy HIGH lists stdlib v1.22.12 with fixed versions 1.23.12 / 1.24.6. Our go.mod is go 1.22; build base image is golang:1.22-alpine.
• K8s misconfigs: Checkov flags map to manifest conditions (namespace missing, no probes, no seccomp, etc.).
• Secrets: Raw & RawV2 include AWS-looking key; Verified=false and is_canary=true in ExtraData → safe training token, still noisy.
• XSS: /echo?input=<script>alert(1)</script> will reflect unescaped input (see fmt.Fprintf(w, "You said: %s", input)).

Remediation (concrete diffs/snippets)

Keep changes minimal, safe, and easy to teach. I’ve split by domain.

1) SCA: upgrade Go toolchain & runtime image

app/Dockerfile (build base) Change base to a fixed Go stream (e.g., 1.24.6):

# build stage
-FROM golang:1.22-alpine AS build
+FROM golang:1.24.6-alpine AS build
WORKDIR /src
COPY . .
RUN --mount=type=cache,target=/go/pkg/mod \
    --mount=type=cache,target=/root/.cache/go-build \
    CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o /out/server ./main.go

(Optionally keep alpine runtime, or use gcr.io/distroless/static:nonroot in a later “golden images” chapter.)

Retest: make build sca → confirm HIGH count drops to 0.

2) App: fix reflected XSS & add basic headers

app/main.go — escape user input and set common headers:

import (
    "fmt"
    "html/template"
    "html"        // add
    "log"
    "net/http"
    "os"
)

mux.HandleFunc("/echo", func(w http.ResponseWriter, r *http.Request) {
    input := r.URL.Query().Get("input")
    w.Header().Set("Content-Type", "text/html; charset=utf-8")
    // Security headers (baseline)
    w.Header().Set("X-Content-Type-Options", "nosniff")
    w.Header().Set("X-Frame-Options", "DENY")
    w.Header().Set("Referrer-Policy", "no-referrer")

    // Optional minimal CSP for demo
    w.Header().Set("Content-Security-Policy", "default-src 'self'")

    // Escape untrusted input to prevent reflected XSS
    fmt.Fprintf(w, "You said: %s", html.EscapeString(input))
})

Retest: make cd && make zap → XSS should no longer be reported (and manual <script> should render as text).

3) Secrets: remove canary key from code (or allowlist)

If it’s in the source (line ~28), delete it or replace with a clearly fake, non-pattern string. If you need it for teaching, add a comment and allowlist it in your TruffleHog config later so CI doesn’t nag students.

Retest: make secrets → verified_secrets: 0, no AWS key finding.

4) K8s Hardening (pass key Checkov rules)

infra/k8s/deployment.yaml

Add namespace, SA token control, seccomp, drop caps, probes, and (optionally) imagePullPolicy:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
  namespace: ship             # add to satisfy CKV_K8S_21
  labels: { app: app }
spec:
  replicas: 1
  selector: { matchLabels: { app: app } }
  template:
    metadata: { labels: { app: app } }
    spec:
      automountServiceAccountToken: false   # CKV_K8S_38
      securityContext:
        seccompProfile:
          type: RuntimeDefault             # CKV_K8S_31
      containers:
        - name: app
          image: ship-securely/app:dev
+         imagePullPolicy: Always           # CKV_K8S_15 (dev-friendly but satisfies rule)
          ports: [{ containerPort: 3000 }]
          securityContext:
            runAsUser: 10001
            runAsGroup: 10001
            runAsNonRoot: true
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
+           capabilities:
+             drop: ["ALL"]                 # CKV_K8S_37 / CKV_K8S_28
          resources:
            requests: { cpu: "50m", memory: "64Mi" }
            limits:   { cpu: "250m", memory: "256Mi" }
+         readinessProbe:
+           httpGet: { path: /healthz, port: 3000 }
+           initialDelaySeconds: 2
+           periodSeconds: 5
+         livenessProbe:
+           httpGet: { path: /healthz, port: 3000 }
+           initialDelaySeconds: 5
+           periodSeconds: 10

NetworkPolicy (satisfy CKV2_K8S_6): add default deny + allow app ingress & DNS egress:

# infra/k8s/netpol/00-default-deny.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: ship
spec:
  podSelector: {}
  policyTypes: ["Ingress", "Egress"]

# infra/k8s/netpol/20-allow-app-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-app-ingress
  namespace: ship
spec:
  podSelector:
    matchLabels: { app: app }
  ingress:
    - from:
        - podSelector: {}   # relax as needed for demo
      ports:
        - protocol: TCP
          port: 3000

# infra/k8s/netpol/10-allow-dns-egress.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-egress
  namespace: ship
spec:
  podSelector: {}
  policyTypes: ["Egress"]
  egress:
    - to:
        - namespaceSelector: {}
      ports:
        - protocol: UDP
          port: 53

(You already have netpol-apply in your full Makefile; for the starter, include these three files and a simple make target later in the book.)

Retest: make iac → Checkov failed set should drop (probes, seccomp, SA token, caps, default ns, netpol).

Note on image digest (CKV_K8S_43): For the local-only starter, we can document digest pinning and enforce it in the “golden images” chapter (students pin a digest when they have a registry). We’ll mark this P2 for now.

Retest (commands to run)

# Rebuild and rescan image after Go upgrade
make build sca

# Re-scan code and secrets after app/main.go changes
make sast secrets

# Re-apply K8s and re-scan IaC
make cd
make iac

# Optional: run ZAP baseline to confirm no reflected XSS
make zap

Capture summaries (before/after) into ARTIFACTS.md, and drop a dated bundle via your evidence target (if present).

Close & Document

• Closed (fixed):

  • CVE-2025-47907 (Go stdlib) — fixed by upgrading to golang:1.24.6-alpine; Trivy HIGH=0.
  • Reflected XSS in /echo — fixed by html.EscapeString and security headers; ZAP/Manual confirm.
  • Checkov: namespace, probes, seccomp, caps drop, SA token, netpol — fixed with manifest edits; make iac clean for these checks.
  • TruffleHog AWS token — removed or allowlisted with clear comment (no verified secrets).

• Accepted/Deferred (documented with reason & revisit date):

  • Image digest pinning (CKV_K8S_43) in local starter — defer to “Golden Images” chapter; add note in text and set a reminder to enforce later.

• Prevention:

  • Keep Go base image current; consider pinning digests in golden-image workflow.
  • Keep Semgrep rules curated; add .semgrepignore if needed for tests/gen code.
  • Maintain .trivyignore for vetted, low-risk CVEs if/when needed.
  • Preserve K8s hardening defaults in base manifests; teach overrides deliberately.

Quick “diff of impact” (what should change in artifacts)

• artifacts/trivy-image.json: HIGH count from 1 → 0.
• artifacts/semgrep.json: still a TLS warning (expected in local dev), but no XSS path (it wasn’t flagged before; now provably mitigated).
• artifacts/trufflehog.json: no AWS detector hit / verified_secrets=0.
• artifacts/checkov.json: failures for CKV_K8S_8/9/31/37/28/38/21/CKV2_K8S_6 drop.
• artifacts/zap/: XSS alert (if any) gone; headers findings likely improved.

Chapter 3. Retesting

Artifacts Delta — Current vs Previous

This table summarizes what changed in the security artifacts after the latest fixes.
Use the “Verify” commands (right column) to reproduce the numbers locally.

Notes

• Trivy: expect High=0 after upgrading the Go toolchain/base image (e.g., golang:1.24.6-alpine).
• Semgrep: our fix removes the reflected XSS behavior (manual validation); your ruleset may not have flagged it originally.
• TruffleHog: canary strings should be removed or allow-listed; we block only on verified credentials.
• Checkov: after adding probes, seccomp, caps drop, SA token hardening, namespace, and basic NetPols, failures should drop significantly.
• ZAP: baseline tends to report header hygiene as Medium/Low; after adding headers (CSP/XFO/XCTO/HSTS*), counts should reduce (HSTS only with TLS).

How to Refresh This Table

1. Capture a “before” snapshot (optional if you already have one):

   rm -rf artifacts_before && mkdir -p artifacts_before
   cp -a artifacts/* artifacts_before/ 2>/dev/null || true
   

2. Apply Fixes and re-run scans:

make build sca
make sast secrets
make iac
make cd
make zap

3. Make delta table

bash scripts/artifacts_delta.sh > artifacts/ARTIFACTS.md
echo "Wrote artifacts/ARTIFACTS.md"

How to reproduce

optional: snapshot current as “before”

rm -rf artifacts_before && mkdir -p artifacts_before cp -a artifacts/* artifacts_before/ 2>/dev/null || true

regenerate table

bash scripts/artifacts_delta.sh > artifacts/ARTIFACTS.md

Makefile convenience target

Add this to your starter Makefile so readers can do make artifacts-delta:

artifacts-delta:
	@bash scripts/artifacts_delta.sh > artifacts/ARTIFACTS.md
	@echo "Wrote artifacts/ARTIFACTS.md"

Chapter 3. Checklist

Codify how you raise bars safely and reproducibly.

Gates & Ratcheting

• Ratcheting plan: What’s your 6-week path from WARN→ENFORCE? (document week-by-week)
• Exit codes: Which tools produce non-zero exit on gate fail (Trivy, Semgrep, Checkov)? Who can override?

Base Images & Digests

• Golden base: Which base images are allowed per language? (e.g., distroless, alpine)
• Digest pinning: Will you pin digests for base & runtime? Where are pins recorded/updated?
• Update cadence: Who bumps base images and how often (weekly, monthly)?

Signing & Provenance

• Signing mode: Offline blob signing vs registry signature — which is feasible in your environment?
• Keys: Who holds cosign.key/cosign.pub? How are they backed up/rotated?

SBOM Diff & Licenses

• Diff trigger: On release candidate or every merge to main?
• License allowlist: What is your current allow/deny? Who approves exceptions?

Mirrors & Caching

• Dependency mirrors: Will you run a local Go proxy (and/or npm cache)? Who owns uptime/space?
• Air-gapped plan: If internet egress is cut, can builds still succeed with mirrors + pinned bases?

Promotion & Evidence

• Promotion rule: Define “promote on pass” criteria (which gates must be green).
• Evidence bundling: Who packages evidence ZIPs and where are they archived?

← Back to the blog