Cruxly › Cloud and DevOps › Cloud & DevOps: CI/CD Pipelines

Build Once, Promote Everywhere: Designing a Pipeline Without Environment Drift

immutable artifacts, digest pinning, and the rebuild trap

Prompt: design a CI/CD pipeline for a service deployed to staging then production. The exact same tested artifact must reach production — no rebuild, no "should be the same" — and the pipeline must fail loudly if that guarantee is ever violated, rather than silently deploying a different artifact than what was tested.

30 minutes first. The naive version builds twice (once per environment) and trusts that identical source produces an identical artifact. That trust is the bug.

Tags move, digests don't

A container registry tag (myapp:v1.4.0, even myapp:prod) is a mutable pointer — nothing stops a later push from retargeting that same tag to a different image. A digest (myapp@sha256:a1b2c3...) is a content hash of the image's actual bytes; it is cryptographically impossible for two different images to share one. Pinning a deployment to a tag means "whatever this tag currently points to, which might not be what I tested." Pinning to a digest means "this exact, specific artifact, and nothing else" — this is the actual mechanism behind "build once, promote everywhere," not a naming convention.

The pipeline's own promotion step should read the digest the build stage produced and pass it forward explicitly, rather than having the staging deploy and the production deploy each independently resolve "the latest build" from the registry — two independent resolutions of "latest" are two chances for the answer to differ, even seconds apart, if another pipeline run is in flight concurrently.

Determinism has to be engineered, not assumed

Pin the base image by digest, not tag, in the Dockerfile itself — FROM python:3.13-slim re-resolves to whatever that tag currently points to on every build, which can silently change the OS packages, security patches, or even the Python patch version baked into a "reproducible" build days or weeks apart. FROM python:3.13-slim@sha256:... removes that variable entirely. The catch: a pinned base image also freezes its CVEs in place, so pair the pin with automated update PRs (Renovate or Dependabot) that bump the digest and run the tests.

Lockfiles only help if the build actually respects them strictly. pip install -r requirements.txt without hash-checking, or a Node install that allows semver ranges to re-resolve, can drift even with a committed lockfile present, if the tool's default behavior permits resolving outside what the lockfile pinned. pip install --require-hashes, npm ci (not npm install), and their equivalents in other ecosystems exist specifically to make the lockfile a hard constraint instead of a suggestion the tool is allowed to override.

CI runner environment drift is the most overlooked source of non-determinism. Two builds of the identical commit, run on CI runner images provisioned a month apart, can differ in system-level compiler versions, CA certificate bundles, or preinstalled tool versions that never show up in the application's own lockfile at all, because they're outside the application's dependency graph entirely. Pinning the CI runner's own container image by digest is the same discipline applied one layer up — the build environment is itself an artifact that needs to be reproducible, not just the thing it builds.

Least privilege for the pipeline itself

A CI pipeline's credentials are, in aggregate, one of the highest-value targets in the whole system, because a single pipeline typically holds registry push access, cloud deployment credentials, and often direct access to production secrets for injecting them at deploy time — a compromise of the CI system itself (a malicious dependency in the build, a compromised third-party GitHub Action, a leaked runner token) can therefore reach further than compromising any single service the pipeline deploys.

Scope credentials per pipeline stage, not one god-credential for the whole pipeline. The build stage needs registry push access and nothing else; it does not need production deployment credentials, and giving it those anyway means a vulnerability in the build step (a malicious postinstall script in a compromised dependency, for instance) has a direct path to production it never needed for its actual job.

Pin third-party CI actions/plugins by commit SHA, not by tag, for exactly the same reason base images get pinned by digest — a popular GitHub Action referenced by a mutable tag (uses: some/action@v2) can have its underlying code changed by the action's maintainer (or an attacker who compromises that maintainer's account) without your pipeline's configuration changing at all, silently running different code on your next build with the same credentials it always had.

How to say it in the interview

"A pipeline that rebuilds per environment is trusting that the same source code plus the same Dockerfile always produces byte-identical output - and that trust breaks the moment a base image tag moves, a dependency resolves differently, or the CI runner's toolchain version drifted between the two builds. The only way to guarantee staging and production ran the exact same artifact is to build it exactly once and reference it everywhere after that by its immutable digest, not its tag."

Checklist

• Pinned deployments to an image digest, not a mutable tag
• Built the artifact exactly once and passed its digest through every promotion stage
• Pinned the Dockerfile's base image by digest, not tag
• Used strict, hash-checked dependency installs (--require-hashes, npm ci) rather than a lockfile the tool can override
• Scoped pipeline credentials per stage rather than one god-credential for the whole pipeline
• Pinned third-party CI actions/plugins by commit SHA, not a mutable version tag

Related lessons

• Deployment Strategies and the Rollback Plan That Survives a Real Incident
• Pipeline Security: The CI System as Its Own Attack Surface