Most container builds are slow for the same two reasons: the cache is thrown away between CI runs, and the build is treated as a black box that emits a tarball nobody can reason about. BuildKit fixes both. Its execution model is a content-addressed graph with a precise cache key, so you can ship that cache to a registry or object store and rehydrate it on a fresh runner. The same engine emits SBOM and SLSA provenance attestations as a side effect of the build, and can rewrite layer timestamps so two independent machines produce byte-identical images. This article wires all of that together: remote cache, a multi-arch builder farm, reproducibility controls, attestations, and the CI plumbing to make it stick. The commands assume Buildx >= 0.13 and BuildKit >= 0.13.
1. BuildKit internals: LLB, frontends, and the cache key
BuildKit does not interpret a Dockerfile line by line. A frontend (the dockerfile.v0 frontend is the default; docker/dockerfile:1 is the upgradable image form) compiles your Dockerfile into LLB – a low-level, content-addressed build graph. Each vertex is an operation (run this command, copy these files, pull this image) and edges are dependencies. Because the graph is a DAG, independent branches run concurrently, and only the vertices whose inputs changed get re-executed.
The cache key is the heart of it. For each vertex BuildKit computes a key from the operation definition and the content digests of its inputs. A COPY is keyed by the digest of the files copied, not their mtimes; a RUN is keyed by the command string plus the digests of every input mount. This is why reordering instructions or touching an unrelated file no longer busts the whole cache – the key only moves when content that actually feeds the step moves.
Two consequences drive everything below:
- Layer order still matters, because a cache miss on vertex N invalidates every vertex downstream of N. Put rarely-changing inputs (base image, OS packages, dependency manifests) early; put your source last.
- The cache is portable precisely because keys are content digests. A key computed on a fresh runner matches a key computed last week, so an exported cache can be imported and reused anywhere.
# syntax=docker/dockerfile:1
FROM golang:1.23-bookworm AS build
WORKDIR /src
# Copy manifests first: this layer's cache key only moves when deps change.
COPY go.mod go.sum ./
RUN --mount=type=cache,target=/go/pkg/mod go mod download
# Source copied last: editing app code does not re-download modules.
COPY . .
RUN --mount=type=cache,target=/go/pkg/mod \
--mount=type=cache,target=/root/.cache/go-build \
CGO_ENABLED=0 go build -trimpath -o /out/app ./cmd/app
FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=build /out/app /app
ENTRYPOINT ["/app"]
2. Remote cache backends: registry, gha, and S3 with mode=max
A docker build on the stock docker driver only keeps cache in the local image store, which a fresh CI runner does not have. Switch to the docker-container driver and you unlock --cache-to/--cache-from exporters that persist the cache externally.
First, the mode distinction, because it is the single biggest hit-rate lever:
| Mode | What is cached | Hit rate | Storage |
|---|---|---|---|
mode=min (default) |
Only layers present in the final image | Lower – intermediate stages are not cached | Smaller |
mode=max |
Every layer, including intermediate build-stage layers | Higher – multi-stage builds reuse compiler/test stages | Larger |
For multi-stage builds (build stage + tiny runtime stage) mode=min caches almost nothing useful, because the expensive compile layers never reach the final image. Use mode=max on any non-trivial build.
Registry backend – the most portable; cache lives as an image alongside your app:
docker buildx create --name kvbuilder --driver docker-container --use --bootstrap
docker buildx build \
--cache-to type=registry,ref=registry.example.com/app/cache,mode=max,image-manifest=true,oci-mediatypes=true \
--cache-from type=registry,ref=registry.example.com/app/cache \
--tag registry.example.com/app:1.4.2 \
--push .
image-manifest=true writes the cache as a real OCI image manifest so registries that reject the older “cache manifest” media type (ECR, some Artifactory configs) still accept it.
GitHub Actions backend (gha) – stores cache in the Actions cache service. Parameters: url (defaults to $ACTIONS_CACHE_URL), token (defaults to $ACTIONS_RUNTIME_TOKEN), scope (defaults to buildkit; give each image a distinct scope), and mode:
docker buildx build \
--cache-to type=gha,scope=app-amd64,mode=max \
--cache-from type=gha,scope=app-amd64 \
--tag registry.example.com/app:1.4.2 --push .
S3 backend – best when you self-host runners and want cache near them. Required params are region and bucket; name defaults to buildkit, mode defaults to min. Credentials are read from the standard AWS environment / config chain, so on a runner with an instance role you pass none:
docker buildx build \
--cache-to type=s3,region=us-east-1,bucket=acme-buildkit-cache,prefix=app/,mode=max \
--cache-from type=s3,region=us-east-1,bucket=acme-buildkit-cache,prefix=app/ \
--tag 1234567890.dkr.ecr.us-east-1.amazonaws.com/app:1.4.2 --push .
Put a lifecycle / TTL policy on the cache location.
mode=maxcache for an active monorepo grows fast; an S3 lifecycle rule expiring objects after 14 days, or a scheduled GC of the registry cache tag, keeps storage bounded without hurting steady-state hit rate.
3. Cache mounts for package managers – and avoiding poisoning
The RUN --mount=type=cache mounts in step 1 are distinct from the layer cache. A cache mount is a writable, persistent directory shared across builds (the Go module cache, ~/.npm, apt lists, pip wheels) that is not committed into the image. It survives even when the surrounding layer is a cache miss, so a one-character source change re-runs go build but reuses the compiled-object cache underneath it.
The footgun is cache poisoning across concurrent or cross-arch builds. Two builds writing the same cache mount path can interleave, and an amd64 build and an arm64 build sharing one apt cache will smear architecture-specific artifacts together. Defend with id and sharing:
RUN --mount=type=cache,id=apt-$TARGETARCH,target=/var/cache/apt,sharing=locked \
--mount=type=cache,id=apt-lists-$TARGETARCH,target=/var/lib/apt/lists,sharing=locked \
apt-get update && apt-get install -y --no-install-recommends ca-certificates
idnamespaces the cache. Keying it on$TARGETARCHgives each architecture its own apt/pip/npm cache – no cross-arch contamination.sharing=lockedserializes writers (one build holds the mount, others wait). The defaultsharedallows concurrent writers, which is fine for append-only caches like Go modules but wrong for apt.sharing=privategives each concurrent build a fresh mount.
For secrets used during a build (registry tokens, private repo credentials) use --mount=type=secret, never a build arg – build args are recorded in provenance (see step 6) and leak into image history.
4. Multi-arch builds: QEMU emulation vs. a native builder farm
A multi-platform image is an OCI index pointing at one manifest per architecture. There are two ways to produce the non-native ones.
QEMU emulation is the zero-infrastructure path: register binfmt handlers once and a single x86 node builds arm64 by emulating it.
# Install binfmt handlers on the host (idempotent; re-run after host reboots).
docker run --privileged --rm tonistiigi/binfmt --install all
docker buildx build --platform linux/amd64,linux/arm64 \
--tag registry.example.com/app:1.4.2 --push .
It is correct but slow. Anything CPU-bound – compiling Go/Rust, running a test suite, transpiling – can run 5-20x slower under emulation, and some toolchains hit emulator edge cases. For trivial images it is fine; for real software it dominates your build time.
A native builder farm routes each platform to real hardware. Create one builder, then --append an arm64 node reachable over an SSH Docker context. Buildx schedules each platform to the node that natively supports it.
# Register a Docker context for the remote arm64 host.
docker context create arm-node --docker "host=ssh://ci@arm-builder.internal"
# Initial node: native amd64 on the local/default engine.
docker buildx create --name farm --driver docker-container \
--platform linux/amd64 --use --bootstrap
# Append the native arm64 node.
docker buildx create --name farm --append \
--node farm-arm64 --platform linux/arm64 \
--driver docker-container arm-node
docker buildx build --builder farm \
--platform linux/amd64,linux/arm64 \
--tag registry.example.com/app:1.4.2 --push .
Each node maintains its own cache; combine the farm with the registry or s3 cache backend so both nodes warm a shared external cache. In practice, native arm64 (Graviton runners, Apple-silicon self-hosted, GitHub’s arm64 runners) is the difference between a 90-second and a 12-minute build for compiled languages.
5. Reproducibility: SOURCE_DATE_EPOCH, pinned digests, rewrite-timestamp
Two builds of the same source should be byte-identical. Three things break that: floating base images, build timestamps baked into layer metadata, and non-deterministic file ordering.
Pin base images by digest. A tag like golang:1.23-bookworm is mutable; pin the digest so the input graph is fixed:
FROM golang:1.23-bookworm@sha256:6dca9f8c0b...e91 AS build
Set SOURCE_DATE_EPOCH. Buildx (>= 0.10) automatically propagates a SOURCE_DATE_EPOCH value from the client environment into the build, and BuildKit clamps the timestamps it controls (image config created, history entries) to that epoch. Derive it from the commit:
export SOURCE_DATE_EPOCH="$(git log -1 --pretty=%ct)"
Rewrite layer file timestamps. SOURCE_DATE_EPOCH alone fixes image config metadata, but files inside layers still carry their original mtimes. The image exporter option rewrite-timestamp=true (BuildKit >= 0.13) clamps every file’s timestamp to the epoch, which is what actually makes layer digests reproducible:
docker buildx build \
--build-arg SOURCE_DATE_EPOCH="$(git log -1 --pretty=%ct)" \
--output type=image,name=registry.example.com/app:1.4.2,push=true,rewrite-timestamp=true \
--provenance=mode=max --sbom=true .
It is off by default because rewriting layers costs time and breaks layer-cache identity from a non-rewritten build, so reserve it for release builds, not every PR. Verify reproducibility by building the same commit on two machines and diffing the resulting digests.
6. Emitting SBOM and SLSA provenance attestations
BuildKit can attach attestations to the image index as a build side effect – no separate scan step, and the SBOM describes exactly what the build produced rather than a re-pull approximation.
- Provenance records how the image was built: source repo and revision, build platform, materials, and (in
max) the full LLB definition. Shorthand--provenance=mode=max, or--attest type=provenance,mode=max,version=v1. Notemode=maxprovenance embeds build-arg values – another reason secrets belong in--mount=type=secret, not build args. - SBOM is a Software Bill of Materials generated by the
buildkit-syft-scannerplugin (built on Anchore Syft). Shorthand--sbom=true, or--attest type=sbom. Swap the scanner withgenerator=.
docker buildx build \
--platform linux/amd64,linux/arm64 \
--provenance=mode=max \
--sbom=true \
--attest type=sbom,generator=docker/buildkit-syft-scanner:latest \
--cache-to type=registry,ref=registry.example.com/app/cache,mode=max,image-manifest=true \
--cache-from type=registry,ref=registry.example.com/app/cache \
--tag registry.example.com/app:1.4.2 \
--push .
Attestations are stored as separate manifests in the OCI image index, referenced from each platform manifest – so a multi-arch image gets a per-platform SBOM and provenance. They travel with the image and can be consumed by an admission policy (cosign / policy-controller verifying the SLSA provenance predicate before a pod is allowed to run).
Verify
Confirm the image is genuinely multi-arch, carries attestations, and is reproducible.
# 1. Confirm both architectures plus attestation manifests in the index.
docker buildx imagetools inspect registry.example.com/app:1.4.2
# 2. Dump the SBOM (top-level) and per-platform.
docker buildx imagetools inspect registry.example.com/app:1.4.2 \
--format '{{ json .SBOM.SPDX }}'
docker buildx imagetools inspect registry.example.com/app:1.4.2 \
--format '{{ json (index .SBOM "linux/arm64").SPDX }}'
# 3. List packages and versions from the SBOM.
docker buildx imagetools inspect registry.example.com/app:1.4.2 \
--format '{{ range .SBOM.SPDX.packages }}{{ .name }}@{{ .versionInfo }}{{ println }}{{ end }}'
# 4. Inspect provenance.
docker buildx imagetools inspect registry.example.com/app:1.4.2 \
--format '{{ json .Provenance }}'
# 5. Prove reproducibility: same commit, two builders, identical digest.
git checkout v1.4.2
export SOURCE_DATE_EPOCH="$(git log -1 --pretty=%ct)"
docker buildx build --output type=oci,dest=a.tar,rewrite-timestamp=true \
--build-arg SOURCE_DATE_EPOCH="$SOURCE_DATE_EPOCH" .
# ...repeat on a second host into b.tar, then compare config+layer digests.
diff <(tar -xOf a.tar index.json) <(tar -xOf b.tar index.json)
To measure cache effectiveness, read the structured build trace.
# Per-vertex CACHED/RUN status reveals which layers missed.
docker buildx build --progress=rawjson . 2>build.log
grep -c '"cached":true' build.log # vertices served from cache
grep -c '"cached":false' build.log # vertices that executed
7. Wiring BuildKit into GitHub Actions, GitLab, and Tekton
GitHub Actions. Use docker/setup-buildx-action (sets up the docker-container driver) and docker/setup-qemu-action only if you lack native arm64 runners. With docker/build-push-action the gha cache url/token are populated automatically; calling gha from a raw buildx command needs crazy-max/ghaction-github-runtime to export the variables.
name: build
on: { push: { tags: ["v*"] } }
permissions:
contents: read
packages: write
id-token: write # for keyless cosign / OIDC if you verify attestations later
jobs:
image:
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v4
- uses: docker/setup-qemu-action@v3
- uses: docker/setup-buildx-action@v3
- uses: docker/login-action@v3
with:
registry: registry.example.com
username: ${{ secrets.REG_USER }}
password: ${{ secrets.REG_TOKEN }}
- id: epoch
run: echo "v=$(git log -1 --pretty=%ct)" >> "$GITHUB_OUTPUT"
- uses: docker/build-push-action@v6
with:
platforms: linux/amd64,linux/arm64
push: true
tags: registry.example.com/app:${{ github.ref_name }}
build-args: SOURCE_DATE_EPOCH=${{ steps.epoch.outputs.v }}
provenance: mode=max
sbom: true
cache-from: type=gha,scope=app
cache-to: type=gha,scope=app,mode=max
outputs: type=image,rewrite-timestamp=true
GitLab CI. No Docker socket on shared runners, so run BuildKit’s buildkitd and use buildctl. Authenticate the cache to the GitLab registry with the job token.
build:
image: moby/buildkit:v0.18.0-rootless
variables:
BUILDKITD_FLAGS: --oci-worker-no-process-sandbox
before_script:
- mkdir -p ~/.docker
- echo "{\"auths\":{\"$CI_REGISTRY\":{\"auth\":\"$(printf '%s:%s' "$CI_REGISTRY_USER" "$CI_REGISTRY_PASSWORD" | base64 -w0)\"}}}" > ~/.docker/config.json
script:
- export SOURCE_DATE_EPOCH="$(git log -1 --pretty=%ct)"
- |
buildctl-daemonless.sh build \
--frontend dockerfile.v0 --local context=. --local dockerfile=. \
--opt build-arg:SOURCE_DATE_EPOCH="$SOURCE_DATE_EPOCH" \
--output type=image,name=$CI_REGISTRY_IMAGE:$CI_COMMIT_TAG,push=true,rewrite-timestamp=true \
--opt attest:provenance=mode=max --opt attest:sbom= \
--export-cache type=registry,ref=$CI_REGISTRY_IMAGE/cache,mode=max,image-manifest=true \
--import-cache type=registry,ref=$CI_REGISTRY_IMAGE/cache
Tekton. The community buildkit Task / a buildkitd sidecar pattern is standard; point --export-cache/--import-cache at the same registry tag so every PipelineRun on ephemeral pods shares cache. The buildctl invocation is identical to the GitLab one above.
Enterprise scenario
A platform team at a fintech ran ~600 service builds/day on GitHub-hosted runners. They had recently added linux/arm64 images for Graviton deployment and turned on QEMU. The Go-heavy services went from ~2-minute builds to 11-18 minutes each under emulation, CI queues backed up past their 20-minute SLA, and runner-minute spend roughly doubled. Their constraint: GitHub-hosted arm64 runners were not yet approved by their security team, and they could not move builds off the GitHub control plane.
The fix was a hybrid farm with shared registry cache. They stood up a small pool of self-hosted Graviton runners inside their VPC (which security had approved, since they were already running Graviton in prod) and registered them as an --append node over an SSH context. The GitHub-hosted amd64 runner remained the build orchestrator; only the arm64 leg shipped to the native node. Both legs read and wrote a single mode=max registry cache in their internal registry, so a cache warmed by one architecture’s shared stages cut the other’s cold time. They moved rewrite-timestamp=true to tag builds only, keeping PR builds on plain mode=max cache for maximum hit rate.
- uses: docker/build-push-action@v6
with:
builder: farm # amd64 local + appended native arm64 node
platforms: linux/amd64,linux/arm64
push: true
tags: reg.internal/app:${{ github.ref_name }}
provenance: mode=max
sbom: true
cache-from: type=registry,ref=reg.internal/app/cache
cache-to: type=registry,ref=reg.internal/app/cache,mode=max,image-manifest=true
Result: p50 multi-arch build dropped to ~3 minutes (native arm64 plus warm cache), CI minutes fell below the pre-arm64 baseline because the amd64 leg also benefited from shared cache, and every released image now carried per-platform SBOM and SLSA provenance that their admission controller verified at deploy.