Multi-Architecture Container Builds with docker buildx bake: Remote Cache, Provenance, and Registry-Native Pipelines

If your container CI is a shell loop calling docker buildx build once per service with a wall of --build-arg, --cache-from, and --platform flags pasted into every job, you already know the failure modes: drift between targets, cache keys nobody can reason about, and a 30-minute pipeline because arm64 is emulated under QEMU. docker buildx bake is the fix. It is a build orchestrator that reads declarative HCL, expands matrices, fans builds out in parallel, and treats the registry as the source of truth for both cache and provenance.

This is a working pipeline, not a tour of flags. Everything assumes a recent Docker Engine with Buildx v0.12+ and BuildKit v0.13+ (docker buildx version).

1. Why bake beats shell loops

bake is to docker buildx build what a build system is to a pile of compiler invocations. You describe targets (one image build), group them, and let bake compute the graph. The wins are concrete:

• One source of truth. Shared base config lives in a target other targets inherits. Change the registry once.
• Parallelism for free. A group builds all its targets concurrently against a single BuildKit daemon, which deduplicates shared layers across them.
• Matrix expansion. One target definition becomes N builds across versions or variants without copy-paste.
• It is just build underneath. Every HCL attribute maps to a buildx build flag, so nothing is hidden or magic.

A bake file can be HCL, JSON, or a Compose file. Use HCL — it has variables, functions, and interpolation that Compose lacks.

2. Set up the builder: QEMU emulation vs native arm64

The default docker driver cannot build multi-platform images or export to a registry cache. You need the docker-container driver, which runs BuildKit in its own container.

# Create a dedicated builder backed by the docker-container driver
docker buildx create \
  --name kv-builder \
  --driver docker-container \
  --driver-opt network=host \
  --bootstrap \
  --use

docker buildx inspect kv-builder

For cross-architecture builds on a single amd64 host, register QEMU so BuildKit can emulate arm64:

# binfmt_misc handlers; --install all wires up every supported arch
docker run --privileged --rm tonistiigi/binfmt --install all

Emulation works but is slow — CPU-bound steps (compilers, npm install running native postinstalls) can run 5-10x slower under QEMU. The production-grade answer is native runners per architecture: an amd64 host builds amd64, a Graviton/Ampere host builds arm64, and you merge the results. You can even attach multiple nodes to one logical builder so a single bake invocation schedules each platform on the node that matches it:

# Append a remote arm64 node to the same builder; bake routes by platform
docker buildx create \
  --name kv-builder \
  --append \
  --node kv-arm64 \
  --platform linux/arm64 \
  ssh://ci@arm64-runner.internal

Rule of thumb: emulation is fine for interpreted-language images and final assembly. For anything that compiles, pay for native arm64 capacity — it is cheaper than the pipeline minutes you burn under QEMU.

3. Write docker-bake.hcl: shared base, inheritance, and matrix

Here is the spine of a real multi-service repo. Variables read from the environment with defaults, a _common base target carries shared config, and concrete targets inherits it.

# docker-bake.hcl
variable "REGISTRY" {
  default = "ghcr.io/kloudvin"
}

variable "TAG" {
  default = "dev"
}

# Comma-separated platform list, overridable in CI
variable "PLATFORMS" {
  default = "linux/amd64,linux/arm64"
}

# Default group: what `docker buildx bake` builds with no args
group "default" {
  targets = ["api", "worker"]
}

# Abstract base. The leading underscore is convention; it is still a
# target, just one you never build directly.
target "_common" {
  context    = "."
  platforms  = split(",", PLATFORMS)
  labels = {
    "org.opencontainers.image.source" = "https://github.com/kloudvin/platform"
  }
}

target "api" {
  inherits   = ["_common"]
  dockerfile = "api/Dockerfile"
  tags       = ["${REGISTRY}/api:${TAG}"]
  args = {
    SERVICE = "api"
  }
}

target "worker" {
  inherits   = ["_common"]
  dockerfile = "worker/Dockerfile"
  tags       = ["${REGISTRY}/worker:${TAG}"]
  args = {
    SERVICE = "worker"
  }
}

Build it:

REGISTRY=ghcr.io/kloudvin TAG=1.4.0 docker buildx bake

bake builds api and worker in parallel, each for both platforms. To build one target: docker buildx bake api.

Matrix expansion

A matrix attribute turns one target into a build per combination. The name field must be unique per cell — derive it from the matrix keys.

target "runtime" {
  inherits = ["_common"]
  name     = "runtime-${tgt.replace(".", "-")}"
  matrix = {
    tgt = ["3.11", "3.12", "3.13"]
  }
  dockerfile = "images/python/Dockerfile"
  tags       = ["${REGISTRY}/python:${tgt}"]
  args = {
    PYTHON_VERSION = tgt
  }
}

docker buildx bake runtime now produces three images, each multi-arch, all concurrent. This single block replaces a nested for loop over versions and platforms.

4. Registry-native remote cache

Inline cache (type=inline, the old BUILDKIT_INLINE_CACHE=1) only stores cache metadata for the final stage, so multi-stage builds miss on intermediate layers. For real CI cache, use type=registry with mode=max, which exports a cache manifest for every layer of every stage to a dedicated tag.

Add cache-to and cache-from to the base target so all targets share the policy:

target "_common" {
  context   = "."
  platforms = split(",", PLATFORMS)

  cache-from = [
    "type=registry,ref=${REGISTRY}/buildcache:api"
  ]
  cache-to = [
    "type=registry,ref=${REGISTRY}/buildcache:api,mode=max,compression=zstd"
  ]
}

Two things that bite people:

• mode=max is non-negotiable for layered builds. The default mode=min only exports the layers in the final image; your build and deps stages get no cache and rebuild every time.
• Cache is per-architecture. A registry cache ref holds a manifest list; BuildKit picks the matching arch entry. You do not need a separate ref per platform, but the cache only helps a platform if that platform was previously exported to it.

compression=zstd shrinks the cache blobs and speeds up restore. If you push images to ECR or another registry that historically lagged on OCI image-index support, also set oci-mediatypes=true and confirm the registry accepts it.

5. Split platforms across native runners, then merge

The fastest pipeline builds each architecture on a native runner and pushes them as separate per-arch tags, then stitches them into one multi-arch manifest with imagetools create. No QEMU, full parallelism across machines.

Per-arch build (one job per runner). Note --set overrides any HCL attribute from the CLI:

# On the amd64 runner
docker buildx bake api \
  --set "*.platform=linux/amd64" \
  --set "api.tags=${REGISTRY}/api:${TAG}-amd64" \
  --set "*.output=type=registry"

# On the arm64 runner
docker buildx bake api \
  --set "*.platform=linux/arm64" \
  --set "api.tags=${REGISTRY}/api:${TAG}-arm64" \
  --set "*.output=type=registry"

The * glob applies to every target; api.tags targets one. Both jobs push by digest.

Merge into a single tag in a final job. imagetools create builds a manifest list referencing the existing per-arch images without re-pulling or rebuilding — it is a registry-side operation:

docker buildx imagetools create \
  --tag ${REGISTRY}/api:${TAG} \
  ${REGISTRY}/api:${TAG}-amd64 \
  ${REGISTRY}/api:${TAG}-arm64

Inspect the result to confirm both platforms are present under one tag:

docker buildx imagetools inspect ${REGISTRY}/api:${TAG}

6. Provenance and SBOM attestations during bake

BuildKit can emit in-toto attestations — SLSA provenance and an SBOM — and push them as referrer manifests alongside the image, all during the build. Add attest entries to the target:

target "api" {
  inherits   = ["_common"]
  dockerfile = "api/Dockerfile"
  tags       = ["${REGISTRY}/api:${TAG}"]
  output     = ["type=registry"]

  attest = [
    "type=provenance,mode=max",
    "type=sbom"
  ]
}

mode=max provenance records the full build: source, materials, the Dockerfile, and build args (so do not bake secrets into args). The equivalent on the CLI is --provenance=mode=max --sbom=true, and you can force it across all targets with --set "*.attest=type=provenance,mode=max".

Attestations only attach when the export target understands the OCI referrers API — that means type=registry (or type=image,push=true). They are dropped silently for type=docker loads into the local engine. Verify after pushing:

# Lists the image plus its attestation manifests
docker buildx imagetools inspect ${REGISTRY}/api:${TAG} --format '{{json .Provenance}}'

This gives you SLSA provenance at build time; sign the resulting digest with Cosign in a later step to complete the chain.

7. Wire bake into CI with a sane cache key strategy

The pattern below uses GitHub Actions with a matrix over architecture, native runners, and a separate merge job. The key decisions: a stable cache ref so warm caches survive across branches, and concurrency to cancel superseded runs.

name: build
on:
  push:
    branches: [main]
    tags: ["v*"]

concurrency:
  group: build-${{ github.ref }}
  cancel-in-progress: true

permissions:
  contents: read
  packages: write
  id-token: write   # required for keyless signing later

jobs:
  build:
    strategy:
      fail-fast: false
      matrix:
        include:
          - platform: linux/amd64
            runner: ubuntu-24.04
            suffix: amd64
          - platform: linux/arm64
            runner: ubuntu-24.04-arm   # native arm64, no QEMU
            suffix: arm64
    runs-on: ${{ matrix.runner }}
    steps:
      - uses: actions/checkout@v4
      - uses: docker/setup-buildx-action@v3
        with:
          driver-opts: network=host
      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - name: Bake (per-arch)
        run: |
          docker buildx bake api \
            --set "*.platform=${{ matrix.platform }}" \
            --set "api.tags=ghcr.io/kloudvin/api:${{ github.sha }}-${{ matrix.suffix }}" \
            --set "*.cache-from=type=registry,ref=ghcr.io/kloudvin/buildcache:api-${{ matrix.suffix }}" \
            --set "*.cache-to=type=registry,ref=ghcr.io/kloudvin/buildcache:api-${{ matrix.suffix }},mode=max" \
            --set "*.output=type=registry"

  merge:
    needs: build
    runs-on: ubuntu-24.04
    steps:
      - uses: docker/setup-buildx-action@v3
      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - name: Merge per-arch into multi-arch tag
        run: |
          docker buildx imagetools create \
            --tag ghcr.io/kloudvin/api:${{ github.sha }} \
            ghcr.io/kloudvin/api:${{ github.sha }}-amd64 \
            ghcr.io/kloudvin/api:${{ github.sha }}-arm64

A per-architecture cache ref (buildcache:api-amd64 vs -arm64) keeps the two runners from clobbering each other’s manifest. Keying the ref by service and arch rather than by commit means the cache stays warm across pushes — BuildKit reuses unchanged layers and only rebuilds what the diff touched.

The GitLab shape is identical in spirit: a parallel:matrix over ARCH, docker buildx bake with the same --set overrides, and a dependent merge stage running imagetools create. Use a project-scoped deploy token for docker login against the GitLab registry.

8. Debug cache misses

When a build that should be cached rebuilds from scratch, work the problem in this order.

First, see the resolved plan. --print evaluates the HCL and prints the final JSON without building. This catches the most common bug: a --set or variable that changed a value participating in the cache key.

docker buildx bake api --print

Compare cache keys across runs. Cache misses almost always trace to an input that changed: a build arg, a base image digest that floated, or a COPY whose context shifted. Run with BuildKit debug output to watch which steps resolve as CACHED versus re-executed:

BUILDKIT_PROGRESS=plain docker buildx bake api 2>&1 | grep -E "CACHED|exporting"

Inspect what the registry cache actually contains. If cache-from finds nothing, confirm the cache tag exists and holds your architecture:

docker buildx imagetools inspect ghcr.io/kloudvin/buildcache:api-amd64

Use the build history. Buildx records recent builds. List them and open one to see the full step-by-step trace, including cache decisions, long after the job finished:

docker buildx history ls
docker buildx history inspect <build-id>

Avoid BUILDKIT_INLINE_CACHE=1 as your primary mechanism — it is mode=min semantics and only carries final-stage cache. Keep registry cache as the source of truth and reserve inline cache for the narrow case where you want consumers of a published image to get some cache without a separate cache ref.

Enterprise scenario

A platform team running ~40 microservices on EKS had a release pipeline that took 34 minutes, dominated by arm64. They had standardized on Graviton nodes for cost, so every image was built linux/amd64,linux/arm64 — but CI ran on amd64 GitHub-hosted runners, so the entire arm64 half went through QEMU. Their Rust and Go services compiled under emulation at roughly a sixth of native speed, and a monorepo change that touched a shared library triggered all 40 builds at once.

The constraint was real cost discipline: they would not run a permanent fleet of self-hosted arm64 runners idling between releases. The fix had three parts. First, they moved arm64 builds onto GitHub’s native ubuntu-24.04-arm hosted runners, eliminating QEMU entirely — the arm64 leg dropped from ~22 minutes to ~4. Second, they converted the per-service shell scripts into a single docker-bake.hcl with a _common base and a matrix over the service list, so the monorepo’s 40 builds ran as one bake group sharing a BuildKit daemon and deduplicating the common base layers. Third, they keyed the registry cache per service and arch and added mode=max, which took the dependency-install stages from cold every run to near-instant.

The merge stayed registry-native — no image ever round-tripped through a runner to be reassembled:

# Final assembly job: one manifest list, zero rebuilds
docker buildx imagetools create \
  --tag $REGISTRY/$SERVICE:$VERSION \
  $REGISTRY/$SERVICE:$VERSION-amd64 \
  $REGISTRY/$SERVICE:$VERSION-arm64

End state: 34 minutes to 9, with SLSA provenance attached at build time via attest = ["type=provenance,mode=max"], and a cache hit rate above 80% on incremental changes. The hidden win was reproducibility — because every target inherited the same base, “works on amd64 but not arm64” drift stopped happening, since both platforms now built from byte-identical instructions.

Verify

Confirm the whole chain end to end before you trust it in production:

# 1. The resolved plan is what you expect (platforms, tags, cache refs)
docker buildx bake api --print

# 2. The published tag is a true multi-arch manifest list
docker buildx imagetools inspect ghcr.io/kloudvin/api:1.4.0
#    -> expect Platform: linux/amd64 AND linux/arm64

# 3. Pull and run the arm64 variant on an arm64 host (or emulated)
docker run --rm --platform linux/arm64 ghcr.io/kloudvin/api:1.4.0 --version

# 4. Provenance and SBOM attestations are attached
docker buildx imagetools inspect ghcr.io/kloudvin/api:1.4.0 \
  --format '{{json .SBOM}}'

# 5. A no-op rebuild is fully cached
BUILDKIT_PROGRESS=plain docker buildx bake api 2>&1 | grep -c CACHED

If step 5 reports zero, your cache key is unstable — go back to --print and diff the inputs.

Checklist

• Builder uses the docker-container driver (not the default docker driver)
• arm64 builds run on native runners; QEMU is reserved for non-compiling images
• Shared config lives in a base target consumed via inherits, not duplicated
• cache-to uses type=registry,mode=max with a stable, per-arch cache ref
• Per-arch images are pushed by tag and merged with imagetools create (no QEMU reassembly)
• attest = ["type=provenance,mode=max", "type=sbom"] is set and exported to a registry
• CI defines concurrency with cancel-in-progress to kill superseded runs
• Cache key is keyed by service and architecture, not by commit SHA
• --print is part of your debugging runbook for cache misses
• No secrets are passed via args (they land in mode=max provenance)