Ansible Lesson 36 of 42

Ansible in Air-Gapped Environments, In Depth: Sealed Networks, Internal Mirrors, Signed EEs & Cross-Boundary Workflows

21 min read Vinod

Ansible in Air-Gapped Environments, In Depth — Sealed Networks, Internal Mirrors, Signed EEs and Cross-Boundary Workflows

Air-gapped automation is not Ansible-with-a-firewall. It is a different operating model: every artefact you depend on must be explicitly imported, scanned, signed, versioned and pinned before it can run. There is no ansible-galaxy collection install amazon.aws from a sealed defence network. There is no pip install from a SCADA segment. There is no ee-supported pulled from quay.io from a regulated banking core. Everything must be brought across a controlled boundary, and the boundary itself becomes the most security-sensitive thing you operate.

This lesson is the specialist guide to running Ansible properly in air-gapped, classified, and high-assurance environments: defence, intelligence, OT/SCADA, banking core, regulated medical, nuclear, and other contexts where the network has no internet egress by deliberate policy.

We will cover three boundary archetypes — firewalled (one-way TLS allowed inbound from a trusted relay), traditional air-gap (sneakernet only), and data-diode (one-way physical link, hardware-enforced) — and the runbook patterns each requires. We will also cover the AAP architecture for sealed networks and how to keep auditable provenance for every collection, EE and module that ever runs in your enclave.

Position in the curriculum. Tier 1–4 fluency is required, plus the Tier 5 compliance lesson (because air-gapped environments are usually also the most regulated). The DR lesson is also useful — air-gapped DR has its own constraints we will cover briefly.

What “air-gapped” really means

The phrase is overused. There are three sharply different network models, and each has different Ansible implications:

  1. Soft air-gap (allow-listed egress). No general internet, but a small allow-list of vendor endpoints (Red Hat CDN, Microsoft Update, vendor APT mirrors) is permitted via an outbound proxy. Ansible mostly works as normal; you point ansible-galaxy at an internal mirror and proxy your collection installs.
  2. Traditional air-gap (no inbound, no outbound). The enclave has zero network connectivity to anything outside. Content arrives via physical media (USB sticks, CDs, removable hard drives) following a chain-of-custody process. Ansible content, EEs, OS packages, and Python wheels must all be pre-staged inside.
  3. One-way (data-diode) air-gap. A hardware diode allows packets to flow in one direction only — typically out of the high-side enclave for monitoring, in to the low-side from a publishing relay. Ansible workflows must be split: control plane and execution plane on the high side; only data products flow out. Updates flow in only via signed bundles.

Most enterprises that say “air-gapped” actually mean #1 with policy that aspires to #2. Real defence and OT environments are #2 or #3. The patterns scale: if you can operate cleanly in a #2/#3 environment, you can trivially operate in #1.

A core principle: everything that enters the enclave must be signed, versioned, scanned, and immutable. A collection version is pinned. An EE image is signed by your build pipeline and verified by AAP before execution. An RHEL repository is mirrored on a known-good schedule with a known-good signature. There is no “latest” anywhere; “latest” is a vector for tampering.

The reference air-gap architecture

This lesson assumes a representative enclave:

This architecture has four key invariants:

  1. No host inside the enclave ever phones out. Even DNS lookups for galaxy.ansible.com should NXDOMAIN to fail closed.
  2. Every artefact has a known provenance: signed by a key the enclave trusts, hash recorded on import.
  3. The boundary is monotonic: things only enter, never leave (except via the diode publishing path).
  4. The boundary is auditable: every import is logged, signed, and reviewable for at least the audit horizon.

Mirroring Galaxy and Automation Hub

Inside the enclave, ansible-galaxy must be redirected to the internal mirror. There are three viable backends:

For anything beyond a lab, use Hub or Pulp. The sync host pulls upstream content and signs it before publishing into the internal mirror.

# sync-host playbook: mirror collections from Galaxy
---
- hosts: sync_host
  tasks:
    - name: Download collections needed inside the enclave
      ansible.builtin.command:
        cmd: >
          ansible-galaxy collection download
          {{ item }}
          -p /var/sync/collections
      loop:
        - amazon.aws:8.0.1
        - community.general:9.4.0
        - ansible.posix:1.5.4
        - community.windows:2.3.0
        - community.postgresql:3.7.0

    - name: GPG sign each tarball
      ansible.builtin.command:
        cmd: gpg --detach-sign --armor --local-user {{ signer_uid }}
              /var/sync/collections/{{ item }}
      loop: "{{ collection_tarballs.stdout_lines }}"

    - name: Compute SHA256 manifest for the bundle
      ansible.builtin.shell: |
        cd /var/sync/collections && sha256sum *.tar.gz *.asc > MANIFEST.sha256
      register: manifest

    - name: Bundle into a single tar for transport
      community.general.archive:
        path: /var/sync/collections
        dest: /var/sync/bundles/collections-{{ ansible_date_time.epoch }}.tar
        format: tar

The bundle then travels to the enclave (sneakernet, jump-host pull, or diode write). On the enclave side:

# inside-enclave playbook: import to Hub
---
- hosts: hub_host
  tasks:
    - name: Verify bundle signature before import
      ansible.builtin.command:
        cmd: gpg --verify {{ bundle_path }}.asc {{ bundle_path }}

    - name: Verify SHA256 manifest
      ansible.builtin.shell: |
        cd /tmp/extracted && sha256sum -c MANIFEST.sha256

    - name: Upload each collection to Automation Hub
      ansible.builtin.command:
        cmd: >
          ansible-galaxy collection publish
          /tmp/extracted/{{ item }}
          -s https://hub.enclave.local/api/galaxy/content/published/
          --token {{ hub_token }}
      loop: "{{ verified_collections }}"
      no_log: true

The chain is: upstream → sync host (signed) → boundary transfer → enclave (verified) → Hub (signed again with internal key) → end-user playbooks. Two signatures with different keys: the upstream signer (Red Hat for certified content, your sync engineer for community content) and the internal Hub signer (your enclave’s CA). End-user playbooks pin to specific versions; AAP enforces signature verification before execution.

# requirements.yml inside enclave
---
collections:
  - name: amazon.aws
    version: "8.0.1"   # exact, not >=
    source: https://hub.enclave.local/api/galaxy/content/published/
    signatures:
      - https://hub.enclave.local/api/galaxy/v3/artifacts/collections/amazon.aws-8.0.1.tar.gz.asc

Building signed Execution Environments offline

Execution Environments (EEs) are container images. Inside an air-gap, they cannot pull base layers from quay.io/ansible at build time, and they cannot fetch Python wheels from pypi.org. You must build them on the sync host (which has curated egress), sign the resulting image, transport it, and import into the enclave registry.

# execution-environment.yml on sync host
version: 3
images:
  base_image:
    name: registry.redhat.io/ansible-automation-platform-2.5/ee-supported-rhel9:latest
dependencies:
  galaxy: requirements.yml          # internal mirror requirements after enclave entry
  python: requirements.txt           # pinned wheels with hashes
  system: bindep.txt
options:
  package_manager_path: /usr/bin/microdnf
  skip_pip_install: false
additional_build_files:
  - src: ./offline-pip-cache         # pre-downloaded wheels with --download
    dest: configs
additional_build_steps:
  prepend_galaxy:
    - ADD _build/configs/offline-pip-cache /opt/wheels
  append_galaxy:
    - RUN pip install --no-index --find-links=/opt/wheels -r requirements.txt

Build steps on the sync host:

# pull and stash all upstream wheels with hashes
pip download -r requirements.txt --dest offline-pip-cache --require-hashes

# build the EE
ansible-builder build -t ee-airgap-base:1.0.0 --container-runtime podman

# sign with cosign (sigstore) using the enclave-trusted key
cosign sign --key cosign-airgap.key registry.local/ee-airgap-base:1.0.0

# export and bundle for transport
podman save -o /var/sync/bundles/ee-airgap-base-1.0.0.tar registry.local/ee-airgap-base:1.0.0

Inside the enclave, the import is the inverse:

# verify cosign signature
cosign verify --key cosign-airgap.pub registry.local/ee-airgap-base:1.0.0

# load image into local registry
podman load -i ee-airgap-base-1.0.0.tar
podman tag registry.local/ee-airgap-base:1.0.0 harbor.enclave.local/ee/airgap-base:1.0.0
podman push harbor.enclave.local/ee/airgap-base:1.0.0

In AAP Controller, configure the registry credential with the enclave Harbor URL, and pin every job template to a specific image tag. Never allow :latest; the entire point of the air-gap is to make every change explicit and reviewed.

RPM, APT, and Python wheel mirroring

A fleet inside an air-gap does not just need Ansible — it needs the OS packages, kernel updates, and Python interpreters that those plays install. A complete sealed environment mirrors:

The sync schedule is its own runbook: weekly full sync of OS repos, daily delta sync of selected pinned content, monthly review of what is being mirrored versus what is needed. Auditors love the “monthly review” cadence; it provides paper trail evidence that the enclave is not slowly accumulating untracked content.

# sync-rpm.yml on sync host (typical pattern)
---
- name: Sync RHEL 9 BaseOS to internal mirror
  ansible.builtin.command:
    cmd: >
      pulp rpm repository sync --name rhel9-baseos
  delegate_to: pulp.dmz.local

- name: Publish and distribute the new version
  ansible.builtin.command:
    cmd: >
      pulp rpm publication create --repository rhel9-baseos --version latest
  delegate_to: pulp.dmz.local

- name: Sign the Pulp metadata with enclave key
  ansible.builtin.command:
    cmd: >
      pulp rpm distribution update
      --name rhel9-baseos
      --metadata-signing-service enclave-rpm-signer
  delegate_to: pulp.dmz.local

Inside the enclave, every host’s dnf.repos.d/ (or apt sources.list.d/) points at the internal Pulp distribution and trusts only the enclave key. There is no upstream URL anywhere on any host.

AAP topology inside the enclave

The Ansible Automation Platform fits cleanly into the air-gapped pattern:

The interesting question is the transfer mechanism for new content. Three patterns:

A. Inbound jump host (most enterprises)

A single jump host has read-only access out to a curated set of vendor mirrors via a tightly controlled forward proxy. It also has write access in to the enclave’s Hub/Pulp/Harbor. Ansible runs on the jump host to pull-and-publish.

[upstream Galaxy]    [upstream registry.redhat.io]
        \                    /
         \                  /
        [Curated outbound proxy]
                  |
              [Jump host] — runs sync playbooks, signs artefacts
                  |
        [Enclave Hub / Harbor / Pulp]
                  |
        [Controller + execution nodes]
                  |
            [target hosts]

B. Sneakernet (true air-gap)

The sync host has no in-enclave connectivity at all. Bundles are written to media (USB, removable HDD), carried physically through a SCIF or controlled-area door, scanned by an Egress Inspection station, and imported on the enclave-side import host. This is slower (often weekly cadence) and requires written procedures and chain-of-custody logs. It is what classified networks actually do.

C. Data diode (one-way physical)

A hardware diode allows traffic in one direction only. The classic OT pattern: low-side (general corporate network) writes to the diode, high-side (the OT/SCADA enclave) reads. Ansible bundles are pushed by the low-side sync host and consumed by the high-side import host, with a UDP-based file-transfer protocol that the diode supports.

The reverse path — high-side outbound for telemetry — uses a separate diode in the opposite direction, often with a syslog/Splunk forwarder or a Prometheus remote-write target on the low side.

Each pattern has different Ansible implications, but the playbook structure is identical: sync → sign → bundle → transport → verify → import → publish. Only the transport mechanism changes.

Cross-boundary playbooks: the “publishing” pattern

Some enclaves need to publish automation results out — for example, an OT enclave that must report telemetry to a corporate IT SIEM, or a defence enclave that publishes summary reports to a less-sensitive system.

The Ansible pattern is the publishing playbook: a job that runs entirely inside the enclave, generates a sanitised artefact (a JSON report, a metric file, a scrubbed log bundle), signs it, and writes it to the diode-out side. The other side picks it up, verifies the signature, and consumes it.

# publishing-playbook.yml — runs inside enclave, writes to diode
---
- hosts: localhost
  tasks:
    - name: Generate sanitised metric bundle
      ansible.builtin.template:
        src: metrics.j2
        dest: /var/spool/diode-out/metrics-{{ ansible_date_time.epoch }}.json

    - name: Strip any classified fields
      ansible.builtin.command:
        cmd: jq 'del(.classified, .pii, .secrets)' /var/spool/diode-out/metrics-*.json

    - name: Sign the bundle
      ansible.builtin.command:
        cmd: >
          gpg --detach-sign --armor --local-user enclave-publisher
          /var/spool/diode-out/metrics-{{ ansible_date_time.epoch }}.json

    - name: Hand off to diode transmitter
      ansible.builtin.command:
        cmd: diode-tx --queue /var/spool/diode-out

Key principles:

Air-gapped collections and module portability

Not every collection works inside an air-gap. The ones that need internet access (e.g., a community module that fetches a script from GitHub at runtime) will fail in surprising ways. Audit your collection list before importing:

A useful sanity check before publishing a new collection version into the enclave Hub:

ansible-galaxy collection install ./community-general-9.4.0.tar.gz \
  --server https://hub.enclave.local/api/galaxy/content/published/ \
  --offline    # FAILS if any role/action calls out

The --offline flag is a poor man’s network sandbox. For real assurance, run the test playbooks for the collection inside a netns with no egress.

Air-gapped leapp upgrades and OS lifecycle

Tier 5 already covered leapp; in an air-gap, the leapp content (target package set, repository pointers, leapp-data) must be present locally. The pattern:

  1. On the sync host, download leapp packages and leapp-data for the source/target combination (e.g., RHEL 8.10 → 9.4) using leapp answer --section satellite_to_repository_mapping --add and the official Red Hat tarballs.
  2. Sign the bundle, transport, verify, import to the enclave Pulp repo.
  3. Configure /etc/leapp/transaction/to_install and /etc/leapp/files/repomap.json to reference enclave-local repository IDs.
  4. Run leapp on enclave hosts as normal.

Without the offline leapp data, leapp upgrade will fail with cryptic errors about missing repositories. Pre-stage it; never improvise.

Compliance overlap: the air-gap as evidence multiplier

A side-effect of running properly in an air-gap is that you are doing most of the security-controls work that your auditors expect anyway:

Ansible plays a key role here: roles/dr_evidence from the DR lesson and roles/compliance_evidence from the compliance lesson both land their bundles in immutable buckets, and an air-gapped equivalent (immutable WORM share, NetApp SnapLock volume, on-prem object store with retention) makes them auditable for the lifetime of the system.

Anti-patterns that destroy air-gap discipline

Frequently asked questions

1. Can I run ansible-core inside an air-gap without any modifications? Yes. ansible-core has no network dependency at runtime beyond what your modules call out to (SSH for Linux, WinRM for Windows). The work is in the content (collections, modules, EEs) and the environment (Python, system packages). Once those are mirrored, ansible-core is happy.

2. How do I update collections on a quarterly cadence inside the enclave? Run the sync host’s pull-and-sign playbook on schedule, transport the bundle, run the enclave’s verify-and-import playbook, then publish a new requirements.yml that pins the new versions. Existing job templates continue to use the old versions until you explicitly bump them; nothing is auto-updated.

3. What about ansible-lint, molecule, and CI tooling — do those work air-gapped? Yes, with the same offline pattern. ansible-lint pulls rule packs from PyPI; mirror them. Molecule pulls container images; mirror them. CI runners (Jenkins, GitLab Runner) are inside the enclave with cached source code and inherited from the EE registry.

4. How do I handle credentials for cloud collections in an air-gapped environment? The cloud endpoint must be reachable from the enclave. For sovereign clouds (AWS GovCloud, Azure Government), this is direct; the enclave is peered. For enterprise patterns where the enclave is truly sealed and cloud is “outside,” you do not run cloud collections at all — you run on-prem/private-cloud collections (community.vmware, kubernetes.core against in-enclave clusters).

5. What’s the right way to handle EE rebuilds for security patches? The sync host rebuilds EEs nightly with the latest patched base layers and pinned content versions. The new image is signed and tagged with both :1.0.<n> (incremental) and :1.0 (latest patched within minor). The enclave imports new tags weekly. Job templates pin to specific full tags during stable periods, and bump in change windows.

6. Can Event-Driven Ansible work across the air-gap? Only one-directionally with discipline. Events from inside the enclave can be processed in-enclave (the typical pattern). Events from outside cannot trigger work inside, because that is a hole in your boundary. If you need cross-boundary events, the diode pattern applies: low-side writes a signed event bundle, high-side picks it up via diode-rx, EDA inside the enclave consumes the bundle.

7. What about secrets management — does Vault work air-gapped? HashiCorp Vault, CyberArk, and the AAP-bundled credential store all work fully air-gapped. Bring up the secrets backend inside the enclave, integrate with internal AD/IdP, and treat it as the only source of credentials. Never sync secrets from a corporate password manager into the enclave; provision separately.

8. How do I provide patching SLAs inside an air-gap? Tighten the sync cadence: critical CVEs trigger an out-of-cycle bundle build, transport, and import within hours of disclosure. Practice the cycle quarterly so the operators know the steps. The “patching window” inside the enclave is 24–72 hours from CVE publication for critical, 7 days for high, 30 days for medium — measured from when the bundle becomes available, not from CVE disclosure.

9. What’s the audit story for a sealed enclave? Every artefact import is a logged event with: source URL, upstream signature hash, internal signature hash, importer identity, timestamp, ticket reference. AAP execution events are logged with EE digest and collection version pins. Together, they let you reconstruct “what code ran on what host, sourced from what upstream version, on what date” — which is what auditors want.

10. What’s the single most underrated air-gap practice? The monthly content review. List every collection, EE, repo, and binary in your internal mirror; ask “do we still use this?”. Aggressively prune. The number of unused, unmaintained, and unwatched dependencies in a typical enterprise is staggering, and each one is a future supply-chain bug. An hour a month of review is the cheapest security improvement available.

Hands-on lab — build a tiny offline collection mirror

This lab simulates an air-gap on a single workstation: one container plays the role of the sync host (with internet), another plays the role of the enclave (network-isolated). You will set up an internal collection mirror, sign it, transfer it, verify it, and run a play from the enclave that uses the mirrored content.

Prerequisites: Podman or Docker, ansible-core ≥ 2.16, gpg.

mkdir -p airgap-lab/{sync,enclave,bundles}
cd airgap-lab
gpg --quick-gen-key 'enclave-sync@kv.local' rsa4096 sign 1y

# 1. Sync host: download a collection
podman run --rm -v $PWD/sync:/work:Z quay.io/ansible/ansible-runner \
  ansible-galaxy collection download community.general:9.4.0 -p /work

# 2. Sign it
gpg --detach-sign --armor --local-user enclave-sync@kv.local \
  sync/community-general-9.4.0.tar.gz

# 3. Build SHA256 manifest
( cd sync && sha256sum *.tar.gz *.asc > MANIFEST.sha256 )

# 4. Bundle for transport
tar cf bundles/airgap-bundle-$(date +%s).tar -C sync .

# 5. Enclave: simulate transport (just copy)
mkdir -p enclave/inbox
cp bundles/airgap-bundle-*.tar enclave/inbox/

# 6. Enclave: extract and verify
cd enclave/inbox && tar xf airgap-bundle-*.tar
gpg --verify community-general-9.4.0.tar.gz.asc community-general-9.4.0.tar.gz
sha256sum -c MANIFEST.sha256

# 7. Enclave: install offline
podman run --rm --network=none \
  -v $PWD:/work:Z quay.io/ansible/ansible-runner \
  ansible-galaxy collection install /work/community-general-9.4.0.tar.gz \
    -p /work/collections

The --network=none flag on the enclave podman run is the lab’s “air-gap”: no network at all. The collection installs cleanly because everything is local. Now run a play from inside that no-network container:

# enclave/test.yml
- hosts: localhost
  collections: [community.general]
  tasks:
    - community.general.archive:
        path: /tmp/foo.txt
        dest: /tmp/foo.tar.gz
        format: gz

Run with --network=none and observe success. You have just replicated, in miniature, the entire air-gap content lifecycle: pull, sign, bundle, transport, verify, install, run.

Glossary

Certification mapping

Next steps

You now know how to run Ansible properly inside a sealed environment, and how to keep the boundary auditable as content rotates. The next specialist lesson covers SAP HANA and NetWeaver automation — a domain where Ansible’s role is to coordinate the most demanding production landscapes (SAP basis tasks, hardware sizing, HANA system replication). Many SAP environments live behind the same regulatory boundaries as defence enclaves; the air-gap discipline you have just learned is part of the foundation.

If you only take one habit from this lesson: NXDOMAIN as a security control. When a misconfigured playbook tries to reach galaxy.ansible.com from inside a real enclave, the failure must be loud, immediate, and obvious — not a slow timeout that hides the bug for an hour.

ansibleair-gapair-gappedprivate-automation-hubsneakernetdata-diodesealed-networkoffline-mirror
Need this built for real?

Vinod is a Senior Cloud Architect (22+ yrs) — available for Azure / AWS / GCP architecture, landing zones, and migrations.

Work with me
VH

Written by Vinod

Senior Cloud Architect with 22+ years of experience. Sharing practical knowledge from production environments.

Comments