Engineering Idempotent Ansible Collections with Molecule Testing

Most Ansible “roles” that ship inside an organization are not idempotent; they merely happen to converge the first time you run them on a clean box. Run them twice and they report changed on a task that did nothing. Run them in check mode and they explode. Run them on RHEL after you wrote them on Ubuntu and a package name is wrong. None of that is caught until production, because the role was never tested — it was demoed once and committed.

The fix is two disciplines that reinforce each other: package your automation as a Collection with explicit contracts (argument specs, defaults, semantic versioning), and prove every role with Molecule across a create -> converge -> idempotence -> verify lifecycle in CI. Idempotence stops being a code-review assertion and becomes a test that fails the build. This is the layout and test matrix I use for collections other teams depend on.

In a nutshell

Molecule is a crash-test rig for your automation. A car maker does not ship a design because it looked fine in the showroom; they strap a dummy in, drive a throwaway car into a wall, and measure what happens — every time, on every trim level. Molecule does the same for an Ansible role: it spins up a brand-new throwaway machine, runs your role against it, then runs it a second time and fails loudly if the role tries to change anything on that second pass. A role that “worked on my laptop” is a car that “looked fine in the showroom” — neither has actually been tested.

The one idea to hold onto: idempotence is convergence that sticks. The first run (converge) is allowed to change the box — that is the role doing its job. The second run (idempotence) must report zero changes, because the box is already in the desired state. If the second run still says changed, the role is lying about what it does, and Molecule turns that lie into a red build instead of a 2 a.m. surprise. Packaging the role in a Collection just gives that tested unit a name, a version, and a contract, so other teams can depend on it safely.

Level: Advanced · Time: ~29 min

Before this lesson you should be comfortable writing a basic Ansible playbook and role (tasks, handlers, variables, templates), running ansible-playbook, and reading a PLAY RECAP. Docker or Podman should be installed if you want to follow the Molecule runs. After it you will be able to:

Scaffold a namespaced Collection and understand what each directory and galaxy.yml field is for.
Write tasks that are genuinely idempotent and check-mode safe, and explain why a bare command never is.
Give a role a typed contract with meta/argument_specs.yml so bad input fails fast.
Stand up a Molecule scenario and run the full create -> converge -> idempotence -> verify -> destroy lifecycle.
Prove idempotence and correct outcome across several Linux distributions in CI, and publish the collection with semantic versioning.

Molecule test sequence for an Ansible role/collection

Read left to right: Molecule provisions a throwaway host, converges your role once, converges it a second time and fails the build unless that pass reports changed=0, asserts the real end state, then destroys the host — so every green run is proof the role both reaches and holds the desired state.

1. Lay out the collection: roles, plugins, module_utils, and galaxy.yml

A collection is a namespaced bundle (namespace.collection) that Ansible resolves as kloudvin.platform.nginx. The directory structure is fixed; scaffold it with ansible-galaxy collection init kloudvin.platform. A real collection fills out that skeleton like this:

kloudvin/platform/
├── galaxy.yml                 # collection metadata + dependencies
├── meta/runtime.yml           # requires_ansible, action/module redirects
├── plugins/
│   ├── modules/healthcheck.py    # -> kloudvin.platform.healthcheck
│   ├── filter/netmask.py         # custom filter plugins
│   └── module_utils/http.py      # shared code imported by modules
├── roles/
│   └── nginx/
│       ├── defaults/main.yml         # lowest-precedence, overridable vars
│       ├── meta/main.yml             # galaxy_info, role dependencies
│       ├── meta/argument_specs.yml   # the role's typed contract
│       ├── tasks/main.yml
│       ├── handlers/main.yml
│       ├── templates/nginx.conf.j2
│       └── molecule/default/         # per-role test scenarios
└── tests/sanity/ignore-2.17.txt   # documented sanity-test exceptions

galaxy.yml is the package manifest. Get the version and dependencies right; everything downstream keys off them:

namespace: kloudvin
name: platform
version: 1.4.0          # semver; bump per the rules in section 8
readme: README.md
authors:
  - Vinod H
description: Reusable platform roles, modules, and filters.
license:
  - MIT
tags:
  - infrastructure
  - web
  - linux
dependencies:
  ansible.posix: ">=1.5.0,<2.0.0"
  community.general: ">=8.0.0"
repository: https://github.com/kloudvin/platform
build_ignore:
  - .github
  - "*.tar.gz"
  - tests/output

meta/runtime.yml declares the minimum control-node Ansible and is required for the sanity tests to pass. Pin a floor you actually test against:

requires_ansible: ">=2.16.0"

Build and inspect the artifact locally before you ever push:

ansible-galaxy collection build           # -> kloudvin-platform-1.4.0.tar.gz
ansible-galaxy collection install kloudvin-platform-1.4.0.tar.gz -p ./collections --force

The dependencies map in galaxy.yml is for collection dependencies (other collections from Galaxy), not Python packages. Python deps go in requirements.txt and tests/requirements.txt; system-level requirements get documented in the README and installed in your test images.

2. Write tasks that are genuinely idempotent

Idempotence is a property of every individual task, not of the playbook as a whole. The contract: running the task when the system is already in the desired state must report ok, not changed, and must not error in check mode. Two anti-patterns break this constantly — command/shell and misuse of changed_when. Prefer a real module over command whenever one exists, because modules report change state honestly:

# Idempotent: ansible.builtin.copy hashes content and only writes on diff.
- name: Deploy nginx config
  ansible.builtin.template:
    src: nginx.conf.j2
    dest: /etc/nginx/nginx.conf
    owner: root
    group: root
    mode: "0644"
    validate: "nginx -t -c %s"   # fail BEFORE replacing a known-good file
  notify: Reload nginx

When you are forced to shell out, you own the change reporting. A bare command is always changed because Ansible cannot know what it did. Gate it with changed_when and make it safe in check mode:

# WRONG: reports changed on every run, breaks --check.
- name: Enable feature flag
  ansible.builtin.command: app-ctl enable telemetry

# RIGHT: idempotent guard + honest change reporting + check-mode safe.
- name: Read current feature flags
  ansible.builtin.command: app-ctl get telemetry
  register: flag_state
  changed_when: false          # a read never changes state
  check_mode: false            # safe to run even in --check (read-only)

- name: Enable telemetry
  ansible.builtin.command: app-ctl enable telemetry
  when: "'enabled' not in flag_state.stdout"
  changed_when: true           # if we got here, we changed something

The pattern is a read task (changed_when: false, check_mode: false), then a write task guarded by when: so it only fires when reality diverges from intent. That is how you make imperative commands behave declaratively. When the side effect is a file, creates/removes is the cheapest guard:

- name: Initialize the database schema once
  ansible.builtin.command: app-ctl db init
  args:
    creates: /var/lib/app/.schema-initialized

Two more rules that catch real bugs:

Never use ignore_errors: true to paper over a non-idempotent task. Fix the task. failed_when exists to define what failure actually means.
Loops over package lists belong in the module, not in with_items. ansible.builtin.package and friends accept a list in name: and converge in a single transaction, which is both faster and atomic.

3. Define the role contract: precedence, defaults, and argument_specs

A role that silently does the wrong thing when a caller fat-fingers a variable is a liability. Two mechanisms make the contract explicit: where variables live (precedence) and a typed spec that validates inputs at runtime. Put every user-tunable variable in defaults/main.yml — the lowest precedence, so callers can override it from group_vars, the play, or -e. Reserve vars/main.yml for values the role author controls and does not want overridden casually (it sits very high in precedence). The simplified order, low to high, that matters day to day:

role defaults  <  inventory/group_vars  <  host_vars  <  play vars
               <  role vars (vars/main.yml)  <  block/task vars  <  extra-vars (-e)

So: tunables in defaults/, internal constants in vars/, and remember that -e beats everything — useful for CI overrides, dangerous if you rely on it for normal config.

meta/argument_specs.yml is the role’s signature. Ansible validates it automatically before the role’s tasks run, producing a clear error instead of a confusing failure 12 tasks deep:

argument_specs:
  main:
    short_description: Install and configure nginx.
    options:
      nginx_worker_processes:
        type: str
        default: "auto"
        description: Value for the worker_processes directive.
      nginx_listen_port:
        type: int
        default: 80
        description: TCP port nginx listens on.
      nginx_server_names:
        type: list
        elements: str
        required: true
        description: server_name entries for the default vhost.
      nginx_ssl:
        type: dict
        required: false
        options:
          cert_path: { type: path, required: true }
          key_path: { type: path, required: true }

With that file present, calling the role with nginx_listen_port: "eighty" fails immediately with a type error — validation runs on role entry, with nothing else to wire up. This is the single highest-leverage file in a shared role.

4. Author custom modules and filter plugins inside the collection

When a task needs real logic — call an API, compute something, enforce a non-trivial idempotent state — write a module, not a 40-line shell block. Modules live in plugins/modules/ and are addressed as namespace.collection.name. The non-negotiables: supports_check_mode set, and changed reported accurately.

#!/usr/bin/python
# plugins/modules/healthcheck.py
from ansible.module_utils.basic import AnsibleModule
from ansible_collections.kloudvin.platform.plugins.module_utils.http import probe


def main():
    module = AnsibleModule(
        argument_spec=dict(
            url=dict(type="str", required=True),
            expected_status=dict(type="int", default=200),
        ),
        supports_check_mode=True,   # mandatory for a testable module
    )
    url = module.params["url"]
    expected = module.params["expected_status"]

    # A health check is a read; it never changes state.
    if module.check_mode:
        module.exit_json(changed=False, msg="check mode: probe skipped")

    status = probe(url)
    if status != expected:
        module.fail_json(msg=f"{url} returned {status}, expected {expected}")

    module.exit_json(changed=False, status=status)


if __name__ == "__main__":
    main()

Shared logic goes in module_utils/ and is imported with the fully qualified ansible_collections.<ns>.<coll>.plugins.module_utils.<mod> path — that exact form is what makes the code resolvable once the collection is installed. Document the module with DOCUMENTATION, EXAMPLES, and RETURN YAML blocks; the sanity tests in section 7 fail without them.

Filter plugins are simpler and keep templates clean. They return a dict of name-to-callable:

# plugins/filter/netmask.py
def cidr_to_netmask(cidr):
    import ipaddress
    return str(ipaddress.ip_network(cidr, strict=False).netmask)


class FilterModule(object):
    def filters(self):
        return {"cidr_to_netmask": cidr_to_netmask}

Used in a template or task as {{ '10.0.0.0/24' | kloudvin.platform.cidr_to_netmask }}.

5. Build the Molecule scenario: create, converge, idempotence, verify

Molecule wraps the test lifecycle. A scenario is a directory under roles/<role>/molecule/<scenario>/ with three files. Install the tooling first:

pip install "molecule>=24.0" molecule-plugins[docker] ansible-lint pytest-testinfra

The molecule.yml defines the driver, the platforms (the throwaway hosts), and which verifier to run:

# roles/nginx/molecule/default/molecule.yml
role_name_check: 1
dependency:
  name: galaxy
driver:
  name: docker
platforms:
  - name: nginx-ubuntu2204
    image: geerlingguy/docker-ubuntu2204-ansible:latest
    pre_build_image: true
    privileged: true
    cgroupns_mode: host
    command: /lib/systemd/systemd
    volumes:
      - /sys/fs/cgroup:/sys/fs/cgroup:rw
provisioner:
  name: ansible
verifier:
  name: testinfra

converge.yml is the playbook Molecule runs to apply your role:

# roles/nginx/molecule/default/converge.yml
- name: Converge
  hosts: all
  tasks:
    - name: Run the nginx role
      ansible.builtin.include_role:
        name: kloudvin.platform.nginx
      vars:
        nginx_server_names:
          - example.test

Now run the lifecycle. The idempotence step is the heart of the whole exercise:

molecule create        # provision the container(s)
molecule converge      # apply the role once
molecule idempotence   # apply AGAIN; FAILS if any task reports changed
molecule verify        # assert final state with the verifier
molecule destroy       # tear down

Or run the entire matrix end to end, which is what CI calls:

molecule test          # destroy -> create -> converge -> idempotence -> verify -> destroy

molecule idempotence works by running converge a second time and parsing the recap: if changed != 0 on any host, it fails the build. This is the mechanical enforcement of section 2. A role that passes converge but fails idempotence is broken, full stop — and now the pipeline says so instead of a reviewer guessing.

The verify stage asserts the outcome, not the steps. With Testinfra you write Python assertions against the live container:

# roles/nginx/molecule/default/tests/test_default.py
def test_nginx_running(host):
    nginx = host.service("nginx")
    assert nginx.is_running
    assert nginx.is_enabled


def test_listening_on_80(host):
    assert host.socket("tcp://0.0.0.0:80").is_listening


def test_config_is_valid(host):
    assert host.run("nginx -t").rc == 0

If you prefer to stay in YAML, set verifier.name: ansible and write a verify.yml playbook using ansible.builtin.assert. Both are first-class; Testinfra reads more cleanly for service/port/file assertions.

6. Cover multiple distros and swap Docker for Podman

The bug you are hunting is the package that is nginx on Debian and on RHEL but pulled from EPEL, or the service that is nginx everywhere but the config path differs. Catch it by adding more platforms to the section 5 scenario (note Rocky’s init path differs):

platforms:
  - name: nginx-ubuntu2204
    image: geerlingguy/docker-ubuntu2204-ansible:latest
    pre_build_image: true
    command: /lib/systemd/systemd
    volumes: ["/sys/fs/cgroup:/sys/fs/cgroup:rw"]
    privileged: true
  - name: nginx-rocky9
    image: geerlingguy/docker-rockylinux9-ansible:latest
    pre_build_image: true
    command: /usr/sbin/init
    volumes: ["/sys/fs/cgroup:/sys/fs/cgroup:rw"]
    privileged: true

Add a Debian 12 entry the same way. molecule test now converges and checks idempotence on all of them. That is where OS-conditional logic earns its keep:

- name: Install nginx (handles per-distro package source)
  ansible.builtin.package:
    name: nginx
    state: present
  # EPEL on EL is set up in an earlier task gated on ansible_os_family.

Switching to Podman for rootless CI runners is a one-line driver change plus the matching plugin (pip install molecule-plugins[podman]):

driver:
  name: podman

The platform definitions are otherwise identical. Podman is the default on modern RHEL-family runners and avoids needing a Docker daemon socket, which matters for least-privilege CI.

Use pre_build_image: true with the geerlingguy/docker-*-ansible images. They ship systemd already working, so service/systemd tasks behave like a real host. Building from a bare ubuntu:22.04 and trying to run systemd inside it is a notorious time sink — don’t.

7. Lint, run sanity tests, and wire up GitHub Actions

Three checks run before merge. ansible-lint enforces idempotency-adjacent rules (no bare command, no latest package versions, FQCN usage) and is configurable:

# .ansible-lint
profile: production      # the strictest built-in profile
exclude_paths:
  - molecule/
  - tests/output/

ansible-lint                       # lint roles and playbooks

The collection sanity tests are Ansible’s own structural checks on your plugins — valid DOCUMENTATION, no Python 2 leftovers, correct argument specs. Run them from inside the installed collection tree:

ansible-test sanity --docker -v

Genuine, justified exceptions are recorded per Ansible version in tests/sanity/ignore-<version>.txt; an empty or absent file means zero ignored failures, which is the goal.

Now make all of it a required check. This workflow lints, runs sanity, and executes the full Molecule matrix:

# .github/workflows/ci.yml
name: ci
on:
  push:
    branches: [main]
  pull_request:

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install ansible-core ansible-lint
      - run: ansible-lint

  molecule:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        scenario: [default]
    steps:
      - name: Check out into the collection path
        uses: actions/checkout@v4
        with:
          path: ansible_collections/kloudvin/platform
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install "molecule>=24.0" molecule-plugins[docker] pytest-testinfra ansible-core
      - name: Run Molecule
        working-directory: ansible_collections/kloudvin/platform/roles/nginx
        run: molecule test -s ${{ matrix.scenario }}

The non-obvious bit: check the repo out into ansible_collections/<namespace>/<name>/. Ansible resolves collections by that path layout, so both ansible-test and any kloudvin.platform.* reference in converge.yml only work when the working tree sits there. Skip this and you get cryptic “collection not found” errors in CI that pass locally.

8. Publish to Galaxy / Automation Hub with semantic versioning

The version in galaxy.yml is a promise to consumers, so follow semver strictly:

MAJOR — a breaking change: you removed a role variable, renamed a module, raised requires_ansible, or changed default behavior. Consumers must read release notes.
MINOR — additive and backward compatible: a new role, a new optional variable with a safe default, a new module.
PATCH — a bug fix that changes no interface.

Build, then publish. To public Galaxy you need an API token from your Galaxy profile:

ansible-galaxy collection build
ansible-galaxy collection publish kloudvin-platform-1.4.0.tar.gz --api-key "$GALAXY_TOKEN"

For private Automation Hub (or any pulp/galaxy_ng server), point at its API and token in ansible.cfg rather than passing flags around:

# ansible.cfg
[galaxy]
server_list = automation_hub

[galaxy_server.automation_hub]
url = https://hub.internal.kloudvin.com/api/galaxy/content/published/
token = <hub-token>

Then ansible-galaxy collection publish kloudvin-platform-1.4.0.tar.gz resolves the server from config. Consumers pin you in their requirements.yml and get reproducible installs:

# requirements.yml
collections:
  - name: kloudvin.platform
    version: ">=1.4.0,<2.0.0"

ansible-galaxy collection install -r requirements.yml

The version range with a major-version ceiling means consumers automatically receive your bug fixes and new roles but never a breaking change without an explicit bump — which is the entire point of publishing with semver instead of telling people to track main.

Verify

Before you tag a release, run the four gates end to end:

ansible-lint                              # 1. production profile clean
ansible-test sanity --docker              # 2. no undocumented ignores
cd roles/nginx && molecule test           # 3. converge + idempotence + verify, all platforms
ansible-galaxy collection build           # 4. artifact builds cleanly

The decisive signal is step 3. A clean molecule test means every platform provisioned, the role converged, a second converge reported zero changes (proven idempotence), and the verifier asserted the real end state. Green across Ubuntu, Rocky, and Debian means the role is safe to publish. As a final smoke test, install the built tarball into a scratch path (ansible-galaxy collection install kloudvin-platform-*.tar.gz -p /tmp/verify --force) and confirm a deliberately bad variable value (nginx_listen_port: "eighty") is rejected with a type error from the argument spec.

Checklist

Going deeper

The real test sequence, in order

Section 5 shows the lifecycle in its abbreviated form — create -> converge -> idempotence -> verify -> destroy. What molecule test actually executes is a longer, configurable test_sequence, and knowing the real order explains several behaviours that otherwise look like magic. The default sequence Molecule ships is:

# the default `test` sequence — defined in the scenario, rarely overridden
test_sequence:
  - dependency
  - cleanup
  - destroy
  - syntax
  - create
  - prepare
  - converge
  - idempotence
  - side_effect
  - verify
  - cleanup
  - destroy

Two details matter. First, test opens with cleanup and destroy: Molecule tears down any leftover instance from a previously aborted run before it builds a fresh one. That is why molecule test always starts from a guaranteed-clean slate, while molecule converge, which you run by hand during development, deliberately reuses the already-running box so your edit-test loop stays fast. Second, dependency, prepare and side_effect are first-class, hookable steps — not the two-file toy from section 5. One thing that is not in the sequence: linting. The old molecule lint command was removed years ago; ansible-lint is its own gate (section 7), and the syntax step here is only ansible-playbook --syntax-check on converge.yml, which catches malformed YAML/Jinja but says nothing about idempotency.

Each step maps to a playbook or a driver action inside the scenario directory:

Step	What runs	Optional	Typical use
`dependency`	`ansible-galaxy` on `requirements.yml`	yes	pull roles/collections the role needs
`create`	`create.yml` (from the driver plugin)	no	provision the throwaway host(s)
`prepare`	`prepare.yml`	yes	seed pre-state the role assumes exists
`converge`	`converge.yml`	no	apply the role — the play under test
`idempotence`	`converge.yml` again	no	second apply; must report `changed=0`
`side_effect`	`side_effect.yml`	yes	inject a reboot/failure to test recovery
`verify`	`verify.yml` or Testinfra tests	no	assert the end state
`cleanup`	`cleanup.yml`	yes	release external resources (delegated)
`destroy`	`destroy.yml` (from the driver plugin)	no	tear the host down

A prepare.yml earns its keep the moment the role assumes something it does not itself install — an EPEL repo, a service account, a fixture file:

# roles/nginx/molecule/default/prepare.yml
- name: Prepare
  hosts: all
  gather_facts: true
  tasks:
    - name: Enable EPEL on EL so nginx resolves
      ansible.builtin.package:
        name: epel-release
        state: present
      when: ansible_os_family == "RedHat"

The rule of thumb: setup that is a precondition of the role belongs in prepare; setup that is the role’s job belongs in the role. Blur that line and your idempotence test quietly starts covering the wrong thing.

How idempotence is actually enforced

The idempotence action is not a separate playbook — it re-runs converge.yml a second time, captures the provisioner’s output, and inspects the PLAY RECAP for every host. If the total changed count across all hosts is anything but zero, the action raises and the build exits non-zero. The only recap that passes looks like this:

# representative — the ONLY acceptable idempotence recap
PLAY RECAP *********************************************************************
nginx-rocky9      : ok=14  changed=0  unreachable=0  failed=0  skipped=3
nginx-ubuntu2204  : ok=14  changed=0  unreachable=0  failed=0  skipped=3

ok can be any number; changed must be 0. When it is not, the debugging move is mechanical: run molecule converge twice by hand and read which task flips to changed on the second pass. That task is the bug, every time. The usual suspects, in rough order of frequency:

a command/shell with no changed_when (always changed);
a template whose rendered output is non-deterministic — an unsorted dict in a loop, a timestamp, {{ ansible_date_time }} baked into the file;
a lineinfile/replace whose regex keeps re-matching its own output;
ansible.builtin.file with a mode that fights the umask, so it re-sets permissions each run;
any package task pinned to state: latest (a new upstream release makes it changed with no code change at all).

Fix the task, not the test. idempotence is section 2’s contract made mechanical: the reviewer no longer has to believe the role is idempotent, the pipeline knows.

Drivers: docker, podman, and delegated

Section 6 swapped Docker for Podman with a one-line change. The third option is the one that unlocks everything else: the built-in default driver, which is delegated — Molecule runs your own create.yml and destroy.yml, so the “throwaway host” can be anything you can provision with Ansible.

Driver	Provided by	Throwaway host is	Reach for it when
`docker`	`molecule-plugins[docker]`	a container	fast local + CI; systemd via `geerlingguy` images
`podman`	`molecule-plugins[podman]`	a rootless container	least-privilege CI; RHEL-family runners
`default` (delegated)	built into Molecule	whatever `create.yml` builds	cloud VMs, Vagrant, LXD, or bare `localhost`

The delegated driver matters because containers cannot test everything: a kernel module, a reboot, a real cloud-init path, an SELinux relabel, or a role that manages the bootloader all want a genuine VM. With driver: name: default you write a create.yml that spins up (say) an EC2 instance or an Azure VM and records its address in Molecule’s instance config, and a destroy.yml that terminates it. Everything downstream — converge, idempotence, verify — is identical. You trade speed for fidelity; keep a fast docker scenario for the inner loop and a slow default scenario for the nightly job.

Scenarios: one role, several test shapes

A scenario is a self-contained molecule.yml + converge.yml (plus optional prepare/verify/side_effect) under roles/<role>/molecule/<name>/. default is just the conventional first one. Add more to test permutations the single happy path never exercises — TLS on, clustered vs standalone, upgrade-from-the-previous-release:

# roles/nginx/molecule/ssl/converge.yml — same role, different contract
- name: Converge
  hosts: all
  tasks:
    - name: Run the nginx role with TLS enabled
      ansible.builtin.include_role:
        name: kloudvin.platform.nginx
      vars:
        nginx_server_names:
          - secure.example.test
        nginx_ssl:
          cert_path: /etc/pki/tls/certs/example.test.crt
          key_path: /etc/pki/tls/private/example.test.key

Run one with molecule test -s ssl, or every scenario with molecule test --all. Each scenario proves idempotence independently, so a role that is idempotent with TLS off but not with TLS on gets caught — which is exactly the branch a hand-demo skips.

Testing a role vs testing a whole collection

A role scenario proves one role in isolation. A collection is a set of roles plus plugins, and it can break in two extra places: roles that interact, and plugins that must pass Ansible’s own structural checks. Cover all three levels:

Per-role Molecule (roles/<r>/molecule/) — behaviour and idempotence of each role, as above.
Collection-level Molecule under extensions/molecule/<scenario>/ — a converge that layers several roles the way a consumer would, catching cross-role assumptions (role B expecting a user that role A creates). The extensions/ location is deliberate: ansible-test scans the collection tree, and keeping test scaffolding there keeps it out of the packaged artifact.
ansible-test for plugins — sanity for structure (section 7) and units/integration for module logic.

At collection level, roles are addressed by their fully-qualified name, never a bare nginx:

# extensions/molecule/default/converge.yml
- name: Converge the whole platform baseline
  hosts: all
  tasks:
    - name: Base OS hardening
      ansible.builtin.include_role:
        name: kloudvin.platform.baseline
    - name: Web tier on top of the baseline
      ansible.builtin.include_role:
        name: kloudvin.platform.nginx
      vars:
        nginx_server_names:
          - example.test

FQCN everywhere is not pedantry: it is what lets Ansible resolve kloudvin.platform.nginx unambiguously when two installed collections both ship a role called nginx.

The CI matrix: distros × scenarios

Section 7 ran one scenario on one runner. Real coverage is a matrix — every supported distro against every scenario — and GitHub Actions expands it for you. Parametrise the image by an environment variable and let the matrix fan out:

# .github/workflows/ci.yml (molecule job)
  molecule:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        distro: [ubuntu2204, rocky9, debian12]
        scenario: [default, ssl]
    env:
      MOLECULE_DISTRO: "${{ matrix.distro }}"
    steps:
      - uses: actions/checkout@v4
        with:
          path: ansible_collections/kloudvin/platform
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: pip
      - run: pip install "molecule>=24.0" molecule-plugins[docker] pytest-testinfra ansible-core
      - name: Molecule
        working-directory: ansible_collections/kloudvin/platform/roles/nginx
        run: molecule test -s ${{ matrix.scenario }}

Molecule interpolates ${VAR} in molecule.yml, so the platform block can read the matrix axis directly — one molecule.yml, six jobs:

# roles/nginx/molecule/default/molecule.yml (platforms only)
platforms:
  - name: nginx-${MOLECULE_DISTRO}
    image: "geerlingguy/docker-${MOLECULE_DISTRO}-ansible:latest"
    pre_build_image: true
    privileged: true
    command: /lib/systemd/systemd
    volumes:
      - /sys/fs/cgroup:/sys/fs/cgroup:rw

fail-fast: false is deliberate — you want to see that the role is broken on Rocky and fine on Ubuntu in one run, not have the first red cell cancel the rest. cache: pip and the checkout-into-ansible_collections/<ns>/<name>/ path (section 7) are the two lines people forget; without the second, every kloudvin.platform.* reference fails to resolve in CI while passing locally.

The three-legged stool: lint, sanity, Molecule

These three gates test different things and none subsumes the others — drop one and a whole class of bug walks through:

Gate	Tool	Catches	Blind to
Static lint	`ansible-lint` (production profile)	bare `command`, `state: latest`, missing FQCN, style	whether the role actually converges
Structure	`ansible-test sanity`	bad `DOCUMENTATION`, Py2 leftovers, arg-spec drift	runtime behaviour of tasks
Behaviour	`molecule test`	non-convergence, non-idempotence, wrong end state	code style, plugin doc structure

Run all three on every PR. Lint is the cheapest and should fail first; Molecule is the slowest and most decisive.

Version and API caveats

Molecule 6.x / the molecule>=24.0 line: drivers other than the delegated default live in molecule-plugins (installed as molecule-plugins[docker]), not in Molecule core. Older guides that pip install molecule-docker predate that split.
molecule lint is gone — lint out-of-band with ansible-lint; Molecule tests behaviour, not style.
The default verifier is now ansible (a verify.yml of assert tasks). Testinfra is opt-in: pip install pytest-testinfra and set verifier: name: testinfra. Both are first-class; this lesson uses Testinfra for the cleaner service/port assertions.
Podman is the default on EL9+ runners; the Docker socket may simply not be there, which is one more reason to keep a Podman scenario green.
requires_ansible in meta/runtime.yml gates the control node, not the target host. Setting it to a floor you do not actually test in CI is a promise you cannot keep.

Practice challenges

Work these in order; each <details> block holds a working answer and the one-line reason it is right.

1. Beginner — make a bare command idempotent. This task reports changed on every run and explodes under --check. Fix it so a second run is ok and check mode is safe:

- name: Enable telemetry
  ansible.builtin.command: app-ctl enable telemetry