Ansible Automation Platform Architecture, In Depth: Controller, Automation Hub & Event-Driven Ansible

In a nutshell

Running one ansible-playbook from your laptop is like one skilled electrician showing up with a toolbox and wiring a single house. It works, and it’s real automation. But now imagine you have to wire a whole city — hundreds of buildings, some behind locked gates, some in another town, some off-grid — with crews who come and go, safety inspectors watching, and only approved parts allowed on site. One electrician with a toolbox doesn’t scale to that. You need a dispatch office.

Ansible Automation Platform (AAP) is the dispatch office for automation. It doesn’t change the wiring (your playbooks, roles, and collections are identical); it surrounds them with the office everyone shares:

• The automation controller is the dispatch desk — it holds the job sheets (job templates), decides who is allowed to run what (RBAC), keeps the schedules and the keys (credentials and inventories), and records who did what, when, against which hosts.
• The private automation hub is the certified-parts warehouse — only approved, signed parts (collections and container images) ever leave the shelf.
• Execution Environments are the standardized toolbox vans — every crew carries the exact same tools and versions, so a job behaves the same whoever runs it.
• Event-Driven Ansible is the alarm system wired straight to the dispatch desk — when a sensor trips at 3 a.m., a job is dispatched automatically, no human paged.
• The automation mesh is the network of relay depots and roads that lets the one central office reach jobs in a locked compound, another city, or an off-grid lab.

Why should a beginner care? Because you already know the hard part — how to write a playbook. AAP is simply how a company runs thousands of them safely: with audit trails, approvals, least-privilege, and a supply chain you can trust. Learn this architecture and you can walk into any AAP environment, point at each moving part, and say exactly what it does and why it’s there.

Level: Advanced · Time: ~45 min

ansible-playbook from a developer’s laptop is automation. It is also, at scale, a problem: who ran it, against which inventory, with which credentials, when? Where do collections come from — and how do you trust them? When a webhook fires at 3 a.m., who runs the runbook? Ansible Automation Platform (AAP) is Red Hat’s answer: the supported, hardened, multi-component product that turns Ansible from “a CLI you trust your operators with” into “a platform your auditors trust.” AAP is not a different language — your roles, collections, and Execution Environments are unchanged — but it surrounds them with Controller (formerly Tower / upstream AWX: web UI + REST API + job execution at scale), Automation Hub (your private collection registry with content signing), and Event-Driven Ansible (EDA) (webhook/kafka/queue-driven autonomous response), all glued together by the Automation Mesh (a peer-to-peer Receptor network that runs jobs near the targets that need them).

This lesson is the architecture tour. It does not teach you how to click around the Controller UI button-by-button — that’s a hands-on exercise. It teaches you the mental model every engineer running AAP needs: which component does what, what an Execution Environment is, where the Mesh’s control/hop/execution nodes sit, what an Automation Hub namespace and repository and signature are, what a rulebook is and how its sources/conditions/actions decide whether to fire, and how the open-source counterparts (AWX and EDA Server) map onto the supported product. We finish with a free, end-to-end lab that brings AWX up on a kind Kubernetes cluster — the Operator-installed deployment Red Hat ships with AAP, just on the upstream image — so you can actually drive a real Controller from a laptop without a Red Hat subscription. Everything targets AAP 2.5+ / AWX 24+ / EDA Server 1.1+ (2026), with FQCN throughout.

Learning objectives

By the end of this lesson you can:

• Distinguish ansible-core, AWX (open source), and AAP (Red Hat-supported), and explain which problems each solves.
• Map the Controller object model end to end: Organisations, Teams, Users, Projects (SCM-backed), Inventories (smart/regular/constructed), Credentials, Credential Types, Execution Environments, Job Templates, Workflow Templates, Surveys, Schedules, Notifications.
• Explain the Automation Mesh node taxonomy — control, hop, execution, hybrid — and when to use each; describe the Receptor protocol and peer relationships.
• Describe Automation Hub: namespaces, repositories (published/rh-certified/community/validated), remote sync, content signing, and how ansible.cfg galaxy_server_list consumes it.
• Describe Event-Driven Ansible: the EDA Server, rulebooks (sources/conditions/actions), the canonical sources (webhook, kafka, azure_service_bus, url_check, journald, aws_sqs), the action types (run_playbook, run_job_template, run_module, post_event, set_fact, debug), and throttling/de-duplication.
• Use the awx CLI and the Controller REST API for Infrastructure-as-Code patterns: defining Projects/Inventories/Job Templates/Schedules in version-controlled code rather than the UI.
• Pick the right topology for your scale: single-node, HA control plane + worker pool, multi-cluster mesh.
• Stand up AWX on Kubernetes (kind + the AWX Operator) and run a Job Template against a free dynamic inventory, end to end, on your laptop.

Prerequisites & where this fits

You should already be comfortable with playbooks, roles, and collections — the collections & Execution Environments lesson is the immediate prerequisite, because AAP’s runtime is an EE. You should know dynamic inventory patterns from the dynamic-inventory lesson — Controller imports plugin-driven inventories the same way the CLI does. You should have run Molecule at least once (the Molecule lesson) so you understand the testing tier the Controller’s content lifecycle expects. Vault familiarity from the vault lesson is helpful because Controller stores secrets in its own encrypted credentials store and integrates with HashiCorp Vault, CyberArk, and Azure Key Vault. In the Ansible Zero-to-Hero programme this is the Advanced/Platform tier capstone: it bridges the CLI Ansible you’ve mastered and the platform your team actually runs in production.

Core concepts

Five mental models carry the whole lesson.

1. AAP is ansible-core + a control plane. Every Job Template, every workflow, every event-driven response eventually shells out to ansible-core running inside an Execution Environment. The platform does not replace Ansible; it manages it — schedules it, audits it, RBACs it, runs it nearer the target. If you understand ansible-playbook you already understand 80% of what Controller does at runtime.

2. The Controller is the API. Everything you do in the UI is a POST/PATCH against /api/v2/.... Every UI tab has a CLI equivalent (awx job_templates create, awx projects update). Treat the UI as a courtesy view; treat the API as the contract. The implication is huge: every Project, Inventory, Job Template, Workflow Template can — and should — be defined in version control and reconciled by code (Terraform’s awx provider, the awx.awx collection’s Ansible modules, or raw awx CLI calls). “Click-ops in the UI” is the AAP anti-pattern.

3. The Mesh moves jobs to where the targets are. The Automation Mesh is a Receptor-based peer-to-peer network. Control nodes host the API, scheduler, and database. Execution nodes actually run ansible-playbook inside an EE. Hop nodes relay traffic across network boundaries (DMZ → trusted, on-prem → cloud). A job submitted to an Instance Group “lands” on whichever execution node has capacity and network reachability to the target inventory. This decouples control (centralised API, single pane of glass) from execution (distributed, near the workload) — and is the answer to “we have ten clusters across three clouds and an air-gapped lab, how do we run Ansible against all of them?”

4. The Automation Hub is your supply chain. Where does community.general come from? In a hobby setup, galaxy.ansible.com. In a regulated setup, Automation Hub — a private registry that hosts your internal collections (the ones written by your platform team), mirrors Red Hat-certified collections from console.redhat.com, syncs validated content (approved patterns), and signs every collection with a sigstore identity. Your laptops, your Controller, and your CI all pull from Hub via galaxy_server_list in ansible.cfg. If a collection is not in Hub, it does not run in production. The Hub is the supply-chain trust boundary.

5. EDA closes the loop. Job Templates are invoked. Rulebooks are triggered. EDA is the missing eventing tier: a rulebook says “when a webhook fires from PagerDuty, AND the alert is high-severity, AND the host is in the prod_db group, run Job Template restart-postgres-replica.” The rulebook is YAML; the EDA Server runs it as a long-running process listening on its sources. EDA turns Ansible from “what an operator runs” into “what the platform runs autonomously.”

Keep these terms straight: AAP (the supported product), AWX (the upstream open-source equivalent of Controller), Controller (the orchestration component, formerly Tower), Automation Hub (the private content registry), EDA (Event-Driven Ansible — the eventing tier), Receptor (the mesh transport), Execution Environment (the OCI image carrying ansible-core + collections), Instance Group (a logical grouping of execution nodes), Project (a Controller object pointing at an SCM repo), Job Template (a Controller object that combines a project + inventory + credential into a runnable thing), Workflow Template (a DAG of Job Templates), Survey (a parameter-prompt UI on top of a JT), Rulebook (the EDA YAML defining sources/conditions/actions).

ansible-core vs AWX vs AAP

A pragmatic rule: if you’re learning, build with AWX (free, identical model). If you’re shipping production at a regulated company, run AAP (the supply-chain and support story is non-negotiable). The core mental models — Controller, Hub, EDA, Mesh — are identical across AWX and AAP at the level you’ll use them in 99% of work.

Controller — the object model

Every Controller object is reachable in the UI and at /api/v2/<resource>/. The hierarchy:

Organisations
├── Teams (RBAC grouping of Users)
├── Users
├── Projects        ← SCM-backed (git, http archive); imports playbooks/roles/collections
├── Inventories     ← static, smart (filter), constructed, dynamic-plugin sources
│   ├── Hosts
│   └── Groups
├── Credentials     ← machine, source-control, vault, cloud (AWS/Azure/GCP/k8s/…), generic
├── Credential Types
├── Execution Environments
├── Instance Groups (which mesh nodes run my jobs)
├── Job Templates           ← project + inventory + credentials + EE + survey + schedule
├── Workflow Templates      ← DAG of JTs with success/failure paths and approval nodes
└── Notifications

Organisations, Teams, Users, RBAC

Organisations are the top-level tenancy boundary — every other object belongs to one. Teams are RBAC groupings; users are added to teams, teams are granted roles on objects. The Controller has a fixed role taxonomy per object type:

Sane defaults: developers get execute on the Job Templates they own and use on the Credentials those JTs need; the platform team gets admin on Projects/Inventories/Credentials; auditors get the org-level auditor role.

Projects

A Project is the Controller’s tracker for a Git repository. Configure:

A Project sync clones the repo onto the control plane and indexes its playbooks, roles, and collection metadata. Job Templates reference a playbook inside a project.

Inventories

A Controller Inventory is the same concept as a CLI inventory, with extras. It can be:

• Regular — a fixed set of hosts/groups, edited in the UI or imported from a file.
• Smart — derived by a host filter across all hosts in the org (groups__name=prod and ansible_facts__os_family=RedHat).
• Constructed — equivalent to the constructed inventory plugin: composes groups across multiple inventory sources.

Each Inventory can have one or more Inventory Sources (the dynamic part):

Every source has its own update schedule (update_on_launch, periodic). Set update_on_launch: true on cloud sources so a Job Template never runs against a stale fleet.

Credentials

Credentials are first-class encrypted secrets. The Controller has a fixed set of built-in Credential Types plus user-defined ones:

A user-defined Credential Type has two parts: an inputs schema (what fields the credential takes) and an injectors block (how those fields are exposed at runtime — env vars, files, extra-vars). This is how you wire HashiCorp Vault, CyberArk, or any other secret store into Controller without writing code.

Execution Environments

An EE is an OCI image carrying a specific ansible-core + collection set + Python deps. Controller runs every job inside an EE. The EE catalogue is per-org; you reference an EE by registry URL and pull credential. The default ee-supported-rhel*-aap* EE ships with AAP and contains the certified collections. Building your own EE is the collections & EE lesson topic — Controller just consumes the resulting image.

Job Templates

A Job Template (JT) is the Controller’s runnable unit. It binds:

A JT with a survey is the canonical “operator self-service” pattern: the platform team writes the playbook, defines a Job Template, attaches a survey of safe parameters (which environment, which app, which version), and grants execute to a developer team. Devs launch the JT through the UI, answer the survey, and the platform team’s playbook runs against the right inventory with the right credentials, audited.

Workflow Templates

A Workflow Template (WT) is a DAG of nodes. Each node is one of:

• A Job Template launch (the common case).
• A Project sync.
• An Inventory source update.
• An approval node — pauses the workflow until a human with the approval role accepts/denies.
• Another Workflow Template (recursive).

Edges are typed: success, failure, always. So a WT can express “sync inventory; on success, run preflight JT; on success, prompt approval; on approval, run deploy JT; on failure of deploy, run rollback JT; on success, notify Slack.” This is the AAP equivalent of CI/CD.

Surveys, Schedules, Notifications

A Survey is a JSON-schema-shaped form attached to a JT or WT, prompted at launch. Schedules are cron-based unattended launchers (extras: rrule for richer recurrence, enabled: false to pause). Notifications are templates (Slack/MS Teams/email/webhook/PagerDuty/Mattermost/IRC) attached to JTs and WTs and triggered on started/success/failure/approved/denied/running.

The awx CLI and the API

pip install awxkit gets you the awx CLI. Login once; from then on every Controller object can be CRUDed from the command line:

awx --conf.host https://controller.example.com login
awx projects list
awx projects create --name "Ops Repo" --organization "Default" --scm_type git \
  --scm_url https://git.example.com/ops/ansible.git --scm_branch main \
  --scm_update_on_launch true
awx job_templates create --name "Deploy Web" --project "Ops Repo" \
  --inventory "Prod AWS" --playbook deploy.yml \
  --execution_environment "ee-supported-rhel9-aap25" \
  --extra_vars '{"app_version":"1.2.3"}'
awx job_templates launch "Deploy Web" --monitor

Treat the CLI as the source of truth for IaC: a awx-config/ directory of YAML or shell scripts that recreates every Project/JT/WT in a clean Controller. Combined with the awx.awx Ansible collection’s modules (awx.awx.tower_project, tower_job_template, etc.), you can reconcile Controller state from a playbook — Controller automating Controller.

Automation Mesh

The Mesh is the data plane. It is a peer-to-peer network of Receptor processes (TLS-mutual-authenticated, latency-tolerant, NAT-friendly). Node roles:

Instance Groups are logical groupings of execution nodes. A Job Template is assigned to an Instance Group; the scheduler picks an idle node in that group with network reachability to the inventory. Common patterns:

• default — the catch-all group containing every execution node.
• aws-runners — execution nodes deployed in AWS, tagged so AWS-only JTs land here (cheaper egress, lower latency).
• dmz-bridge — hop nodes that bridge a corporate-DMZ JT into a customer’s VPC.
• gpu-heavy — execution nodes with GPUs, for ML provisioning workflows.

Peer relationships form the topology graph: each node’s config lists which other nodes it dials and which it accepts dials from. The mesh tolerates link failures — if a hop is down, traffic re-routes through alternative peers.

A typical small AAP install has one hybrid control node + two execution nodes + a hop node at the customer-VPC bridge — a topology that handles thousands of concurrent jobs and survives single-node loss. The clustered AAP install scales out by adding more control nodes (HA with a shared Postgres) and more execution nodes (just more workers).

Automation Hub

Hub is the private registry. It is a Pulp-based service that hosts collections in repositories:

A Namespace is the Hub equivalent of a GitHub organisation: my_namespace.my_collection. A namespace owner controls who can publish.

Content Signing: Hub signs every published collection with a configured signing key (sigstore-compatible). When ansible-galaxy collection install pulls from Hub, it verifies the signature against a public key stored locally or in a Controller GPG Public Key credential. Unsigned collections are rejected if signature_required is set. This is the supply-chain integrity story: a malicious collection cannot land on production unless it carries a signature from a key your CI trusts.

Remote sync lets Hub mirror upstream sources on a schedule. Configure a Remote (the upstream URL + auth + auth header) and a Repository (which content to pull). Schedule it. Air-gapped installs use this with an HTTP proxy that holds the Red Hat manifest token, fetching once into a staging Hub and never letting console.redhat.com reach production directly.

Consuming Hub from a CLI client (laptop, CI, Controller EE build) is just an ansible.cfg galaxy_server_list:

[galaxy]
server_list = my_hub, rh_certified, community

[galaxy_server.my_hub]
url = https://hub.example.com/api/galaxy/content/published/
token = ${HUB_TOKEN}

[galaxy_server.rh_certified]
url = https://hub.example.com/api/galaxy/content/rh-certified/
token = ${HUB_TOKEN}

[galaxy_server.community]
url = https://hub.example.com/api/galaxy/content/community/
token = ${HUB_TOKEN}

The order matters: ansible-galaxy collection install walks the list in order. Put your published collections first so an internal collection that shadows an upstream name (rare but legitimate) wins.

Event-Driven Ansible (EDA)

EDA is a separate component (the EDA Server) that runs rulebooks: long-running YAML processes that listen on sources, evaluate conditions, and fire actions. Architecturally it is pub/sub for Ansible.

A rulebook, dissected

# rulebooks/restart-on-pagerduty.yml
- name: Restart Postgres replica on PagerDuty alert
  hosts: all
  sources:
    - ansible.eda.webhook:
        host: 0.0.0.0
        port: 5000
        token: "{{ EDA_PAGERDUTY_TOKEN }}"
  rules:
    - name: High-severity alert on a prod_db host
      condition: >-
        event.payload.event.event_type == 'incident.triggered' and
        event.payload.event.data.incident.urgency == 'high' and
        event.payload.event.data.incident.tags is contains 'prod_db'
      throttle:
        once_within: 10 minutes
        group_by_attributes:
          - event.payload.event.data.incident.id
      action:
        run_job_template:
          name: "Restart Postgres Replica"
          organization: "Default"
          job_args:
            extra_vars:
              incident_id: "{{ event.payload.event.data.incident.id }}"
              host: "{{ event.payload.event.data.incident.tags | select('match', '^host:') | first | replace('host:', '') }}"

Every rulebook has three sections:

Sources — the eventing inputs. Each source is a long-running coroutine in the EDA Server. Common sources from ansible.eda:

Conditions — Jinja-shaped predicates over the event payload. Conditions can chain (and/or), reference nested fields (event.payload.foo.bar), use built-in operators (is contains, is regex, is search, is defined), and reference rule-local state (vars).

Actions — what to do when a condition fires. The action set:

Throttling & de-duplication — EDA has built-in throttle: blocks (once_within, once_after, group_by_attributes) that prevent a noisy source from firing the same action a hundred times. Pair with group_by_attributes to scope the throttle key to the meaningful identity (an incident ID, a host name) rather than the whole rule. Without throttling, a flapping monitor will DoS your Controller.

EDA Server runs rulebooks as long-running processes managed via a rulebook activations API on the Controller. Rulebooks live in Decision Environments (DE) — an EE-shaped image that ships ansible-rulebook + the ansible.eda collection + your custom event sources. The DE is to EDA what an EE is to Controller.

Topology cheat-sheet

The Mesh’s flexibility is what makes any of these workable: control plane stays small (one HA cluster), execution scales linearly by adding nodes, hop nodes solve every network-segmentation problem.

Going deeper

Everything above is the model an operator needs. This section is the why-under-the-hood an architect needs — the AAP 2.4→2.5 shift, the supported collections, the job lifecycle, the mesh internals, and the HA boundaries. It is where the “Advanced” in this lesson’s level actually lives.

The AAP 2.4 → 2.5 shift: the platform gateway

Before 2.5, Controller, Hub, and EDA were three separate web applications: each had its own login page, its own port, its own RBAC store, and its own URL. Operators juggled three sets of users and three sessions. AAP 2.5 introduced the platform gateway — the aap-gateway service — as a single front door for the whole platform. The gateway:

• Terminates TLS and brokers authentication once (local, LDAP, SAML, OIDC/social), then federates the identity to Controller, Hub, and EDA.
• Presents one unified navigation and one RBAC model — users, teams, and role assignments are managed at the gateway and propagate down.
• Proxies the component APIs. The services still exist behind it (/api/controller/v2/, /api/galaxy/, /api/eda/v1/); the gateway adds its own /api/gateway/v1/.

If you learned AAP 2.4, the single biggest muscle-memory change in 2.5 is that authentication and RBAC moved up to the gateway — you no longer create a user separately in Controller and Hub. AWX has no gateway; it remains a single self-contained service, which is one of the few places the upstream and supported products visibly diverge. When you read “AAP 2.5 topology,” expect a gateway (often HA behind a load balancer, backed by Redis) sitting in front of the control plane.

awx.awx vs ansible.controller — the supported-collection caveat

The CLI and playbook examples earlier use the community awx CLI and the awx.awx collection, which target AWX. On supported AAP the equivalent modules ship in the Red Hat-certified ansible.controller collection. Same parameters, supported names:

The tower_* names still resolve as aliases for backwards compatibility, but they are legacy — do not write new content against them. Rule of thumb: on AAP use ansible.controller.*; on AWX use awx.awx.*. A minimal, idempotent “Controller-as-code” task on AAP:

- name: Declare a project on AAP (supported collection)
  hosts: localhost
  connection: local
  gather_facts: false
  tasks:
    - name: Ensure the Ops Repo project exists
      ansible.controller.project:
        name: Ops Repo
        organization: Default
        scm_type: git
        scm_url: https://git.example.com/ops/ansible.git
        scm_branch: main
        state: present

A job’s lifecycle, and the capacity algorithm

When a JT is launched, Controller writes a Unified Job row and the job enters pending. The task manager evaluates dependencies (does the project need a sync? does an inventory source need an update?), moves the job to waiting, then assigns it to an Instance Group with free capacity and dispatches it over the mesh to an execution node — where ansible-runner invokes ansible-playbook inside the EE and streams events back. The state machine:

new → pending → waiting → running → successful | failed | error | canceled

Capacity is not “one job per node.” Each instance advertises a capacity in forks, derived from its CPU and memory; a job consumes roughly min(forks, hosts) + 1 of it. The scheduler packs jobs onto an instance until its capacity is booked. That is precisely why “No execution capacity” appears: the instance group’s summed capacity is fully allocated, not that every node is on fire. You trade throughput for headroom with per-instance capacity_adjustment (slide between the CPU-derived and memory-derived numbers) and by adding execution nodes to the group.

Receptor internals + work signing

The mesh binary is Receptor. Nodes speak a mutually-TLS-authenticated protocol and form a graph from each node’s declared peers. Two properties matter for security and reliability:

• Work signing. A control node cryptographically signs each unit of work with a private key; the execution node verifies the signature with the matching public key before running it. A compromised mesh link therefore cannot inject a rogue job — the execution node runs only work issued by a trusted control node.
• Work units and re-routing. A job is streamed as a Receptor work unit (stdin = the runner payload, stdout = the event stream) that flows back over the same peer path. If a hop goes down, the mesh re-routes through an alternative peer. receptorctl status on any node prints the known mesh graph and the advertised work types — your first stop when a node is “stuck pending.”

EE and DE image resolution

An EE reference is registry/namespace/name:tag plus a pull policy (always / missing / never) and an optional Container Registry credential. Images are built by ansible-builder from an execution-environment.yml (a base image plus dependencies: — galaxy collections, pip packages, system RPMs). Controller pulls the EE onto the execution node before the run. Decision Environments are the same idea for EDA: an ansible-builder-built image carrying ansible-rulebook + ansible.eda + any custom event-source plugins. Two production rules: pin tags (never let a floating latest decide what runs in prod), and mirror the images into Hub so air-gapped execution nodes can pull them without reaching the internet.

EDA activation lifecycle + delivery semantics

A rulebook does not run by itself. You create a Rulebook Activation — a project + a rulebook + a Decision Environment + a restart policy (always / on-failure / never) — which the EDA Server launches as a long-lived ansible-rulebook worker. Because most sources deliver at least once, an activation restart can re-deliver an event, so the golden rule is: make actions idempotent (a run_job_template that is safe to run twice) and lean on throttle + group_by_attributes for de-duplication. Activations expose their running/stopped state and logs under /api/eda/v1/activations/; treat a crash-looping activation exactly like a crash-looping Deployment — read the logs, fix the DE or the condition, restart.

HA control-plane internals

A clustered Controller splits its work into web processes (the API/UI, fronted by the gateway/load balancer) and task processes (the dispatcher + task manager) spread across N control nodes. They coordinate through one external HA Postgres — the single source of truth for RBAC, schedules, and encrypted credentials — and, in 2.5, through Redis for the gateway and websocket layer. There is no leader you elect and babysit: any control node can serve the API and any can dispatch; Postgres is the coordination point. That makes Postgres HA (Patroni, or a managed RDS/Cloud SQL) the real availability boundary of the whole platform — if the database is down, the control plane is down, no matter how many control nodes you have. Execution capacity scales orthogonally: add execution nodes to instance groups and the control plane never grows.

Hands-on lab: AWX on a kind cluster

You can drive a real Controller end-to-end on a laptop with AWX and kind. No Red Hat subscription, no cloud bill.

1. Bring up kind.

brew install kind kubectl   # or use the project's apt/yum repos on Linux
kind create cluster --name awx
kubectl config use-context kind-awx

2. Install the AWX Operator.

kubectl apply -k "https://github.com/ansible/awx-operator/config/default?ref=2.x"
kubectl create namespace awx

3. Deploy AWX itself.

# awx.yaml
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx
  namespace: awx
spec:
  service_type: nodeport
  nodeport_port: 30080

kubectl apply -f awx.yaml
kubectl -n awx get pods --watch     # wait until awx-... is Running (5–10 minutes)

4. Get the admin password.

kubectl -n awx get secret awx-admin-password -o jsonpath="{.data.password}" | base64 -d; echo

5. Open the UI.

kubectl -n awx port-forward svc/awx-service 8080:80
# now visit http://localhost:8080 and log in as admin / <the password>

6. Drive it from the CLI.

pip install awxkit
awx --conf.host http://localhost:8080 --conf.username admin \
    --conf.password "$(kubectl -n awx get secret awx-admin-password -o jsonpath='{.data.password}' | base64 -d)" \
    login
awx projects list
awx organizations list

7. Create a Project + Inventory + JT entirely from CLI.

# A public, well-known role repo to keep the lab free
awx projects create --name "demo" --organization Default --scm_type git \
  --scm_url https://github.com/ansible/ansible-tower-samples.git --scm_branch master \
  --wait

awx inventories create --name "localhost" --organization Default
awx hosts create --name "localhost" --inventory "localhost" \
  --variables '{"ansible_connection": "local"}'

awx job_templates create --name "demo-hello" --project "demo" \
  --inventory "localhost" --playbook hello_world.yml \
  --execution_environment "AWX EE (latest)" \
  --extra_vars '{"who": "Wave-2 student"}'

awx job_templates launch "demo-hello" --monitor

The --monitor flag streams the live job output to your terminal — same logs as the UI’s job pane. You just exercised the Controller object model end-to-end from a Linux/Mac shell.

8. Add a webhook-driven rulebook (EDA upstream, optional).

pip install ansible-rulebook ansible-runner
ansible-galaxy collection install ansible.eda

# rulebook.yml
- name: Trigger AWX JT on local webhook
  hosts: all
  sources:
    - ansible.eda.webhook:
        host: 0.0.0.0
        port: 5050
  rules:
    - name: any POST to /endpoint
      condition: event.endpoint == "endpoint"
      action:
        run_job_template:
          name: "demo-hello"
          organization: "Default"

EDA_CONTROLLER_URL=http://localhost:8080 \
EDA_CONTROLLER_TOKEN=<personal-token-from-AWX-UI> \
ansible-rulebook --rulebook rulebook.yml -i ansible/inventory.yml --verbose

# in another shell:
curl -X POST http://localhost:5050/endpoint -d '{"hello":"world"}' -H content-type:application/json

The rulebook fires, the action runs, AWX picks up the JT launch — the same pattern AAP’s EDA Server runs at scale, just managed manually here.

9. Cleanup.

kind delete cluster --name awx

You now have hands-on familiarity with every Controller object, the awx CLI, and an EDA rulebook firing into Controller — without spending a cent.

Practice challenges

Six graded exercises, beginner → advanced. Try each before opening the solution. They build on the mental models above; the last two are EX374-shaped.

Challenge 1 (Beginner) — Name the layers. In one sentence each, say what ansible-core, AWX, and AAP give you, and name the component AAP ships as a supported product that AWX does not.

<details> <summary>Solution</summary>

• ansible-core = the CLI (ansible-playbook and friends) — no UI, audit, or scheduling.
• AWX = the open-source Controller: web UI + REST API + RBAC + scheduling on Kubernetes, community-supported.
• AAP = the Red Hat-supported product: Controller + private automation hub + Event-Driven Ansible (EDA Server) + automation mesh + certified content + support.
• The clearest “AAP-only products”: Automation Hub and the EDA Server (plus support and certified content).

Why it matters: the boundary tells you what you must self-assemble on AWX — a Galaxy/Pulp of your own instead of Hub, and upstream ansible-rulebook instead of the EDA Server. </details>

Challenge 2 (Beginner) — Which credentials? A Job Template must SSH to RHEL hosts as deploy (become root), read an AWS EC2 dynamic inventory, and decrypt a vaulted secrets.yml. List the Controller credentials you attach.

<details> <summary>Solution</summary>

Attach three credentials to the JT:

1. Machine — SSH user deploy, private key, become method sudo.
2. Amazon Web Services — for the amazon.aws.aws_ec2 inventory source.
3. Vault — the vault password (ideally labelled with a vault_id).

Controller injects all three at runtime.

Why it matters: each credential type maps to a distinct runtime need — connection, cloud API, decryption — and none of them belongs in extra_vars, where it would land in the audit log. </details>

Challenge 3 (Intermediate) — Controller as code (CLI). Using the awx CLI, create a git-backed Project, a localhost Inventory holding one local host, and a Job Template that runs site.yml.

<details> <summary>Solution</summary>

awx projects create --name "Ops Repo" --organization Default --scm_type git \
  --scm_url https://git.example.com/ops/ansible.git --scm_branch main --wait

awx inventories create --name "local" --organization Default
awx hosts create --name "localhost" --inventory "local" \
  --variables '{"ansible_connection": "local"}'

awx job_templates create --name "site" --project "Ops Repo" \
  --inventory "local" --playbook site.yml \
  --execution_environment "AWX EE (latest)"

Why it matters: the UI is a courtesy view; the API is the contract. Anything you can click you can script, so the whole Controller state can live in version control instead of drifting in someone’s browser. </details>

Challenge 4 (Intermediate) — Controller as code (playbook). Rewrite Challenge 3’s Job Template as an idempotent task using the AAP-supported collection, with credentials attached.

<details> <summary>Solution</summary>

- name: Reconcile the Deploy Web job template
  hosts: localhost
  connection: local
  gather_facts: false
  tasks:
    - name: Ensure the job template exists (idempotent)
      ansible.controller.job_template:
        name: "Deploy Web"
        project: "Ops Repo"
        inventory: "Prod AWS"
        playbook: site.yml
        execution_environment: "ee-supported-rhel9-aap25"
        credentials:
          - "Prod SSH"
          - "AWS Inventory"
        ask_variables_on_launch: false
        state: present
        controller_host: "{{ controller_host }}"
        controller_username: "{{ controller_user }}"
        controller_password: "{{ controller_password }}"
        validate_certs: true

Why it matters: state: present makes the task converge — re-running reconciles the JT to the declared shape instead of creating a duplicate, which is exactly what a CI reconcile job needs. On AWX, swap ansible.controller.job_template for awx.awx.job_template. </details>

Challenge 5 (Advanced) — An EDA rulebook. Write a rulebook: an Alertmanager source; fire only when severity == "critical" and group == "prod_db"; launch the restart-postgres-replica Job Template; throttle to once per 10 minutes per alertname.

<details> <summary>Solution</summary>

- name: Restart the prod DB replica on a critical Alertmanager alert
  hosts: all
  sources:
    - ansible.eda.alertmanager:
        host: 0.0.0.0
        port: 5001
  rules:
    - name: Critical alert on a prod_db host
      condition: >-
        event.alert.labels.severity == "critical" and
        event.alert.labels.group == "prod_db"
      throttle:
        once_within: 10 minutes
        group_by_attributes:
          - event.alert.labels.alertname
      action:
        run_job_template:
          name: restart-postgres-replica
          organization: Default
          job_args:
            extra_vars:
              alertname: "{{ event.alert.labels.alertname }}"

Why it matters: the ansible.eda.alertmanager source emits one event per alert (fields under event.alert); the throttle + group_by_attributes de-dupes a flapping alert down to one action per alertname per window; and run_job_template keeps the actual restart audited on the Controller mesh rather than running loose inside the EDA worker. Output shown is representative — no control node ran here. </details>

Challenge 6 (Advanced) — Design the mesh. You run workloads in AWS, Azure, on-prem VMware, and one air-gapped lab. Sketch the node roles and instance groups so each job runs near its targets and the air-gapped lab is reachable.

<details> <summary>Solution</summary>

• Control plane: an HA cluster of control nodes (or one hybrid in a small install) in the management DC, behind the AAP 2.5 gateway + load balancer, with external HA Postgres.
• Execution nodes tagged into Instance Groups: aws (execution nodes inside the AWS VPC), azure (inside the Azure VNet), vmware (on-prem, near vCenter). JTs target the group nearest their inventory, so jobs run local — lower latency, less egress.
• Air-gapped lab: place an execution node inside the lab and reach it through a hop node at the boundary (the only thing that crosses the segmentation). The control plane dispatches over the hop; nothing else routes in.
• Supply chain: a staging Hub in the connected DC mirrors console.redhat.com, one-way-synced to an air-gapped Hub in the lab. EEs/DEs are built and signed on the staging side, then mirrored.

Why it matters: Instance Groups pin work to reachable, cost-appropriate nodes; hop nodes solve network segmentation without opening the control plane to every network; and the control plane stays a small HA cluster while execution scales out per environment. </details>

Common beginner mistakes

These are misconceptions, not symptoms — the wrong mental model that trips people new to the platform. (For symptom → cause → fix, see the troubleshooting table below.)

• “AAP is a new Ansible — I’ll have to rewrite my playbooks.” Your roles, collections, and playbooks run unchanged; AAP executes them inside an Execution Environment. Right model: AAP wraps and manages ansible-core; it does not replace the language.
• “I’ll just set everything up by clicking around the UI.” Click-ops drifts, can’t be code-reviewed, and can’t be rebuilt after a disaster. Right model: the UI is a view, the API is the contract — define Projects/JTs/WTs in git and reconcile them with ansible.controller (AAP) or awx.awx (AWX).
• “AWX and AAP are different tools I’d have to relearn.” Same code, same object model. AAP is supported AWX plus Hub, EDA, Mesh, and certified content. Learn one and you can drive the other.
• “EDA runs my automation.” EDA only decides and triggers; run_job_template hands the real run to Controller on the mesh, audited. Right model: EDA is the doorbell, Controller is the house.
• “A rulebook is fine without a throttle.” A flapping source fires the same action hundreds of times and can DoS Controller. Right model: every production rulebook has throttle + group_by_attributes scoped to the real identity (incident, host, alertname).
• “I’ll just pass the secret in extra_vars.” extra_vars is the highest-precedence var and it lands in the job’s audit record in plain sight. Right model: attach a Credential and expose it through a Credential Type’s injectors (env var or file) — never as an extra-var.
• “Control nodes and execution nodes are the same; I’ll just add servers.” Control hosts the API/scheduler/DB and does not run jobs; execution nodes run EEs; hop nodes only relay. Right model: scale execution by adding execution nodes to instance groups, and keep the control plane a small HA cluster.
• “Hub is optional — production can pull from galaxy.ansible.com.” That makes an internet registry your supply chain, unsigned and unpinned. Right model: Hub is the trust boundary — signed, versioned, mirrored. If it isn’t in Hub, it doesn’t run in prod.

Common mistakes & troubleshooting

JT runs against the wrong inventory — you set ask_inventory_on_launch: true but the launcher accepted the default. Decide whether the JT’s inventory is fixed (most cases) or selectable (rare); fixed JTs are safer.

Project sync hangs — wrong scm_credential (no read access to the repo) or scm_branch doesn’t exist. Run awx projects update <name> --monitor and watch the trace.

Survey answers don’t reach the play — the survey variable name doesn’t match the play’s extra_vars reference (they must be exact). Surveys produce extra-vars, which are highest-precedence Ansible vars; review the variables-precedence lesson.

Mesh node “stuck pending” — the Receptor handshake failed (TLS cert mismatch, peer not in the allow-list). awx-manage list_instances and the Receptor logs on the node tell you why.

Hub install signature failure — your Controller’s signature_validation_credential references a GPG key that doesn’t match Hub’s signing key. Either re-publish the collection, or update the credential. Never disable signature validation in production.

EDA rulebook fires repeatedly — your condition: matches every event, not just the relevant one, and there is no throttle:. Tighten the condition; add a throttle: { once_within: <period>, group_by_attributes: [...] }.

“No execution capacity” — the JT’s Instance Group has no idle execution nodes. Either add nodes, or assign the JT to a different group with capacity. The awx instance_groups list command shows utilisation.

Click-ops drift — someone edited a JT in the UI; CI’s IaC reconciliation now overwrites it. Either lock the UI editing rights to platform admins only, or make the IaC reconciliation idempotent and accept the drift as a routine “platform owns this object” check.

Best practices

• Treat the Controller as code. Every Project/Inventory/JT/WT/Survey/Schedule/Notification lives in version-controlled YAML reconciled by the awx.awx collection or Terraform’s awx provider. The UI is a courtesy.
• Layer credentials: Machine credential for SSH; Cloud credential for the inventory plugin; Vault credential for any vaulted vars in the play. Combine on a JT, do not stuff secrets into extra_vars.
• Tag JTs by intent: a purpose: provision / purpose: rolling-deploy / purpose: rotate-secrets taxonomy in JT names makes the UI navigable at scale.
• Survey for safety: every operator-facing JT has a survey constraining choices to the safe ones (drop-down environments, integer with min/max, regex-validated app names).
• Use Workflow Templates with approval nodes for any JT that touches prod — humans gate the deploy, not the JT.
• Run Hub even at small scale. The published repo gives your team a single source of internal collections, signed, version-pinned.
• Sign your collections and require signature validation in Controller. Make supply-chain integrity a default, not an option.
• Use Instance Groups to keep cloud jobs on cloud-side execution nodes — saves egress, cuts latency.
• EDA throttling is mandatory. A noisy source without throttle: will DoS Controller.
• Air-gap with a staging Hub. Never let production-tier hosts reach console.redhat.com or galaxy.ansible.com directly.
• Schedule project syncs explicitly rather than update_on_launch: true for hot-path JTs — update_on_launch adds wall-clock delay to every launch.

Security notes

AAP is the automation control plane. It holds privileged credentials for every system it touches; protect it like a privileged-access workstation.

• TLS everywhere: API, UI, mesh, Hub, EDA. Use cert-manager (or your enterprise PKI). No HTTP outside localhost.
• SSO + MFA: integrate with SAML/OIDC; require MFA for any human user.
• RBAC scoped tightly: developers get execute on the JTs they own, use on the credentials those JTs need. Nobody gets admin on the org by default.
• Credential injection only: never echo a credential into extra_vars. Use a Custom Credential Type with injectors: so secrets reach the EE as env vars or files, not as extra-vars in the audit log.
• Audit log retention: Controller writes a row per job + per task to its database; archive these per your compliance regime.
• Signed collections, signed playbooks: enable Hub content signing and Controller’s signature_validation_credential on every Project. A tamper of main does not run in production.
• EDA token rotation: rulebook activations carry a Controller token. Use short-lived personal tokens, never service-account static tokens in YAML.
• Mesh peer ACLs: only intended nodes peer; a rogue Receptor cannot dial in.
• Backups: the Controller database holds RBAC, schedules, and encrypted credentials. Back it up encrypted; rehearse restore quarterly. Hub’s Pulp content is bulky but reproducible from upstream syncs; back up the metadata, not necessarily the content.

Interview & exam questions

1. What’s the difference between AAP, AWX, and Controller? AAP = the supported product; AWX = the upstream open-source project that becomes Controller; Controller = the AAP component that orchestrates jobs (formerly called Tower).
2. What does an Execution Environment do? It’s the OCI image containing ansible-core + collections + Python deps that Controller runs every job inside; the contract between Controller and Ansible.
3. Control vs hop vs execution nodes? Control = API + scheduler + DB; execution = runs jobs in EEs; hop = relays mesh traffic across network boundaries with no execution capacity.
4. What is an Instance Group? A logical grouping of execution nodes; JTs target an Instance Group and the scheduler picks an idle node in it.
5. What’s a Project, in Controller terms? A tracked SCM repo (usually git) from which JTs draw playbooks and configs.
6. What does a Survey do? Prompts the JT launcher for parameters via a JSON-schema-shaped form; answers become extra-vars at runtime.
7. What’s an Automation Hub Repository? A logical collection of collections — published (yours), rh-certified, community, validated. Each can be served, signed, and synced separately.
8. How does content signing work? Hub signs every published collection with a sigstore-compatible key; clients verify against a public key (Controller’s GPG Public Key credential or ~/.ansible/galaxy_keys); unsigned content is rejected.
9. What’s a rulebook, in EDA? A YAML file with sources (event inputs), conditions (predicates), and actions (run JTs/playbooks/modules); EDA Server runs it as a long-running process.
10. Difference between run_playbook and run_job_template? run_playbook runs the playbook inside EDA Server’s worker. run_job_template calls Controller’s API to launch a JT — the production-grade path because the JT runs in the mesh, audited, with all the RBAC and credential handling.
11. Why is throttling necessary in EDA? Without it, a noisy source (a flapping monitor) fires the same action repeatedly and overwhelms Controller. throttle: { once_within: <period>, group_by_attributes: [...] } scopes the dedup to the meaningful identity.
12. What’s the awx CLI good for? Treating the Controller as code: every object can be CRUDed from a shell script or CI job; the UI is convenience, the API is the contract.
13. How do you scale Controller for HA? Three control nodes behind a load balancer, an external HA Postgres, multiple execution nodes per region/cloud, hop nodes at every network boundary.
14. What’s a Decision Environment? The EDA equivalent of an Execution Environment — the container image carrying ansible-rulebook + ansible.eda + your event sources.
15. What did the AAP 2.5 platform gateway change? It unified authentication, RBAC, and navigation for Controller, Hub, and EDA behind a single front door (/api/gateway/v1/), so you log in once and manage users/roles in one place instead of three. AWX has no gateway.
16. On AAP, which collection manages Controller objects? ansible.controller (Red Hat-certified); awx.awx is the AWX equivalent, and the tower_* module names are deprecated aliases.

Quick check

• AAP = Controller + Automation Hub + EDA + Mesh (+ Lightspeed) behind the 2.5 platform gateway. AWX = upstream Controller, no gateway, no Hub product, no EDA Server.
• Controller objects: Org → Teams/Users → Projects/Inventories/Credentials/EEs → Job Templates / Workflow Templates → Schedules / Notifications / Surveys.
• Mesh nodes: control / hybrid / execution / hop. Instance Groups select where a JT runs; capacity is measured in forks.
• Hub repositories: published (yours), rh-certified, community, validated. Always sign; always verify.
• awx CLI = the API made shell-friendly. Treat Controller objects as IaC — ansible.controller on AAP, awx.awx on AWX.
• EDA = sources → conditions → actions, with throttling. run_job_template is the production action; activations are at-least-once, so keep actions idempotent.
• Topology scales by adding execution + hop nodes; control plane stays small (HA pair/triple) and Postgres HA is the real availability boundary.

Exercise

Stand up the AWX-on-kind lab and:

1. Define one Project, one Inventory, one Job Template entirely from the awx CLI (no UI clicks).
2. Add a Survey to the JT prompting for target_env (drop-down: dev/staging) and app_version (regex ^\d+\.\d+\.\d+$).
3. Build a Workflow Template that runs a preflight JT, an approval node, then the deploy JT, with a Slack notification on success/failure.
4. Mirror a community collection (e.g. community.general) into your local Hub if you’ve stood one up; otherwise document the steps.
5. Write a rulebook that listens on a webhook and fires run_job_template against your demo JT; throttle to once per minute per source IP.
6. Verify that an unsigned collection cannot install when signature_required is set; sign it; verify it does install.
7. Document in your repo’s README.md the exact commands a new engineer types to recreate the AWX state from scratch (the IaC story).

Certification mapping

• EX374 (Developing Automation with Ansible Automation Platform) — the AAP exam. Every section of this lesson maps directly: Controller objects, Mesh topology, Automation Hub, EDA rulebooks. Expect to write a rulebook from scratch, configure a JT with a Survey, and use Hub-served signed collections.
• EX467 (Managing AAP) — the operations-focused exam. Heavier on Mesh, HA, backup/restore, Hub remote sync, EDA Server install. This lesson gives you the model; pair with hands-on AAP installer experience for EX467.
• EX358 (Cloud Automation with Ansible) — references AAP as the orchestrator for cloud roles; the Controller half of this lesson is its prerequisite.
• Beyond exams, every senior Ansible job description in 2026 lists “experience with AAP / AWX, including EDA” as a discriminator. This is the lesson that lets you say yes.

Glossary

• AAP — Ansible Automation Platform; Red Hat’s supported product.
• AWX — the open-source upstream of Controller.
• Controller — the orchestration component (API, UI, scheduler); formerly Tower.
• Automation Hub — the private collection registry.
• EDA / Event-Driven Ansible — the eventing tier; runs rulebooks in the EDA Server.
• Automation Mesh — the Receptor-based peer-to-peer execution network.
• Receptor — the mesh transport protocol/binary.
• Control node — hosts API/scheduler/DB; no execution.
• Execution node — runs jobs inside EEs; no API.
• Hop node — relays mesh traffic; no execution.
• Hybrid node — control + execution on one machine.
• Instance Group — logical grouping of execution nodes; JTs target a group.
• Execution Environment (EE) — OCI image with ansible-core + collections.
• Decision Environment (DE) — OCI image with ansible-rulebook + ansible.eda for EDA.
• Project — Controller object pointing at an SCM repo.
• Inventory — Controller object holding hosts/groups; can be regular/smart/constructed; sources can be plugin-driven.
• Credential — encrypted Controller secret; types include Machine/Source-Control/Vault/Cloud/Network/Custom.
• Credential Type — the schema for a Credential; user-defined types let you wire arbitrary secret stores.
• Job Template (JT) — combines Project + Inventory + Credentials + EE + Survey + Schedule into a runnable thing.
• Workflow Template (WT) — DAG of JTs/syncs/approvals/sub-workflows with success/failure edges.
• Survey — JSON-schema-like prompt attached to a JT/WT, surfaced as extra-vars at launch.
• Schedule — cron/rrule recurrence on a JT/WT.
• Notification — Slack/Teams/email/webhook/PagerDuty fired on job state changes.
• Hub Repository — published/rh-certified/community/validated.
• Hub Namespace — the publisher container (namespace.collection).
• Content Signing — sigstore-compatible signature attached to every Hub collection.
• Remote Sync — Hub mirror of an upstream Galaxy/Hub.
• Rulebook — YAML document with sources, conditions, actions; the unit of EDA.
• Source / Condition / Action — the rulebook’s three sections.
• Throttle — EDA dedup rule that prevents an action from firing too often.
• awx CLI — awxkit-shipped client that CRUDs every Controller object via the API.
• awx.awx collection — Ansible modules for managing Controller objects from a play (AWX).
• ansible.controller collection — the Red Hat-certified modules for managing Controller objects on AAP (ansible.controller.job_template, …); supported successor to awx.awx and the deprecated tower_* modules.
• Platform gateway — AAP 2.5’s single front door: one URL/login, unified RBAC and navigation across Controller, Hub, and EDA; adds /api/gateway/v1/. AWX has none.
• ansible-runner — the library control/execution nodes use to invoke ansible-playbook inside an EE and stream events back.
• ansible-builder — the tool that builds Execution/Decision Environment images from an execution-environment.yml.
• Rulebook Activation — a running instance of a rulebook (project + rulebook + Decision Environment + restart policy) managed by the EDA Server; sources are at-least-once, so keep actions idempotent.
• Capacity / forks — Controller’s per-instance execution headroom; a job consumes ~min(forks, hosts)+1; “No execution capacity” means the instance group is fully booked.
• Work signing — Receptor signs each work unit so an execution node runs only jobs cryptographically issued by a trusted control node.
• Pull policy — how an execution node fetches an EE image (always/missing/never).
• Unified Job — the Controller database row for any run (job, project update, inventory update, workflow) with its new→pending→waiting→running→… state machine.
• Pulp — the content-management engine under Automation Hub (repositories, remotes, sync, signing).
• sigstore — the signing ecosystem Hub’s content signing is compatible with.

Next steps

You now have the architecture model of AAP: how Controller, Hub, EDA, and the Mesh fit together; what every Controller object is for; how rulebooks turn events into runs; and how the platform’s supply-chain story (signed collections in Hub, Decision Environments for EDA, Execution Environments for Controller) anchors trust in production. The platform is now a substrate you can reason about — but the substrate only matters when fed by quality content. The natural sequels close that loop: ship tested, packaged content to Hub by combining the Molecule lesson (proof of correctness) with the collections & EE lesson (the build-and-publish path); feed the platform with live fleets by using the patterns from Dynamic Inventory & Secure Secrets inside Controller’s Inventory Sources; and operate at scale with Tuning Ansible for Speed & Scale and Delegation, Strategies & Rolling Updates — the levers that compose into Workflow Templates that complete in minutes rather than hours. AAP is where every other Advanced lesson cashes in.