Ansible for AWS, In Depth: amazon.aws & community.aws, IAM Auth, Multi-Account & Tagging at Scale

In a nutshell

If you have already met Terraform, hold two pictures in your head. Terraform is the construction company that erects the building and keeps the master blueprint — its state file — of every wall and pipe. Ansible is the property manager who runs the building after it is built: unlocking the doors each morning, changing the filters, rotating the keys, and reacting the moment a smoke alarm goes off. Both are essential, and they work at different altitudes. This lesson is about the property-manager job — operating a live AWS estate from Ansible — not about being a second construction company.

The amazon.aws collection is the property manager’s set of proper, labelled tools. The tempting alternative — shelling out to the AWS CLI inside shell: tasks — is like leaving Post-it notes for a human to go and click buttons: nothing remembers what was already done, so the same instruction run twice does the work twice. A native module such as amazon.aws.s3_bucket is idempotent: ask for the bucket, and if it already exists exactly as described, nothing happens (changed=0). Declare the end state, run it as often as you like, converge safely — that single property is the whole reason to reach for modules over CLI glue.

Everything a module does eventually becomes a call through boto3, the AWS Python SDK. That means Ansible inherits boto3’s credential rules for free: it looks for keys in environment variables, then a profile, then — best of all — an identity attached to the machine itself (an EC2 instance role, or IRSA inside EKS) so there is no secret on disk to leak. Get that mental model — modules over CLI, declare-then-converge, and let the machine’s own identity do the auth — and the rest of this lesson is detail.

Level: Expert · Time: ~40 min · You need: comfort with playbooks, variables, roles, and dynamic inventory. A free LocalStack lab at the end lets you drive every module with no AWS bill.

Ansible-on-AWS confuses people who have already met Terraform. The two tools overlap, but they answer different questions. Terraform asks “what does my AWS estate look like, as code?” — its job is to converge a desired state across hundreds of resources, with a state file that tracks every dependency. Ansible asks “what should this thing do, right now?” — its job is to make targeted, often imperative changes to live infrastructure: drain an instance, snapshot an RDS database, rotate a security group rule because an alert just fired, run a one-off migration, deploy an application into boxes that already exist. Both are correct; they live at different altitudes. The job of this lesson is to teach you the AWS modules deeply enough that you stop reaching for the AWS CLI in shell: tasks (the most common anti-pattern in Ansible-on-AWS) and start using the native modules that are idempotent, check-mode-aware, and --diff-friendly.

We start by drawing the Ansible vs Terraform line cleanly so you know when to reach for which tool, then walk the amazon.aws and community.aws collections module-by-module: the EC2 family (ec2_instance, ec2_vpc_net, ec2_vpc_subnet, ec2_vpc_route_table, ec2_security_group, elb_application_lb, autoscaling_group), the data-plane modules (s3_bucket, s3_object, rds_instance, rds_cluster), and the IAM/identity modules (iam_role, iam_user, iam_policy). We cover the AWS auth chain end to end — environment variables (AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY/AWS_SESSION_TOKEN), ~/.aws/credentials profiles, IAM Roles for EC2 when the control node is an EC2 instance, IRSA when it is a pod inside EKS, AWS SSO, and STS assume-role chains for multi-account access — and the cookbook patterns each one needs. We re-meet the amazon.aws.aws_ec2 dynamic inventory plugin from a deeper angle than the dynamic inventory lesson, focusing on the AWS-specific knobs (include_filters, iam_role_arn, regions: [aws-global], hostnames with tag:Name/private-ip-address/public-dns-name, keyed_groups per tags.Environment/placement.availability_zone/instance_type). We finish on multi-account patterns with assume_role_arn, tagging strategy that turns a 2,000-instance fleet into manageable groups, idempotency and check-mode behaviour for the awkward AWS modules (ec2_instance in particular), and packaging an AWS-aware Execution Environment for AAP. Everything targets current Ansible (ansible-core 2.17+, the amazon.aws 8+ and community.aws 8+ collections, 2026), uses FQCN throughout, and ends with a free hands-on lab that uses LocalStack so you can drive real amazon.aws.* modules without a real AWS bill.

Learning objectives

After this lesson you can:

• Articulate exactly when Ansible-on-AWS beats Terraform and vice versa, and stop the “should I use Ansible or Terraform here?” debate at the door.
• Drive the EC2/VPC/IAM/RDS/S3 module families with full option matrices and idempotent behaviour.
• Plumb AWS credentials safely with env vars, profiles, IAM Roles for EC2, IRSA, AWS SSO, and STS assume-role.
• Configure the aws_ec2 dynamic inventory plugin with the AWS-specific knobs, and explain when it caches and when it doesn’t.
• Operate a multi-account AWS estate with one playbook by passing assume_role_arn (or per-task aws_profile).
• Use a tagging strategy so your inventory fans out into clean cross-cutting groups (tag_Environment_prod, tag_Role_web, az_eu_west_1a).
• Ship an AWS-aware Execution Environment that includes boto3, botocore, the two collections, and the AWS CLI for the few command: cases that need it.

Prerequisites & where this fits

You should already be comfortable with playbooks and tasks, variables and the precedence rules, Jinja templating, roles and collections, and the dynamic inventory lesson (because every real AWS run uses dynamic inventory). The companion expert lessons that compound here are Ansible for Kubernetes — many AWS plays target EKS — Ansible for Containers — for ECR/ECS/Fargate adjacency — and Hybrid Orchestration — when AWS is one of three or four targets in a single workflow. In the Ansible Zero-to-Hero programme this is the Cloud expert (AWS) lesson and a textbook EX374-grade topic.

Core concepts

Five mental models carry the whole lesson.

1. Ansible-on-AWS is operations, not provisioning. Terraform builds the estate and tracks state; Ansible operates it. The ideal split is “Terraform builds VPC + subnets + EKS + RDS skeleton” then “Ansible runs every day to deploy apps, rotate keys, snapshot databases, scale ASGs, drain instances, react to events.” If you find yourself building a 200-resource VPC in pure Ansible, stop — you’ve crossed into Terraform’s lane and you’re losing state, drift detection, and dependency-graphing for nothing.

2. The AWS auth chain is shared with boto3. Every amazon.aws.* module ultimately constructs a boto3 client, so the credential resolution order is the standard AWS SDK chain: explicit module params → env vars → ~/.aws/credentials profile → instance metadata (IMDSv2) for EC2 → IRSA for EKS pods → SSO/SSO-OIDC. You almost never put credentials in module args. You configure the environment and let the chain resolve.

3. STS assume-role is the multi-account primitive. A single playbook running with one set of base credentials can hop into 20 AWS accounts by setting assume_role_arn per task or per host. The pattern is “one centralised automation account, with an IAM role in every spoke account that trusts it.” Ansible’s job is to call sts:AssumeRole per target and use the temporary credentials.

4. Tags are the inventory. aws_ec2’s keyed_groups turns every tag and every cloud field into an Ansible group. A consistent tag schema (Environment, Role, Owner, CostCenter) is what turns “all hosts” into clean cross-cutting groups (tag_Environment_prod, tag_Role_web). The minute your tag schema is inconsistent, your dynamic inventory becomes useless.

5. EC2 is not a regular Ansible target — it is both a target and a thing-you-create. Most of this lesson lives in the second world: ec2_instance creates an instance. Once it exists, Ansible’s normal SSH model targets it (via aws_ec2 inventory). Don’t confuse the two phases — ec2_instance runs from localhost (with connection: local) against the AWS API; later plays run with connection: ssh against the running instance. The transition between the two is what wait_for_connection handles.

Keep these terms straight: amazon.aws (Red-Hat-supported AWS collection — the one you should default to), community.aws (community-maintained extras — aws_eks_cluster_info, etc.), boto3/botocore (the Python SDK every module uses — must be installed in the EE), the SDK auth chain (env → profile → instance role → IRSA → SSO), assume_role_arn (per-task multi-account hop), aws_ec2 plugin (dynamic inventory; lives in amazon.aws), IAM Role for EC2 (control-node identity for self-hosted), IRSA (control-node identity for EKS-hosted), connection: local (used for every AWS API task — the work runs on the control node, not on a target).

Ansible vs Terraform vs CloudFormation

Beginners almost always arrive with the same question — “we already have Terraform (or CloudFormation); why would I point Ansible at AWS at all?” Here is the whole answer in one table. The three tools are not rivals fighting for the same job; they sit at different altitudes and compose.

The line to internalise: Terraform or CloudFormation builds the estate and remembers it; Ansible runs the estate day after day. The healthiest large shops use both — terraform apply (or a CloudFormation stack) stands up the VPC, subnets, EKS control plane, and RDS skeleton once; then an Ansible job runs every hour to deploy application versions, take RDS snapshots before a migration, rotate a security-group rule an alert just demanded, scale an Auto Scaling Group for a sale, drain an instance for patching, and bake a fresh AMI. Notice that none of those day-2 verbs is “converge 200 resources to a desired state” — that is the giveaway that you are in Ansible’s lane, not Terraform’s.

Ansible can provision (this lesson is full of state: present), and for a handful of resources reacting to a live event that is exactly right. The anti-pattern is rebuilding a 200-resource VPC as a pile of Ansible tasks — you throw away the dependency graph, the state file, and the plan/apply safety that Terraform or CloudFormation give you, and get nothing back in return.

amazon.aws vs community.aws

Two collections cover ~all of AWS:

Both come from the same upstream organisation; the split exists so the supported core remains stable while community modules iterate fast. Install both:

# requirements.yml
collections:
  - name: amazon.aws
    version: ">=8.0.0"
  - name: community.aws
    version: ">=8.0.0"

ansible-galaxy collection install -r requirements.yml
pip install boto3 botocore

The AWS auth chain

This is the single most important table in the lesson. The order Ansible (via boto3) resolves credentials:

The rule of thumb in production: use the lowest-numbered source that doesn’t require a static secret. Order 4 (instance role) and order 5 (IRSA) are the gold standards because there are no keys on disk.

Pattern A — control node has an instance role

Run AAP on EC2 with an instance role. Every amazon.aws.* task uses IMDSv2 transparently. Zero credentials in inventory or vault.

- name: Provision a security group (uses instance-role creds)
  amazon.aws.ec2_security_group:
    name: web-sg-prod
    description: "Web tier"
    vpc_id: vpc-0abc123
    region: eu-west-1
    rules:
      - proto: tcp
        ports: [443]
        cidr_ip: 0.0.0.0/0
    state: present
  delegate_to: localhost
  connection: local

Pattern B — IRSA for EKS-hosted Container Groups

AAP Container Groups can be configured so the EE pod’s ServiceAccount is annotated with an IAM Role ARN. The pod’s pod-identity webhook injects AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE, and boto3 picks them up. Same playbook, no code change.

Pattern C — multi-account assume-role

The control node holds base credentials (instance role or IRSA). Per task, hop into the account you want:

- name: Snapshot RDS in dev account
  amazon.aws.rds_cluster_snapshot:
    db_cluster_identifier: dev-app
    db_cluster_snapshot_identifier: dev-app-{{ ansible_date_time.iso8601_basic_short }}
    region: eu-west-1
    assume_role:
      role_arn: arn:aws:iam::111122223333:role/AnsibleAutomation
      role_session_name: ansible-snap-dev
  delegate_to: localhost
  connection: local

For long plays in one account, set the role at the play level via env vars + community.aws.sts_assume_role:

- name: Hop into account 111122223333 once
  community.aws.sts_assume_role:
    role_arn: arn:aws:iam::111122223333:role/AnsibleAutomation
    role_session_name: ansible-{{ ansible_date_time.iso8601_basic }}
  register: assumed

- name: Run all subsequent tasks in that account
  amazon.aws.ec2_instance_info:
    region: eu-west-1
  environment:
    AWS_ACCESS_KEY_ID: "{{ assumed.sts_creds.access_key }}"
    AWS_SECRET_ACCESS_KEY: "{{ assumed.sts_creds.secret_key }}"
    AWS_SESSION_TOKEN: "{{ assumed.sts_creds.session_token }}"

EC2 family — the headline modules

A canonical instance-creation task with wait_for_connection:

- name: Launch a web instance
  amazon.aws.ec2_instance:
    name: web-eu-1a-{{ deploy_id }}
    region: eu-west-1
    image_id: "{{ web_ami_id }}"
    instance_type: t3.medium
    vpc_subnet_id: "{{ subnet_eu_1a }}"
    security_groups: [web-sg-prod]
    key_name: ops
    tags:
      Environment: prod
      Role: web
      Owner: platform
      Deploy: "{{ deploy_id }}"
    state: running
    wait: true
    wait_timeout: 300
  delegate_to: localhost
  connection: local
  register: launched

- name: Add to the in-memory inventory and wait for SSH
  ansible.builtin.add_host:
    name: "{{ item.public_ip_address | default(item.private_ip_address) }}"
    groups: just_launched
    ansible_user: ec2-user
  loop: "{{ launched.instances }}"
  delegate_to: localhost

- name: Wait for SSH on each new box
  ansible.builtin.wait_for_connection:
    timeout: 300
  delegate_to: "{{ item }}"
  loop: "{{ groups['just_launched'] }}"

Notice the pattern: API tasks run delegate_to: localhost + connection: local; once a box exists, Ansible switches to its real connection.

IAM, S3, RDS

- name: IAM role for the web tier
  amazon.aws.iam_role:
    name: web-instance-role
    assume_role_policy_document: "{{ lookup('file', 'trust/web.json') }}"
    managed_policies:
      - AmazonSSMManagedInstanceCore
      - CloudWatchAgentServerPolicy
    state: present
  delegate_to: localhost

- name: S3 bucket for app logs (with lifecycle)
  amazon.aws.s3_bucket:
    name: prod-app-logs
    region: eu-west-1
    versioning: true
    public_access:
      block_public_acls: true
      block_public_policy: true
      ignore_public_acls: true
      restrict_public_buckets: true
    encryption: AES256
    tags:
      DataClass: logs
    state: present
  delegate_to: localhost

- name: Postgres RDS for staging
  amazon.aws.rds_instance:
    id: stg-app-db
    engine: postgres
    engine_version: "16.3"
    db_instance_class: db.t4g.medium
    allocated_storage: 50
    storage_type: gp3
    master_username: app
    master_user_password: "{{ vault_rds_pw }}"
    db_subnet_group_name: stg-db
    vpc_security_group_ids: ["{{ db_sg }}"]
    backup_retention_period: 7
    deletion_protection: true
    state: present
    region: eu-west-1
  delegate_to: localhost

aws_ec2 dynamic inventory — AWS-specific knobs

The cross-cutting plugin schema (hostnames, compose, keyed_groups, groups, strict) is covered in the dynamic inventory lesson. The AWS-specific knobs:

# inventory/prod.aws_ec2.yml
plugin: amazon.aws.aws_ec2

# Multi-region in one inventory file
regions:
  - eu-west-1
  - eu-west-2
  - us-east-1

# Multi-account via assume-role (per-source!)
iam_role_arn: arn:aws:iam::111122223333:role/AnsibleInventory

# Pre-filter on the AWS side — saves API calls
include_filters:
  - tag:Environment: ["prod"]
  - instance-state-name: ["running"]

exclude_filters:
  - tag:Decommission: ["true"]

# Hostnames priority (first that exists wins)
hostnames:
  - tag:Name
  - private-dns-name

# Useful per-host variables
compose:
  ansible_host: private_ip_address
  env: tags.Environment | default('unknown')
  role: tags.Role | default('unknown')
  cost_center: tags.CostCenter | default('unknown')

# Cross-cutting groups
keyed_groups:
  - key: tags.Role
    prefix: role
  - key: tags.Environment
    prefix: env
  - key: placement.availability_zone
    prefix: az
  - key: instance_type
    prefix: type
  - key: vpc_id
    prefix: vpc

# Named groups via expressions
groups:
  prod_eu: tags.Environment == 'prod' and placement.region.startswith('eu-')
  needs_patch: tags.Patched is not defined or tags.Patched != 'true'

# Caching — critical for big fleets
cache: true
cache_plugin: jsonfile
cache_connection: /var/cache/ansible_inventory
cache_timeout: 600

include_filters is the most performance-impactful knob: it’s a server-side filter, so a 10,000-instance account becomes a 200-result query. Without it, every play hits the AWS API for every instance.

Multi-account inventory — one file per account

# inventory/account-prod.aws_ec2.yml
plugin: amazon.aws.aws_ec2
regions: [eu-west-1, us-east-1]
iam_role_arn: arn:aws:iam::111111111111:role/AnsibleInventory
hostnames: [tag:Name]
keyed_groups:
  - { key: "'prod'", prefix: account }
  - { key: tags.Role, prefix: role }

# inventory/account-stg.aws_ec2.yml
plugin: amazon.aws.aws_ec2
regions: [eu-west-1]
iam_role_arn: arn:aws:iam::222222222222:role/AnsibleInventory
hostnames: [tag:Name]
keyed_groups:
  - { key: "'stg'", prefix: account }
  - { key: tags.Role, prefix: role }

Then point ansible.cfg at the directory; both files are merged automatically:

[defaults]
inventory = ./inventory/

[inventory]
enable_plugins = amazon.aws.aws_ec2, amazon.aws.aws_rds, constructed

Tagging strategy that scales

A consistent tag schema is the single highest-leverage thing you can do for Ansible-on-AWS:

Enforce the schema with AWS Config rules (built-in required-tags) so an instance without Environment is non-compliant within minutes of launch. Ansible relies on the tags being present and correct; AWS Config makes that contract enforceable.

Idempotency & check-mode for the awkward modules

The two known sharp edges:

• ec2_instance without name: is not idempotent — every run launches a new instance. Always set a unique name: (typically <role>-<az>-<deploy_id>).
• autoscaling_group with replace_all_instances: true is a real action; check-mode cannot fully simulate it. Always test in stg first.

Hands-on free lab — LocalStack

LocalStack is a fake AWS that runs in Docker. The amazon.aws.* modules treat it as real AWS via an endpoint override.

docker run -d --name localstack -p 4566:4566 -e SERVICES=ec2,s3,iam,rds,sts localstack/localstack

# expose the override to boto3
export AWS_ACCESS_KEY_ID=test
export AWS_SECRET_ACCESS_KEY=test
export AWS_DEFAULT_REGION=eu-west-1
export AWS_ENDPOINT_URL=http://localhost:4566

mkdir aws-lab && cd aws-lab
ansible-galaxy collection install amazon.aws community.aws
pip install boto3 botocore

cat > play.yml <<'EOF'
- hosts: localhost
  gather_facts: false
  tasks:
    - name: VPC
      amazon.aws.ec2_vpc_net:
        name: lab-vpc
        cidr_block: 10.42.0.0/16
        state: present
      register: vpc

    - name: Subnet
      amazon.aws.ec2_vpc_subnet:
        vpc_id: "{{ vpc.vpc.id }}"
        cidr: 10.42.1.0/24
        az: eu-west-1a
        tags: { Name: lab-subnet }
        state: present
      register: subnet

    - name: SG
      amazon.aws.ec2_security_group:
        name: lab-sg
        description: lab
        vpc_id: "{{ vpc.vpc.id }}"
        rules:
          - proto: tcp
            ports: [22, 80]
            cidr_ip: 0.0.0.0/0
        state: present

    - name: S3 bucket
      amazon.aws.s3_bucket:
        name: lab-bucket-{{ 9999 | random }}
        state: present

    - name: Show what we made
      ansible.builtin.debug:
        msg: "VPC {{ vpc.vpc.id }} / subnet {{ subnet.subnet.id }}"
EOF

ansible-playbook play.yml --diff
ansible-playbook play.yml --diff   # second run — changed=0

Tear down:

docker rm -f localstack

Going deeper

Everything above is enough to be productive. This section is for the reader who has to make Ansible-on-AWS fast, safe, and audit-clean at scale — the material that separates “it works on my laptop” from “it runs 4,000 instances across 30 accounts from AAP.”

How a module actually talks to AWS

Every amazon.aws.* and community.aws.* module is built on a shared base class (AnsibleAWSModule) that constructs a boto3 client from the resolved credentials and region, then makes typed API calls. Three consequences follow:

• boto3/botocore must be importable in the same Python that runs the module — i.e. inside the Execution Environment, not just on the control host. ImportError: No module named boto3 always means “wrong interpreter / EE,” never “wrong playbook.”
• Retries are boto3’s, not Ansible’s. The SDK’s standard/adaptive retry mode handles throttling (RequestLimitExceeded, Throttling) with exponential backoff. You tune it with the AWS_RETRY_MODE and AWS_MAX_ATTEMPTS env vars (or a botocore config), not with Ansible’s retries:/until:. Reaching for a task-level until: loop to beat throttling is a smell — raise AWS_MAX_ATTEMPTS instead.
• region: selects the endpoint. Under the hood it becomes https://ec2.eu-west-1.amazonaws.com. For GovCloud/FIPS you override with endpoint_url: or the AWS_ENDPOINT_URL* variables — the same mechanism the LocalStack lab uses to point at http://localhost:4566.

Idempotency internals — how changed is really decided

An idempotent AWS module runs a GET-diff-mutate cycle: it describes the current resource, compares field-by-field against your parameters, and calls the mutating API only for what differs. Two knobs decide how aggressive the diff is:

• purge_tags (default true on most tag-aware modules): with it on, a tag present on the resource but absent from your tags: dict is removed. This surprises people who tag out-of-band — an SSM or cost tool adds LastScanned, your next Ansible run strips it, and the two systems fight forever (changed=1 every run). Set purge_tags: false when another system co-owns tags.
• purge_rules / purge_rules_egress on ec2_security_group (default true): a rule not in your task is deleted. This is what makes the module converge an SG to exactly your list — powerful, and a foot-gun if a human added an emergency rule in the console that your next run silently removes.

The lesson’s rule “ec2_instance without name: is not idempotent” is this same mechanism: with no Name tag and no instance_ids:, the GET step has no key to match on, so the module can never find the previous instance and always creates a new one.

Check-mode, --diff, and the modules that can’t fully simulate

--check asks every module “what would you change?” A well-written AWS module runs the GET + diff and reports changed without calling the mutating API. But some actions are inherently unsimulatable: autoscaling_group with replace_all_instances: true is a runtime rollout, not a config diff, so check-mode can show the launch-template change but not the instance churn. Treat any module whose check-mode the docs mark “partial” as test-in-staging-only for its destructive paths. Pair --check with --diff to see the actual field-level before/after on the resources that support it (security-group rules and tags diff beautifully; opaque blobs like a launch template’s user-data do not).

Long-running AWS calls: async, throttle, forks

Some AWS operations take minutes (RDS creation, AMI copy, ASG rollout). Two scaling levers:

• Fire-and-forget with async: + poll: 0 for genuinely long, independent calls, then collect with ansible.builtin.async_status. Combined with wait: false on the module, you launch twenty RDS restores in parallel instead of serially.
• Cap concurrency against a shared API with throttle: on a task, and tune forks globally. More forks does not mean faster against AWS — past a point you just trigger Throttling and boto3 backs everything off. On big fleets, a throttle: on the API-heavy task plus serial: on the rolling play beats cranking forks to 100.

STS, session duration, and role chaining

Assume-role credentials are temporary. Two caveats bite in production:

• Chaining caps at one hour. If role A assumes role B (a chained assume-role — which any multi-account hop through an intermediary is), the maximum session is 3,600 s regardless of the role’s MaxSessionDuration. A play that runs 90 minutes will start failing with ExpiredToken mid-run. Fix: shorten the play, re-assume, or assume directly rather than through a chain.
• role_session_name is your audit trail. It lands in every CloudTrail AssumeRole event. Set it to something like aap-job-{{ tower_job_id | default('local') }} so “which automation run touched this account at 03:14?” is a one-line CloudTrail query, not a forensic project.

Inventory at scale: caching, --flush-cache, and multiple sources

On a 10,000-instance estate the aws_ec2 plugin is your biggest performance surface. include_filters does a server-side DescribeInstances filter so AWS returns 200 rows, not 10,000. cache: true with a jsonfile plugin then means the next play reuses the result with zero API calls until cache_timeout. When you know the fleet just changed (you launched instances earlier in the same workflow), run with --flush-cache or you will target a stale picture. Multiple *.aws_ec2.yml files under one inventory/ directory are merged, which is how one control node inventories thirty accounts — one file per account, each with its own iam_role_arn.

Version and collection caveats (2026)

• FQCN is mandatory — the short ec2/s3 module names were removed years ago; always write amazon.aws.ec2_instance.
• amazon.aws 8.x/9.x target ansible-core 2.15+ and a current boto3/botocore floor; pin both the collection and the SDK in your EE so a Galaxy refresh can’t move under you.
• Some services migrated between collections over time (EKS/ECS info modules live in community.aws). If a module “disappeared,” check whether it moved collections rather than assuming it was deleted.
• boto2 is long gone — anything on the internet using the ancient ec2 module with boto (not boto3) is a decade stale; ignore it.

Common mistakes & troubleshooting

ImportError: No module named boto3. The Execution Environment doesn’t have boto3/botocore installed. Bake them in via the EE’s python_requirements (or pip install in your venv for local dev).

Credentials work in aws s3 ls but not in Ansible. You’re using SSO and Ansible’s process didn’t inherit the profile. Run aws sso login --profile X then export AWS_PROFILE=X in the same shell you run Ansible from.

ec2_instance keeps creating new instances. You forgot the name: (or instance_ids:) parameter. Without an identity key, the module is not idempotent.

Inventory returns 0 hosts. Either: (a) enable_plugins doesn’t include amazon.aws.aws_ec2; (b) the file isn’t named *.aws_ec2.yml; © include_filters excludes everything; (d) credentials are wrong (no sts:GetCallerIdentity permission to even list).

assume_role_arn works in CLI but Ansible says AccessDenied. The role’s trust policy must allow sts:AssumeRole from the Ansible automation account’s role/user, not the user’s own ARN.

autoscaling_group with replace_all_instances: true triggered an outage in stg. It’s a real action; it terminates instances. Use replace_batch_size: 1 and health_check_grace_period: 300, or do blue/green by creating a new ASG and shifting the ALB target group over.

Slow inventory. No cache:, no include_filters. Add both. A 1000-instance account with caching disabled hits AWS APIs on every play.

shell: aws ec2 describe-instances … everywhere. This is the cardinal sin. Replace every one with amazon.aws.ec2_instance_info. The module is idempotent, returns structured data, supports check-mode, and is --diff-friendly.

Common beginner mistakes

These are mindset traps, not symptom lookups (those are in the troubleshooting table above). Each is a wrong mental model a beginner arrives with, followed by the correction.

• “Ansible replaces Terraform, so I’ll build my whole VPC in Ansible.” The wrong model is treating Ansible as another provisioning-and-state tool. Ansible has no state file and no dependency graph; it converges individual resources, not an entire estate. Right model: Terraform/CloudFormation builds and remembers the estate; Ansible operates it and provisions only the handful of resources that react to live events.

• “I already know the AWS CLI, so shell: aws ec2 ... is fine.” The wrong model is that a command is a command. But shell: is opaque to Ansible: it is never idempotent (it runs every time, always changed), returns a blob of text you have to parse, ignores --check, and shows nothing in --diff. Right model: for every aws <service> <verb> there is a native module (amazon.aws.ec2_instance_info, s3_object, …) that is idempotent, returns structured data, and honours check-mode.

• “Credentials belong in the playbook (or in group_vars).” The wrong model is that auth is configuration you write down. Long-lived keys in a repo are the single most common AWS audit finding. Right model: credentials come from the environment via the SDK chain — ideally an instance role or IRSA so there is no secret anywhere — and any secret that must exist lives in Vault, never in plain vars.

• “connection: local is optional boilerplate I can drop.” The wrong model is that an AWS task runs “on the host like any other.” There is no host — an ec2_instance call is an HTTPS request to the AWS API that must run on the control node. Omit delegate_to: localhost + connection: local and Ansible tries to SSH into a machine just to make an API call, which is nonsensical (and either fails or, worse, runs against the wrong box).

• “ec2_instance is idempotent, so re-running is safe.” The wrong model is that idempotency is automatic. It is idempotent only when the module can find the existing resource — which for ec2_instance means a name: (Name tag) or instance_ids:. Without a match key, every run launches a brand-new instance and your account fills with duplicates. Right model: give every managed instance a stable identity key.

• “More forks = faster AWS runs.” The wrong model is that AWS parallelism scales with Ansible parallelism. Past a modest point, extra forks just hammer a rate-limited API; boto3 hits Throttling and backs everything off, so you go slower. Right model: tune the boto3 retry env vars, use throttle: on API-heavy tasks, and use async for long calls — concurrency against AWS is governed by API limits, not fork count.

• “Tags are just metadata / labels for humans.” The wrong model is that tags are cosmetic. In Ansible-on-AWS the tag schema is the inventory and often the security boundary — keyed_groups turns tags into groups, and tag-based IAM turns tags into permissions. Right model: design Environment/Role/Owner/CostCenter deliberately and enforce them with AWS Config; sloppy tags mean a useless dynamic inventory.

Best practices

• Pick the line. Terraform builds the estate; Ansible operates it. If you’re writing Ansible to provision a 50-resource VPC, you’ve crossed the line.
• Use the SDK auth chain. Never put long-lived keys in module args. Prefer instance role / IRSA / assume-role.
• Always set name: on ec2_instance. No exceptions.
• Always delegate_to: localhost + connection: local on AWS API tasks. (Or set it once at the play level.)
• Tag schema first. Enforce with AWS Config. Inventory follows.
• Cache the inventory. cache: true with a 5-10 minute timeout on a directory cache plugin.
• One inventory file per account. Merge via inventory: directory.
• Pin collection versions. amazon.aws 8.x will not break your plays mid-flight.
• Build an AWS EE with boto3, botocore, awscli, amazon.aws, community.aws. Pin AAP job templates to it.
• Mesh execution nodes inside the VPC. Latency to the AWS regional endpoint matters; cross-region from on-prem will time out on big fleets.

Security notes

• No long-lived access keys. Instance role, IRSA, SSO, and STS assume-role are the supported patterns. Long-lived keys are a finding in any modern audit.
• Enable IMDSv2 on every instance you create (metadata_options: { http_tokens: required }); the legacy IMDSv1 is exploitable from in-pod processes.
• Tag-based authorisation in IAM is your friend: write the role policy so ec2:* is allowed only on resources tagged Owner == ${aws:PrincipalTag/Team}. Ansible’s tag schema becomes the security boundary.
• CloudTrail every assume-role call. Set role_session_name: to a meaningful per-job string; that string lands in CloudTrail and gives you “which AAP job hopped into this account” forensics.
• Vault any RDS / DocumentDB / ElastiCache passwords with Ansible Vault or AAP credential plugins backed by Secrets Manager.
• Never pass aws_secret_key as a module param — it ends up in process args and (worse) in Ansible’s verbose log output. Use the env / profile / instance-role chain.
• Block public-S3 by default. public_access: block on every s3_bucket task. Make exceptions explicit and code-reviewed.
• Air-gap-friendly EE. Build the AWS EE locally, push to Private Automation Hub, pin AAP by digest. EKS-hosted execution pods can pull through ECR.

Interview & exam Q&A

Q1. When does Ansible-on-AWS beat Terraform? For operations: deployments, snapshots, key rotation, draining, event-driven response. Terraform converges desired state; Ansible operates the running estate. The two compose — Terraform builds, Ansible runs.

Q2. What’s the recommended way to authenticate Ansible to AWS in production? The lowest-friction credential-free path that fits the host: instance role (Controller on EC2), IRSA (Container Group on EKS), SSO (laptops), assume-role (multi-account hops). Long-lived access keys are an audit finding.

Q3. How do you run one playbook across 20 AWS accounts? Each task (or each play) sets assume_role_arn to a per-account AnsibleAutomation role whose trust policy allows the central automation account’s principal. Or use community.aws.sts_assume_role once and inject the credentials via environment: for subsequent tasks.

Q4. Why is ec2_instance not idempotent without name:? The module identifies an existing instance by its name: tag (or explicit instance_ids:). Without either, every run launches a new instance — there is no way for the module to “find” the previous one.

Q5. Difference between amazon.aws and community.aws? amazon.aws is Red-Hat-supported and covers the core services (EC2/VPC/IAM/RDS/S3/ELB/ASG/Lambda/Route53). community.aws is community-maintained and covers the long tail (EKS, ECS, MSK, Glue, Athena, MWAA). You install both.

Q6. How does the aws_ec2 inventory plugin save API calls on big fleets? Two levers: include_filters does a server-side filter (e.g. only tag:Environment=prod), so a 10,000-instance account becomes a 200-result query; cache: true with a jsonfile cache plugin and a 5-minute timeout means subsequent plays don’t hit the API at all until the cache expires.

Q7. Why must AWS API tasks delegate_to: localhost + connection: local? Because there is no “host” to connect to — you’re calling an HTTPS API. The work runs on the control node. Set connection: local at the play level if every task in the play is an API task; per-task delegation is for mixed plays.

Q8. What’s the difference between IAM Roles for EC2 and IRSA? IAM Roles for EC2 attach an IAM role to an EC2 instance; processes on the instance get temporary creds via IMDSv2. IRSA (IAM Roles for Service Accounts) attaches an IAM role to a Kubernetes ServiceAccount in EKS; pods using that ServiceAccount get temporary creds via OIDC + STS. Both produce credential-free identity.

Q9. How do you write Ansible against multiple AWS regions safely? Either region: per task (clean, explicit), or set AWS_DEFAULT_REGION per play via environment:. Avoid setting it globally in ansible.cfg; per-play scoping is what lets one playbook hit multiple regions.

Q10. What’s a sane tag schema? Required: Environment, Role, Owner, CostCenter. Recommended: Project, PatchGroup, BackupPolicy. Enforce with AWS Config rules. Make untagged instances literally non-compliant within minutes of launch.

Q11. How does check-mode behave for autoscaling_group? Partial — it can show the would-be config diff, but replace_all_instances is a real runtime action and check-mode cannot fully simulate it. Always test in stg, never replace_all in prod from a fresh playbook.

Q12. How do you handle EKS pod-level credentials when AAP runs Container Groups in EKS? The Container Group config sets a ServiceAccount that’s annotated with eks.amazonaws.com/role-arn. The pod-identity webhook injects AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE; boto3 picks them up. No code change needed.

Q13. What’s the safest way to manage RDS passwords? Don’t store them in playbooks. Either Vault-encrypt them, or (better) generate them at instance creation, write to AWS Secrets Manager, and have rds_instance reference the Secrets Manager-backed variable via a lookup plugin or AAP credential plugin.

Q14. How do you build a production AWS EE? ansible-builder with dependencies.yml listing amazon.aws, community.aws; requirements.txt listing boto3, botocore, awscli; system deps (gcc, python3-devel for any C extensions). Push to Private Automation Hub, sign, pin AAP job templates by digest.

Quick check

1. Which two collections cover ~all of AWS?
2. What is the boto3 credential resolution order Ansible inherits?
3. What single tag schema lever has the biggest impact on inventory clarity?
4. Why do all AWS API tasks need connection: local?
5. How do you make ec2_instance idempotent?

(Answers: amazon.aws + community.aws; explicit args → env → profile → IMDSv2 → IRSA → SSO → AssumeRoleWithWebIdentity; Environment (or any single, consistently-applied tag); because there’s no SSH target — the work is an HTTPS API call running on the control node; set name: (or instance_ids:) on every call.)

Practice challenges

Six graded drills, beginner → advanced. Run them against the LocalStack lab above so there is no AWS bill — export the four AWS_* vars plus AWS_ENDPOINT_URL=http://localhost:4566 first. Try each before opening the solution.

Challenge 1 — Install and verify the collections (Beginner)

Get amazon.aws and community.aws installed with boto3/botocore, and prove they are present.

<details> <summary>Solution</summary>

ansible-galaxy collection install amazon.aws community.aws
pip install boto3 botocore
ansible-galaxy collection list | grep -E 'amazon.aws|community.aws'

Why: every module builds a boto3 client, so the SDK and the collections must both live in the same interpreter — which is exactly what an Execution Environment bakes in. </details>

Challenge 2 — One idempotent, locked-down S3 bucket (Beginner)

Write a single task that creates a versioned bucket with all public access blocked, and confirm the second run reports changed=0.

<details> <summary>Solution</summary>

- name: Private, versioned log bucket
  amazon.aws.s3_bucket:
    name: lab-logs-{{ 9999 | random }}
    versioning: true
    encryption: AES256
    public_access:
      block_public_acls: true
      block_public_policy: true
      ignore_public_acls: true
      restrict_public_buckets: true
    state: present
  delegate_to: localhost
  connection: local

Why: the public_access block is declarative, so a correct second run finds nothing to change — public-by-default S3 is a classic breach, and blocking it in code makes the safe state the reproducible one. </details>

Challenge 3 — Kill a shell: aws anti-pattern (Beginner → Intermediate)

You inherit shell: aws ec2 describe-instances --filters Name=tag:Role,Values=web. Replace it with a native, idempotent equivalent that registers structured data and debugs the instance IDs.

<details> <summary>Solution</summary>

- name: List web instances (structured, check-mode safe)
  amazon.aws.ec2_instance_info:
    region: eu-west-1
    filters:
      "tag:Role": web
      instance-state-name: running
  delegate_to: localhost
  connection: local
  register: web

- name: Show the IDs
  ansible.builtin.debug:
    msg: "{{ web.instances | map(attribute='instance_id') | list }}"

Why: _info modules are read-only and idempotent, return typed data you can filter with Jinja, and honour --check — none of which a shell: blob does. </details>

Challenge 4 — A real aws_ec2 inventory with groups and caching (Intermediate)

Write prod.aws_ec2.yml that filters server-side to running prod instances, builds keyed_groups for Role/Environment/AZ, and caches. Verify the groups with ansible-inventory --graph.

<details> <summary>Solution</summary>

# inventory/prod.aws_ec2.yml
plugin: amazon.aws.aws_ec2
regions: [eu-west-1]
include_filters:
  - tag:Environment: ["prod"]
  - instance-state-name: ["running"]
keyed_groups:
  - { key: tags.Role, prefix: role }
  - { key: tags.Environment, prefix: env }
  - { key: placement.availability_zone, prefix: az }
compose:
  ansible_host: private_ip_address
cache: true
cache_plugin: jsonfile
cache_connection: /var/cache/ansible_inventory
cache_timeout: 600

ansible-inventory -i inventory/prod.aws_ec2.yml --graph

Why: include_filters turns a whole-account scan into a small server-side query, and keyed_groups is what makes hosts: role_web and hosts: env_prod work — the tag schema becomes the inventory. </details>

Challenge 5 — Multi-account snapshot with assume-role (Advanced)

From one control identity, take an RDS snapshot in a different account by assuming a role, and make the session name traceable in CloudTrail.

<details> <summary>Solution</summary>

- name: Assume the automation role in the target account
  community.aws.sts_assume_role:
    role_arn: arn:aws:iam::111122223333:role/AnsibleAutomation
    role_session_name: "aap-snap-{{ lookup('pipe', 'date +%Y%m%d%H%M%S') }}"
  register: assumed
  delegate_to: localhost
  connection: local

- name: Snapshot RDS in that account with the temporary creds
  amazon.aws.rds_instance_snapshot:
    db_instance_identifier: app-db
    db_snapshot_identifier: "app-db-{{ lookup('pipe', 'date +%Y%m%d%H%M') }}"
    region: eu-west-1
    state: present
  environment:
    AWS_ACCESS_KEY_ID: "{{ assumed.sts_creds.access_key }}"
    AWS_SECRET_ACCESS_KEY: "{{ assumed.sts_creds.secret_key }}"
    AWS_SESSION_TOKEN: "{{ assumed.sts_creds.session_token }}"
  delegate_to: localhost
  connection: local

Why: one base identity + sts_assume_role is how a single playbook operates many accounts without a key per account; the role_session_name is what lets CloudTrail answer “which job did this?”. </details>

Challenge 6 — Idempotent instance, then target it via dynamic inventory (Advanced)

Launch an instance with a stable name: and IMDSv2 required, register it in-memory, then run a follow-up play against it over SSH — and prove the launch play is idempotent.

<details> <summary>Solution</summary>

- name: Provision from the control node
  hosts: localhost
  connection: local
  gather_facts: true
  tasks:
    - name: Launch (idempotent — matched by Name tag)
      amazon.aws.ec2_instance:
        name: "web-eu1a-{{ deploy_id }}"
        region: eu-west-1
        image_id: "{{ web_ami_id }}"
        instance_type: t3.micro
        vpc_subnet_id: "{{ subnet_id }}"
        security_groups: [lab-sg]
        metadata_options:
          http_tokens: required        # IMDSv2 only
        tags: { Role: web, Environment: prod }
        state: running
        wait: true
      register: launched

    - name: Register the new box in-memory
      ansible.builtin.add_host:
        name: "{{ item.public_ip_address | default(item.private_ip_address) }}"
        groups: just_launched
        ansible_user: ec2-user
      loop: "{{ launched.instances }}"

- name: Configure the new box over SSH
  hosts: just_launched
  gather_facts: false
  tasks:
    - name: Wait for SSH
      ansible.builtin.wait_for_connection:
        timeout: 300
    - name: Prove reachability
      ansible.builtin.ping:

Why: the name: gives the module a match key so a re-run is changed=0 instead of a second instance; http_tokens: required closes the IMDSv1 hole every new instance should be born without — and the two-play shape is the connection: local → connection: ssh handover in miniature. </details>

Exercise

Stand up the LocalStack lab. Then:

1. Build a prod.aws_ec2.yml inventory with regions, include_filters (tag Environment=prod, instance-state running), full keyed_groups for Role/Environment/AZ/instance_type, and caching.
2. Write a play that creates a VPC, subnet, SG, and 2 instances tagged Role=web, Environment=prod. Use delegate_to: localhost properly.
3. Add a follow-up play targeting tag_Role_web from the dynamic inventory; have it run wait_for_connection then ansible.builtin.debug the inventory_hostname.
4. (Stretch) Add community.aws.sts_assume_role at the start of the play (against your own account for the lab; the API call still works locally) and run subsequent tasks with the assumed creds.
5. Run with --check --diff. Then for real. Then again — changed=0.

Certification mapping

Glossary

• amazon.aws — Red-Hat-supported core AWS collection.
• community.aws — community-maintained AWS extras.
• boto3 / botocore — the Python SDK every AWS module uses; must be in the EE.
• SDK auth chain — explicit args → env vars → profile → IMDSv2 → IRSA → SSO → AssumeRoleWithWebIdentity.
• IAM Role for EC2 — instance identity via IMDSv2.
• IRSA — IAM Roles for Service Accounts; pod identity in EKS via OIDC.
• assume_role_arn — multi-account hop primitive; per-task or per-play.
• aws_ec2 plugin — dynamic inventory plugin in amazon.aws.
• include_filters — server-side filter on the aws_ec2 plugin (the most performance-impactful knob).
• connection: local — used for AWS API tasks; the work runs on the control node.
• Idempotency — a task you can run repeatedly for the same end state; a correct AWS module reports changed=0 on the second run.
• _info modules — read-only modules (ec2_instance_info, s3_bucket_info) that describe resources and return structured data; the native replacement for shell: aws … describe-….
• purge_tags / purge_rules — module options (default true) that remove tags/SG-rules not present in your task, so the module converges to exactly your declared list.
• Check-mode (--check) — a dry run; a module reports would-change without mutating. “Partial” for runtime actions like ASG instance replacement.
• wait_for_connection — waits until a freshly launched instance is SSH-reachable before the next play configures it.
• add_host — adds a just-created host to the in-memory inventory for a later play in the same run.
• IMDSv2 — token-required instance metadata service (metadata_options: { http_tokens: required }); the legacy IMDSv1 is exploitable and should be disabled.
• STS / assume-role — AWS Security Token Service; issues temporary credentials when you assume an IAM role (the multi-account primitive).
• Execution Environment (EE) — the container image AAP runs plays in; must contain the collections plus boto3/botocore.
• LocalStack — a fake AWS in Docker; point boto3 at it with AWS_ENDPOINT_URL to run amazon.aws.* modules with no bill.
• Throttling / backoff — AWS API rate-limiting; boto3 retries with exponential backoff (tune via AWS_MAX_ATTEMPTS), which is why more forks ≠ faster.
• keyed_groups — aws_ec2 option that turns each tag/field value into an Ansible group (role_web, env_prod, az_eu_west_1a).

Next steps

You can now drive AWS from Ansible. The same shape — collection + auth chain + dynamic inventory + tagging strategy — repeats for the other clouds. Continue with Ansible for Azure and Ansible for GCP, then Ansible for Kubernetes for EKS-native ops, and finally Hybrid Multi-Cloud Orchestration to compose all three in a single workflow.