Ansible for Kubernetes, In Depth: kubernetes.core, k8s, Helm Charts, Manifests & Operator-Style Workflows

In a nutshell

Kubernetes is a robot warehouse. You hand it a stack of index cards — YAML manifests — that say “keep three copies of this app running and expose it on this port,” and the warehouse makes reality match the cards: restarting, rescheduling, and self-healing without you. The everyday way to hand over a card is kubectl apply -f card.yaml: one human, one card, one cluster, one time.

Ansible is the warehouse-opening checklist. Opening a new location isn’t one card — it’s a repeatable procedure: pour the foundation (create the cluster), bolt in the shelving and forklifts (Helm charts like Prometheus and an ingress controller), issue staff badges and door locks (namespaces and RBAC), and only then start feeding in the recipe cards. Ansible runs that whole procedure the same way every time — and, the real point, across fifty warehouses at once, plus the things outside the warehouse walls that Kubernetes can’t see: DNS, load balancers, cloud IAM.

The kubernetes.core collection is the set of Ansible modules that speak fluent Kubernetes. k8s applies and deletes manifests; k8s_info reads and waits; helm drives chart releases; k8s_scale, k8s_drain, and friends handle the day-2 chores. Every one of them talks straight to the cluster’s REST API — no kubectl binary in the middle — using the exact same kubeconfig you already have. If you can write a manifest and you know a little Ansible, you can automate an entire fleet of clusters from one playbook.

Level: Intermediate → Advanced · Time: ~55 min · You’ll need: a throwaway cluster (kind / k3d / Minikube) and Tier 1–3 Ansible fluency.

Kubernetes engineers tend to fall into two camps: the GitOps purists who think every cluster change should flow through ArgoCD or Flux, and the platform engineers who need to manage 300 clusters across 12 cloud accounts and want a single tool that can talk to all of them. Ansible serves the second group exceptionally well — and, perhaps surprisingly, it serves the first group too, as the orchestrator that creates the cluster, bootstraps ArgoCD, and manages the bits of infrastructure (load balancers, DNS, IAM) that GitOps can’t touch.

This lesson covers the kubernetes.core collection in EX374-grade depth: the k8s module’s states and how they map to kubectl apply/create/delete/replace/patch, manifest templating patterns, Helm chart management with helm and helm_info, operator-style “wait for ready” workflows, multi-cluster inventory patterns, RBAC, namespace lifecycle, and the line between “Ansible should do this” and “let GitOps do this.” If you are coming from kubectl apply -f and shell scripts, this lesson is your upgrade path.

Learning Objectives

By the end you will be able to:

• Install kubernetes.core and the matching Python client libraries on the control node.
• Use kubernetes.core.k8s to apply, create, replace, delete, and patch Kubernetes objects with full idempotence.
• Use k8s_info, k8s_exec, k8s_log, k8s_scale, k8s_drain, and k8s_rollback for read, exec, log, scale, drain, and rollback operations.
• Render manifests from Jinja2 templates with environment-specific values using lookup('template', ...) patterns.
• Manage Helm chart releases with kubernetes.core.helm (install, upgrade, rollback, uninstall) and read state with helm_info.
• Authenticate to multiple clusters using kubeconfig contexts, in-cluster ServiceAccount tokens, or short-lived OIDC tokens.
• Build operator-style workflows: create a CR, wait for status conditions, reconcile drift.
• Combine Ansible (cluster bootstrap, IAM, networking, day-2 ops) with ArgoCD/Flux (application delivery) without overlap.

Prerequisites

• Working knowledge of Kubernetes concepts: pods, deployments, services, namespaces, RBAC, custom resources.
• Comfortable reading and writing YAML manifests.
• Tier 1–3 Ansible fluency: roles, inventories, vault.
• A test cluster: kind, k3d, Minikube, or a real cluster you can break.

Mental Model: How Ansible Talks to Kubernetes

1. The K8s API is just a REST server — Ansible is an HTTP client

Every Kubernetes module in kubernetes.core is a thin Python wrapper around the official kubernetes Python client, which itself is a generated client for the cluster’s REST API. Whether you kubectl apply -f deploy.yaml or k8s: definition: ..., the wire-level call is the same: a PATCH or POST to /apis/apps/v1/namespaces/<ns>/deployments. This means anything kubectl can do, Ansible can do — and the auth model is identical: kubeconfig with contexts, or in-cluster ServiceAccount tokens.

2. state: maps to kubectl verbs

The k8s module exposes three state values — present, absent, and patched — plus two switches, apply: true (server-side apply) and force: true (full replace), that decide exactly which kubectl verb the wire call becomes. state: present is the implicit “create if missing, update if exists” behaviour that kubectl apply does. One module, the whole apply/create/replace/patch/delete surface, full idempotence — Ansible’s k8s module is essentially kubectl apply with idempotence guarantees.

3. Manifests are Jinja2-rendered, not pre-baked

The right way to template manifests for multiple environments isn’t envsubst or kustomize — it’s lookup('template', 'deploy.yaml.j2'). You write one deploy.yaml.j2 with {{ replicas }}, {{ image }}, {{ namespace }}, then call it from a play with environment-specific vars. This is more powerful than Kustomize overlays for complex templating (loops over service ports, conditional sidecars), simpler than Helm for one-off applications.

4. Helm is a first-class concept, not a shell-out

kubernetes.core.helm runs Helm chart releases without shelling out to the helm binary on the target. It tracks the chart name, version, values, and namespace, and produces idempotent install/upgrade/uninstall behavior. You can drive Helm-packaged apps from Ansible exactly as you would drive container deployments from Compose — same playbook structure, same idempotence guarantees.

5. Ansible and GitOps are complements, not competitors

Ansible owns the cluster lifecycle (create cluster, install CNI, bootstrap ArgoCD, configure ingress controller IAM), while GitOps owns app delivery (the manifests of your microservices). The test for “is this Ansible or GitOps?” is: if I redeploy the cluster from scratch, who creates this object first? That object belongs in Ansible. Everything else (your apps, your team’s workloads) belongs in Git, watched by ArgoCD/Flux.

Setting Up the Control Node

kubernetes.core requires the kubernetes Python client and (for Helm) the helm binary on the control node. The collection itself comes from Galaxy.

# Install the Python client
python3 -m pip install --user 'kubernetes>=29.0.0' 'PyYAML>=6.0' 'jsonpatch>=1.33'

# Install the collection
ansible-galaxy collection install kubernetes.core

# Install Helm (control node)
curl -fsSL https://get.helm.sh/helm-v3.14.0-linux-amd64.tar.gz | tar -xz
sudo mv linux-amd64/helm /usr/local/bin/

# Verify
ansible-galaxy collection list kubernetes.core
helm version

For multi-architecture environments (control node on macOS or arm64), match the helm binary to your control node OS — the target is the K8s API, not a specific OS, so target architecture doesn’t matter.

Authentication: Three Patterns

Pattern A: Kubeconfig file (most common)

- name: Apply a namespace using kubeconfig
  kubernetes.core.k8s:
    kubeconfig: ~/.kube/config
    context: prod-cluster
    state: present
    definition:
      apiVersion: v1
      kind: Namespace
      metadata:
        name: payments

This reads the standard kubeconfig with your contexts. To target multiple clusters, set context: per task or per play.

Pattern B: In-cluster ServiceAccount (Ansible runs inside K8s)

When Ansible runs inside a pod (e.g. as a Kubernetes Job, or inside an Ansible Automation Platform execution environment that runs as a pod):

- name: Apply a manifest using in-cluster auth
  kubernetes.core.k8s:
    state: present
    definition: "{{ lookup('file', 'manifest.yaml') | from_yaml }}"
  # No kubeconfig — auto-detects /var/run/secrets/kubernetes.io/serviceaccount

The pod’s ServiceAccount token must have RBAC for whatever you’re applying. This is the pattern AAP uses when its execution environments run as Kubernetes pods.

Pattern C: OIDC / short-lived tokens

For zero-long-lived-credential setups (EKS with IRSA, GKE Workload Identity, AKS Workload Identity, or any OIDC provider):

- name: Get short-lived token from OIDC
  ansible.builtin.command:
    cmd: aws eks get-token --cluster-name prod
  register: eks_token
  changed_when: false

- name: Apply with token
  kubernetes.core.k8s:
    api_key: "{{ (eks_token.stdout | from_json).status.token }}"
    host: "https://prod.eks.us-east-1.amazonaws.com"
    ca_cert: /etc/eks/prod-ca.crt
    state: present
    definition: "{{ lookup('template', 'deploy.yaml.j2') | from_yaml }}"

Use this pattern when your control node runs in a cloud workload that has cluster-targeted IAM permissions but no static kubeconfig.

The kubernetes.core.k8s Module — Module-by-Module

k8s — the workhorse

Manages any Kubernetes object. Three state values — present, absent, patched — combined with the apply: and force: switches, plus a definition (inline) or src (file path) parameter. The table below maps each combination to its kubectl equivalent:

- name: Apply a namespace
  kubernetes.core.k8s:
    state: present
    definition:
      apiVersion: v1
      kind: Namespace
      metadata:
        name: payments
        labels:
          team: payments
          env: prod

- name: Apply a deployment from a templated file
  kubernetes.core.k8s:
    state: present
    definition: "{{ lookup('template', 'roles/payments/templates/deploy.yaml.j2') | from_yaml }}"
    namespace: payments
    apply: true   # Equivalent to `kubectl apply` (server-side merge)

- name: Replace (not merge) a configmap — full overwrite
  kubernetes.core.k8s:
    state: present
    force: true          # PUT/replace the whole object instead of merging
    definition: "{{ lookup('file', 'config.yaml') | from_yaml }}"

- name: Strategic-merge-patch a deployment's image
  kubernetes.core.k8s:
    state: patched
    kind: Deployment
    namespace: payments
    name: api
    definition:
      spec:
        template:
          spec:
            containers:
              - name: api
                image: registry.example.com/api:v1.2.3

- name: Delete a deployment
  kubernetes.core.k8s:
    state: absent
    kind: Deployment
    namespace: payments
    name: deprecated-service

Note apply: true — when present, the module uses server-side apply (the modern kubectl apply behavior), which is the right default for most cases because it preserves fields that other controllers (autoscalers, admission webhooks) have written.

k8s_info — the gather-facts equivalent

- name: Get all pods in payments namespace
  kubernetes.core.k8s_info:
    kind: Pod
    namespace: payments
  register: payments_pods

- name: Show pod names
  ansible.builtin.debug:
    msg: "{{ payments_pods.resources | map(attribute='metadata.name') | list }}"

- name: Get a specific deployment with full status
  kubernetes.core.k8s_info:
    kind: Deployment
    namespace: payments
    name: api
  register: api_deploy

- name: Wait for at least 3 ready replicas
  ansible.builtin.assert:
    that:
      - api_deploy.resources[0].status.readyReplicas >= 3

k8s_exec — exec into pods

- name: Run a smoke test inside a pod
  kubernetes.core.k8s_exec:
    namespace: payments
    pod: "{{ payments_pods.resources[0].metadata.name }}"
    container: api
    command: /bin/sh -c "curl -s http://localhost:8080/health | grep ok"
  register: smoke
  failed_when: smoke.rc != 0

k8s_log — read pod logs

- name: Get last 100 lines of a pod's logs
  kubernetes.core.k8s_log:
    namespace: payments
    name: api-7d8f4c5b9-xkj2p
    tail_lines: 100
  register: api_logs

k8s_scale — scale workloads

- name: Scale the api deployment to 5 replicas
  kubernetes.core.k8s_scale:
    api_version: apps/v1
    kind: Deployment
    name: api
    namespace: payments
    replicas: 5
    wait: true
    wait_timeout: 120

k8s_drain — cordon and drain a node

- name: Drain node before maintenance
  kubernetes.core.k8s_drain:
    name: ip-10-0-1-15.ec2.internal
    state: drain
    delete_options:
      force: true
      ignore_daemonsets: true
      grace_period: 30

k8s_rollback — roll back deployments

- name: Rollback the api deployment one revision
  kubernetes.core.k8s_rollback:
    api_version: apps/v1
    kind: Deployment
    name: api
    namespace: payments

Helm Lifecycle with kubernetes.core.helm

Helm releases are first-class Ansible resources. The collection ships helm, helm_info, helm_repository, helm_template, and helm_plugin.

- name: Add a Helm repo
  kubernetes.core.helm_repository:
    name: prometheus-community
    repo_url: https://prometheus-community.github.io/helm-charts
    state: present

- name: Install kube-prometheus-stack with custom values
  kubernetes.core.helm:
    release_name: monitoring
    chart_ref: prometheus-community/kube-prometheus-stack
    chart_version: 56.6.2
    release_namespace: monitoring
    create_namespace: true
    values:
      grafana:
        adminPassword: "{{ vault_grafana_admin }}"
        ingress:
          enabled: true
          hosts:
            - grafana.example.com
      prometheus:
        prometheusSpec:
          retention: 30d
          storageSpec:
            volumeClaimTemplate:
              spec:
                accessModes: [ReadWriteOnce]
                resources:
                  requests:
                    storage: 100Gi
    wait: true
    wait_timeout: 10m

- name: Get state of installed release
  kubernetes.core.helm_info:
    release_name: monitoring
    release_namespace: monitoring
  register: release_state

- name: Rollback if a check failed
  kubernetes.core.helm:
    release_name: monitoring
    release_namespace: monitoring
    release_state: rolled_back
    revision: 3
  when: release_state.failed

kubernetes.core.helm is idempotent — running it twice with the same chart_version and values results in changed: false. Bumping chart_version or changing values triggers an upgrade. Setting release_state: absent uninstalls.

Manifest Templating Patterns

Pattern 1: Inline definition (small objects)

- name: Create a ConfigMap
  kubernetes.core.k8s:
    state: present
    definition:
      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: app-config
        namespace: "{{ app_namespace }}"
      data:
        DATABASE_URL: "{{ db_connection_string }}"
        LOG_LEVEL: "{{ log_level }}"

Pattern 2: Templated file (deployments, custom resources)

# templates/deploy.yaml.j2
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ app_name }}
  namespace: {{ app_namespace }}
spec:
  replicas: {{ replicas | default(2) }}
  selector:
    matchLabels:
      app: {{ app_name }}
  template:
    metadata:
      labels:
        app: {{ app_name }}
        version: {{ image_tag }}
    spec:
      containers:
        - name: {{ app_name }}
          image: {{ image_repo }}:{{ image_tag }}
          ports:
            {% for port in app_ports %}
            - containerPort: {{ port }}
            {% endfor %}
          resources:
            requests:
              cpu: {{ cpu_request | default('100m') }}
              memory: {{ memory_request | default('128Mi') }}

# task in role
- name: Apply deployment
  kubernetes.core.k8s:
    state: present
    definition: "{{ lookup('template', 'deploy.yaml.j2') | from_yaml }}"
    apply: true

Pattern 3: Multi-document YAML (deployment + service + ingress in one file)

- name: Apply a multi-doc manifest
  kubernetes.core.k8s:
    state: present
    definition: "{{ lookup('template', 'app.yaml.j2') | from_yaml_all | list }}"
    apply: true

from_yaml_all parses ----separated YAML documents into a list, then the module applies each.

Operator-Style Workflows

A pattern that’s common in platform automation: create a Custom Resource, then wait for a status condition before continuing.

- name: Create an Argo CD Application
  kubernetes.core.k8s:
    state: present
    definition:
      apiVersion: argoproj.io/v1alpha1
      kind: Application
      metadata:
        name: payments-api
        namespace: argocd
      spec:
        project: payments
        source:
          repoURL: https://github.com/example/payments-api
          targetRevision: HEAD
          path: deploy/k8s
        destination:
          server: https://kubernetes.default.svc
          namespace: payments
        syncPolicy:
          automated:
            prune: true
            selfHeal: true

- name: Wait for the Application to be Synced and Healthy
  kubernetes.core.k8s_info:
    api_version: argoproj.io/v1alpha1
    kind: Application
    namespace: argocd
    name: payments-api
  register: app
  until:
    - app.resources[0].status.sync.status == 'Synced'
    - app.resources[0].status.health.status == 'Healthy'
  retries: 30
  delay: 10

This is the operator-style “create then reconcile” pattern. Same approach works for Cert-Manager Certificates, Crossplane Compositions, and any other CRD that exposes status conditions.

Multi-Cluster Inventory

For platform teams managing 50+ clusters, define them in inventory:

# inventories/all-clusters/clusters.yml
all:
  children:
    prod_clusters:
      hosts:
        prod-us-east:
          k8s_context: prod-us-east
          k8s_region: us-east-1
        prod-eu-west:
          k8s_context: prod-eu-west
          k8s_region: eu-west-1
        prod-ap-south:
          k8s_context: prod-ap-south
          k8s_region: ap-south-1
    nonprod_clusters:
      hosts:
        dev:
          k8s_context: dev
        staging:
          k8s_context: staging

Then run a play that targets prod_clusters and uses delegate_to: localhost (since K8s API calls don’t run on the cluster — they run on the control node):

- hosts: prod_clusters
  gather_facts: false
  tasks:
    - name: Apply baseline namespace policy to every prod cluster
      kubernetes.core.k8s:
        kubeconfig: ~/.kube/config
        context: "{{ k8s_context }}"
        state: present
        definition: "{{ lookup('template', 'baseline-policy.yaml.j2') | from_yaml }}"
      delegate_to: localhost

delegate_to: localhost is the key — Ansible iterates over the cluster inventory but executes each task locally, just changing the context: per cluster.

Hands-on Free Lab: Bootstrap a kind Cluster, Install ArgoCD, Deploy an App

Free, runs on your laptop, takes 10 minutes.

# Prerequisites
brew install kind helm  # macOS, or use your distro equivalent
mkdir -p ~/ansible-k8s-lab && cd ~/ansible-k8s-lab

# Create a kind cluster
cat > kind-config.yaml <<'EOF'
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
  - role: worker
  - role: worker
EOF
kind create cluster --name ansible-lab --config kind-config.yaml

# Build the inventory
cat > inventory.yml <<'EOF'
all:
  hosts:
    cluster_lab:
      ansible_connection: local
      k8s_context: kind-ansible-lab
EOF

# Build the playbook
cat > site.yml <<'EOF'
---
- hosts: cluster_lab
  gather_facts: false
  collections:
    - kubernetes.core
  tasks:

    - name: Create monitoring namespace
      kubernetes.core.k8s:
        context: "{{ k8s_context }}"
        state: present
        definition:
          apiVersion: v1
          kind: Namespace
          metadata:
            name: monitoring

    - name: Add prometheus-community Helm repo
      kubernetes.core.helm_repository:
        name: prometheus-community
        repo_url: https://prometheus-community.github.io/helm-charts

    - name: Install kube-prometheus-stack
      kubernetes.core.helm:
        kubeconfig: ~/.kube/config
        context: "{{ k8s_context }}"
        release_name: monitoring
        chart_ref: prometheus-community/kube-prometheus-stack
        chart_version: 56.6.2
        release_namespace: monitoring
        wait: true
        wait_timeout: 5m
        values:
          grafana:
            adminPassword: "labpassword"

    - name: Add argo-cd Helm repo
      kubernetes.core.helm_repository:
        name: argo
        repo_url: https://argoproj.github.io/argo-helm

    - name: Install Argo CD
      kubernetes.core.helm:
        kubeconfig: ~/.kube/config
        context: "{{ k8s_context }}"
        release_name: argocd
        chart_ref: argo/argo-cd
        chart_version: 6.7.1
        release_namespace: argocd
        create_namespace: true
        wait: true
        wait_timeout: 5m

    - name: Deploy a sample app via templated manifest
      kubernetes.core.k8s:
        context: "{{ k8s_context }}"
        state: present
        definition: "{{ lookup('template', 'demo-app.yaml.j2') | from_yaml_all | list }}"

    - name: Wait for sample app to have 2 ready replicas
      kubernetes.core.k8s_info:
        context: "{{ k8s_context }}"
        kind: Deployment
        namespace: default
        name: hello
      register: hello_deploy
      until: hello_deploy.resources[0].status.readyReplicas | default(0) >= 2
      retries: 20
      delay: 5
EOF

# Build the manifest template
mkdir -p templates
cat > templates/demo-app.yaml.j2 <<'EOF'
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: hello
  template:
    metadata:
      labels:
        app: hello
    spec:
      containers:
        - name: hello
          image: gcr.io/google-samples/hello-app:2.0
          ports:
            - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: hello
  namespace: default
spec:
  selector:
    app: hello
  ports:
    - port: 80
      targetPort: 8080
EOF

# Run the playbook
ansible-playbook -i inventory.yml site.yml

# Verify
kubectl get pods -A
kubectl --context kind-ansible-lab port-forward -n monitoring svc/monitoring-grafana 3000:80 &
# Open http://localhost:3000 (admin / labpassword)

# Tear down
kind delete cluster --name ansible-lab

This is the complete bootstrap pattern: namespace lifecycle, Helm charts (Prometheus + ArgoCD), and a templated manifest deployment — exactly the workflow you’d use to provision a real cluster.

Going deeper

Everything above gets the job done. This section is for the reader who has to defend these playbooks in production — how idempotence actually works, when it flaps, and the knobs that separate a demo from a fleet.

How k8s decides “changed”

kubernetes.core.k8s is not a shell wrapper. For every task it (1) GETs the live object, (2) computes the difference against your definition, and (3) sends a write only if that diff is non-empty — otherwise it reports changed: false. Which kind of diff depends on your switches:

• Client-side (default state: present) — the module builds a strategic-merge-patch locally and PATCHes it. Server-defaulted fields you didn’t specify are left alone.
• Server-side apply (apply: true) — the module sends your object as an apply request under a field manager (default name ansible). The API server records which manager owns which field in metadata.managedFields. If another manager (an autoscaler, a mutating webhook, a human with kubectl) already owns a field you’re also setting, the apply conflicts and fails — deliberately. You resolve it explicitly:

- name: Apply and take ownership of contested fields
  kubernetes.core.k8s:
    state: present
    apply: true
    server_side_apply:
      field_manager: platform-ansible
      force_conflicts: true          # take ownership of conflicting fields
    definition: "{{ lookup('template', 'deploy.yaml.j2') | from_yaml }}"

This is why a task can report changed: true on every run: a controller keeps rewriting a field you also declare. The fix is not to spam changed_when: false — it is to stop declaring the field you don’t own (let the HPA own replicas), or to take ownership deliberately with force_conflicts.

Waiting properly: wait_condition vs. until

There are two ways to block until something is ready, and they are not interchangeable:

• wait: true on k8s / k8s_scale / k8s_info waits for the built-in readiness of well-known kinds — a Deployment’s Available condition, a Pod’s Ready, a Job’s completion — with no polling loop to write.
• wait_condition: lets you wait on any status condition by type / status / reason, which is how you wait on CRDs and cluster-scoped objects:

- name: Wait until the CRD is Established before creating any CR
  kubernetes.core.k8s_info:
    api_version: apiextensions.k8s.io/v1
    kind: CustomResourceDefinition
    name: certificates.cert-manager.io
    wait: true
    wait_condition:
      type: Established
      status: "True"
    wait_timeout: 120

• until: / retries: / delay: is the fallback for status fields that aren’t Kubernetes “conditions” at all — a chart-installed app that reports status.phase: Running, an Argo CD Application’s status.sync.status. It polls, it works on anything, it is more verbose. Reach for wait_condition first; reach for until when the value you need isn’t a condition.

Check mode, diff, and server dry-run

kubernetes.core.k8s honours --check and --diff. In check mode with apply: true, the module performs a server-side dry-run (?dryRun=All) so admission webhooks and validation run without persisting — a far more trustworthy preview than a local guess. You can force that dry-run outside check mode too:

- name: Validate a manifest against admission control without applying
  kubernetes.core.k8s:
    state: present
    apply: true
    dry_run: true
    definition: "{{ lookup('template', 'deploy.yaml.j2') | from_yaml }}"
  register: preview

Run ansible-playbook --check --diff site.yml and you get a field-level diff of every object the play would touch — the single most useful habit for changes to a live cluster.

Deleting safely: finalizers and propagation

state: absent returns as soon as the API accepts the delete — not when the object is actually gone. Two knobs matter. wait: true blocks until the object disappears, and delete_options controls cascade behaviour:

- name: Delete a deployment (and its ReplicaSets/Pods) and block until gone
  kubernetes.core.k8s:
    state: absent
    api_version: apps/v1
    kind: Deployment
    namespace: payments
    name: deprecated
    wait: true
    delete_options:
      propagationPolicy: Foreground   # children removed before the parent reports gone
      gracePeriodSeconds: 30

The classic trap is a namespace stuck in Terminating: a finalizer on some object inside it is blocking deletion, and wait: true will (correctly) hang. Find and clear the finalizer at its source — never blindly strip finalizers on a shared cluster.

Impersonation and multi-tenancy

Platform teams often run one automation identity that acts as a tenant to stay inside that tenant’s RBAC. The auth args support it directly:

- name: Apply as a scoped tenant identity (requires the impersonate verb)
  kubernetes.core.k8s:
    impersonate_user: system:serviceaccount:team-a:deployer
    impersonate_groups:
      - team-a:writers
    state: present
    definition: "{{ lookup('template', 'app.yaml.j2') | from_yaml }}"

Your automation identity needs the impersonate verb; every action is then evaluated — and audit-logged — as the impersonated subject.

Templating inside the module vs. lookup('template')

Two equivalent ways to render Jinja into a manifest:

• definition: "{{ lookup('template', 'deploy.yaml.j2') | from_yaml }}" — render on the controller, parse to a dict, hand the dict to the module. Explicit, easy to debug (| from_yaml_all | list for multi-doc), and you can pipe the result to debug before it hits the cluster.
• The module’s own template: parameter — hand k8s a template path and it renders internally. Slightly terser, but you lose that “print it first” escape hatch.

Prefer lookup('template') while you’re learning — you can always inspect the rendered dict before it is applied.

Performance and scale

Every kubernetes.core task runs on the control node (that’s why multi-cluster plays use delegate_to: localhost), so throughput is bounded by control-node API round-trips, not by target CPUs:

• A play that applies 1,000 objects serially is 1,000+ round-trips. Group related objects into multi-document templates (from_yaml_all) so one task applies many.
• forks parallelises across hosts, not across a single task’s loop items — a loop of 500 manifests still runs serially within one host. Split large workloads across cluster-hosts and let forks fan out.
• The Python client keeps a discovery cache under ~/.kube/cache; on a cold control node the first call to each API group pays a discovery cost. Long-lived controllers (AWX/AAP execution environments) amortise this; short-lived CI runners pay it every time.
• Against a single cluster, too much parallelism trips the API server’s rate limiter — prefer serial: 1 / throttle: 1 for bulk writes to one cluster and scale out across clusters instead.

Collection and version caveats

• kubernetes.core is the current name; it was renamed from community.kubernetes in 2021. Migrate old community.kubernetes.k8s references to the FQCN kubernetes.core.k8s.
• The k8s module needs the kubernetes Python client (plus PyYAML, and jsonpatch for JSON-patch operations) installed in the interpreter Ansible uses — frequently not the one your login shell’s python3 resolves to. Confirm with ansible-config dump | grep INTERPRETER_PYTHON.
• Keep the Python client within roughly one minor version of the cluster (the same ±1 skew guidance as kubectl). A very old client against a new cluster can miss newer API versions silently.
• The helm, helm_info, helm_repository, helm_template, and helm_plugin modules shell out to a Helm v3 binary on the control node — there is no Tiller, and Helm v2 is unsupported. k8s, by contrast, needs no external binary at all.

Common Mistakes & Troubleshooting

1. “Failed to import the required Python library (kubernetes)” The kubernetes Python client isn’t installed in the Python interpreter Ansible uses. Run ansible-config dump | grep INTERPRETER_PYTHON to find it, then <that python> -m pip install kubernetes.

2. apply: true keeps reporting changed: true A controller (autoscaler, admission webhook) is rewriting fields you specify. Use server-side apply with field manager owner-tracking, or remove the field from your manifest and let the controller own it.

3. k8s_info returns empty list when the resource exists Check the api_version — listing pods is v1 (no group) but listing deployments is apps/v1. Wrong API version returns empty silently.

4. Helm install fails with “context deadline exceeded” The chart has CRDs and the cluster takes time to register them. Increase wait_timeout: 10m or split into two tasks: install with wait: false, then helm_info with until: status.status == 'deployed'.

5. “Unable to recognize” error on a CRD The CRD wasn’t applied before the CR. Either install the CRD in a separate task with wait: true, or use Helm (which applies CRDs first), or pass --validate=false (last resort).

6. Multi-cluster play hits the same cluster repeatedly You forgot delegate_to: localhost. Without it, Ansible tries to SSH to the inventory hostname (which isn’t a real host in K8s inventory).

7. RBAC errors when running from inside the cluster The pod’s ServiceAccount doesn’t have the required RBAC. Create a ClusterRole / Role with the verbs you need (get, list, create, update, delete) on the resources you touch, and bind it to the ServiceAccount.

Best Practices

• Use server-side apply (apply: true). It’s the modern default and plays nicely with operators that own fields.
• Pin chart versions explicitly. chart_version: 56.6.2, never empty or “latest” — Helm chart updates can break clusters.
• Wait for readiness with until/retries/delay. “Apply succeeded” doesn’t mean “the workload is healthy.”
• Use roles for cluster baselines. A cluster_baseline role that creates namespaces, RBAC, network policies, and ingress is reusable across all clusters.
• Don’t manage every CR with Ansible. App manifests belong in Git, watched by ArgoCD/Flux. Ansible should manage the cluster, not the apps (with exceptions for cluster-level CRs like ArgoCD Applications themselves).
• Use k8s_info before k8s for read-modify-write. The lookup-then-patch pattern is safer than blind replaced.
• Keep manifests small. A 500-line Deployment template is unmaintainable. Split into Deployment, Service, ConfigMap, Ingress as separate templates.
• Test in dry-run mode. kubernetes.core.k8s supports check mode for most operations — combine with --diff for change preview.

Security Notes

• Never commit kubeconfigs to Git. Even encrypted, they grant cluster admin and rotate poorly.
• Use short-lived tokens (OIDC, IRSA, Workload Identity) in CI/CD. Long-lived ServiceAccount tokens are an audit finding.
• Scope ServiceAccounts to the minimum RBAC. A namespace-scoped Role beats a ClusterRole every time.
• Enable audit logging on the K8s API server. Every Ansible action becomes an audit log entry — invaluable for incident response.
• Pin Helm chart versions and review chart changelogs. Helm charts often include CRDs and ClusterRoles that grant cluster-wide privileges.
• Vault-encrypt Helm values containing secrets — vault_grafana_admin, vault_db_password, etc. Helm releases pass values to the chart’s templates, which can render them into Secret resources.

Common beginner mistakes

These are mental-model errors, not error messages — the traps that make Kubernetes automation feel unpredictable until the model clicks. (For symptom→fix recipes, see “Common Mistakes & Troubleshooting” above.)

“I have to choose: Ansible or kubectl or GitOps.” They are three layers, not three rivals. kubectl is your interactive debugger (poke one object, tail one log). Ansible is the procedure that stands a cluster up and manages the platform plus the world outside the cluster (DNS, load balancers, IAM). GitOps continuously reconciles your apps. The right question is never “which tool” but “which layer does this change live in?”

“The k8s task went green, so my app is running.” A green apply means the API server accepted your desired state — nothing more. The Deployment may still be pulling images, failing readiness probes, or wedged on a missing Secret. “Applied” and “healthy” are different facts; always follow a write with wait: true or a k8s_info + until on readyReplicas/conditions before you call it done.

“Force-replace is just a safer, more thorough apply.” force: true does a full PUT — it overwrites the whole object, discarding fields other controllers own and bumping resourceVersion, which can trigger a needless rollout or drop an autoscaler’s replicas. Server-side apply (apply: true) is the safe default; reach for force only when you intend to blow away out-of-band changes.

“Ansible SSHes into the cluster nodes to do this.” No connection is made to any node. kubernetes.core modules are HTTP clients that run on the control node and talk to the API server. In a cluster inventory, a “host” is just a label carrying a context: — which is why multi-cluster plays need delegate_to: localhost (or connection: local). Treating cluster-hosts like SSH targets is the number-one source of “why is it timing out on SSH / hitting the same cluster” confusion.

“I’ll manage every app manifest in Ansible, forever.” Ansible enforces state only when the playbook runs. Between runs, anything can drift and nothing corrects it. Apps that change often want a controller that reconciles continuously — that’s ArgoCD/Flux. Keep fast-moving workloads in Git; keep the cluster baseline (namespaces, RBAC, ingress controller, the GitOps engine itself) in Ansible.

“The collection just uses my terminal’s kubectl/helm and kubeconfig.” The kubernetes Python client does read KUBECONFIG / ~/.kube/config and the current context by default — but k8s never calls the kubectl binary, and the Python libraries must live in the interpreter Ansible selects, which often differs from your shell’s python3. “It works in my terminal but the playbook says the library is missing” is almost always this. Be explicit with kubeconfig: / context: and confirm INTERPRETER_PYTHON.

“Idempotent means it must report changed: false after the first run.” Idempotent means converging to the declared state, not never reporting a change. If a mutating webhook or an HPA keeps rewriting a field you also declare, honest re-runs will keep showing changed: true. That’s a signal to fix ownership (stop declaring the field, or use force_conflicts), not something to paper over with a blanket changed_when: false.

“Secrets in Helm values: are just variables.” A value passed to a chart is frequently rendered into a Secret object and can surface in task output and logs. Pull them from Vault ({{ vault_grafana_admin }}) and set no_log: true on the task. A plaintext admin password in a committed values block is a credential leak, not a config default.

Practice challenges

Six exercises that climb from “first namespace” to “operator-style reconcile.” Run them against a throwaway kind / k3d cluster. Each solution is one defensible answer, not the only one.

Challenge 1 — Idempotent namespace (beginner)

Create a namespace demo carrying the labels team=platform and env=lab. Run your play twice and confirm the second run reports changed: false.

<details> <summary>Solution</summary>

- name: Ensure the demo namespace exists with labels
  kubernetes.core.k8s:
    state: present
    definition:
      apiVersion: v1
      kind: Namespace
      metadata:
        name: demo
        labels:
          team: platform
          env: lab

Why: state: present is create-or-update; the module diffs against the live object, so an unchanged second run is a no-op (changed: false). </details>

Challenge 2 — Read-only inventory of a namespace (beginner)

Without changing anything, list every Deployment in kube-system and print each name alongside its ready-replica count.

<details> <summary>Solution</summary>

- name: Gather Deployments in kube-system
  kubernetes.core.k8s_info:
    api_version: apps/v1
    kind: Deployment
    namespace: kube-system
  register: ksys

- name: Report name and ready replicas
  ansible.builtin.debug:
    msg: >-
      {{ item.metadata.name }} =
      {{ item.status.readyReplicas | default(0) }}/{{ item.spec.replicas }}
  loop: "{{ ksys.resources }}"
  loop_control:
    label: "{{ item.metadata.name }}"

Why: k8s_info is the read-only “gather” module; it never mutates the cluster, so it is safe to run anywhere, including under --check. </details>

Challenge 3 — Templated deploy with a real readiness gate (intermediate)

Render a Deployment from a Jinja template with replicas and image_tag as variables, apply it with server-side apply, then block until every replica is ready.

<details> <summary>Solution</summary>

# templates/web.yaml.j2 → name: web, image nginx:{{ image_tag }}, replicas {{ replicas }}
- name: Apply the web deployment (server-side)
  kubernetes.core.k8s:
    state: present
    apply: true
    namespace: demo
    definition: "{{ lookup('template', 'web.yaml.j2') | from_yaml }}"

- name: Wait until the Deployment reports Available
  kubernetes.core.k8s_info:
    api_version: apps/v1
    kind: Deployment
    namespace: demo
    name: web
    wait: true
    wait_condition:
      type: Available
      status: "True"
    wait_timeout: 120

Why: apply: true is server-side apply (field-tracked); the built-in Available condition is a truer “it’s up” signal than a bare apply, which only means the spec was accepted. </details>

Challenge 4 — Pinned Helm release, verified (intermediate)

Install the podinfo chart at a pinned version into its own namespace (creating the namespace), wait for it, then assert via helm_info that the release status is deployed.

<details> <summary>Solution</summary>

- name: Add the podinfo Helm repo
  kubernetes.core.helm_repository:
    name: podinfo
    repo_url: https://stefanprodan.github.io/podinfo

- name: Install podinfo at a pinned version
  kubernetes.core.helm:
    release_name: podinfo
    chart_ref: podinfo/podinfo
    chart_version: "6.7.0"   # representative — check the current chart version
    release_namespace: demo-podinfo
    create_namespace: true
    wait: true
    wait_timeout: 5m

- name: Read the release state
  kubernetes.core.helm_info:
    release_name: podinfo
    release_namespace: demo-podinfo
  register: podinfo_state

- name: Assert the release is deployed
  ansible.builtin.assert:
    that:
      - podinfo_state.status.status == 'deployed'
    fail_msg: "podinfo release is {{ podinfo_state.status.status | default('missing') }}"

Why: pinning chart_version makes the install reproducible; wait: true blocks on rollout, but you still verify intent with helm_info — “waited” and “deployed” are separate facts. </details>

Challenge 5 — Zero-downtime node maintenance (advanced)

Given a node name in target_node, cordon and drain it (evicting pods but ignoring DaemonSets and respecting PodDisruptionBudgets), then uncordon it. Make the play safe to re-run whatever the node’s starting state.

<details> <summary>Solution</summary>

- name: Drain the node for maintenance
  kubernetes.core.k8s_drain:
    name: "{{ target_node }}"
    state: drain
    delete_options:
      ignore_daemonsets: true
      delete_emptydir_data: true
      wait_timeout: 120
  # k8s_drain uses the eviction API, so it honours PodDisruptionBudgets

- name: Do the maintenance here — patch, reboot, etc.
  ansible.builtin.debug:
    msg: "Node {{ target_node }} is drained and unschedulable"

- name: Uncordon the node when done
  kubernetes.core.k8s_drain:
    name: "{{ target_node }}"
    state: uncordon

Why: k8s_drain evicts through the eviction API, so it respects PDBs instead of hard-killing pods; state: uncordon is idempotent, so re-running restores schedulability from any starting state. </details>

Challenge 6 — Operator-style reconcile with a CRD (advanced)

Apply a CRD, wait until it is Established, then create a Custom Resource and poll until its custom status.phase reads Ready, failing with a clear message after a bounded wait.

<details> <summary>Solution</summary>

- name: Install the CRD
  kubernetes.core.k8s:
    state: present
    apply: true
    src: files/widget-crd.yaml

- name: Wait for the CRD to be Established
  kubernetes.core.k8s_info:
    api_version: apiextensions.k8s.io/v1
    kind: CustomResourceDefinition
    name: widgets.example.com
    wait: true
    wait_condition:
      type: Established
      status: "True"
    wait_timeout: 60

- name: Create the Widget CR
  kubernetes.core.k8s:
    state: present
    apply: true
    definition:
      apiVersion: example.com/v1
      kind: Widget
      metadata:
        name: demo-widget
        namespace: demo
      spec:
        size: large

- name: Wait until the operator marks the Widget Ready
  kubernetes.core.k8s_info:
    api_version: example.com/v1
    kind: Widget
    namespace: demo
    name: demo-widget
  register: widget
  until: (widget.resources[0].status.phase | default('')) == 'Ready'
  retries: 30
  delay: 10

Why: a CRD must be Established before its CRs will register, so you gate on wait_condition; the CR’s custom status.phase isn’t a built-in Kubernetes condition, so you fall back to until / retries polling. </details>

Q&A — 13 Questions

Q1. Does kubernetes.core.k8s use kubectl under the hood? No. It uses the kubernetes Python client to talk directly to the K8s API. No kubectl binary required on either control node or target.

Q2. What’s the difference between state: present and apply: true? state: present creates if missing, updates if exists (client-side merge). state: present + apply: true uses server-side apply, which is the modern equivalent of kubectl apply and tracks field ownership.

Q3. Should I use Helm or templated manifests? Helm for distributed packages (Prometheus, ArgoCD, ingress controllers) — anyone with Helm can install them. Templated manifests for your own apps where you don’t need Helm’s chart distribution model.

Q4. Can I use Ansible and ArgoCD on the same cluster? Yes — and you should. Ansible manages cluster baseline (namespaces, RBAC, ingress controllers, ArgoCD itself). ArgoCD manages app delivery (your workloads). They don’t fight if you draw the line clearly.

Q5. What’s the difference between k8s_drain and kubectl drain? None functionally. k8s_drain is the Ansible wrapper. Use it when you want drain-then-do-something workflows in a playbook, e.g. drain a node, run package upgrades, uncordon.

Q6. How do I roll back a Helm release? kubernetes.core.helm with release_state: rolled_back and a revision: parameter. Use helm_info to find prior revisions.

Q7. Can I template a manifest that includes secrets? Yes — pull values from Ansible Vault ({{ vault_db_password }}) and render into a Secret resource. The rendered manifest is in-memory only (Ansible doesn’t write it to disk unless you do).

Q8. What if a CRD doesn’t exist when I try to apply a CR? k8s will fail. Apply the CRD first (Helm, k8s with the CRD definition, or operator-installed), wait for it (k8s_info with until), then apply the CR.

Q9. How do I run a one-off Job and wait for it to complete? Apply the Job manifest with state: present, then k8s_info with until: status.succeeded == 1 and a generous retries/delay. There’s no “wait for Job” shorthand.

Q10. Can I use Ansible to provision the cluster itself? Indirectly — use Ansible to call Terraform/Pulumi (or community.general.terraform module) which creates the cluster, then use kubernetes.core to manage objects inside it. Don’t try to write “create EKS cluster” in pure Ansible — there’s no module for it (use Terraform).

Q11. What’s the right way to manage ConfigMaps with binary data? Use kubernetes.core.k8s with definition.binaryData.<key>: "{{ lookup('file', 'path') | b64encode }}". Don’t use data: for binary — it’s UTF-8 only.

Q12. How does Ansible handle K8s API rate limiting? The Python client respects the API server’s rate-limit responses. For high-volume operations (apply 1000 manifests), use forks: 1 and serial: 1 — parallel Ansible against the same cluster trips the rate limiter.

Q13. Should I use the --dry-run=server mode? Yes — kubernetes.core.k8s supports dry_run: true (server-side dry-run) which validates against admission controllers without actually applying. Combine with --check for full preview.

Quick Check

1. Which collection ships the k8s module?
2. What does apply: true do?
3. What’s the equivalent of kubectl get pods in Ansible?
4. How do you scale a deployment to 5 replicas?
5. What’s the recommended pattern for multi-cluster plays?
6. How do you wait for a Helm release to be ready?
7. What’s the difference between state: replaced and state: patched?
8. When is delegate_to: localhost required for K8s plays?

Exercise

Build a complete role cluster_baseline that:

1. Creates standard namespaces (monitoring, logging, ingress-nginx, cert-manager, argocd).
2. Applies baseline NetworkPolicies (default-deny, allow-monitoring).
3. Installs cert-manager via Helm with a Let’s Encrypt ClusterIssuer.
4. Installs ingress-nginx via Helm with cloud-specific load balancer annotations (parameterize by cloud_provider: aws|gcp|azure).
5. Installs argocd via Helm with an admin OIDC integration.
6. Applies a baseline ResourceQuota and LimitRange to every namespace.
7. Includes a validate.yml task list that confirms each component is healthy.

Test it on a kind cluster locally, then deploy it to two real cloud clusters (one EKS, one GKE) and confirm the same role bootstraps both with only the cloud_provider var changing.

Cert Mapping

• EX374 — K8s automation is a major component. Expect to bootstrap a cluster baseline, install Helm releases, and apply manifests under time pressure.
• CKA / CKAD — Not directly tested, but the K8s concepts (namespaces, RBAC, custom resources) overlap heavily.
• CKS — RBAC and admission policy management via k8s is a common automation target.

Glossary

• kubeconfig — A YAML file containing cluster, user, and context entries. The standard auth mechanism for kubectl and Ansible.
• Context — A named pairing of (cluster, user, namespace). kubectl config use-context switches between them.
• CRD — Custom Resource Definition. Extends the K8s API with new resource types.
• CR — Custom Resource. An instance of a CRD.
• Server-side apply (SSA) — The modern kubectl apply mode where the API server tracks field ownership.
• Helm release — A named installation of a Helm chart in a cluster.
• Operator pattern — Custom controller + CRD that manages a complex application.
• GitOps — A delivery model where the desired state lives in Git and a controller (ArgoCD/Flux) reconciles cluster state to match.
• kubernetes.core — The Ansible collection whose modules speak to the Kubernetes API (k8s, k8s_info, helm, and friends). Formerly named community.kubernetes.
• FQCN (fully-qualified collection name) — The full namespace.collection.module path, e.g. kubernetes.core.k8s. Preferred over a bare k8s so Ansible always resolves the intended module.
• k8s module — The workhorse that creates, patches, replaces, and deletes any Kubernetes object.
• k8s_info module — The read-only query module; the Kubernetes equivalent of gathering facts.
• Field manager — The name the API server records as the owner of each field under server-side apply. A conflict arises when two managers claim the same field.
• apply: true — Send the object as a server-side apply request so the API server tracks field ownership; the modern kubectl apply behaviour.
• force: true — Replace the whole object (PUT) instead of merging — overwrites out-of-band changes and other managers’ fields.
• wait_condition — A block that waits for a named status condition (type / status / reason); the clean way to wait on CRDs and cluster-scoped objects.
• Idempotence — Applying the same desired state repeatedly converges to the same result; not the same thing as “never reports changed.”
• delegate_to: localhost — Runs a task on the control node instead of the inventory host; required for K8s plays because API calls originate from the control node, not the cluster.
• In-cluster auth / ServiceAccount token — When Ansible runs inside a pod, it authenticates with the pod’s mounted ServiceAccount token instead of a kubeconfig.
• OIDC / IRSA / Workload Identity — Cloud mechanisms that hand a workload a short-lived, auto-rotated token instead of a static credential.
• Helm chart / repository / revision — A versioned, templated package of manifests / an index of downloadable charts / a numbered point in a release’s history you can roll back to.
• values — The parameters passed to a chart’s templates at install/upgrade time (image tags, replica counts, ingress hosts, secrets).
• Finalizer — A key on an object that blocks its deletion until a controller clears it; the usual cause of a namespace stuck Terminating.
• PodDisruptionBudget (PDB) — A policy capping how many pods of a workload may be voluntarily evicted at once; k8s_drain respects it.
• Cordon / drain / uncordon — Mark a node unschedulable / evict its pods / make it schedulable again — the node-maintenance cycle.

Next Steps

You can now drive Kubernetes from Ansible — manifests, Helm releases, custom resources, multi-cluster — and integrate cleanly with GitOps tools. The next lesson covers Ansible for containers (Docker and Podman): the community.docker and containers.podman collections, Compose-style multi-container deployments, container image building, registry pushing, and the patterns that let Ansible drive container hosts the way it drives any other infrastructure.