Ansible Interview Questions and Answers

Answer

Agentless means there is no persistent daemon to install, patch and monitor on the managed node. It does not mean nothing runs there. For a normal task, the controller opens an SSH session, copies the module (a self-contained Python payload) into a temporary directory under the remote user's home, usually ~/.ansible/tmp/ansible-tmp-<timestamp>, executes it, reads a single JSON document off stdout, and deletes the temp directory.

So a Linux managed node needs three things: SSH reachability, credentials that work non-interactively, and a Python interpreter for anything except the raw and script modules. Windows nodes are different, they use WinRM or PSRP with PowerShell modules rather than Python. The controller itself must be Linux, macOS or WSL, native Windows is not a supported controller.

The classic first-day failure is 'MODULE FAILURE ... /usr/bin/python: not found' on a minimal cloud image, which is why bootstrap plays use ansible.builtin.raw to install python3 before anything else touches the host. Two more real-world gotchas: password-based SSH auth fails with 'you must install the sshpass program' unless sshpass is on the controller, and on SELinux hosts file-related modules need the SELinux Python bindings present or they refuse to set contexts. Interviewers ask this to see whether you understand that agentless is a deployment convenience, not magic, and that the node still has hard prerequisites you have to plan for during onboarding.

# inventory.ini
[web]
web01.example.com
web02.example.com ansible_port=2222

[web:vars]
ansible_user=deploy
ansible_ssh_private_key_file=~/.ssh/deploy.pem

# Prove SSH and Python are both working
# ansible web -i inventory.ini -m ansible.builtin.ping

# Bootstrap a node that has no Python yet (raw needs no interpreter)
# ansible web -i inventory.ini -b -m ansible.builtin.raw \
#   -a 'apt-get update && apt-get install -y python3'

Answer

Ansible does interpreter discovery at connection time. The default INTERPRETER_PYTHON setting is 'auto', which means the controller inspects the distribution and release reported by the node and picks the interpreter that distribution ships, for example /usr/bin/python3 on modern Ubuntu and /usr/libexec/platform-python or /usr/bin/python3 on RHEL family systems. The result lands in the discovered_interpreter_python fact and is reused for the rest of the run.

If the distribution is unknown, you get the noisy warning about no default interpreter being available and a fallback path. You override it by setting ansible_python_interpreter, per host, per group, or globally, and this matters more often than beginners expect. The common production case is a node where the application runs inside a virtualenv, and someone writes an ansible.builtin.pip task without specifying the virtualenv, so packages install into the system interpreter and the app never sees them.

Another case is a hardened image where the platform Python is deliberately minimal and lacks the libraries a module needs. Set interpreter_python = auto_silent in ansible.cfg once you have verified the discovery result across your fleet, it suppresses the deprecation-style warning without hiding real errors. Interviewers use this question to check whether you have actually debugged an 'ImportError' on a managed node rather than only read the getting-started guide.

# group_vars/legacy_rhel.yml
ansible_python_interpreter: /usr/bin/python3.9

# ansible.cfg
[defaults]
interpreter_python = auto_silent

# Installing into an application virtualenv, not the system Python
- name: Install app requirements into the venv
  ansible.builtin.pip:
    requirements: /srv/app/requirements.txt
    virtualenv: /srv/app/.venv
    virtualenv_command: /usr/bin/python3 -m venv

# Inspect what discovery actually chose
# ansible web -m ansible.builtin.setup -a 'filter=ansible_python*'

Answer

ansible-core is the engine: the ansible-playbook, ansible, ansible-galaxy, ansible-doc, ansible-config and ansible-inventory binaries, the plugin loader, the connection and strategy plugins, and exactly one bundled collection, ansible.builtin. The ansible package on PyPI is a batteries-included distribution that pins one ansible-core version plus several hundred community and vendor collections (community.general, community.crypto, amazon.aws, ansible.posix, cisco.ios and so on). The version numbers are deliberately different: the community package moves one major per core minor, so ansible 9 shipped ansible-core 2.16, ansible 10 shipped 2.17, ansible 11 shipped 2.18, and the pattern continues.

That is why 'ansible --version' prints a core version that looks nothing like what you pip installed. Which one you use is a real engineering decision. Teams that want a small, auditable dependency set install ansible-core and declare their collections explicitly in requirements.yml, which gives reproducible builds and lets you pin community.general to a version you have tested.

Teams that want everything available on a jump host install the full package. Recent ansible-core releases also keep raising the Python floor on both sides, the controller requirement has moved up through 3.10 and 3.11 while the managed-node floor moved to 3.8 or higher, so an upgrade is never purely cosmetic. Check the porting guide before bumping either, because collection majors also drop deprecated modules.

# Minimal, reproducible setup
pip install 'ansible-core==2.18.*'

# requirements.yml
collections:
  - name: community.general
    version: '>=9.0.0,<10.0.0'
  - name: ansible.posix
    version: 1.5.4
  - name: amazon.aws
    version: '>=8.0.0'

# Install into a project-local path so CI is hermetic
# ansible-galaxy collection install -r requirements.yml -p ./collections

# What am I actually running?
# ansible --version
# ansible-galaxy collection list

Answer

Since the 2.10 split, almost everything except a small core set lives in collections, and a module is addressed by its fully qualified collection name (FQCN) in the form namespace.collection.module. Short names like copy or yum still work because the plugin loader falls back to a search order: anything listed in the play's collections keyword, then ansible.legacy, then ansible.builtin. That fallback is exactly the problem. ansible.legacy also picks up local library/ directories, so a stray copy.py in your repo silently shadows the real module, and a short name can resolve differently depending on which collections are installed on the machine running the play.

Writing ansible.builtin.copy removes the ambiguity, makes grep and code review reliable, and is enforced by the ansible-lint fqcn rules in the production profile. It also matters for portability into execution environments, where the collection set is baked into an image and you want failures at build time rather than at 2 AM. Practical workflow: declare every collection you use in requirements.yml, install with ansible-galaxy collection install -r requirements.yml -p ./collections, set collections_path in ansible.cfg, and use FQCNs everywhere including in roles and handlers. When you inherit a legacy repo, ansible-lint can autofix most short names with the --fix flag, which is usually a single mechanical commit worth doing before you start real work on the codebase.

- name: Configure web tier
  hosts: web
  become: true
  tasks:
    - name: Drop the nginx config
      ansible.builtin.template:
        src: nginx.conf.j2
        dest: /etc/nginx/nginx.conf
        owner: root
        mode: '0644'
        validate: 'nginx -t -c %s'

    - name: Open the firewall (collection module, not core)
      ansible.posix.firewalld:
        service: https
        permanent: true
        state: enabled
        immediate: true

Answer

Both formats describe the same object model: hosts, groups, group children and variables. INI is terse and fine for a flat list, YAML wins the moment you need nested variables or structured data. In INI you write groups as [web], nested groups as [prod:children], and group variables under [web:vars], which forces every variable to be a flat scalar.

In YAML you get real dictionaries and lists, so a group variable can be a list of virtual hosts or a dict of tuning parameters without stuffing JSON into a string. Two implicit groups always exist: all, which contains every host, and ungrouped, which contains hosts that belong to no other group. Host ranges work in both formats, web[01:05].example.com expands to five hosts with zero padding preserved.

In real repositories the inventory is usually a directory rather than a file, so inventories/prod/hosts.yml sits next to group_vars/ and host_vars/, and you point -i at the directory. The command that saves you during interviews and outages is ansible-inventory: --graph shows group membership as a tree, --list dumps the resolved JSON, and --host prints every variable that resolved for one host including where precedence landed. Pair that with --limit patterns, which support wildcards, intersection with & and exclusion with !, for example --limit 'web:!web03.example.com'. Interviewers often hand you a broken inventory and watch whether you reach for ansible-inventory or start guessing.

# inventories/prod/hosts.yml
all:
  children:
    web:
      hosts:
        web[01:04].prod.internal:
      vars:
        nginx_worker_processes: auto
        vhosts:
          - {name: api, port: 8080}
          - {name: admin, port: 8081}
    db:
      hosts:
        db01.prod.internal:
          postgres_role: primary
        db02.prod.internal:
          postgres_role: replica
    prod:
      children:
        web:
        db:

# ansible-inventory -i inventories/prod --graph
# ansible-inventory -i inventories/prod --host db02.prod.internal

Answer

Idempotency means running the playbook a second time produces zero changed tasks because the system is already in the declared state. Well-written modules achieve this by reading current state first: ansible.builtin.user checks /etc/passwd before touching it, ansible.builtin.copy compares checksums, ansible.builtin.dnf asks the package database. ansible.builtin.command and ansible.builtin.shell cannot do that, because Ansible has no idea what your command does, so they unconditionally report changed. That is not a cosmetic problem.

Every changed result fires any notify attached to the task, so a shell task can restart nginx on every run forever, and your 'zero changes' smoke test in CI stops being a signal. There are three correct fixes, in order of preference. First, use a real module, there is almost always one.

Second, give command a guard: creates: /opt/app/.installed makes it skip entirely when the path exists, and removes: does the inverse. Third, when the command is genuinely a read-only probe, set changed_when: false, and when it signals status through exit codes or stdout, express that with failed_when and changed_when explicitly. Also remember that command does not go through a shell, so pipes, redirects, globs and environment expansion silently do not work, which is the single most common 'but it works when I paste it into bash' bug. ansible-lint flags both patterns with no-changed-when and command-instead-of-module, and reviewers at Red Hat-style shops treat those warnings as blocking.

# Bad: changed on every run, restarts nginx forever
- name: Build the asset bundle
  ansible.builtin.shell: npm run build
  notify: restart nginx

# Better: guarded and explicit
- name: Build the asset bundle
  ansible.builtin.command:
    cmd: npm run build
    chdir: /srv/app
    creates: /srv/app/dist/manifest.json
  notify: restart nginx

# Read-only probe: never a change
- name: Read the running schema version
  ansible.builtin.command: /usr/local/bin/appctl schema-version
  changed_when: false
  failed_when: schema.rc not in [0, 2]
  register: schema

Answer

First, config resolution: Ansible locates one ansible.cfg (it does not merge multiple files) and layers environment variables and command-line flags on top. Then it loads the inventory sources, applies group_vars and host_vars, and parses the playbook and any imported files at parse time. For each play it computes the target host list from the hosts pattern intersected with --limit, then runs the implicit setup task unless gather_facts is false.

Tasks then execute in order, and this is the part candidates get wrong: under the default linear strategy, Ansible runs task one on up to `forks` hosts in parallel, waits for every host to finish that task, and only then starts task two. The forks value (default 5) is a concurrency window, not a per-host thread pool, so a single slow host stalls the whole play at that task. Any host that fails is removed from the play for the remaining tasks, but the rest carry on unless you set any_errors_fatal or max_fail_percentage.

Notified handlers queue up and flush at the very end of the play, once, in the order they are defined in the handlers section. If serial is set, the play is chopped into batches and this entire sequence including handler flush repeats per batch. Finally the PLAY RECAP prints ok, changed, unreachable, failed, skipped, rescued and ignored per host. Knowing that unreachable and failed are different counters, and that rescued comes from block/rescue, is a small detail that signals real operational experience.

# See the batching and timing for yourself
# ANSIBLE_STDOUT_CALLBACK=yaml \
# ANSIBLE_CALLBACKS_ENABLED=profile_tasks,timer \
# ansible-playbook -i inventories/prod site.yml --forks 25 --diff

# PLAY RECAP output columns
# web01 : ok=24 changed=3 unreachable=0 failed=0 skipped=5 rescued=0 ignored=1
#   ok         = tasks that ran without change
#   changed    = tasks that modified state (these fire notify)
#   unreachable= SSH / connection failure, not a task failure
#   rescued    = failures caught by a rescue: block

Answer

The ansible binary (as opposed to ansible-playbook) runs a single module against an inventory pattern without writing a file. It is the right tool for three things: discovery, one-off operational commands, and emergency response. Discovery means questions like 'which of these 300 hosts is still on kernel 5.4', answered with a setup filter or a command plus a grep.

Operational means restarting a service across a tier or clearing a cache. Emergency means pushing a hotfix package during an incident when you do not have time for a PR. The syntax is ansible <pattern> -i <inventory> -m <fqcn> -a '<module args>' with -b for become, -f for forks, --limit to narrow, -C for check mode and -D for diff.

Free-form -a arguments are parsed as key=value pairs, so anything with spaces needs quoting, and complex structures need the JSON form. The limits are what interviewers want you to name: nothing is version controlled, nothing is reviewed, there is no record of what was run beyond your shell history, and it is trivially easy to fat-finger a pattern and hit prod instead of staging. Mature teams route ad-hoc work through AWX or Ansible Automation Platform job templates instead, so every run is logged against a user with an audit trail. The habit to build is running with --check --diff first, then re-running for real once the diff looks right.

# Which hosts still need a reboot after patching?
ansible all -i inventories/prod -b -m ansible.builtin.command \
  -a 'needs-restarting -r' --limit 'app:!app07' -f 30

# Fleet-wide fact query
ansible all -i inventories/prod -m ansible.builtin.setup \
  -a 'filter=ansible_kernel'

# Always dry-run a mutating ad-hoc first
ansible web -i inventories/prod -b -C -D \
  -m ansible.builtin.service -a 'name=nginx state=restarted'

Answer

Facts are the data Ansible collects about a managed node by running the ansible.builtin.setup module as an implicit first task in every play. They land in the ansible_facts dictionary and cover the OS family and version, network interfaces and IPs, CPU and memory, mounts, virtualisation type, and more. Because INJECT_FACTS_AS_VARS still defaults to true, you can read them either as ansible_facts['distribution'] or as the top-level ansible_distribution, and modern style guides prefer the namespaced form since the flat variables pollute the global namespace and can collide with your own.

Gathering is not free. On a large fleet the setup module is a full Python run plus network and mount enumeration on every host, and it can dominate playbook wall time. Three levers fix this.

First, gather_subset lets you request only what you need, for example ['!all', '!min', 'network'] skips the expensive hardware and mount probing. Second, gather_facts: false on plays that do not need facts at all, with an explicit setup task later if one role needs them. Third, fact caching persists the setup output to jsonfile or Redis for a configured timeout, so subsequent playbooks in the same window skip gathering entirely. Custom facts are the other half of the story: drop an INI or JSON file into /etc/ansible/facts.d/ on the node and it appears under ansible_local, which is how teams expose application-specific state (deployed version, tenant id, hardware tier) to their playbooks.

# ansible.cfg
[defaults]
gathering = smart
fact_caching = redis
fact_caching_connection = redis.internal:6379:0
fact_caching_timeout = 7200

# Play-level trimming
- name: Deploy application
  hosts: app
  gather_facts: true
  gather_subset:
    - '!all'
    - '!min'
    - network
    - distribution
  tasks:
    - name: Branch on OS family
      ansible.builtin.debug:
        msg: "{{ ansible_facts['distribution'] }} {{ ansible_facts['distribution_version'] }}"

    - name: Read a custom fact placed by the base role
      ansible.builtin.debug:
        msg: "{{ ansible_local.app.deploy.version | default('unknown') }}"

Answer

Ansible checks four locations in strict order and uses the first one it finds: the file named by the ANSIBLE_CONFIG environment variable, ./ansible.cfg in the current working directory, ~/.ansible.cfg, then /etc/ansible/ansible.cfg. The critical detail is that these are not merged. The first file wins completely, so a project ansible.cfg silently disables every setting you carefully wrote in ~/.ansible.cfg.

There is also a security behaviour worth knowing: Ansible ignores an ansible.cfg found in a world-writable current directory, because otherwise cd-ing into a shared temp directory would let anyone hijack your plugin paths, and it prints a warning when it does so. The settings that earn their keep in production are inventory (so you stop typing -i), forks (the default of 5 is far too low for any real fleet), host_key_checking (leave it on and manage known_hosts properly rather than turning it off), pipelining under [ssh_connection] which removes one SSH round trip per task, ssh_args with ControlMaster and ControlPersist for connection reuse, retry_files_enabled = false so you stop littering the repo, stdout_callback = yaml for readable multi-line output, and callbacks_enabled = profile_tasks when you are hunting slowness. Use ansible-config dump --only-changed to see exactly what differs from defaults on whatever machine you are on, and ansible-config list to discover the environment variable that corresponds to any setting, which is how you override things cleanly inside CI containers.

# ansible.cfg
[defaults]
inventory = inventories/prod
forks = 40
host_key_checking = True
retry_files_enabled = False
stdout_callback = yaml
callbacks_enabled = profile_tasks, timer
interpreter_python = auto_silent
collections_path = ./collections
roles_path = ./roles:./galaxy_roles

[ssh_connection]
pipelining = True
ssh_args = -o ControlMaster=auto -o ControlPersist=300s -o PreferredAuthentications=publickey
control_path = /tmp/ansible-%%h-%%p-%%r

# ansible-config dump --only-changed

Answer

become: true tells Ansible to escalate privileges for a task or play. become_method selects the mechanism (sudo by default, plus su, doas, pbrun, pfexec, and runas on Windows), become_user names the target account (root by default), and become_flags passes extra arguments. Passwords come from -K / --ask-become-pass interactively, or from the ansible_become_password variable, which must live in a vault file and never in plaintext group_vars. The subtlety worth knowing, and a frequent interview question, is escalating from one unprivileged user to a different unprivileged user, for example connecting as deploy and running a task as postgres.

Ansible writes the module payload to a temp directory owned by the connecting user, and the target user cannot read it, so you get 'Failed to set permissions on the temporary files Ansible needs to create when becoming an unprivileged user'. There are three fixes. The clean one is installing the acl package on the managed node so Ansible can grant a POSIX ACL on the temp files.

The second is going through root, connect as deploy, become root, and let root become the target user. The third, allow_world_readable_tmpfiles = true, makes the temp files readable by everyone for the duration of the task, and should be treated as a last resort because module arguments can contain secrets. Also remember that become and pipelining interact: sudo configured with requiretty breaks pipelining, and that failure surfaces as a confusing tty error rather than an obvious permissions message.

- name: Database maintenance
  hosts: db
  become: true            # escalate to root for the play
  tasks:
    - name: Ensure acl is present so unprivileged become works
      ansible.builtin.package:
        name: acl
        state: present

    - name: Run a psql query as the postgres user
      become_user: postgres
      community.postgresql.postgresql_query:
        db: appdb
        query: SELECT count(*) FROM pg_stat_activity;
      register: activity
      changed_when: false

# group_vars/db/vault.yml (ansible-vault encrypted)
# ansible_become_password: "..."

Answer

A handler is a task that only runs if something notified it, and only at the end of the play. Any task that reports changed and carries notify: <handler name> queues that handler. Ansible deduplicates by name, so ten config templates all notifying 'restart nginx' produce exactly one restart.

Two behaviours surprise people. First, handlers run in the order they are defined in the handlers section, not the order they were notified, which matters if you have 'reload systemd' and 'restart app' and the reload must come first. Second, and this is the real production trap, if any task in the play fails after the notification, the play aborts and the queued handlers never run.

So a run that rewrote nginx.conf but failed on a later unrelated task leaves the new config on disk with the old config still loaded in memory, and the next run reports ok for the template task (the file is already correct), so it never notifies again and the restart never happens. The system stays broken silently until someone reboots. The fixes are force_handlers: true at play level or --force-handlers on the command line, which runs queued handlers even after a failure, and meta: flush_handlers placed immediately after a critical config block to force an early flush.

Use listen: to have several handlers respond to one logical event, which is cleaner than notifying three handler names from every task. With serial, handlers flush at the end of each batch, not once at the end of the whole play.

- name: Configure the app tier
  hosts: app
  become: true
  force_handlers: true
  tasks:
    - name: Install unit file
      ansible.builtin.template:
        src: app.service.j2
        dest: /etc/systemd/system/app.service
        mode: '0644'
      notify: app config changed

    - name: Flush before the smoke test so we test the new config
      ansible.builtin.meta: flush_handlers

    - name: Smoke test
      ansible.builtin.uri:
        url: http://127.0.0.1:8080/healthz
        status_code: 200

  handlers:
    - name: Reload systemd
      listen: app config changed
      ansible.builtin.systemd_service:
        daemon_reload: true

    - name: Restart app
      listen: app config changed
      ansible.builtin.systemd_service:
        name: app
        state: restarted

Answer

Ansible documents twenty-two precedence levels, but a working engineer needs to know roughly eight of them cold, lowest to highest. Role defaults (roles/x/defaults/main.yml) are the floor, and that is deliberate: defaults exist to be overridden, so every tunable in a role belongs there. Above that come inventory file variables, then inventory group_vars/all, then inventory group_vars/<group>, then playbook-adjacent group_vars with the same ordering, then host_vars (which always beat group vars for the same host), then host facts and cacheable set_facts, then play vars and vars_files, then role vars (roles/x/vars/main.yml, which are hard to override and should be reserved for role internals), then block and task vars, then include_vars, then set_fact, and finally extra vars passed with -e, which beat absolutely everything and cannot be overridden from inside the play.

Two rules resolve most confusion. Child groups beat parent groups, so a variable in group_vars/prod_web overrides the same key in group_vars/all. And when two groups at the same depth define the same variable, the tie is broken alphabetically unless you set ansible_group_priority on the group, which is the documented fix for the classic 'why does the value from group_a keep winning' question. The practical convention is: put every tunable in role defaults, override per environment in group_vars, override per machine in host_vars, and reserve -e for CI pipelines and break-glass runs.

# roles/nginx/defaults/main.yml   (lowest, meant to be overridden)
nginx_worker_connections: 1024

# inventories/prod/group_vars/all.yml
nginx_worker_connections: 4096

# inventories/prod/group_vars/edge.yml  (child group wins over all)
nginx_worker_connections: 16384

# inventories/prod/host_vars/edge03.yml (host beats group)
nginx_worker_connections: 32768

# Break-glass, beats everything above
# ansible-playbook site.yml -e nginx_worker_connections=65536

# Prove which value won for one host
# ansible edge03 -m ansible.builtin.debug -a 'var=nginx_worker_connections'

Answer

Ansible looks for group_vars/ and host_vars/ in two places: next to the inventory source, and next to the playbook. Inventory-adjacent is the right choice for anything environment specific, because it keeps prod and staging values physically separate and makes the blast radius of an edit obvious in a pull request. A directory named group_vars/web is treated as a group_vars file too, and every YAML file inside it is loaded and merged alphabetically, which is the standard way to split plain configuration from secrets: group_vars/web/main.yml holds the readable values and group_vars/web/vault.yml holds the ansible-vault encrypted ones.

A convention that pays off immediately is prefixing every encrypted key with vault_ and referencing it from an unencrypted variable, so db_password: '{{ vault_db_password }}' lives in main.yml while only the vault_ key is inside the encrypted file. Now grep still tells you which variables exist and where they are consumed, even though the value is unreadable. Filenames must match group and host names exactly, and this bites people with hostnames that contain dots: host_vars/web01.prod.internal.yml only matches if the inventory_hostname is written the same way.

Keep an inventories/<env>/ directory per environment holding hosts.yml, group_vars/ and host_vars/, share roles and playbooks at the repository root, and never let a staging value leak into a file that prod also reads. The one directory that deserves suspicion in review is group_vars/all, because everything you put there applies to every host in every play.

inventories/
  prod/
    hosts.yml
    group_vars/
      all/
        main.yml
        vault.yml        # ansible-vault encrypted
      web/
        main.yml
      db/
        main.yml
        vault.yml
    host_vars/
      db01.prod.internal.yml
  staging/
    hosts.yml
    group_vars/...
roles/
playbooks/
  site.yml

# group_vars/db/main.yml
postgres_password: "{{ vault_postgres_password }}"
postgres_max_connections: 400

# group_vars/db/vault.yml (encrypted)
# vault_postgres_password: hunter2

Answer

loop is the current keyword and takes a plain list. The old with_* family (with_items, with_dict, with_subelements, with_fileglob and friends) still works because each one maps to a lookup plugin, but it is deprecated in style guides and ansible-lint flags it, so new code uses loop plus a filter or an explicit lookup/query call. The mechanical difference that catches people: with_items flattens one level of nested lists, loop does not, so porting a with_items over a list of lists needs an explicit | flatten.

Inside the loop the current element is item, and register captures a results list rather than a single value, which is why you often iterate the registered results in a following task. loop_control is the quality-of-life layer. label controls what is printed per iteration, and it matters enormously when looping over dictionaries containing passwords or long JSON, because without it the whole structure is dumped into the log. index_var exposes the zero-based position, pause inserts a delay between iterations (useful for rate-limited APIs), loop_var renames item so nested include_tasks loops do not shadow the outer variable, and extended: true adds ansible_loop with first, last, index and length. Separately, until with retries and delay is the retry loop: the task runs, the until expression is evaluated against the registered result, and it repeats up to retries times. That is how you wait for a service to come healthy without shelling out to a bash while loop.

- name: Create service accounts
  ansible.builtin.user:
    name: "{{ item.name }}"
    groups: "{{ item.groups | default(omit) }}"
    password: "{{ item.hash }}"
    state: present
  loop: "{{ service_accounts }}"
  loop_control:
    label: "{{ item.name }}"      # never log item.hash
    index_var: idx

- name: Wait for the app to report healthy
  ansible.builtin.uri:
    url: http://127.0.0.1:8080/healthz
    status_code: 200
  register: health
  until: health.status == 200
  retries: 30
  delay: 2

- name: Report any user task that changed
  ansible.builtin.debug:
    msg: "created {{ item.item.name }}"
  loop: "{{ users.results | selectattr('changed') | list }}"
  loop_control:
    label: "{{ item.item.name }}"

Answer

The when keyword is already evaluated as a Jinja2 expression, so wrapping it in {{ }} means Ansible templates the string first and then evaluates the result, which usually still works but is redundant and occasionally changes the semantics. ansible-lint flags it with the jinja and no-jinja-when rules. Write when: app_enabled, not when: '{{ app_enabled }}'. A list under when is an implicit AND, which reads far better than chaining with and, and you can nest an or inside one list element with parentheses.

Robust conditionals need three defensive habits. First, guard for existence with is defined, is not defined, or the default filter, because a bare reference to an undefined variable raises 'The task includes an option with an undefined variable' at runtime rather than skipping. Second, be explicit about booleans: YAML happily gives you the string 'false' from an extra var passed as -e enabled=false, and a non-empty string is truthy, so use | bool to coerce.

Recent ansible-core releases have made this stricter, the rewritten templating engine no longer quietly accepts a non-boolean result from a conditional, so patterns that worked for years can now raise an error on upgrade, which is one of the most common breakages teams hit when moving forward a core version. Third, prefer facts over inference: when: ansible_facts['os_family'] == 'RedHat' is clearer and safer than sniffing for the presence of a file. Also remember that when on a block applies to every task inside it, and that when combined with loop is evaluated per item, not once for the whole task.

- name: Install the RHEL-only agent
  ansible.builtin.dnf:
    name: monitoring-agent
    state: present
  when:
    - ansible_facts['os_family'] == 'RedHat'
    - ansible_facts['distribution_major_version'] | int >= 8
    - monitoring_enabled | default(false) | bool

- name: Skip cleanly when the variable was never set
  ansible.builtin.debug:
    msg: "tenant is {{ tenant_id }}"
  when: tenant_id is defined and tenant_id | length > 0

- name: Per-item conditional inside a loop
  ansible.builtin.file:
    path: "{{ item.path }}"
    state: directory
    mode: '0750'
  loop: "{{ app_dirs }}"
  when: item.create | default(true) | bool

Answer

copy ships a file byte for byte; template renders it through Jinja2 on the controller first, substituting variables, facts, filters and control structures, then transfers the result. Any config that varies per host or per environment should be a .j2 template, and any file that is genuinely static should be copy, because a template that contains no Jinja is just a slower copy with more moving parts. Both modules compare checksums, so both are idempotent and only report changed when the rendered content or the file attributes actually differ.

The options that matter in production are validate, backup and mode. validate runs a command against a temporary copy of the rendered file before it replaces the real one, with %s substituted for the temp path, so 'nginx -t -c %s' or 'visudo -cf %s' or 'sshd -t -f %s' turns a syntax error into a failed task instead of a service that will not start on the next restart. Skipping validate on sshd_config is how people lock themselves out of a fleet. backup: true keeps a timestamped copy next to the original, which is your cheap rollback. Always set mode explicitly, because the default depends on the remote umask and you will eventually ship a 0644 file containing a password.

Two rendering flags are worth knowing: trim_blocks and lstrip_blocks clean up the blank lines that Jinja control structures leave behind, and they can be set per task rather than globally. Finally, --diff shows the rendered delta before you commit to it, and template plus --check --diff is the safest review loop that exists in Ansible.

- name: Render sshd_config safely
  ansible.builtin.template:
    src: sshd_config.j2
    dest: /etc/ssh/sshd_config
    owner: root
    group: root
    mode: '0600'
    backup: true
    validate: '/usr/sbin/sshd -t -f %s'
    trim_blocks: true
    lstrip_blocks: true
  notify: restart sshd

# templates/sshd_config.j2
# Port {{ ssh_port | default(22) }}
# PermitRootLogin {{ 'no' if harden_ssh | bool else 'prohibit-password' }}
# {% for net in ssh_allowed_networks %}
# AllowUsers *@{{ net }}
# {% endfor %}

# Review before applying:
# ansible-playbook site.yml --check --diff --limit bastion01

Answer

ansible-vault encrypts files (or single values) with AES-256 using a key derived from a password, and Ansible transparently decrypts them at runtime when the right password is supplied. The commands you use are create, edit, view, encrypt, decrypt, rekey, and encrypt_string. Passwords come from --ask-vault-pass, --vault-password-file pointing at a file, or a vault id, and a vault id can point at an executable script rather than a static file, which is how you fetch the password from AWS Secrets Manager or HashiCorp Vault inside CI instead of storing it on disk.

Multiple vault ids let different teams hold different keys: --vault-id prod@prompt --vault-id dev@dev-pass.txt, with the label recorded in the file header so Ansible picks the right key automatically. The workflow problem with whole-file encryption is that diffs are useless in code review, since every change looks like a wall of new ciphertext. Two things fix that.

First, the group_vars/<group>/vault.yml split, so only the secrets file is opaque and everything else diffs normally. Second, encrypt_string for one-off values, which produces an inline !vault block you can paste into an otherwise readable YAML file. You can also register a vault diff helper in .gitattributes so git diff decrypts on the fly for people who hold the key. Two rules for production: never commit the vault password itself, and rotate with rekey whenever someone leaves the team, because everyone who ever cloned the repo still has the ciphertext.

# Create and edit an encrypted vars file
ansible-vault create inventories/prod/group_vars/db/vault.yml
ansible-vault edit  inventories/prod/group_vars/db/vault.yml

# Encrypt one value for inline use
ansible-vault encrypt_string --vault-id prod@prompt \
  'S3cretP@ss' --name 'vault_postgres_password'

# Rotate the key across every encrypted file
ansible-vault rekey inventories/prod/group_vars/*/vault.yml

# CI: fetch the password from a secret manager, never from disk
# cat vault-pass.sh
#   #!/bin/sh
#   aws secretsmanager get-secret-value --secret-id ansible/prod \
#     --query SecretString --output text
# ansible-playbook site.yml --vault-id prod@./vault-pass.sh

Answer

import_* is static and include_* is dynamic, and every practical difference follows from that. import_tasks, import_role and import_playbook are processed at parse time: Ansible reads the referenced file before the play starts and splices its tasks into the play as if you had typed them inline. include_tasks and include_role are processed at runtime: the file is read only when execution reaches that line. Consequences, in the order interviewers ask about them. Tags: a tag on an import propagates down to every imported task, so --tags nginx picks them up, but a tag on an include applies only to the include statement itself, so the included tasks are skipped entirely unless they carry their own tags.

This is the single most common 'my tags do not work' bug. Loops: you cannot loop an import, because the loop variable does not exist at parse time; include_tasks loops fine and is the standard way to run a task file once per item. Conditionals: a when on an import is copied onto every imported task individually, while a when on an include is evaluated once and skips the whole file.

Variables: an import cannot reference a variable in its filename if that variable is only known at runtime; an include can. Debugging: imports show every task in --list-tasks output, includes show only the include statement, which makes imports easier to audit. Rule of thumb: default to import for structure you know at parse time, and reach for include only when you genuinely need runtime behaviour like looping over a list of task files.

# Static: tags reach every task inside common.yml
- name: Base configuration
  ansible.builtin.import_tasks: tasks/common.yml
  tags: [base]

# Dynamic: needed because we loop and use a runtime variable
- name: Configure each tenant
  ansible.builtin.include_tasks: "tasks/tenant_{{ item.tier }}.yml"
  loop: "{{ tenants }}"
  loop_control:
    loop_var: item
    label: "{{ item.name }}"

# Tag trap: this runs NOTHING with --tags deploy
- ansible.builtin.include_tasks: tasks/deploy.yml
  tags: [deploy]
# Fix: apply_tags or switch to import_tasks
- ansible.builtin.include_tasks:
    file: tasks/deploy.yml
    apply:
      tags: [deploy]
  tags: [deploy]

Answer

A role is a directory with a fixed set of well-known subdirectories, each auto-loaded so you never write a path. tasks/main.yml is the entry point, handlers/main.yml holds handlers, defaults/main.yml holds the lowest-precedence tunables, vars/main.yml holds high-precedence role internals, files/ holds static files for copy, templates/ holds .j2 files, meta/main.yml holds metadata and dependencies, and tests/ or molecule/ holds the test scenarios. ansible-galaxy init <name> scaffolds all of it. There are three ways to pull one role into another. Listing it under dependencies in meta/main.yml makes it run before the dependent role every time, always, with no conditional control, and Ansible deduplicates it by default (allow_duplicates: true overrides that) so a role listed as a dependency by five other roles still runs once. import_role splices the role in statically at the point you write it, so tags and ordering behave predictably. include_role pulls it in at runtime, so you can loop it, gate it with a when that skips the whole role, and pass different variables per invocation.

Prefer meta dependencies only for genuine hard prerequisites that every consumer needs, for example a 'common' role that installs the package repositories. Prefer import_role for ordinary composition inside a role, because the ordering is visible in the file. Prefer include_role when the decision to run the role is data driven. The failure mode to name in an interview is meta dependency chains: they are invisible in the playbook, they run before your role's own tasks including its pre-checks, and deduplication means a dependency that needs different variables in two places silently runs with only the first set.

roles/webapp/
  defaults/main.yml
  vars/main.yml
  tasks/main.yml
  handlers/main.yml
  templates/app.conf.j2
  files/ca-bundle.crt
  meta/main.yml
  molecule/default/

# roles/webapp/meta/main.yml
dependencies:
  - role: common
    vars:
      common_repos: [epel]

# Data-driven composition inside tasks/main.yml
- name: Apply the right TLS role for this environment
  ansible.builtin.include_role:
    name: "tls_{{ tls_provider }}"
  when: tls_enabled | bool

- name: Hard prerequisite, ordering visible in the file
  ansible.builtin.import_role:
    name: firewall

Answer

block groups tasks so that keywords like when, become, tags and ignore_errors apply to all of them at once, and it enables structured error handling. If any task in the block fails, execution jumps to rescue, and whatever happens the always section runs afterwards. Inside rescue you get ansible_failed_task (the task object that failed) and ansible_failed_result (its result dict), which is what you use to log something useful rather than a generic 'deployment failed'.

A host that goes through rescue successfully is reported as rescued in the PLAY RECAP and is not counted as failed, which means a rescue that swallows a real problem hides it from your CI gate. ignore_errors: true is a blunter tool: it marks the task as ignored and moves to the very next task, with no cleanup path and no signal that anything went wrong beyond the ignored counter. Use it only for genuinely optional operations, like a best-effort cache warm. When you need conditional tolerance, failed_when is better than ignore_errors because it encodes the actual success criteria, for example failed_when: result.rc not in [0, 2] for a command whose exit code 2 means 'nothing to do'.

The production pattern interviewers look for is a deploy block with a rescue that rolls back to the previous release symlink and an always that re-enables the host in the load balancer, so a failed deploy never leaves a node drained and broken. Note that rescue does not catch unreachable hosts, only task failures, and that block does not create a variable scope, so set_fact inside a block is visible after it.

- name: Deploy with rollback
  block:
    - name: Drain from the load balancer
      community.general.haproxy:
        state: disabled
        host: "{{ inventory_hostname }}"
        backend: app
      delegate_to: "{{ groups['lb'][0] }}"

    - name: Switch the release symlink
      ansible.builtin.file:
        src: "/srv/releases/{{ release_id }}"
        dest: /srv/current
        state: link
      notify: restart app

    - name: Smoke test
      ansible.builtin.uri:
        url: http://127.0.0.1:8080/healthz
      retries: 10
      delay: 3
      register: probe
      until: probe.status == 200

  rescue:
    - name: Roll back to the previous release
      ansible.builtin.file:
        src: "{{ previous_release_path }}"
        dest: /srv/current
        state: link
    - name: Fail loudly with the real cause
      ansible.builtin.fail:
        msg: "rollback done, failed at: {{ ansible_failed_task.name }}"

  always:
    - name: Always re-enable in the load balancer
      community.general.haproxy:
        state: enabled
        host: "{{ inventory_hostname }}"
        backend: app
      delegate_to: "{{ groups['lb'][0] }}"

Answer

delegate_to runs a task on a different host from the one currently being processed, while keeping all the current host's variables in scope. That combination is the point: when Ansible is processing web03 and you delegate an haproxy task to the load balancer, the task body can still reference inventory_hostname and it resolves to web03, so you drain exactly the right backend. Typical uses are load-balancer drain and enable, DNS record updates, creating a monitoring silence, and calling an external API from the controller with delegate_to: localhost (the older local_action is the same thing with different syntax). run_once makes a task execute on only the first host of the current batch, with the result copied to all hosts in the play so subsequent when checks behave consistently.

It is how you run a database migration exactly once during a rolling deploy of ten app servers. Combining run_once with delegate_to: localhost is the standard 'do this once, on the controller' idiom. The subtle one is delegate_facts.

By default, facts gathered or set during a delegated task are attributed to the original host, not the delegate. Setting delegate_facts: true stores them against the delegate instead, which is what you want when you deliberately gather facts about a machine that is not in the current play, for example querying the database primary from within the app play. Two gotchas worth mentioning: delegate_to with serial delegates per batch, and run_once inside a serial play runs once per batch, not once overall, which surprises people writing migrations.

- name: Rolling app deploy
  hosts: app
  serial: 2
  tasks:
    - name: Run schema migration exactly once, from the controller
      ansible.builtin.command: /usr/local/bin/migrate --up
      delegate_to: localhost
      run_once: true
      changed_when: true

    - name: Drain this host from HAProxy (task runs on the LB)
      community.general.haproxy:
        state: disabled
        backend: app_pool
        host: "{{ inventory_hostname }}"
        wait: true
      delegate_to: "{{ groups['lb'][0] }}"

    - name: Gather facts about the DB primary, stored against the DB host
      ansible.builtin.setup:
        gather_subset: ['!all', 'min']
      delegate_to: "{{ groups['db_primary'][0] }}"
      delegate_facts: true
      run_once: true

Answer

serial converts a play into a sequence of batches. serial: 2 processes two hosts at a time, serial: '25%' uses a proportion, and a list like [1, 5, '30%'] gives you a canary ramp: one host first, then five, then thirty percent of what remains. Every batch runs the whole play including its handler flush before the next batch starts, which is what makes a rolling deploy actually rolling rather than a big-bang change with pauses. By default a failing host is simply dropped from the remaining tasks and the other hosts continue, and Ansible only aborts the play when every host in a batch has failed, which is almost never the policy you want in production. max_fail_percentage sets the threshold: if the percentage of failed hosts in a batch exceeds that number, the play stops before the next batch, so a bad artifact hits one canary rather than the entire fleet.

Note it is strictly greater than, so max_fail_percentage: 0 with serial: 1 means any single failure halts the play, which is the standard canary setting. any_errors_fatal: true is the strictest option: the moment any host fails a task, the play stops for all hosts at the end of that task. Use it for orchestration where partial application is worse than no application, for example a coordinated cluster upgrade. The combination interviewers want to hear is serial with a canary ramp, max_fail_percentage: 0 on the first batch, a real health check inside the batch (not just 'service started'), and force_handlers so a mid-batch failure does not leave a config written but not loaded.

- name: Canary then ramp
  hosts: app
  serial:
    - 1        # canary
    - 5
    - '30%'
  max_fail_percentage: 0
  force_handlers: true
  tasks:
    - name: Deploy the new artifact
      ansible.builtin.unarchive:
        src: "{{ artifact_url }}"
        dest: "/srv/releases/{{ release_id }}"
        remote_src: true
      notify: restart app

    - ansible.builtin.meta: flush_handlers

    - name: Real health check, not just 'service is running'
      ansible.builtin.uri:
        url: "http://{{ ansible_host }}:8080/healthz"
        status_code: 200
        return_content: true
      register: hc
      until: hc.json.version == release_id
      retries: 20
      delay: 3

Answer

The strategy plugin decides how Ansible schedules tasks across hosts. linear, the default, is a lockstep barrier: every host runs task N, Ansible waits for the slowest one, then everyone starts task N+1. That barrier is what makes playbooks predictable, since you know all hosts are configured before the handler flush, but it also means one host stuck on a slow package download holds up hundreds of others. free removes the barrier entirely: each host races through the whole task list as fast as it can, so total wall time is roughly the slowest single host rather than the sum of the slowest per task. The cost is that ordering guarantees across hosts disappear, so anything that assumes 'all web servers are updated before we touch the LB' breaks. host_pinned is the middle ground: it assigns hosts to worker slots and lets each worker complete a host fully before picking up the next, which is useful when a single host must be finished before another starts on the same worker. forks is orthogonal and controls how many hosts are worked on concurrently, defaulting to 5, which is the single most impactful number in most Ansible installations.

Raising it to 25 or 50 is usually free, but it is bounded by controller CPU, memory (each fork is a process), and the open file descriptor limit, since every host holds SSH sockets. throttle works the other way, capping concurrency for one specific task regardless of forks, which is how you avoid hammering a shared artifact server or a rate-limited cloud API from 200 hosts at once. Mention serial too: serial limits the batch, forks limits parallelism within it.

# Independent hosts, no cross-host ordering needed: use free
- name: Patch the fleet
  hosts: all
  strategy: free
  gather_facts: false
  tasks:
    - name: Apply security updates
      ansible.builtin.dnf:
        name: '*'
        state: latest
        security: true

    - name: Do not melt the internal mirror
      ansible.builtin.get_url:
        url: "{{ artifact_url }}"
        dest: /tmp/app.tar.gz
      throttle: 10

# ansible.cfg
# [defaults]
# forks = 50
# strategy = linear
#
# Controller-side limits that actually bind:
# ulimit -n 8192   (one fork holds several file descriptors)

Answer

Measure first. Enable the profile_tasks and timer callbacks so you get per-task durations and a total, and run with --forks set high enough that you are measuring the work rather than the queue. Then work down the usual suspects in order of payoff.

Concurrency: forks defaults to 5, so raising it to 40 or 50 is often a five to eight times improvement on its own, bounded by controller CPU, memory per forked process and the open file descriptor limit. SSH: enable pipelining, which removes a file transfer and an extra SSH round trip per task (it requires requiretty to be disabled in sudoers), and set ControlMaster=auto with ControlPersist so the TCP and auth handshake happens once per host instead of once per task. On a 200-task playbook those two changes alone routinely halve the runtime.

Facts: gathering is a full Python run on every host, so cache facts to jsonfile or Redis with gathering = smart, and trim gather_subset to what you actually reference. Strategy: if hosts are independent, switch to free so a slow host does not gate everyone else. Task design: replace loops that call a package module once per item with a single call passing the whole list, since ansible.builtin.dnf and apt both accept lists and one transaction beats fifty.

Push long-running work to async with poll: 0 and reap it later with async_status. Cut wasteful debug tasks and drop verbosity in CI. Finally, if the bottleneck is genuinely per-task Python startup on the node, the mitogen strategy plugin can help, though it is third party and lags core releases, so most teams exhaust the built-in levers first.

# ansible.cfg tuned for a 400-host fleet
[defaults]
forks = 50
gathering = smart
fact_caching = jsonfile
fact_caching_connection = /var/tmp/ansible_facts
fact_caching_timeout = 3600
callbacks_enabled = profile_tasks, timer

[ssh_connection]
pipelining = True
ssh_args = -o ControlMaster=auto -o ControlPersist=600s -o ServerAliveInterval=30
control_path_dir = /tmp/.ansible-cp

# One transaction instead of fifty
- name: Install the base package set
  ansible.builtin.dnf:
    name: "{{ base_packages }}"    # a list, not a loop
    state: present

# Fire and forget, reap later
- name: Kick off the long reindex
  ansible.builtin.command: /usr/local/bin/reindex --full
  async: 3600
  poll: 0
  register: reindex

Answer

By default a task blocks the fork until the module returns, and the SSH connection is held open the whole time. async changes that: Ansible starts the module in the background on the managed node via the async_wrapper, writes a job file under ~/.ansible_async/, and returns immediately with an ansible_job_id. The poll value decides what happens next. With poll greater than zero, Ansible keeps checking every poll seconds until the job finishes or the async timeout expires, which frees the SSH connection between checks and is how you survive tasks that outlive a network hiccup.

With poll: 0 it is fire and forget, the task returns instantly and you reap the result later with the ansible.builtin.async_status module, using until with retries to wait for finished == 1. The fire-and-forget pattern is what lets you start a slow operation on 200 hosts simultaneously and then collect results, instead of waiting serially. The gotchas are real. async requires a writable home directory for the remote user, and it does not work with unprivileged become unless the temp-file permissions are sorted out.

With poll: 0 Ansible does not clean up the job file, so long-lived hosts accumulate them. A reboot in the middle destroys the job record, so combine async with ansible.builtin.reboot rather than rolling your own. The task result from a poll: 0 task contains no module output at all, only the job id, so anything you want to assert on must come from the async_status result. And async on a delegated or local task behaves differently from what people expect, because the job file lives on the delegate.

- name: Start a long database reindex on every shard
  ansible.builtin.command: /usr/local/bin/reindex --full
  async: 7200        # give up after 2 hours
  poll: 0            # do not wait
  register: reindex_job
  changed_when: false

- name: Do other useful work while it runs
  ansible.builtin.include_tasks: tasks/monitoring.yml

- name: Wait for the reindex to finish
  ansible.builtin.async_status:
    jid: "{{ reindex_job.ansible_job_id }}"
  register: reindex_result
  until: reindex_result.finished
  retries: 240
  delay: 30

- name: Clean up the job record
  ansible.builtin.async_status:
    jid: "{{ reindex_job.ansible_job_id }}"
    mode: cleanup

Answer

Static inventory files stop working the moment autoscaling exists. An inventory plugin queries the source of truth at runtime instead. For AWS you use amazon.aws.aws_ec2, enabled in ansible.cfg under [inventory] enable_plugins, and configured in a file whose name must end in aws_ec2.yml or aws_ec2.yaml, which is a rule people trip over constantly.

The plugin authenticates with the standard boto3 chain (environment variables, shared credentials file, or better, an IAM instance profile or an assumed role), then lists instances filtered by whatever you specify. The three configuration keys that do the real work are filters, keyed_groups and compose. filters narrows the instance set, typically by tag and by instance-state-name: running, so terminated instances do not linger in the inventory. keyed_groups builds groups automatically from instance attributes, so a prefix of tag_Role over tags.Role yields groups like tag_Role_web that your playbooks can target. compose sets host variables using Jinja over the instance data, and the most common one is ansible_host, where you choose private_ip_address for hosts reached over a VPN or direct connect and public_ip_address only when you genuinely go over the internet. Turn on cache with a jsonfile or Redis backend and a sensible cache_timeout, otherwise every playbook run makes a fresh set of DescribeInstances calls and you will meet EC2 API throttling during an incident, exactly when you least want to.

Verify with ansible-inventory -i aws_ec2.yml --graph before you trust it. The same pattern applies to azure.azcollection.azure_rm, google.cloud.gcp_compute and the constructed plugin, which builds derived groups from any inventory source.

# inventories/prod/aws_ec2.yml
plugin: amazon.aws.aws_ec2
regions:
  - ap-south-1
filters:
  instance-state-name: running
  'tag:Environment': production
keyed_groups:
  - key: tags.Role
    prefix: role
    separator: '_'
  - key: placement.availability_zone
    prefix: az
compose:
  ansible_host: private_ip_address
  ansible_user: "'ec2-user' if 'amzn' in image_id else 'ubuntu'"
hostnames:
  - tag:Name
  - private-dns-name
cache: true
cache_plugin: jsonfile
cache_connection: /var/tmp/ansible_inventory_cache
cache_timeout: 600

# ansible.cfg
# [inventory]
# enable_plugins = amazon.aws.aws_ec2, constructed, yaml, ini

# ansible-inventory -i inventories/prod/aws_ec2.yml --graph

Answer

register captures the full result dictionary of a task, including rc, stdout, stdout_lines, changed, failed and any module-specific keys, and it is scoped to the host that ran the task. If the task loops, you get a results list instead of a single result, which is why so many playbooks reference something.results. register overwrites on every run of that task, so registering inside a loop of includes needs care. set_fact creates a named variable at runtime for the current host, sits very high in precedence (above almost everything except extra vars), and persists for the rest of the play. Use it to compute derived values once instead of repeating a long Jinja expression in five places, and to normalise data before feeding it to a role.

Play vars and vars_files are static, parsed up front, and cannot depend on runtime results, so they are for configuration, not computation. The cacheable: true flag on set_fact is the part interviewers probe. Normally a set_fact lives only for the current play.

With cacheable: true, the value is written into the fact cache alongside gathered facts, so a later playbook run in the same cache window can read it without recomputing, and it becomes visible as a real fact rather than a play-scoped variable. That is useful for expensive lookups (a license key fetched from an API, a computed cluster topology) but dangerous for anything volatile, because a stale cached fact is much harder to debug than a recomputed one. Also remember set_fact is per host: computing something on one host does not make it visible on another unless you read it back through hostvars.

- name: Read the currently deployed version
  ansible.builtin.slurp:
    src: /srv/current/VERSION
  register: version_file
  changed_when: false

- name: Normalise it once, use it everywhere
  ansible.builtin.set_fact:
    deployed_version: "{{ version_file.content | b64decode | trim }}"
    needs_deploy: "{{ (version_file.content | b64decode | trim) != release_id }}"

- name: Cache an expensive lookup across playbook runs
  ansible.builtin.set_fact:
    cluster_topology: "{{ topology_api.json }}"
    cacheable: true

- name: Read another host's fact
  ansible.builtin.debug:
    msg: "primary is on {{ hostvars[groups['db_primary'][0]].deployed_version }}"

Answer

Magic variables are set by Ansible itself and cannot be overridden by user variables. inventory_hostname is the name of the host as written in the inventory, and inventory_hostname_short is everything before the first dot, which is what you usually want in config files. ansible_host is the address actually connected to, which is frequently different from inventory_hostname when you are using a dynamic inventory or SSH aliases. groups is a dictionary of every group to its member list, and group_names is the list of groups the current host belongs to, which is how you write when: "'canary' in group_names". hostvars is the important one: it is a dict keyed by hostname giving access to every variable and fact of every host in the inventory, and it is how one host reads another host's data. The classic use is templating an upstream block in an nginx config from the members of the app group, or writing a cluster peer list. The trap is that hostvars only contains facts for hosts whose facts have actually been gathered in this run, so if your play targets only the web group and you try to read a db host's ansible_default_ipv4, you get an undefined error unless you either gathered facts against db in an earlier play, enabled fact caching, or ran a delegated setup with delegate_facts. Also useful: ansible_play_hosts (hosts still active in the current play, failures removed), ansible_play_batch (hosts in the current serial batch), ansible_play_hosts_all (the original list), playbook_dir, role_path and inventory_dir for path building, and ansible_check_mode for behaving differently under --check.

# Gather facts for everyone first so hostvars is populated
- name: Collect facts fleet-wide
  hosts: all
  gather_facts: true
  tasks: []

- name: Render the LB config from live app hosts
  hosts: lb
  tasks:
    - name: Write upstream block
      ansible.builtin.template:
        src: upstream.conf.j2
        dest: /etc/nginx/conf.d/upstream.conf
        mode: '0644'
      notify: reload nginx

# templates/upstream.conf.j2
# upstream app_pool {
# {% for h in groups['app'] %}
#   server {{ hostvars[h]['ansible_default_ipv4']['address'] }}:8080;
# {% endfor %}
# }

- name: Only the canary group gets the beta flag
  ansible.builtin.lineinfile:
    path: /etc/app/flags
    line: 'beta=true'
  when: "'canary' in group_names"

Answer

--check runs the playbook in dry-run mode: modules that support check mode report what they would change without touching the system. --diff shows the actual content delta for file-like modules (template, copy, lineinfile, blockinfile), which is where the real value is, since 'would change' without showing what is far less useful than a unified diff. Together they are the closest thing Ansible has to terraform plan, and the closest is doing a lot of work in that sentence, because check mode has genuine holes. First, command and shell do not execute at all in check mode, they are skipped, so any task whose result feeds a later conditional produces an undefined or empty register and the rest of the play makes decisions on garbage.

Second, tasks that depend on state created earlier in the same run report misleading results, because the earlier task did not actually create anything: installing a package then templating a file into a directory that package would have created shows a failure or a false change. Third, not every module implements check mode; those run for real or fail, and the module documentation is the only reliable source. The controls you have are check_mode: false on a specific task to force it to run even under --check (correct for read-only probes so the register is populated), check_mode: true to force a task to never make changes even in a normal run, and the ansible_check_mode variable to branch explicitly. In practice teams run --check --diff --limit one-host in CI on every pull request as a smoke gate, accept that it is imperfect, and back it with Molecule for real verification.

- name: Read current config (must run even under --check)
  ansible.builtin.command: /usr/local/bin/appctl config dump
  register: current_config
  check_mode: false
  changed_when: false

- name: Render the config, diffable under --diff
  ansible.builtin.template:
    src: app.yml.j2
    dest: /etc/app/app.yml
    mode: '0640'

- name: Skip destructive work during a dry run
  ansible.builtin.command: /usr/local/bin/appctl migrate --apply
  when: not ansible_check_mode

# CI gate on every pull request
# ansible-playbook site.yml --check --diff --limit staging-web01

Answer

The filters that carry most real playbooks are a small set. default(value) supplies a fallback, and default(value, true) also replaces empty strings and other falsy values, which is what you usually want when a variable might be defined but blank. default(omit) is special: omit is a magic sentinel that makes Ansible drop the parameter entirely, so the module applies its own default instead of receiving None. That is the correct way to write an optional module argument, and the difference matters, passing owner: None to the file module is not the same as not passing owner at all. combine(other) merges dictionaries, and combine(other, recursive=True) merges nested ones, which is how you layer a per-host override dict on top of a role default dict without losing sibling keys. selectattr and rejectattr filter a list of dicts by an attribute test, map('attribute', 'name') projects a single field out, and chaining them gives you list comprehensions in YAML. dict2items and items2dict convert between mappings and lists so you can loop over a dictionary, and subelements handles the parent-child case such as users and their SSH keys. For data conversion, from_json, to_nice_json, from_yaml and to_nice_yaml handle marshalling, b64decode pairs with the slurp module, and ansible.builtin.regex_replace and regex_search cover the string work. community.general.json_query gives you JMESPath for deeply nested API responses. One habit worth building: when a Jinja expression grows past a line, compute it once with set_fact and give it a name, because a three-line filter chain inside a module argument is unreadable and untestable.

- name: Optional arguments done correctly
  ansible.builtin.file:
    path: /srv/app
    state: directory
    owner: "{{ app_owner | default(omit) }}"
    mode: "{{ app_mode | default('0755') }}"

- name: Layer per-host tuning over role defaults, keeping nested keys
  ansible.builtin.set_fact:
    jvm_opts: "{{ jvm_defaults | combine(jvm_overrides | default({}), recursive=True) }}"

- name: Only the hosts tagged as primary
  ansible.builtin.debug:
    msg: "{{ shards | selectattr('role', 'eq', 'primary') | map(attribute='host') | list }}"

- name: Loop over a dictionary
  ansible.builtin.lineinfile:
    path: /etc/app/env
    regexp: "^{{ item.key }}="
    line: "{{ item.key }}={{ item.value }}"
  loop: "{{ app_env | dict2items }}"
  loop_control:
    label: "{{ item.key }}"

Answer

A filter transforms data you already have: it takes a value on the left of the pipe and returns a new value. A lookup fetches data from somewhere outside the play, and critically it runs on the controller, not on the managed node. That single fact answers most lookup questions. lookup('ansible.builtin.file', '/etc/motd') reads the file on the machine running ansible-playbook, not on the target, which is why people are confused when the content does not match what is on the server (the module you want there is slurp or a delegated read). lookup('ansible.builtin.env', 'AWS_PROFILE') reads the controller's environment, lookup('ansible.builtin.pipe', 'git rev-parse HEAD') shells out on the controller, and lookup('community.hashi_vault.hashi_vault', ...) or amazon.aws.aws_secret pulls secrets from an external manager at runtime, which is the modern alternative to committing an ansible-vault file. query() (aliased as lookup with wantlist=True) is the version you want when looping, because plain lookup returns a comma-joined string by default while query always returns a real list.

Two behaviours to flag in an interview. First, lookups are evaluated every time the expression is templated, so a lookup inside a loop or a template runs repeatedly and a password lookup can generate a different value on each evaluation, which is why lookup('password', ...) writes to a file to stay stable. Second, lookup failures are fatal by default; pass errors='warn' or errors='ignore' when the source is genuinely optional. Lazy evaluation also means a lookup inside a skipped task's argument may still be evaluated, so do not rely on when to guard an expensive or side-effecting lookup.

- name: Read a secret from HashiCorp Vault at runtime
  ansible.builtin.set_fact:
    db_password: "{{ lookup('community.hashi_vault.hashi_vault',
                     'secret=secret/data/prod/db:password',
                     url='https://vault.internal:8200') }}"
  no_log: true

- name: Loop over matching files on the CONTROLLER
  ansible.builtin.copy:
    src: "{{ item }}"
    dest: "/etc/pki/trust/{{ item | basename }}"
    mode: '0644'
  loop: "{{ query('ansible.builtin.fileglob', 'files/certs/*.crt') }}"

- name: Stamp the deploy with the controller's git SHA
  ansible.builtin.set_fact:
    release_id: "{{ lookup('ansible.builtin.pipe', 'git rev-parse --short HEAD') }}"

- name: Optional lookup that must not abort the play
  ansible.builtin.debug:
    msg: "{{ lookup('ansible.builtin.file', '/opt/optional.txt', errors='warn') }}"

Answer

Molecule is the standard test harness for roles and collections. A scenario lives in molecule/<name>/ and consists of molecule.yml (driver, platforms, provisioner), converge.yml (a playbook that applies the role), and verify.yml (assertions, either an Ansible playbook using ansible.builtin.assert or a Testinfra suite). molecule test runs the full sequence: dependency, cleanup, destroy, syntax, create, prepare, converge, idempotence, side_effect, verify, cleanup, destroy. During development you use the individual commands instead, molecule converge to apply, molecule login to shell into the instance, and molecule verify to re-run assertions, because a full test cycle rebuilds containers every time.

The default driver in recent Molecule versions is delegated, with container drivers supplied by the molecule-plugins package, so most teams run Docker or Podman containers as platforms; some run cloud instances for anything that needs systemd behaviour containers cannot fake. The idempotence step is the one to talk about: Molecule runs converge a second time and fails the scenario if any task reports changed. That single check catches the entire class of bugs that shell tasks, missing creates guards and non-idempotent lineinfile regexes introduce, and it is why Molecule catches things ansible-lint cannot.

Multiple scenarios matter in practice: one per supported platform (a rocky9 scenario and an ubuntu2404 scenario) so a role that claims cross-distro support actually proves it. In CI, run molecule test in a matrix over scenarios. The practical caveat is that container platforms cannot test kernel modules, firewalld, reboots, or real systemd units without a privileged systemd-enabled image, so keep a small set of VM-based scenarios for the roles where that matters.

# molecule/default/molecule.yml
driver:
  name: docker
platforms:
  - name: rocky9
    image: geerlingguy/docker-rockylinux9-ansible:latest
    command: /usr/sbin/init
    privileged: true
    cgroupns_mode: host
    volumes:
      - /sys/fs/cgroup:/sys/fs/cgroup:rw
provisioner:
  name: ansible
  config_options:
    defaults:
      callbacks_enabled: profile_tasks
verifier:
  name: ansible

# molecule/default/verify.yml
- name: Verify
  hosts: all
  tasks:
    - name: nginx is enabled and running
      ansible.builtin.service_facts:
    - ansible.builtin.assert:
        that:
          - ansible_facts.services['nginx.service'].state == 'running'
          - ansible_facts.services['nginx.service'].status == 'enabled'

# molecule converge && molecule verify && molecule test

Answer

ansible-lint is a static analyser for playbooks, roles and collections. It groups rules into profiles that get progressively stricter: min, basic, moderate, safety, shared and production. That layering is the answer to the legacy-repo question, because switching a five-year-old repository straight to production yields hundreds of violations and everyone turns the tool off.

Start at min, get the build green, commit, then raise one profile at a time. The rules worth knowing by name are fqcn (short module names), no-changed-when (command and shell without a changed_when or creates guard), risky-file-permissions (a file or template task with no mode, which inherits the remote umask), risky-shell-pipe (a shell pipeline without pipefail, so a failure in the middle of the pipe is invisible), command-instead-of-module, no-free-form (key=value module arguments instead of proper YAML), jinja (spacing and templating style), name (every task needs a descriptive name starting with a capital), latest (package state: latest, which is non-deterministic), and var-naming. ansible-lint also runs yamllint underneath, so YAML formatting problems surface in the same report. Configuration lives in .ansible-lint, where you set the profile, skip_list for rules you genuinely reject, warn_list for rules you are migrating toward, and exclude_paths for vendored roles you did not write.

Inline exceptions use a # noqa comment with the rule id and should carry a reason. In CI, run ansible-lint alongside ansible-playbook --syntax-check (which only catches parse errors, not semantics) and yamllint, and gate the pull request on all three. ansible-lint --fix mechanically resolves a good fraction of fqcn and formatting issues.

# .ansible-lint
profile: moderate
exclude_paths:
  - galaxy_roles/
  - collections/
skip_list:
  - yaml[line-length]
warn_list:
  - latest
  - risky-shell-pipe
enable_list:
  - args

# Inline exception with a reason
- name: Legacy vendor installer, no module exists
  ansible.builtin.shell: /opt/vendor/install.sh   # noqa: command-instead-of-module
  args:
    creates: /opt/vendor/.installed

# CI gate
# yamllint .
# ansible-lint --profile moderate
# ansible-playbook site.yml --syntax-check
# ansible-playbook site.yml --check --diff --limit staging

Answer

no_log: true suppresses a task's arguments and its result from the standard output and from the callback plugins, including at higher verbosity, and it is the right default on any task that handles a credential. What it does not do is what candidates miss. First, the registered variable still contains the secret, so a later ansible.builtin.debug on that register prints it in full, and no_log on the earlier task does nothing to stop that.

Second, no_log applies to the task, not to the loop label, so a loop over user records with passwords will print the whole item structure unless you also set loop_control.label. Third, secrets that end up written into a file on the managed node are visible to anyone who can read that file, so mode matters as much as no_log, and a template task without an explicit mode gets the remote umask. Fourth, if the module itself echoes the secret into a command line, the value appears in the process table on the node and possibly in shell history and auditd logs, which is why passing credentials via environment or a file is safer than via argv.

Fifth, ANSIBLE_KEEP_REMOTE_FILES=1, used for debugging, leaves the module payload including its arguments on disk. Beyond no_log, the layered controls are: keep secrets in ansible-vault or better in an external manager fetched by lookup at runtime, restrict who holds the vault password, set display_args_to_stdout to false, avoid -vvv in CI logs, and make sure your AWX or AAP job output retention policy does not archive credential material. Rotate anything that ever appeared in a build log; treat that as compromised rather than as an embarrassment to be quietly ignored.

- name: Create the database user
  community.postgresql.postgresql_user:
    name: appuser
    password: "{{ vault_app_db_password }}"
    db: appdb
  no_log: true

# Leak: no_log above does nothing for this
- name: Debug
  ansible.builtin.debug:
    var: db_user_result      # contains the password

- name: Loop safely over credentialed items
  community.general.htpasswd:
    path: /etc/nginx/.htpasswd
    name: "{{ item.user }}"
    password: "{{ item.password }}"
    mode: '0640'
  loop: "{{ basic_auth_users }}"
  no_log: true
  loop_control:
    label: "{{ item.user }}"

- name: Write a credential file with explicit permissions
  ansible.builtin.template:
    src: pgpass.j2
    dest: /root/.pgpass
    owner: root
    mode: '0600'

Answer

Terraform and Ansible solve adjacent problems and the honest answer is that most teams run both. Terraform is declarative provisioning with a state file: it records what it created, computes a plan as a diff against that state, and can destroy resources it no longer wants. That state file is the source of its power and its operational pain (locking, drift, imports, and the blast radius of a corrupted state).

Ansible has no state file. It is procedural at the top level (an ordered task list) with declarative modules underneath, and it converges the machine toward a described state every time it runs, discovering current state per task. That makes Ansible excellent at configuring things inside a machine or a device, and mediocre at lifecycle management of cloud resources, because it cannot tell you 'this security group exists but nobody declared it'.

The common division of labour is Terraform for VPCs, instances, managed databases and IAM, Ansible for OS hardening, package and service configuration, application deployment, network device configuration and day-two operations like patching and certificate rotation. Against Puppet and Chef the difference is architectural. Those are pull-based with an agent on each node that periodically checks in with a master and enforces a catalog, which gives continuous drift correction but requires you to run and secure that infrastructure.

Ansible is push-based and agentless, so it is far easier to adopt on machines you do not fully control (network gear, appliances, customer environments) and gives you explicit ordering, which matters enormously for orchestration across multiple machines. The trade-off is that nothing corrects drift between runs unless you schedule the runs yourself, which is what AWX, Ansible Automation Platform or ansible-pull are for.

# Terraform provisions, then hands off via a dynamic inventory
# terraform apply -auto-approve
# ansible-playbook -i inventories/prod/aws_ec2.yml site.yml

# Ansible is the right tool once you are inside the box
- name: Harden and configure
  hosts: tag_Role_web
  become: true
  roles:
    - cis_hardening
    - nginx
    - app_deploy

# Scheduled convergence without an agent (cron on the node)
# ansible-pull -U https://git.internal/ops/infra.git \
#   -i localhost, local.yml --vault-password-file /etc/ansible/.vault

Answer

A module is a standalone Python program that Ansible copies to the managed node and executes. Correctness has a short checklist. Use AnsibleModule from ansible.module_utils.basic, declare an argument_spec with type, required, default and choices so Ansible validates inputs for you, and mark credential arguments no_log=True in the spec so they never reach the log even if the caller forgets.

Pass supports_check_mode=True and honour module.check_mode by returning the predicted changed value without mutating anything, because a module that silently makes changes under --check is a bug that will eventually cost someone an outage. Use mutually_exclusive, required_if and required_together in the constructor rather than hand-rolled validation. The body must be idempotent: read current state first, compare against desired state, and only act on a genuine difference, then exit with module.exit_json(changed=..., ...) or module.fail_json(msg=...).

Never print to stdout, because Ansible parses stdout as the module's JSON result and a stray print corrupts it. Shell out with module.run_command(), which handles quoting and returns rc, stdout and stderr, rather than subprocess directly. Ship the module inside a collection under plugins/modules/ with a DOCUMENTATION, EXAMPLES and RETURN block, since ansible-doc and the sanity tests both read them and ansible-test sanity fails without them. Before writing one at all, check whether an existing module plus a filter solves the problem, and whether an action plugin (which runs on the controller) is a better fit for something that mostly manipulates data or talks to an API.

#!/usr/bin/python
# plugins/modules/app_feature_flag.py
# (DOCUMENTATION / EXAMPLES / RETURN blocks omitted here for space)
from ansible.module_utils.basic import AnsibleModule
import json, os

def main():
    module = AnsibleModule(
        argument_spec=dict(
            name=dict(type='str', required=True),
            state=dict(type='str', default='present', choices=['present', 'absent']),
            path=dict(type='str', default='/etc/app/flags.json'),
            token=dict(type='str', no_log=True, default=None),
        ),
        supports_check_mode=True,
    )
    p = module.params
    current = {}
    if os.path.exists(p['path']):
        with open(p['path']) as fh:
            current = json.load(fh)

    desired = dict(current)
    if p['state'] == 'present':
        desired[p['name']] = True
    else:
        desired.pop(p['name'], None)

    changed = desired != current
    if changed and not module.check_mode:
        with open(p['path'], 'w') as fh:
            json.dump(desired, fh, indent=2)
    module.exit_json(changed=changed, flags=desired)

if __name__ == '__main__':
    main()

Answer

An Execution Environment is a container image that packages ansible-core, a pinned set of collections, the Python libraries those collections need, and any system packages, so that a playbook runs identically on a laptop, in CI and on an Automation Controller node. It exists because the classic failure is environmental: a role works locally because someone pip installed boto3 and a specific community.general version years ago, and it fails in CI with an obscure import error. Pinning requirements.yml helps, but collections also carry Python dependencies (boto3, kubernetes, pywinrm, netaddr, jmespath) and sometimes system libraries, and none of that is captured by an Ansible-level pin.

You build an EE with ansible-builder from an execution-environment.yml that references a base image and three dependency files: requirements.yml for collections, requirements.txt for Python, and bindep.txt for system packages. ansible-builder create generates a Containerfile and context, ansible-builder build produces the image, and you push it to your registry the same way you would any other artifact. You run playbooks against it with ansible-navigator run, which mounts your project directory into the container and gives you an interactive TUI over the run, or with plain podman/docker in CI, or by pointing an Automation Controller job template at the image. The operational discipline that matters: version the EE image and reference it by digest or an immutable tag in the job template, rebuild it on a schedule so CVEs in the base image get picked up, and run your Molecule suite inside the same EE so what you test is what you ship. Teams treat the EE as the deployable unit of their automation, not the git repository.

# execution-environment.yml
version: 3
images:
  base_image:
    name: quay.io/ansible/awx-ee:latest
dependencies:
  galaxy: requirements.yml
  python: requirements.txt
  system: bindep.txt
additional_build_steps:
  append_final:
    - RUN ansible-galaxy collection list

# requirements.txt
# boto3>=1.34
# jmespath
# netaddr

# bindep.txt
# openssh-clients [platform:rpm]
# rsync [platform:rpm]

# Build and run
# ansible-builder build -t registry.internal/ops/ee-prod:2026.08 -v 3
# ansible-navigator run site.yml -i inventories/prod \
#   --execution-environment-image registry.internal/ops/ee-prod:2026.08 -m stdout

Answer

Event-Driven Ansible flips the trigger model. A playbook is something a human or a scheduler runs. A rulebook is a long-running process that subscribes to event sources and reacts.

You run it with ansible-rulebook, and the rulebook YAML has three sections: sources, rules and actions. A source is a plugin that emits events, and the ansible.eda collection ships webhook, kafka, alertmanager, file_watch, journald and url_check sources among others; you can write your own in Python as an async generator. Rules match against event data with a condition expression, and there is a small but real condition language including boolean logic, string and numeric comparison, list membership, and multi-event correlation with all() and any() over a defined timeframe, which is how you express 'fire only if we saw three failures within five minutes'.

Actions include run_playbook, run_module, run_job_template (handing off to Automation Controller so you get the audit trail and credential handling), set_fact, post_event and debug. The underlying rules engine is Drools, which is why conditions look declarative rather than Jinja-like, and why the condition syntax does not use {{ }}. The realistic use cases are remediation and enrichment: an Alertmanager webhook fires on disk pressure and a rulebook runs a log-rotation playbook against just that host, or a Kafka topic of configuration-drift events triggers a targeted re-converge. The operational cautions are the important part of the answer: rulebooks are stateful long-running processes that need supervision and restart policy, automatic remediation needs a circuit breaker so a flapping alert does not run a playbook every thirty seconds, and every action should be idempotent because you will receive duplicate events.

# rulebooks/disk_pressure.yml
- name: Remediate disk pressure alerts
  hosts: all
  sources:
    - ansible.eda.alertmanager:
        host: 0.0.0.0
        port: 5001
        data_alerts_path: alerts
        data_host_path: labels.instance

  rules:
    - name: Rotate logs when disk is critical
      condition: >-
        event.alert.labels.alertname == 'DiskWillFillIn4Hours' and
        event.alert.labels.severity == 'critical'
      action:
        run_job_template:
          name: Emergency log rotation
          organization: Platform
          job_args:
            limit: "{{ event.alert.labels.instance }}"

    - name: Escalate only after three failures in five minutes
      condition:
        all:
          - event.alert.labels.alertname == 'AppHealthCheckFailed'
        timeout: 5 minutes
      action:
        run_playbook:
          name: playbooks/escalate.yml

# ansible-rulebook -r rulebooks/disk_pressure.yml -i inventory.yml --verbose

Answer

A collection is the packaging unit for modules, plugins, roles and playbooks. The layout is fixed: galaxy.yml at the root declaring namespace, name, version, dependencies and build_ignore; plugins/modules/ for modules; plugins/filter/, plugins/lookup/, plugins/inventory/ and plugins/action/ for other plugin types; plugins/module_utils/ for shared Python; roles/ for roles; playbooks/ for playbooks; docs/ and meta/runtime.yml, which declares the minimum ansible-core version and any redirects for renamed modules. The test story is ansible-test, run from inside the collection directory. ansible-test sanity is the gate everyone hits first: it runs pep8, pylint, import checks, validate-modules (which cross-checks your DOCUMENTATION block against the actual argument_spec and fails on mismatches, missing types, or an undocumented option), YAML lint and shebang checks. ansible-test units runs pytest against tests/unit/, where you exercise module logic by patching AnsibleModule and asserting on exit_json payloads. ansible-test integration runs tests/integration/targets/<name>/tasks/main.yml against a real target, and the convention is to include an idempotency assertion by running the same tasks twice and checking changed is false the second time.

Add --docker to any of these to run inside a controlled container and --python to pin the interpreter, which matters because sanity tests must pass across every Python version your collection claims to support. Publishing is ansible-galaxy collection build followed by publish with an API token, or uploading the tarball to a private Automation Hub. Version with semver and treat a major bump as the only place you are allowed to remove a module or change a default, because consumers pin ranges in requirements.yml.

# galaxy.yml
namespace: goodspace
name: platform
version: 2.3.0
readme: README.md
authors: [Platform Team]
dependencies:
  ansible.posix: '>=1.5.0,<2.0.0'
build_ignore:
  - .github
  - tests/output

# meta/runtime.yml
requires_ansible: '>=2.16.0'

# tests/integration/targets/app_feature_flag/tasks/main.yml
- name: Enable the flag
  goodspace.platform.app_feature_flag:
    name: new_checkout
    state: present
  register: first

- name: Enable it again (must be idempotent)
  goodspace.platform.app_feature_flag:
    name: new_checkout
    state: present
  register: second

- ansible.builtin.assert:
    that:
      - first is changed
      - second is not changed

# ansible-test sanity --docker default -v
# ansible-test units --docker default --python 3.11
# ansible-test integration --docker default app_feature_flag
# ansible-galaxy collection build && ansible-galaxy collection publish *.tar.gz

Answer

A filter plugin is a Python file exposing a FilterModule class with a filters() method that returns a dict mapping filter names to callables. A lookup plugin subclasses LookupBase and implements run(self, terms, variables=None, **kwargs) returning a list. Both execute on the controller inside the Ansible process, so they are fast, but they also mean anything they import must be present on the controller (or in your Execution Environment), not on the managed node.

Discovery is the part people get wrong. The legacy paths still work: a filter_plugins/ or lookup_plugins/ directory next to the playbook or inside a role is auto-loaded, and filter_plugins can also be listed in ansible.cfg. But the modern and correct home is inside a collection, at plugins/filter/<name>.py and plugins/lookup/<name>.py, referenced by FQCN as namespace.collection.filtername.

Collection-scoped plugins are namespaced, so two collections can both ship a to_cidr filter without colliding, whereas two roles each dropping a filter_plugins/util.py into the same run will silently shadow one another and the winner depends on load order. That shadowing bug is a genuinely nasty one to debug, which is why interviewers ask. Write plugins when a Jinja expression has grown into something you cannot read or unit test, or when you need real Python logic (parsing a vendor config format, computing subnet allocations, normalising an API payload). Keep them pure functions with no side effects, unit test them with plain pytest since they are ordinary Python, and document them with a DOCUMENTATION block so ansible-doc -t filter works and ansible-test sanity passes.

# plugins/filter/netutil.py
from ansible.errors import AnsibleFilterError
import ipaddress

def usable_hosts(cidr, reserve=4):
    try:
        net = ipaddress.ip_network(cidr, strict=False)
    except ValueError as exc:
        raise AnsibleFilterError('invalid cidr %s: %s' % (cidr, exc))
    return max(net.num_addresses - reserve, 0)

def to_ptr(addr):
    return ipaddress.ip_address(addr).reverse_pointer

class FilterModule(object):
    def filters(self):
        return {'usable_hosts': usable_hosts, 'to_ptr': to_ptr}

# Usage with FQCN, no shadowing possible
# - ansible.builtin.debug:
#     msg: "{{ '10.20.0.0/22' | goodspace.platform.usable_hosts }}"
#
# - ansible.builtin.debug:
#     msg: "{{ ansible_default_ipv4.address | goodspace.platform.to_ptr }}"
#
# ansible-doc -t filter goodspace.platform.usable_hosts

Answer

Upgrades are never cosmetic, and the porting guide is the document that saves you. The changes that bite hardest, roughly in the order teams hit them: the 2.10 collections split moved most modules out of core, so short names now resolve through ansible.legacy and can behave differently depending on which collections are installed, which is why FQCN is enforced by the production lint profile. The bare include keyword, deprecated for years in favour of include_tasks and import_tasks, has been removed, and any playbook still using it fails to parse.

The warn parameter on command and shell was removed, so tasks passing it now error on an unknown argument. Python floors keep rising on both sides: recent core minors have pushed the controller requirement through 3.10 and 3.11 and the managed-node floor to 3.8 and above, which means an ansible-core bump can strand your oldest CentOS boxes and force you to pin ansible_python_interpreter or keep an older controller image for legacy estates. The most disruptive recent change is the rewritten templating engine introduced in the 2.19 line, which added data tagging and made template evaluation much stricter: undefined values propagate more visibly instead of silently rendering as empty, and conditionals that evaluate to something other than a real boolean now raise an error rather than being coerced.

Playbooks that relied on when: some_string or on an undefined variable quietly becoming empty will fail after that upgrade. Collection majors are the other axis, since community.general and amazon.aws remove deprecated modules on major bumps independently of core. The safe process is: pin everything, upgrade core and collections in separate pull requests, run the full Molecule matrix, and read both the core porting guide and each collection's changelog.

# Patterns that stop working after a modern core upgrade

# 1. Removed keyword
- include: tasks/setup.yml          # removed, use import_tasks / include_tasks

# 2. Removed argument
- ansible.builtin.command:
    cmd: /usr/bin/updatedb
    warn: false                     # removed argument, now an error

# 3. Non-boolean conditional, now rejected
- ansible.builtin.debug: {msg: hi}
  when: app_mode                    # 'app_mode' is a string, not a bool
# Fix:
  # when: app_mode | default('') | length > 0

# 4. Silent undefined, now loud
#   msg: "{{ maybe_missing }}"      -> use | default('') explicitly

# Upgrade safely
# pip install 'ansible-core==2.18.*'
# ansible-playbook site.yml --syntax-check
# ansible-lint --profile production
# molecule test --all

Answer

At a few hundred hosts you tune ansible.cfg. At a few thousand you change the architecture. Start with the controller itself: every fork is a Python process holding SSH sockets, so forks is bounded by CPU, RAM and the file descriptor limit, and pushing forks to 200 on a four-core box makes things slower, not faster, because the processes fight for CPU while SSH handshakes queue.

Measure the point where increasing forks stops reducing wall time and stop there. Then remove work: fact caching in Redis so gathering happens once per cache window rather than once per playbook, trimmed gather_subset, and inventory caching so a dynamic inventory plugin is not making thousands of cloud API calls per run (which will get you throttled). Then shard: the free strategy removes the per-task barrier, and --limit lets you split the fleet into slices you can run in parallel from separate workers.

That sharding is exactly what Automation Controller (the commercial AWX) does natively with job slicing, which splits an inventory across N jobs distributed over execution nodes. That is the architectural answer interviewers want: a single controller does not scale linearly, so you move to a mesh. In Ansible Automation Platform the control plane schedules work onto execution nodes, with hop nodes relaying through network boundaries over the Receptor mesh, so the SSH fan-out happens close to the targets rather than from one machine crossing regions.

Container groups run the same jobs as Kubernetes pods. Add smart inventories to avoid loading the whole fleet for a job that touches one tier, keep a fact cache shared across nodes, and instrument with callback plugins that ship per-task timings somewhere queryable, because at this scale you cannot eyeball the output.

# Shard from CI across four workers
# worker N of 4:
ansible-playbook site.yml -i inventories/prod \
  --limit "$(ansible-inventory -i inventories/prod --list \
    | jq -r '.app.hosts[]' | awk 'NR % 4 == 1' | paste -sd, -)" \
  --forks 40

# ansible.cfg for a large fleet
[defaults]
forks = 40
gathering = smart
fact_caching = redis
fact_caching_connection = redis.internal:6379:0
fact_caching_timeout = 7200
inventory_cache_enabled = True
inventory_cache_plugin = jsonfile
inventory_cache_timeout = 900

[ssh_connection]
pipelining = True
ssh_args = -o ControlMaster=auto -o ControlPersist=900s

# Controller-side limit that actually binds
# ulimit -n 16384

Answer

Narrow first, then instrument. Reproduce against one failing host with --limit so you are not scrolling through 300 hosts of output, and switch to the yaml or debug stdout callback so multi-line stderr is readable instead of being crushed into a single escaped line. Then raise verbosity deliberately: -v adds task results, -vv adds task paths, -vvv adds the full module arguments and connection details, and -vvvv prints the exact SSH command line, which lets you copy it and run it by hand to separate an Ansible problem from an SSH, sudo or MTU problem.

If the module itself is misbehaving, ANSIBLE_KEEP_REMOTE_FILES=1 leaves the payload on the node so you can execute it manually and read the traceback (remember it also leaves the arguments, including secrets, on disk, so clean up afterward). For a failure deep in a long play, --start-at-task skips ahead, --step prompts before each task, and --list-tasks or --list-hosts confirms what would run before you commit. The interactive debugger is underused: set debugger: on_failed on a task or play and Ansible drops into a prompt on failure where you can inspect task.args, task_vars and result, edit an argument, and type redo to retry the task in place, which turns a twenty-minute reproduce loop into seconds.

For differences that are host specific rather than code specific, the answer is almost always in the data, so compare ansible-inventory --host between a working and a failing node and diff the gathered facts, because the usual culprits are a different OS minor version, a missing group membership, a stale cached fact, or a host whose ansible_host resolves somewhere unexpected. Finally, ansible-navigator can replay a run from its artifact file, which is how you debug a production failure after the fact without re-running anything.

# Narrow and make the output readable
ANSIBLE_STDOUT_CALLBACK=yaml ansible-playbook site.yml \
  -i inventories/prod --limit app037 -vvv --diff

# Reproduce the transport layer by hand (copy the command -vvvv printed)
# ssh -vvv -o ControlMaster=auto deploy@app037 'echo ok'

# Interactive debugger on the suspect task
- name: Render the app config
  ansible.builtin.template:
    src: app.yml.j2
    dest: /etc/app/app.yml
  debugger: on_failed
# At the (debug) prompt:
#   p task.args
#   p task_vars['ansible_facts']['distribution_version']
#   task.args['dest'] = '/tmp/app.yml'
#   redo

# Compare a good host against a bad one
# diff <(ansible-inventory -i inventories/prod --host app001) \
#      <(ansible-inventory -i inventories/prod --host app037)

# Resume a long play after a fix
# ansible-playbook site.yml --start-at-task 'Render the app config'

Answer

The design has six parts and each one maps to a specific Ansible feature. First, immutable releases: unpack each build into /srv/releases/<release_id> and switch a symlink, so rollback is another symlink swap rather than a re-download, and the file module makes both operations idempotent. Second, batching: serial with a canary ramp such as [1, 5, '25%'] plus max_fail_percentage: 0, so a bad artifact hits one node.

Third, traffic control: drain the host from the load balancer with delegate_to pointed at the LB, wait for connections to finish, then re-enable, and put the re-enable in an always block so a failed deploy never leaves a node drained. Fourth, real verification: a uri health check with until that asserts the response reports the new release id, not just that the port is open, plus meta: flush_handlers before the check so you are testing the restarted process rather than the old one. Fifth, rollback: wrap the batch in block/rescue that restores the previous symlink and fails loudly with ansible_failed_task in the message.

Sixth, database changes: expand-and-contract migrations run once with run_once and delegate_to, written so the old and new application versions can both run against the schema during the rollout, because a rolling deploy means both versions are live simultaneously. Proving re-runnability is the other half of the answer, and it is mechanical: run the playbook twice in CI and fail the job if the recap reports any changed, run Molecule with its idempotence step for every role, gate pull requests on --check --diff against a staging host, and keep command and shell tasks either guarded with creates or annotated with changed_when so they cannot manufacture false changes. A deploy you cannot run twice safely is not a deploy, it is a one-shot script with YAML syntax.

- name: Zero-downtime app deploy
  hosts: app
  become: true
  serial: [1, 5, '25%']       # canary, then ramp
  max_fail_percentage: 0      # any failure stops the rollout
  force_handlers: true
  tasks:
    - block:
        - name: Drain this host from the load balancer
          community.general.haproxy:
            state: disabled
            backend: app_pool
            host: "{{ inventory_hostname }}"
            wait: true
          delegate_to: "{{ groups['lb'][0] }}"

        - name: Point /srv/current at the new release
          ansible.builtin.file:
            src: "/srv/releases/{{ release_id }}"
            dest: /srv/current
            state: link
          notify: restart app

        - ansible.builtin.meta: flush_handlers

        - name: Assert the NEW version is serving, not just the port
          ansible.builtin.uri: {url: 'http://127.0.0.1:8080/healthz', return_content: true}
          register: hc
          until: hc.json.release == release_id
          retries: 20
          delay: 3

      rescue:
        - name: Roll back the symlink
          ansible.builtin.file: {src: '{{ previous_release_path }}', dest: /srv/current, state: link}
          notify: restart app
        - ansible.builtin.fail: {msg: 'rolled back at {{ ansible_failed_task.name }}'}

      always:
        - name: Re-enable in the load balancer, whatever happened
          community.general.haproxy: {state: enabled, backend: app_pool, host: '{{ inventory_hostname }}'}
          delegate_to: "{{ groups['lb'][0] }}"