Ansible and Infrastructure as Code: Idempotency and Best Practices

The first Ansible playbook I put into production was a mess of shell: tasks that ran fine the first time and broke the second. It appended a line to a config file. Run it once, one line. Run it three times, three lines. Nginx stopped starting. That was the day idempotency stopped being a word in the docs and became something I cared about.

If you take one idea away from this, make it this: a good playbook is safe to run again. And again. The tenth run should report the same thing as the second, nothing changed, because the machine was already in the state you asked for.

What idempotency actually means

Idempotency is the property that applying an operation once has the same result as applying it many times. Set a file's mode to 0644. It is already 0644? Then the module does nothing and reports ok. It was 0600? The module changes it and reports changed. Either way you end up in the same place, and the module tells you honestly which of the two happened.

That honesty is the point. In infrastructure work you are almost never building a machine from scratch; you are reconciling a running box against a description of how it should look. Run the playbook against a hundred servers where sixty are already correct, and you want those sixty to come back green and untouched. Without idempotency you cannot tell "I fixed drift" apart from "I trampled a config that was already fine."

How modules get you this for free

Ansible's native modules converge on a state, not on steps. You do not tell ansible.builtin.copy to copy a file; you tell it the file should have this content and these permissions. It checks, then acts only if reality disagrees.

- name: Ensure nginx config is present
  ansible.builtin.template:
    src: nginx.conf.j2
    dest: /etc/nginx/nginx.conf
    owner: root
    group: root
    mode: "0644"
  notify: Reload nginx

Run that on a box that already has the rendered file and it reports ok. Change the template and rerun, and only then do you see changed, and only then does the handler fire. Compare it to the trap I fell into:

# Don't do this. Appends every single run.
- name: Add upstream to config
  ansible.builtin.shell: echo "upstream app { server 10.0.0.5; }" >> /etc/nginx/nginx.conf

shell and command have no idea what state they are supposed to produce. They run the string you gave them, every time, and report changed every time because from their point of view they did do work. That is the root of most non-idempotent playbooks I have reviewed.

Making command and shell behave

Sometimes you genuinely need to shell out. There is no module for the weird vendor CLI, or the operation is a one-shot. You can still make those tasks honest with creates, removes, and changed_when.

- name: Initialize the database schema once
  ansible.builtin.command: /opt/app/bin/migrate --init
  args:
    creates: /var/lib/app/.schema_initialized

- name: Check cluster health without ever reporting changed
  ansible.builtin.command: /usr/local/bin/cluster status
  register: cluster
  changed_when: false
  failed_when: "'healthy' not in cluster.stdout"

The creates guard means the migration runs once and is skipped forever after, because the marker file exists. changed_when: false tells Ansible this is a read, not a write, so it never pollutes your change count. These knobs turn a lying task into a truthful one. If you use shell, use them.

Check mode and diff before you touch anything

Before I apply a playbook to something I care about, I run it in check mode. --check does a dry run: modules report what they would change without changing it. Pair it with --diff and you see the exact lines a template would rewrite.

ansible-playbook site.yml --check --diff --limit web-prod-01

This is only as good as your modules, though. A shell task can't predict its own effect, so it either gets skipped or lies to you in check mode. Yet another reason to prefer modules: they make your dry run mean something.

Structure: roles and reuse

A single 800-line playbook is a smell. Once one grows past a screen or two, break it into roles. A role bundles tasks, handlers, templates, defaults, and variables under a fixed directory layout, and you include it by name. The nginx role sets up nginx; the app role deploys the app; the site playbook just lists which roles run on which hosts.

The payoff is reuse. The same postgres role serves your staging and production inventories with different variables. Keep roles focused on one thing, put sane values in defaults/main.yml, and let callers override what they need. Roles pulled from Ansible Galaxy or a private git ref keep teams from reinventing the same package-and-service dance.

Variables and inventory

Ansible's variable precedence is deep, and fighting it causes real pain. The short version: role defaults sit at the bottom, group vars above them, host vars above those, and command-line -e extra vars win over everything. Set broad defaults low and narrow overrides high.

Structure inventory around that. Group hosts by role and environment (webservers, dbservers, prod, staging) and let group_vars/ and host_vars/ carry the values. A host in both webservers and prod picks up both files. Keep environment differences in group vars, not in the tasks, so the same role runs everywhere.

Handlers and notify

Handlers are how you avoid needless restarts. A handler only runs if a task notifys it, and only once at the end of the play no matter how many tasks poked it.

tasks:
  - name: Deploy vhost
    ansible.builtin.template:
      src: vhost.conf.j2
      dest: /etc/nginx/conf.d/app.conf
    notify: Reload nginx

handlers:
  - name: Reload nginx
    ansible.builtin.service:
      name: nginx
      state: reloaded

Change the template on ten hosts and nginx reloads on those ten. Change nothing and nginx is left alone. That is idempotency at the service level, and the reason I never put a bare service: restarted in the middle of a task list.

Secrets with Vault

Do not commit plaintext secrets, and do not pass them around as extra vars in your shell history. Ansible Vault encrypts values or whole files at rest with a password or key.

ansible-vault encrypt_string 's3cr3t-db-pass' --name db_password

That drops an encrypted blob into a vars file. Playbooks reference db_password normally; Ansible decrypts at runtime when you supply the vault password. Encrypt individual strings inside otherwise readable vars files so diffs stay meaningful, and keep the vault password in a real secret store, not next to the repo.

Testing: molecule and the idempotence check

The single most useful test for a role is the idempotence test, and it is almost free. Run the role, then run it again and assert the second run reports zero changes. Anything that comes back changed on the second pass is a non-idempotent task hiding somewhere.

Molecule automates this: it spins up a container, applies the role, and runs the converge step twice.

# molecule/default/molecule.yml (sketch)
scenario:
  test_sequence:
    - create
    - converge      # first apply
    - idempotence   # second apply must report no changes
    - verify
    - destroy

That idempotence step is the guardrail. Wire it into CI and a shell task without changed_when will fail the build, which is exactly what you want it to do.

Ansible vs Terraform

People treat these as rivals. They are not. Terraform provisions infrastructure: it talks to cloud APIs, tracks state, and creates or destroys the VMs, networks, and load balancers. Ansible configures things that already exist. It SSHes into a box and reconciles packages, files, and services. If you build a Terraform module and need it to survive real use, we wrote up Terraform module patterns separately.

The clean split we use: Terraform stands up the machines and the network; Ansible dresses them once they exist. Where they overlap, I lean on Ansible for the in-box config and keep Terraform on the API-driven provisioning. Terraform's state file is its source of truth. Ansible has no state file, which is exactly why idempotency matters so much for it: the running machine is the only truth it has.

Anti-patterns we've stopped tolerating

• shell and command for everything. If a module exists, use it. Reach for shell only when nothing else fits, and guard it.
• Tasks with no idempotence. If your second run is not green, something is wrong, even if the box looks fine.
• One giant playbook. Break it into roles before it breaks you.
• Restarts scattered through the task list instead of handlers.
• Secrets in plaintext or in extra vars. Encrypt them.

The call we'd make

Write for the second run, not the first. Prefer modules, and when you must shell out, tell Ansible the truth with creates and changed_when. Split work into small roles, keep environment differences in inventory, gate restarts behind handlers, and put an idempotence test in CI so the machine catches the drift you will not. Use Terraform to build the boxes and Ansible to configure them, and stop pretending either one wants to do the other's job. Do that and your playbooks become boring, which in infrastructure is the highest compliment there is.