Ansible and Infrastructure as Code: Idempotency and Best Practices
KU
Kiril Urbonas
4 months ago • 8 min read•Updated 7 hours ago•
Idempotent Ansible means you can run a playbook twice and the second run does nothing. Here is how we get there, and where we stopped fighting it.
The first Ansible playbook I put into production was a mess of shell: tasks that ran fine the first time and broke the second. It appended a line to a config file. Run it once, one line. Run it three times, three lines. Nginx stopped starting. That was the day idempotency stopped being a word in the docs and became something I cared about.
If you take one idea away from this, make it this: a good playbook is safe to run again. And again. The tenth run should report the same thing as the second, nothing changed, because the machine was already in the state you asked for.
Idempotency is the property that applying an operation once has the same result as applying it many times. Set a file's mode to 0644. It is already 0644? Then the module does nothing and reports ok. It was 0600? The module changes it and reports changed. Either way you end up in the same place, and the module tells you honestly which of the two happened.
That honesty is the point. In infrastructure work you are almost never building a machine from scratch; you are reconciling a running box against a description of how it should look. Run the playbook against a hundred servers where sixty are already correct, and you want those sixty to come back green and untouched. Without idempotency you cannot tell "I fixed drift" apart from "I trampled a config that was already fine."
Ansible's native modules converge on a state, not on steps. You do not tell ansible.builtin.copy to copy a file; you tell it the file should have this content and these permissions. It checks, then acts only if reality disagrees.
- name: Ensure nginx config is present
ansible.builtin.template:
src: nginx.conf.j2
dest: /etc/nginx/nginx.conf
owner: root
group: root
mode: "0644"
notify: Reload nginx
Run that on a box that already has the rendered file and it reports ok. Change the template and rerun, and only then do you see changed, and only then does the handler fire. Compare it to the trap I fell into:
# Don't do this. Appends every single run.
- name: Add upstream to config
ansible.builtin.shell: echo "upstream app { server 10.0.0.5; }" >> /etc/nginx/nginx.conf
shell and command have no idea what state they are supposed to produce. They run the string you gave them, every time, and report changed every time because from their point of view they did do work. That is the root of most non-idempotent playbooks I have reviewed.
Sometimes you genuinely need to shell out. There is no module for the weird vendor CLI, or the operation is a one-shot. You can still make those tasks honest with creates, removes, and changed_when.
- name: Initialize the database schema once
ansible.builtin.command: /opt/app/bin/migrate --init
args:
creates: /var/lib/app/.schema_initialized
- name: Check cluster health without ever reporting changed
ansible.builtin.command: /usr/local/bin/cluster status
register: cluster
changed_when: false
failed_when: "'healthy' not in cluster.stdout"
The creates guard means the migration runs once and is skipped forever after, because the marker file exists. changed_when: false tells Ansible this is a read, not a write, so it never pollutes your change count. These knobs turn a lying task into a truthful one. If you use shell, use them.
Before I apply a playbook to something I care about, I run it in check mode. --check does a dry run: modules report what they would change without changing it. Pair it with --diff and you see the exact lines a template would rewrite.
ansible-playbook site.yml --check --diff --limit web-prod-01
This is only as good as your modules, though. A shell task can't predict its own effect, so it either gets skipped or lies to you in check mode. Yet another reason to prefer modules: they make your dry run mean something.
A single 800-line playbook is a smell. Once one grows past a screen or two, break it into roles. A role bundles tasks, handlers, templates, defaults, and variables under a fixed directory layout, and you include it by name. The nginx role sets up nginx; the app role deploys the app; the site playbook just lists which roles run on which hosts.
The payoff is reuse. The same postgres role serves your staging and production inventories with different variables. Keep roles focused on one thing, put sane values in defaults/main.yml, and let callers override what they need. Roles pulled from Ansible Galaxy or a private git ref keep teams from reinventing the same package-and-service dance.
Ansible's variable precedence is deep, and fighting it causes real pain. The short version: role defaults sit at the bottom, group vars above them, host vars above those, and command-line -e extra vars win over everything. Set broad defaults low and narrow overrides high.
Structure inventory around that. Group hosts by role and environment (webservers, dbservers, prod, staging) and let group_vars/ and host_vars/ carry the values. A host in both webservers and prod picks up both files. Keep environment differences in group vars, not in the tasks, so the same role runs everywhere.
Handlers are how you avoid needless restarts. A handler only runs if a task notifys it, and only once at the end of the play no matter how many tasks poked it.
tasks:
- name: Deploy vhost
ansible.builtin.template:
src: vhost.conf.j2
dest: /etc/nginx/conf.d/app.conf
notify: Reload nginx
handlers:
- name: Reload nginx
ansible.builtin.service:
name: nginx
state: reloaded
Change the template on ten hosts and nginx reloads on those ten. Change nothing and nginx is left alone. That is idempotency at the service level, and the reason I never put a bare service: restarted in the middle of a task list.
Do not commit plaintext secrets, and do not pass them around as extra vars in your shell history. Ansible Vault encrypts values or whole files at rest with a password or key.
ansible-vault encrypt_string 's3cr3t-db-pass' --name db_password
That drops an encrypted blob into a vars file. Playbooks reference db_password normally; Ansible decrypts at runtime when you supply the vault password. Encrypt individual strings inside otherwise readable vars files so diffs stay meaningful, and keep the vault password in a real secret store, not next to the repo.
The single most useful test for a role is the idempotence test, and it is almost free. Run the role, then run it again and assert the second run reports zero changes. Anything that comes back changed on the second pass is a non-idempotent task hiding somewhere.
Molecule automates this: it spins up a container, applies the role, and runs the converge step twice.
# molecule/default/molecule.yml (sketch)
scenario:
test_sequence:
- create
- converge # first apply
- idempotence # second apply must report no changes
- verify
- destroy
That idempotence step is the guardrail. Wire it into CI and a shell task without changed_when will fail the build, which is exactly what you want it to do.
People treat these as rivals. They are not. Terraform provisions infrastructure: it talks to cloud APIs, tracks state, and creates or destroys the VMs, networks, and load balancers. Ansible configures things that already exist. It SSHes into a box and reconciles packages, files, and services. If you build a Terraform module and need it to survive real use, we wrote up Terraform module patterns separately.
The clean split we use: Terraform stands up the machines and the network; Ansible dresses them once they exist. Where they overlap, I lean on Ansible for the in-box config and keep Terraform on the API-driven provisioning. Terraform's state file is its source of truth. Ansible has no state file, which is exactly why idempotency matters so much for it: the running machine is the only truth it has.
shell and command for everything. If a module exists, use it. Reach for shell only when nothing else fits, and guard it.Write for the second run, not the first. Prefer modules, and when you must shell out, tell Ansible the truth with creates and changed_when. Split work into small roles, keep environment differences in inventory, gate restarts behind handlers, and put an idempotence test in CI so the machine catches the drift you will not. Use Terraform to build the boxes and Ansible to configure them, and stop pretending either one wants to do the other's job. Do that and your playbooks become boring, which in infrastructure is the highest compliment there is.
Get the latest tutorials, guides, and insights on AI, DevOps, Cloud, and Infrastructure delivered directly to your inbox.
A practical risk-management framework for release timing, Friday deployment policies, progressive delivery, and how elite teams protect reliability and people.
A practical GitHub Actions monorepo CI guide built around a real scaling problem: long queues, noisy failures, and developers waiting 40 minutes for feedback.
Explore more articles in this category
The Backstage demo always wows leadership. Then six months later the catalog has 400 stale entries and nobody trusts it. Here's what got ours to actually stick.
We inherited 200-odd AWS resources built by hand over four years, with no state file anywhere. Here's how import blocks and a generation workflow got them under Terraform without a rebuild.
A single ALTER TABLE took a lock and stalled every write for 40 seconds during peak traffic. Expand-contract is how we stopped shipping outages.
Evergreen posts worth revisiting.
