# How to Stop Anyone From Deleting Your Production Database

URL: https://www.devopsness.com/blog/prevent-deleting-production-database
Author: Kiril Urbonas
Published: 2026-10-29
Updated: 2026-10-29
Categories: Infrastructure
Tags: Terraform, Reliability, Postgres, Database, Disaster Recovery

Summary: GitLab lost six hours of data to one command on the wrong host. Linear lost five hours to one TRUNCATE CASCADE in a reviewed migration. The six guardrails that make that kind of mistake hard to make and cheap to recover from.

---
# How to Stop Anyone From Deleting Your Production Database

Nobody deletes a production database on purpose. In the well-known cases, a competent engineer runs a reasonable command in the wrong place, or a reviewed change does more than anyone expected. So the fix isn't "be more careful". It's making the destructive path hard to take by accident, and making recovery fast when it happens anyway.

Two public postmortems show both failure shapes.

**GitLab, January 31, 2017.** While rebuilding replication late at night, an engineer meant to wipe the data directory on the secondary and ran the command on the primary. About 300 GB was gone before they stopped it. Then the backups failed one after another: `pg_dump` backups had been silently failing because of a version mismatch, failure emails were rejected, and disk snapshots weren't enabled for the database servers. GitLab restored from a six-hour-old LVM snapshot that happened to exist, lost about six hours of data, and was down for around 18 hours.

**Linear, January 24, 2024.** A migration used `TRUNCATE TABLE <new_table> CASCADE` to clear two new tables. `CASCADE` also truncated every table with a foreign key pointing at them, which wiped production data in issue descriptions, comments and more. The change had passed local tests and code review. Linear restored over 99% of lost data within 36 hours, partly by replaying their sync engine's action log.

## The six guardrails, layered

| Layer | Guardrail | Would have stopped |
|---|---|---|
| Access | No standing production write access for humans | Commands run on the wrong host |
| Cloud | Deletion protection on the instance | Console or CLI deletes |
| IaC | `prevent_destroy` on stateful resources | Plans that replace or destroy databases |
| Database | Application roles without `TRUNCATE` or `DROP` | Destructive statements in app code and migrations |
| Review | Automated linting of migrations | `TRUNCATE ... CASCADE`, `DROP COLUMN`, unsafe locks |
| Recovery | Point-in-time restore, tested on a schedule | Everything else |

No single layer is enough. Each one fails sometimes; together they turn a career-defining incident into a bad afternoon.

## Make the dangerous path inconvenient

**Access.** Day-to-day credentials should be read-only. Production write access comes through a break-glass role that requires a reason, expires, and is logged. Make production visibly different too: a red shell prompt, distinct hostnames, and no identical-looking `db1` and `db2` that differ only by role.

**Deletion protection.** On Amazon RDS, an instance with deletion protection can't be deleted until someone deliberately turns it off. Enabling it doesn't cause an outage:

```bash
$ aws rds modify-db-instance \
    --db-instance-identifier prod-orders \
    --deletion-protection \
    --apply-immediately
```

**Terraform.** Add `lifecycle { prevent_destroy = true }` to every stateful resource, and `deletion_protection = true` on the `aws_db_instance`. Know the gap: Terraform's documentation is explicit that `prevent_destroy` doesn't help if someone removes the resource block from the configuration. That's why the cloud-level protection matters too. If a plan ever wants to replace your database, our guide to [a Terraform plan that wants to recreate production](/blog/terraform-plan-replace-production-database) explains why and how to stop it.

## Take destructive SQL away from the application

Linear's first remediation was revoking `TRUNCATE` from production database users. Your application role doesn't need it, and neither do most migrations:

```sql
-- PostgreSQL: application and migration roles keep DML, lose the destructive verbs
REVOKE TRUNCATE ON ALL TABLES IN SCHEMA public FROM app_user;
ALTER DEFAULT PRIVILEGES IN SCHEMA public REVOKE TRUNCATE ON TABLES FROM app_user;
-- DROP requires table ownership, so don't let app_user own the tables
ALTER TABLE orders OWNER TO schema_owner;
```

Then lint migrations in CI. A migration linter for PostgreSQL such as Squawk flags dangerous operations before a human reviewer has to spot them, which matters because reviewers read application code carefully and skim migrations.

## Recovery is the guardrail that always runs

Backups you haven't restored are a hope, not a plan. GitLab had five backup and replication mechanisms, and the only one that saved them was an LVM snapshot never designed for disaster recovery. Linear had never tested point-in-time recovery, so they chose a full-backup restore they trusted, which lost more data than PITR would have.

Schedule a restore drill: restore last night's backup and a point-in-time target to a new instance, run a few integrity queries, and time it. Our [backup restoration testing](/blog/database-backup-restoration-testing-restores) drill is the version we run quarterly. One more fact worth knowing before you need it: AWS's documentation says that if an RDS instance is deleted unintentionally, you should contact AWS Support immediately, because recovery may be possible for up to six days after the deletion request.

## Questions people ask

### How do I prevent accidental deletion of an RDS database?

Turn on deletion protection with `aws rds modify-db-instance --deletion-protection`, set `deletion_protection = true` and `prevent_destroy = true` in Terraform, and keep automated backups with a retention period you've tested restoring from. Then remove standing delete permissions from human IAM roles, so a console click needs a break-glass role.

### What should I do right after deleting a production database by accident?

Stop further damage first: pause writes or jobs that could make it worse. Don't overwrite anything you might restore from. Restore to a new instance from point-in-time recovery or the latest backup, verify it, then cut over. On RDS, contact AWS Support immediately, since recovery may be possible for up to six days.

### Does Terraform prevent_destroy fully protect a database?

No. It blocks plans that would destroy the resource while the block is in your configuration, but Terraform's documentation notes it doesn't apply if the resource block is removed. Pair it with cloud-level deletion protection and code review on removals of stateful resources.

## The decision, concretely

- **No deletion protection on production databases today?** Turn it on this week; it's one command and causes no downtime.
- **Application role can TRUNCATE or owns its tables?** Revoke and transfer ownership; it's the cheapest fix for Linear's failure mode.
- **Engineers hold production write credentials by default?** Move to read-only plus a logged break-glass role.
- **Never restored a backup end to end?** Do it before anything else on this list.

## The call we'd make

Start with the restore drill, because it's the only guardrail that covers mistakes you haven't imagined yet. Then add deletion protection and revoke destructive privileges, both of which take minutes. The goal isn't an engineer who never makes a mistake, it's a system where one mistake can't cost you a day of customer data.

## Sources

- [GitLab: Postmortem of database outage of January 31 (2017)](https://about.gitlab.com/blog/postmortem-of-database-outage-of-january-31/)
- [Linear: Incident on January 24th, 2024](https://linear.app/now/linear-incident-on-jan-24th-2024)
- [AWS: Deleting a DB instance (Amazon RDS)](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_DeleteInstance.html)
- [HashiCorp: Terraform lifecycle meta-argument](https://developer.hashicorp.com/terraform/language/meta-arguments/lifecycle)