Dev/staging is spun down — what changed and how to bring it back

As of 2026-08-05 the dev/staging tier is intentionally at rest. Every Terraform-managed dev/staging ECS service sits at desiredCount: 0 and phenom-dev-postgres has been downsized from db.m5.large to db.t4g.micro.

Nothing was destroyed. No terraform destroy, no dropped databases, no deleted buckets. This is a pause, and the way back is one line.

Tracked in phenom-infra#246, shipped in PR #247.

Why

July 2026 AWS spend was $1,043.25. Roughly $292/mo of that was dev-attributable, for an environment nobody was actively developing against.

What is still running, on purpose

The database. It is downsized, not stopped — and that distinction is the whole design.

A stopped RDS instance cannot accept Hasura migrations. Our workflow is staging-first, so migrations would queue unapplied and the staging schema would silently drift from production for as long as the environment slept. Scale-up would then land on something weeks stale that needed repairing before it could be used. Keeping the instance alive at db.t4g.micro costs a few dollars a month and keeps the migration path open, so staging stays in sync while it rests.

AWS also force-restarts a stopped instance after 7 days, so stopping is not even a durable resting state.

db.t4g.micro is sufficient precisely because the services are at zero — nothing queries that database except migrations.

The dev ALB and the two dev-VPC NAT gateways also stay, about $88/mo combined. They keep the environment resurrectable in one command rather than rebuildable: the ALB preserves DNS and listener configuration, and the staging chat provisioner Lambda is VPC-attached. Worth revisiting only if the spin-down becomes long-term.

Bringing it back

One value, in environments/development/locals.tf:

locals {
  dev_service_count = 0   # 1 to bring the environment back
}

Set it to 1, open a PR, merge. dev-infra-ci auto-applies on merge when the plan is 0-destroy. Down is the same move in reverse. Both directions go through a PR, so there is an audit trail either way.

Why it is in Terraform rather than done by hand

dev-infra-ci auto-applies on merge whenever a plan destroys nothing — and scaling a service to zero is an in-place update, not a destroy. The 0-destroy seatbelt does not catch it.

A console-side spin-down would therefore have been silently reverted by the next merge touching environments/development/**, and would have kept billing until somebody noticed. Codifying it inverts that: CI now enforces the resting state, and the daily drift check keeps the config honest at zero replicas.

Known gaps

phenom-dev-nest-ops is still running. It is an out-of-band ECS service with no Terraform resource and no state entry — created 2026-04-06, untagged — so it cannot be scaled from the switch above. About $18/mo. It needs to be imported and managed, or deleted if abandoned.

phenom-dev-storage was already broken. It showed desiredCount: 1 with runningCount: 0, failing to place and costing nothing. Scaling it to zero tidied the config but saved nothing real; the underlying failure is a separate issue.

Staging chat will not work regardless of scale. The staging Synapse admin token is still a placeholder and CHAT_ROOM_ID is empty — see phenom-infra#240, closed as not-planned.

What to expect while it rests

dev-nest, api-staging, and staging chat will not serve traffic. Production is entirely unaffected — the production plan showed zero ECS and zero RDS changes, and prod was verified still serving 200 on nest.thephenom.app and api.thephenom.app immediately after the apply.

Container images are not exercised at zero replicas, so a broken image would only surface on scale-up. If the environment stays down for a long stretch, a periodic scale-up smoke test — monthly, or gated before a release — is worth the few minutes.