Development Environment Scale Up / Scale Down Runbook

The staging tier rests at zero to save roughly $63/month of Fargate, and is raised when someone needs to validate against it.

The command

cd phenom-infra
export CLOUDFLARE_API_TOKEN=...        # from wherever you keep secrets

scripts/dev-env-scale.sh up            # dry run: shows the plan, changes nothing
scripts/dev-env-scale.sh up --apply    # does it
scripts/dev-env-scale.sh down --apply
scripts/dev-env-scale.sh status        # report only

Dry run is the default. --apply is required to touch anything, and a dry run reverts its own edits before exiting.

If CLOUDFLARE_API_TOKEN is unset the script prompts for it at a terminal, and fails immediately in CI rather than hanging. It is required because the Cloudflare provider is configured in this environment; without it Terraform builds an incomplete plan that only fails minutes later at apply.

After up, re-run deploy-hasura-development on phenom-backend. The script prints this reminder and deliberately does not do it: the Hasura engine is one of the services that was at zero, so migrations could not apply while the tier rested.

Commit the Terraform edits in a PR afterwards. --apply is the break-glass path; dev-infra-ci auto-applying on merge is the normal one.

Checking without changing anything

scripts/dev-env-status.sh

The third code exists so a missing credential is never reported as healthy. Safe to wire into CI or a cron.

Why a command rather than a checklist

dev_service_count in environments/development/locals.tf looks like the environment switch. It is one line that scales four ECS services, and treating it as the whole story has failed three distinct ways:

It does not resize the database. db_instance_class is a separate value in main.tf. Flipping dev_service_count brings services up against a resting database, so “scaled up” and “usable” diverge with nothing reconciling them.

It does not cover phenom-dev-nest-ops. That service was created out-of-band on 2026-04-06, is absent from Terraform state, and is invisible to any plan.

It reports desiredCount, not runningCount. On 2026-09-08 the environment read as fully up while three of five services sat at runningCount: 0. Terraform was satisfied, nothing was running, and it billed for seventeen days.

That last one is the whole design principle: desiredCount is what Terraform asked for, runningCount is what happened, and only the second is the environment.

What the command does, in order

The RDS resize goes first because services that come up against a db.t4g.micro run badly, and the symptom looks like an application bug rather than a sizing one.

A failed apply does not abort the run. Terraform applies in dependency order and does not roll back, so a mid-apply failure leaves the tier partially scaled. The script says so plainly, finishes the remaining steps rather than leaving it half-done, verifies, and exits non-zero regardless of what the probe reports.

Instance sizes

Staging deliberately does not match production. It serves at most a handful of people testing.

apply_immediately = true is set for development only, so a resize starts at once with the brief interruption that implies. Production keeps the default and resizes in a planned window. Without it the resize sits in PendingModifiedValues while terraform apply reports success and the account keeps running the old class.

What stays up on purpose

If a service will not place

running=0 against desired=1 is a placement failure, not a slow start:

aws ecs describe-services --cluster phenom-dev-cluster \
  --services <name> --query 'services[0].events[0:3].[createdAt,message]' --output text

• CannotPullContainerError / 401 — the task pulls from a registry it cannot authenticate to. phenom-staging-synapse pulls a private Docker Hub image while production pulls the same app from ECR. Fix the image reference, not the credential.
• Starts then dies — crash loop. Events say “unable to consistently start tasks”; the reason is in the task’s CloudWatch log group.

Known open problems

Live as of 2026-09-15. The probe flags all of these, so a non-zero exit today is expected rather than a broken script:

• phenom-dev-storage fails to place. Tracked in phenom-infra#246.
• phenom-staging-synapse / -admin fail to place on the image-pull 401 above.
• phenom-dev-file-validator has been Inactive since 2026-06-06: “AWS Lambda does not have permission to access the provided code artifact.” Its S3 notification still fires into a function that cannot start, so staging uploads are never validated or tagged. It also fails every terraform apply against this environment, which is why the scale command has to survive a failed apply.

Related

• Development Environments & Workflow
• RDS Dev
• API Staging