Development Environments & Workflow Runbook

What it is

The Phenom platform runs two fully separated environments: a staging tier (also called the development tier) and a production tier. They share the same architecture — NEST app on Cloudflare Pages → API (Hasura GraphQL + Auth) on AWS → RDS PostgreSQL — but every layer is a distinct deployment with its own URL, its own database, and its own branch. Nothing in staging touches production data, and nothing reaches production without an explicit, reviewed promotion.

This runbook is the orientation page for the per-asset runbooks. It exists so an engineer can see the whole topology at once, know exactly which URL belongs to which tier, and follow the enforced workflow for shipping a change safely.

Environment URLs

Every Phenom asset, by tier. Use these to confirm you are pointed at the right environment before you deploy, query, or test.

Cognito is shared. Both tiers authenticate against the same Cognito pool (us-east-1_knEL7cqS3). A user account therefore exists in both environments; only the data behind the API differs. Treat auth as cross-environment and never assume a staging login is isolated from production identity.

Databases are private. Neither RDS instance has a public endpoint. All SQL access goes through the phenom-oneoff-sql ECS Fargate task inside the VPC — see the RDS dev and RDS prod runbooks.

How each environment works

Staging / development tier

The staging tier is where all integration testing, QA, and pre-production verification happens.

• dev-nest.thephenom.app is built by Cloudflare Pages from the develop branch of Phenom-Backend. Every push to develop triggers an automatic build.
• It calls the staging API at api-staging.thephenom.app, which runs the same Hasura GraphQL + Auth stack as production but against the phenom-dev-postgres database.
• Staging runs on the phenom-dev-cluster ECS cluster and stores non-production data only.

Full detail: Dev NEST runbook · API staging runbook · RDS dev runbook.

Production tier

The production tier serves real users and is the source of truth for all platform data.

• nest.thephenom.app is built by Cloudflare Pages from the main branch of Phenom-Backend. Merging to main triggers an automatic production build.
• It calls the public API at api.thephenom.app, backed by the phenom-prod-postgres database (Multi-AZ, automated backups). This is a P0 data store.
• Production runs on the phenom-prod-cluster ECS cluster.

Full detail: NEST runbook · API production runbook · RDS prod runbook.

Topology

flowchart TB
    subgraph STAGING["Staging / development tier"]
        direction TB
        DN["dev-nest.thephenom.app<br/>(CF Pages · develop branch)"]
        AS["api-staging.thephenom.app<br/>(Hasura + Auth · phenom-dev-cluster)"]
        DDB[("phenom-dev-postgres<br/>(non-prod data)")]
        DN --> AS --> DDB
    end

    subgraph PROD["Production tier"]
        direction TB
        N["nest.thephenom.app<br/>(CF Pages · main branch)"]
        AP["api.thephenom.app<br/>(Hasura + Auth · phenom-prod-cluster)"]
        PDB[("phenom-prod-postgres<br/>(source of truth · P0)")]
        N --> AP --> PDB
    end

    COG["Cognito pool us-east-1_knEL7cqS3 (shared)"]
    COG -.auth.-> AS
    COG -.auth.-> AP

    STAGING ==>|"PR: develop → main (reviewed)"| PROD

Enforced workflow

These rules are mandatory. They keep the environment separation real instead of aspirational.

Branch policy

The Phenom-Backend repository has exactly two long-lived branches: develop (development / staging) and main (production). There is no prod or master branch.

• All work is done off the development (develop) branch, never directly on main. Engineers cut feature branches from develop, then merge them back into develop via PR.
• Production is advanced only by merging develop → main through a reviewed GitHub Pull Request. Never push directly to main; the main-merge is what triggers the production deploy, so it is the audit trail and the human gate.
• A change must be validated on staging (dev-nest / api-staging) before the promotion PR is opened.

# Day-to-day: feature branch → develop (auto-deploys to dev-nest / staging)
git checkout develop && git pull
git checkout -b feature/<issue>-<short-desc>
# ...work...
git push origin feature/<issue>-<short-desc>   # open PR into develop

# Promote to production: develop → main via reviewed PR (never a direct push)
gh pr create --base main --head develop \
  --title "Release: $(date +%Y-%m-%d)" \
  --body "Production release — validated on dev-nest/staging"
# After approval + merge to main, Cloudflare Pages builds nest.thephenom.app automatically.

Branch names are verified against the repo. develop → staging and main → production are the actual branches and the names wired into the Cloudflare Pages deploy triggers. If a branch is ever renamed, the Cloudflare Pages production-branch setting must be updated in lockstep, or deploys will stop firing.

Environment enforcement

• Development builds use the staging environment only. dev-nest must point NEXT_PUBLIC_API_URL at https://api-staging.thephenom.app, which is backed by phenom-dev-postgres. A development build must never read or write the production database.
• Only production uses the production environment. nest points at https://api.thephenom.app → phenom-prod-postgres. Nothing else may target the production API or production database for routine work.
• Verify the API binding in the Cloudflare Pages environment variables for the matching project (phenom-backend-dev vs phenom-backend-prod) before promoting.

Hasura migrations

Every new Hasura migration must satisfy both of these rules before it is merged:

1. Idempotent. Re-running the migration must be safe. Use IF NOT EXISTS / IF EXISTS guards, CREATE OR REPLACE, and ADD COLUMN IF NOT EXISTS, so applying it twice (or applying it after a partial failure) never errors.
2. Includes the table metadata. A migration that creates or alters a table must also ship the corresponding Hasura metadata — track the table, plus its relationships and permissions — so staging and production stay structurally identical. A schema change without its metadata is incomplete.

Apply to staging first, verify, then to production — never the reverse:

cd hasura/

# 1) Staging first
hasura migrate apply  --endpoint https://api-staging.thephenom.app \
  --admin-secret "$HASURA_ADMIN_SECRET_STAGING" --database-name default
hasura metadata apply --endpoint https://api-staging.thephenom.app \
  --admin-secret "$HASURA_ADMIN_SECRET_STAGING"

# 2) Verify on staging (dev-nest + a test query), then production
hasura migrate apply  --endpoint https://api.thephenom.app \
  --admin-secret "$HASURA_ADMIN_SECRET" --database-name default
hasura metadata apply --endpoint https://api.thephenom.app \
  --admin-secret "$HASURA_ADMIN_SECRET"

Example of an idempotent, metadata-complete migration shape:

-- up.sql — safe to re-run
CREATE TABLE IF NOT EXISTS public.sighting_report (
  id          uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  reporter_id uuid NOT NULL,
  created_at  timestamptz NOT NULL DEFAULT now()
);
ALTER TABLE public.sighting_report
  ADD COLUMN IF NOT EXISTS reviewed boolean NOT NULL DEFAULT false;

Then ensure the table is tracked and its permissions/relationships are present in Hasura metadata (metadata/databases/default/tables/) and applied with hasura metadata apply.

Infrastructure changes (Terraform in phenom-infra)

All infrastructure must be defined as code in the phenom-infra repository using Terraform. No infrastructure is created, changed, or deleted by hand in the AWS or Cloudflare consoles. Console click-ops produces drift that Terraform will later try to revert, causes staging and production to diverge silently, and leaves no review trail. If you find a resource that was created manually, import it into Terraform and reconcile — don’t leave it out-of-band.

What “infrastructure” covers here: VPC and networking, ECS Fargate clusters and services, the ALB and target groups, RDS PostgreSQL instances, ECR repositories, AWS Secrets Manager entries, IAM roles, Cognito, WAF/security rules, and the Cloudflare resources that are codified (cloudflare-*.tf). Everything the Phenom stack runs on lives in phenom-infra.

Repository layout — environments are separate Terraform roots that consume shared modules, which is what keeps staging and production isolated yet structurally identical:

phenom-infra/
├── environments/
│   ├── development/   # staging tier root — backend.tf (remote state), main.tf, locals.tf, *.tf per service
│   └── production/    # production tier root — same shape, plus cognito.tf, ecs.tf, secrets.tf, security-waf.tf, cloudflare-*.tf
└── modules/           # reusable building blocks consumed by BOTH environments
    ├── networking/    # VPC, subnets, NAT, security groups
    ├── alb/           # load balancer + target groups
    ├── ecs/           # Fargate cluster + services
    ├── rds/           # PostgreSQL
    ├── drop/          # Phenom Drop media pipeline
    ├── chat-*/        # Synapse / Hasura-lite / MCP chat stack
    └── ci-cd/         # pipeline resources

Workflow — change the module or environment config, plan, get it reviewed, then apply. Develop in the development environment first; apply to production only after it is proven in development.

# 1) Make the change in modules/ and/or the relevant environments/ root, on a feature branch.

# 2) Plan against development (staging) FIRST and review the diff.
cd environments/development
terraform init
terraform plan          # review every add/change/destroy carefully

# 3) Open a PR with the plan output. Apply development after approval.
terraform apply

# 4) Promote the same change to production only after development is verified.
cd ../production
terraform init
terraform plan          # production CI (.github/workflows/prod-infra-ci.yml) also gates this
terraform apply

Rules that make the policy enforceable:

• No console click-ops. Every resource is authored in Terraform. Manual changes are drift and must be imported back into code.
• Modules are the unit of reuse. New infrastructure is added as (or to) a module under modules/, then referenced from each environment root — never copy-pasted between development and production.
• State is remote. Each environment’s backend.tf configures remote state; never run Terraform with local-only state.
• Production is gated by CI. Changes to the production root flow through .github/workflows/prod-infra-ci.yml; review the plan before apply.
• development before production. Prove an infra change in the development environment, then promote the identical change to production.

Verify the separation is intact

# Staging app is live and talking to the staging API
curl -si https://dev-nest.thephenom.app | head -n 1            # Expect: HTTP/2 200
curl -sf -X POST https://api-staging.thephenom.app/v1/graphql \
  -H "Content-Type: application/json" \
  -d '{"query":"{ __typename }"}' | grep __typename

# Production app is live and talking to the production API
curl -si https://nest.thephenom.app | head -n 1               # Expect: HTTP/2 200
curl -sf -X POST https://api.thephenom.app/v1/graphql \
  -H "Content-Type: application/json" \
  -d '{"query":"{ __typename }"}' | grep __typename

Common failure modes

Related

• Runbooks index
• Dev NEST runbook · NEST runbook
• API staging runbook · API production runbook
• RDS dev runbook · RDS prod runbook