Infrastructure & Service Map

Complete map of all services, APIs, databases, and credentials that power the N.E.S.T. (Nexus for Evidence, Screening, and Tracking) platform. Use this when debugging cross-service issues or onboarding new team members.

Sensitive Infrastructure: This page documents credential locations and service endpoints. Access is restricted to INT team members via Cloudflare Access.

Service Architecture

graph LR
    Browser(["Browser<br/>N.E.S.T. SPA"])
    CFA(["CF Access<br/>OAuth Gate"])
    Pages(["CF Pages<br/>Static Assets"])
    Worker(["nest-api Worker<br/>/api/* &amp; /health"])
    Cognito(["AWS Cognito<br/>User Pool"])
    Hasura(["Hasura GraphQL<br/>api-staging.thephenom.app"])
    RDS(["PostgreSQL<br/>AWS RDS"])
    S3(["AWS S3<br/>Drop Media"])
    GitHub(["GitHub App<br/>Org Teams"])
    Synapse(["Synapse<br/>chat-staging.thephenom.app"])

    Browser -->|"HTTPS"| CFA
    CFA -->|"Gate entry"| Pages
    Browser -->|"Cognito SDK"| Cognito
    Cognito -->|"JWT"| Browser
    Browser -->|"Bearer JWT"| Worker
    Worker -->|"Forward JWT"| Hasura
    Hasura -->|"SQL"| RDS
    Worker -->|"Pre-signed URLs"| S3
    Worker -->|"App JWT"| GitHub
    Cognito -.->|"JWT SSO"| Synapse

    style Browser fill:#1a1a2e,stroke:#e94560,color:#fff,rx:12,ry:12
    style CFA fill:#16213e,stroke:#0f3460,color:#fff,rx:12,ry:12
    style Pages fill:#16213e,stroke:#0f3460,color:#fff,rx:12,ry:12
    style Worker fill:#0f3460,stroke:#e94560,color:#fff,rx:12,ry:12
    style Cognito fill:#1a1a2e,stroke:#e94560,color:#fff,rx:12,ry:12
    style Hasura fill:#1a1a2e,stroke:#533483,color:#fff,rx:12,ry:12
    style RDS fill:#1a1a2e,stroke:#533483,color:#fff,rx:12,ry:12
    style S3 fill:#1a1a2e,stroke:#533483,color:#fff,rx:12,ry:12
    style GitHub fill:#1a1a2e,stroke:#533483,color:#fff,rx:12,ry:12
    style Synapse fill:#1a1a2e,stroke:#e94560,color:#fff,rx:12,ry:12

                ┌─────────────────────────────────────────────┐
                │              User's Browser                  │
                │  ┌─────────────────────────────────────────┐ │
                │  │  N.E.S.T. SPA (React + CesiumJS)       │ │
                │  │  - Cognito sign-in (idToken in store)   │ │
                │  │  - Sends Bearer JWT on ALL /api/* calls │ │
                │  └──────────┬──────────────────────────────┘ │
                └─────────────┼────────────────────────────────┘
                              │
                ┌─────────────▼──┐  ┌────────────────┐
                │ AWS Cognito    │  │   nest-api     │
                │ User Pool      │  │   CF Worker    │
                │ phenom-dev-    │  │                │
                │ local          │  │  Verifies JWT  │
                └────────────────┘  │  Forwards JWT  │
                                    │  to Hasura     │
                                    └────┬───────────┘
                                         │
                          ┌──────────────┼──────────────┐
                          │              │              │
                ┌─────────▼─────┐  ┌─────▼─────┐  ┌────▼──────┐
                │ Hasura GraphQL│  │  AWS S3   │  │ GitHub    │
                │ via Cloudflare│  │ (drop     │  │ App API   │
                │ → ALB         │  │  media)   │  │ (teams)   │
                └───────┬───────┘  └───────────┘  └───────────┘
                        │
                ┌───────▼───────┐
                │  PostgreSQL    │
                │  (AWS RDS)     │
                └────────────────┘


The Worker forwards the user's Cognito JWT to Hasura on every `/api/*` call. Both the Worker and Hasura verify the same JWT against Cognito JWKS independently — no admin secret in the request path. All data (events, lists, shares, users, notifications) is served from Hasura/Postgres.

---

## Deployments

| Site | URL | Platform | Deploys From |
|------|-----|----------|-------------|
| **Production UI** | nest.thephenom.app | CF Pages (`nest-phenom`) | Merge to `main` via CI |
| **Production API** | nest-api Worker (prod) | CF Workers (`nest-api-prod`) | Merge to `main` via CI |
| **Dev UI** | dev-nest.thephenom.app | CF Pages (`dev-nest-phenom`) | Merge to `develop` via CI |
| **Dev API** | nest-api Worker | CF Workers (`nest-api`) | Merge to `develop` via CI |

### CI/CD Pipeline (`nest-ci.yml`)

All deploys are automated via GitHub Actions:

1. **Layer 1** — Typecheck & Lint (all pushes + PRs to `admin_sandbox/**`)
2. **Layer 2** — Unit Tests (depends on Layer 1)
3. **Dev Deploy** — On merge to `develop`: deploys Worker + Pages to dev-nest
4. **Prod Deploy** — On merge to `main`: deploys Worker (`--env production`) + Pages to nest


### Required DNS Records

If DNS records go missing, the service becomes unreachable. Verify these exist:

| Record | Type | Target | Proxied |
|--------|------|--------|---------|
| `nest.thephenom.app` | CNAME | `phenom-backend.pages.dev` | Yes |
| `dev-nest.thephenom.app` | CNAME | `dev-nest-phenom.pages.dev` | Yes |

**Note**: Worker Routes do NOT require separate DNS records — they intercept requests on existing domains. See [#198](https://github.com/Phenom-earth/phenom-backend/issues/198) for the 2026-04-09 DNS incident.

---

## Authentication

### AWS Cognito (Primary)

| Setting | Value |
|---------|-------|
| **Pool name (dev)** | `phenom-dev-local` |
| **Pool ID (dev)** | `us-east-1_AkG9mnbjA` |
| **Pool name (staging)** | `phenom-staging` |
| **Pool ID (staging)** | `us-east-1_n8gO6SbP6` |
| **Region** | `us-east-1` |
| **JWKS URL** | `https://cognito-idp.us-east-1.amazonaws.com/{pool-id}/.well-known/jwks.json` |
| **Frontend SDK** | `amazon-cognito-identity-js` |
| **Worker verification** | `nest-api/src/auth.ts` → `verifyCognitoJwt()` |
| **Hasura verification** | `HASURA_GRAPHQL_JWT_SECRET` env var, points at the same JWKS |

The frontend client IDs are scoped to specific apps (`phenom-dev-hasura-client`, `phenom-dev-web-chat`, etc.) and visible in the AWS console under each pool's "App integration" tab.

The full request chain is documented in [the platform overview](https://int-docs.thephenom.app/docs/project/nest/#authentication-flow).

### Cloudflare Access (Login Gate)

CF Access still gates entry to N.E.S.T. domains. Once past the CF Access gate, all API calls use Cognito Bearer tokens — no more `CF_Authorization` cookie auth for API routes.

| Setting | Value |
|---------|-------|
| **Team domain** | `thephenom-app.cloudflareaccess.com` |
| **IdP type** | GitHub OAuth App |
| **OAuth App Client ID** | `Ov23li8PtxSU0LskOBU9` |
| **OAuth App secret** | `pass sanmarcsoft/github-oauth/phenom-oauth-client-secret` |
| **Callback URL** | `https://thephenom-app.cloudflareaccess.com/cdn-cgi/access/callback` |
| **CF Account** | Pavestar (`pass cloudflare/phenom-account-id`) |

**Access Apps** (all use the same IdP):

| App | Domain | Policy |
|-----|--------|--------|
| nest | nest.thephenom.app | GitHub login required |
| Dev NEST Dashboard | dev-nest.thephenom.app | GitHub login + IP 84.112.12.104 (Matt), Phenom-Earth/INT team |
| The Phenom App Docs | int-docs.thephenom.app | GitHub login required |
| phenom-backend Pages | *.phenom-backend.pages.dev | GitHub login required |

### GitHub App (Team Data)

| Setting | Value |
|---------|-------|
| **App name** | N.E.S.T. Team |
| **App ID** | `3085870` |
| **Client ID** | `Iv23li5FmbWUV3ME5ZOt` |
| **Client secret** | `pass sanmarcsoft/github-oauth/phenom-nest-client-secret` |
| **Purpose** | Pull INT team member profiles from `Phenom-earth` GitHub org |
| **NOT used for** | CF Access login (that's the OAuth App above) |




CRITICAL: The OAuth App and GitHub App are DIFFERENT things. The OAuth App (Ov23li...) controls CF Access login. The GitHub App (Iv23li...) pulls team data. Never overwrite the CF Access IdP with GitHub App credentials — this breaks production login for ALL sites.




### Cognito User Pool (Unified Auth)

NEST and Synapse chat share a single Amazon Cognito user pool. Users authenticate once and access both services without re-entering credentials.

| Setting | Value |
|---------|-------|
| **User Pool ID** | `us-east-1_n8gO6SbP6` |
| **Client ID** | `3krb3en56c5qhga7vomtetbkui` (phenom-dev-web-chat, public client) |
| **Client name** | `phenom-dev-web-chat` |
| **Region** | `us-east-1` |
| **Shared with** | Synapse chat (`chat-staging.thephenom.app`, `chat.thephenom.app`, `chat-testing.thephenom.app`) |




Pool Unification (#202): NEST and Synapse chat were unified onto pool us-east-1_n8gO6SbP6 to enable single sign-on across services. Both VITE_COGNITO_USER_POOL_ID and VITE_COGNITO_CLIENT_ID are injected by CI and must match this pool. See the CI/CD Environment Variables section for the full list of required secrets.




---

## Data Sources

### Hasura GraphQL (Primary — all N.E.S.T. data)

| Setting | Value |
|---------|-------|
| **Staging endpoint** | `https://api-staging.thephenom.app/v1/graphql` |
| **ALB DNS** | `phenom-dev-alb-1007680551.us-east-1.elb.amazonaws.com` |
| **Cloudflare DNS** | `api-staging` CNAME → ALB (proxy enabled, Flexible SSL) |
| **Target group** | `phenom-dev-graphql-tg` (port 8080) |
| **Database** | PostgreSQL 17.4 on AWS RDS |
| **Schema migrations** | `phenom-backend/hasura/migrations/default/` |
| **Metadata** | `phenom-backend/hasura/metadata/` |
| **Auth** | Cognito JWT (forwarded from the user, not an admin secret) |
| **Hasura deploy workflow** | `deploy-hasura-development.yaml` (on PR merge to `develop`) |

**Tables used by `nest-api`**:
- **Events**: `phenom`, `drops`, `phenom_media` (via `/api/events`)
- **Lists**: `lists`, `list_items` (CRUD via `/api/lists`)
- **Sharing**: `list_shares`, `item_shares` (via `/api/shares`, `/api/lists/:id/share`)
- **Users**: `users` (via `/api/users`, `/api/users/me`)
- **Transcriptions**: `transcriptions` (cached video transcription/translation)

**Other tables in the schema**: `phenom_coords`, `phenom_sensor_data`, `phenom_shoots`, `phenom_categories`, `phenom_behavior_types`, `behavior_types`, `categories`, `roles`. See `hasura/metadata/databases/default/tables/` for the full list.

**Backfill from Firestore**: `phenom-backend/hasura/scripts/migrate_firebase_to_postgres.py` is the idempotent migration script (Cognito user resolution included). Re-runnable any time to catch new Firestore writes from the legacy Buzzard mobile app while it is being phased out.

### Firebase Firestore (Retired)

| Collection | Source | Status |
|------------|--------|--------|
| `phenom` | Buzzard (legacy mobile app) | Data migrated to Hasura |
| `phenom/{id}/shoots` | Buzzard | Data migrated to Hasura |
| `drops` | Phenom Drop web app | Data migrated to Hasura |

**Firebase Project**: `phenom-7ee1a`
**Status**: `firebase.ts` has been deleted from nest-api. The `FIREBASE_SERVICE_ACCOUNT` Worker secret should be removed from Cloudflare. The backfill script can still be re-run to capture any final Buzzard writes during sunset.

### AWS S3 (Drop Media)

| Setting | Value |
|---------|-------|
| **Bucket** | `phenom-dev-media-staging` |
| **Region** | `us-east-1` |
| **URL format** | `https://phenom-dev-media-staging.s3.us-east-1.amazonaws.com/{s3Key}` |
| **Access** | Public GET for `uploads/*` prefix, restricted by Referer |

**CORS Configuration** (`docs/s3-cors.json`) — two rules as of 2026-03-14:

**Rule 1 — N.E.S.T. origins** (GET, HEAD):
- `https://nest.thephenom.app`
- `https://dev-nest.thephenom.app`
- `https://phenom.matthewstevens.org`

**Rule 2 — Drop origins** (GET, PUT, HEAD; exposes `ETag` header):
- `https://www.thephenom.app`
- `https://thephenom.app`

> Drop origins require PUT for presigned URL uploads and HEAD for integrity checks. The `ETag` exposed header is needed so the client can verify upload success.

**Bucket Policy Referers** (`docs/s3-bucket-policy.json`):
- `https://nest.thephenom.app/*`
- `https://dev-nest.thephenom.app/*`
- `https://phenom.matthewstevens.org/*`




Adding a new domain: If you deploy N.E.S.T. or Drop to a new domain, you MUST update the S3 CORS config (the appropriate rule for the site type) AND the bucket policy to include the new origin. N.E.S.T. domains need GET + HEAD; Drop domains need GET + PUT + HEAD. Without this, media loading or uploads will fail with CORS errors.




### Cloudflare D1 (Retired)

D1 SQLite (`nest-db`, ID `c7929d36-0f8e-4343-8f18-99bee03e326e`) previously stored lists, shares, and users. All tables have been migrated to Hasura/Postgres via the `20260409100000_nest_tables` migration. The D1 binding has been removed from `wrangler.toml`.

---

## API Endpoints

### nest-api (Cloudflare Worker)

Lives at: `admin_sandbox/nest-api/` (TypeScript, deployed via `wrangler deploy`).

All routes require `Authorization: Bearer <Cognito JWT>`. The Worker verifies the JWT and forwards it to Hasura for row-level security.

| Endpoint | Method | Purpose |
|----------|--------|---------|
| `/health` | GET | Health check (no auth) |
| `/api/events` | GET | Phenom + drop events from Hasura, with presigned drop media URLs |
| `/api/media/sign` | GET | Generate presigned S3 GET URL for an `uploads/*` key |
| `/api/lists` | GET / POST | List or create user's curated lists |
| `/api/lists/:id` | PATCH / DELETE | Rename or delete a list |
| `/api/lists/:id/items` | GET / POST | Items in a list |
| `/api/lists/:id/items/:itemId` | DELETE | Remove item from list |
| `/api/lists/:id/share` | POST | Share list with team member |
| `/api/lists/:id/share/:userId` | DELETE | Unshare list |
| `/api/shares` | POST | Share individual item (phenom or drop) |
| `/api/shares/inbox` | GET | Items shared with me |
| `/api/shares/:id/seen` | PATCH | Mark share as seen |
| `/api/users` | GET | All known users |
| `/api/users/me` | GET / PATCH | Current user profile + notification counts |
| `/api/notifications/count` | GET | Unseen share counts |
| `/api/teams` | GET | GitHub org teams (requires GitHub App) |
| `/api/teams/:slug/members` | GET | Team members |

### Drop Upload API (AWS Lambda)

| Setting | Value |
|---------|-------|
| **Endpoint** | `https://lxd3gpzlph.execute-api.us-east-1.amazonaws.com/development/upload/generate-url` |
| **Purpose** | Generate presigned S3 PUT URL for drop upload |
| **Auth** | Upload password (server-side only) |

### Drop Backend (Python)

| Setting | Value |
|---------|-------|
| **Repo** | `phenom-drop/backend/server.py` |
| **Endpoints** | `/api/drop/send-password`, `/api/drop/verify-password`, `/api/drop/upload` |
| **Purpose** | Email verification, hash registry, S3 upload proxy |

---

## Drop Upload Flow

User (phenom-drop web app) │ ├─1─▶ Verify C2PA credentials (client-side) │ ├─2─▶ POST /api/drop/send-password │ → Backend sends 6-char OTP via Brevo email │ ├─3─▶ POST /api/drop/verify-password │ → Backend validates OTP + registers hash │ ├─4─▶ POST /api/drop/upload │ → Backend proxies to AWS Lambda │ → Lambda returns presigned S3 PUT URL │ → Backend writes metadata to the drops table in Hasura/Postgres │ ├─5─▶ PUT to S3 presigned URL (direct upload from browser) │ → File stored at s3://phenom-dev-media-staging/uploads/… │ → S3 CORS Rule 2 allows PUT from Drop origins │ └─6─▶ Backend records mediaUrl + s3Key → N.E.S.T. reads drops from Hasura via /api/events → Worker presigns the s3_key on demand if mediaUrl is null


**Note** (2026-03-14): `computeFileHash()` in `www/web/drop.html` now uses streaming SHA-256 (chunked reads via `ReadableStream`) for files >100MB. This avoids loading the entire file into memory at once, preventing browser tab crashes on large uploads.

---

## Credential Locations (GPG Pass)

| Path | Purpose |
|------|---------|
| `cloudflare/phenom-account-id` | CF account ID (Pavestar) |
| `cloudflare/phenom-zone-id` | thephenom.app zone ID |
| `cloudflare/phenom-api-token` | CF API token (limited perms) |
| `verifieddit/CLOUDFLARE_API_KEY` | Global API Key (works across accounts) |
| `verifieddit/CLOUDFLARE_AUTH_EMAIL` | Auth email for Global API Key |
| `sanmarcsoft/github-oauth/phenom-oauth-client-id` | OAuth App for CF Access |
| `sanmarcsoft/github-oauth/phenom-oauth-client-secret` | OAuth App secret |
| `sanmarcsoft/github-oauth/phenom-nest-client-id` | GitHub App for team data |
| `sanmarcsoft/github-oauth/phenom-nest-client-secret` | GitHub App secret |

---

## CesiumJS Globe Configuration

### Imagery Layers (bottom to top)

| Layer | Provider | Visibility |
|-------|----------|-----------|
| Day imagery | Esri World Imagery (satellite) | `dayAlpha=1.0`, `nightAlpha=0.0` |
| Night imagery | NASA VIIRS Black Marble 2012 | `dayAlpha=0.0`, `nightAlpha=1.0` |
| Political borders | CartoDB dark_only_labels | Toggled via toolbar button |
| Google 3D Tiles | Google Maps Tile API | Hidden above 500km altitude |

### Google Maps Tile API

| Setting | Value |
|---------|-------|
| **API Key** | `AIzaSyDLi52bh9UXdiqZows69Z41S6qWnsPwzII` |
| **Endpoint** | `https://tile.googleapis.com/v1/3dtiles/root.json?key=...` |
| **Loaded via** | `Cesium3DTileset.fromUrl()` |
| **Night dimming** | `imageBasedLightingFactor` adjusted per frame by sun angle |

### Day/Night Effects

| Effect | Mechanism | Scope |
|--------|-----------|-------|
| Globe day/night | `globe.enableLighting = true` + `SunLight` | Base globe surface |
| Atmosphere | `dynamicAtmosphereLighting` + `dynamicAtmosphereLightingFromSun` | Sky limb |
| 3D Tile dimming | `imageBasedLightingFactor` per frame | Google 3D Tiles only |
| Night vision | CSS `hue-rotate + brightness + desaturate` filter | Entire canvas below 500km |
| Tile visibility | `tileset.show = height < 500km` | Reveals day/night imagery at global view |

### Clock

- Default: current real time (`JulianDate.now()`)
- On marker click: jumps to recording timestamp
- On deselect: resets to real time
- `shouldAnimate = false` — time only changes on marker selection

---

## Build & Deploy Checklist

Before deploying any changes:

1. **Google Maps key** is hardcoded in `CesiumGlobe.tsx` (not an env var)
2. **S3 CORS** must include the deployment domain
3. **CF Access** must have an app + policy for the deployment domain
4. **OAuth App callback** must be `https://thephenom-app.cloudflareaccess.com/cdn-cgi/access/callback`
5. **Worker secrets** must be set via `wrangler secret put`: `GITHUB_APP_ID`, `GITHUB_APP_PRIVATE_KEY`, `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` (no admin secret needed — all Hasura queries use the caller's Cognito JWT)
6. **GitHub environment** `Development` must have `HASURA_ENDPOINT` variable set to `https://api-staging.thephenom.app`

---

**Last Updated**: 2026-04-17

Feedback

Was this page helpful?

Glad to hear it! Please tell us how we can improve.

Sorry to hear that. Please tell us how we can improve.