Deployment Guide

Terraform deployment procedures, Docker image builds, DNS configuration, and post-deployment setup for the Phenom Chat system.

Reconciling to the canonical page. The single source of truth for chat services is Chat Services (validated 2026-05-27). Parts of this page may still describe the retired dual-implementation design. Chat is now Matrix/Synapse only behind an mTLS-locked origin. Where this page disagrees with the canonical page, the canonical page wins.

This guide covers deploying and updating the Phenom Chat infrastructure. It assumes familiarity with Terraform, AWS CLI, and Docker.

Prerequisites

Tool Version Purpose
Terraform >= 1.0 Infrastructure as code
AWS CLI v2 AWS resource management
Docker Latest Image builds (on ai.matthewstevens.org only)
SSH access ai alias configured Build server for Docker images

Terraform Modules Overview

The chat system is composed of five Terraform modules, all under modules/ in the phenom-infra repository:

Module Purpose Key Resources
chat-shared Shared Cognito clients and secrets Cognito OIDC client (for Synapse), Cognito agent client (for MCP), Secrets Manager secret
chat-link-preview Link preview resolution Lambda function, IAM role, CloudWatch log group
chat-hasura-lite Chat schema and Hasura metadata SQL migration script, metadata setup script (run as ECS tasks)
chat-synapse Synapse homeserver and Admin UI ECS services (Synapse + Admin UI), ALB rules, security groups, RDS schema, Synapse config in Secrets Manager
chat-mcp-server MCP tool server ECS service, ALB rules, target group

Module Dependency Graph

graph LR SHARED["chat-shared
Cognito clients
Secrets Manager"] LP["chat-link-preview
Lambda function"] HL["chat-hasura-lite
SQL migration
Hasura metadata"] SYN["chat-synapse
Synapse + Admin UI
ECS services"] MCP["chat-mcp-server
MCP server
ECS service"] SHARED --> SYN SHARED --> MCP LP --> HL HL --> MCP style SHARED fill:#151515,color:#e0e0e0,rx:30 style MCP fill:#121010,color:#a5e3e8,rx:30

Database Migrations

SQL Migration (Hasura Lite Schema)

The schema migration file is at modules/chat-hasura-lite/sql/001-chat-schema.sql. It creates all five chat tables, indexes, triggers, and seeds the three rooms.

To run the migration as an ECS task:

# 1. Register a one-shot task definition using postgres:16-alpine
aws ecs register-task-definition \
  --family phenom-dev-chat-migration \
  --network-mode awsvpc \
  --requires-compatibilities FARGATE \
  --cpu 256 --memory 512 \
  --execution-role-arn <ecs-task-execution-role-arn> \
  --container-definitions '[{
    "name": "migration",
    "image": "postgres:16-alpine",
    "command": ["psql", "-h", "<rds-host>", "-U", "<db-user>", "-d", "<db-name>", "-f", "/sql/001-chat-schema.sql"],
    "essential": true,
    "logConfiguration": {
      "logDriver": "awslogs",
      "options": {
        "awslogs-group": "/ecs/phenom-dev-chat-migration",
        "awslogs-region": "us-east-1",
        "awslogs-stream-prefix": "migration"
      }
    }
  }]'

# 2. Run the task
aws ecs run-task \
  --cluster phenom-dev-cluster \
  --task-definition phenom-dev-chat-migration \
  --launch-type FARGATE \
  --network-configuration '{
    "awsvpcConfiguration": {
      "subnets": ["<private-subnet-1>", "<private-subnet-2>"],
      "securityGroups": ["<ecs-tasks-sg>"],
      "assignPublicIp": "DISABLED"
    }
  }'

Monitor the task in CloudWatch logs to confirm successful completion.


Docker Image Builds

Registry: ECR, pinned by digest (both environments)

Staging and production Synapse run the same image from the in-account ECR repository phenom-synapse, pinned by digest in Terraform:

  • production: environments/production/main.tf, module chat_synapse, synapse_image
  • staging: environments/development/main.tf, module chat_synapse_staging, synapse_image

ECS pulls ECR images through the task execution role (AmazonECSTaskExecutionRolePolicy), so neither environment sets a registry credential (dockerhub_credentials_arn = ""). The admin UI uses the public awesometechnologies/synapse-admin image and needs no credential either.

Admin UI per environment. Production serves the Synapse Admin UI at chat-admin.thephenom.app, behind Cloudflare Access. Staging does not run it (enable_admin_ui = false, phenom-infra#327): since the #94 lockdown nothing forwards to its target group there, so ECS rejected every update to the service.

Until 2026-09-25, staging pulled applepublicdotcom/phenom-synapse:testing from a private Docker Hub repo with phenom-dev-dockerhub-credentials. Those credentials went bad, and staging Synapse and its admin UI both stopped with CannotPullContainerError ... 401 Unauthorized. Moving staging to prod’s ECR image removed the Docker Hub dependency entirely (phenom-infra#325).

Synapse wrapper image

The custom Synapse image includes a startup wrapper that injects homeserver.yaml from the SYNAPSE_CONFIG_YAML environment variable (sourced from Secrets Manager). Its source is modules/chat-synapse/docker.

To ship a new build:

  1. Build for linux/amd64 and push to ECR:

    aws ecr get-login-password --region us-east-1 \
      | docker login --username AWS --password-stdin 657033058608.dkr.ecr.us-east-1.amazonaws.com
    docker buildx build --platform linux/amd64 --push \
      -t 657033058608.dkr.ecr.us-east-1.amazonaws.com/phenom-synapse:<tag> \
      modules/chat-synapse/docker
    
  2. Get the pushed digest from aws ecr describe-images --repository-name phenom-synapse.

  3. Set that digest as synapse_image in the staging module first. Merge, verify staging, then do the same for production.

Changing the digest replaces the task definition. The pipeline counts that as a destroy (it deregisters the old revision), so the dev and prod auto-applies skip it and it ships through a confirmed workflow_dispatch apply.

MCP server (decommissioned)

The chat MCP server was decommissioned in phenom-infra#260 in both environments. Its source still lives at modules/chat-mcp-server/app, but no service runs it and CI no longer builds or pushes its image.


DNS Configuration

Chat uses Cloudflare CNAME records pointing to the ALB:

Record Type Value Proxy
chat-staging.thephenom.app CNAME phenom-dev-alb-c4a9c9XX.us-east-1.elb.amazonaws.com DNS only (grey cloud)
chat.thephenom.app CNAME phenom-prod-alb-1196696419.us-east-1.elb.amazonaws.com DNS only (grey cloud)

chat.thephenom.app is production. It is served by the prod cluster on the prod ALB. Mobile clients should target it. chat-staging is the official staging tier (isolated ECS / Cognito OIDC / Synapse logical DB) for pre-prod work.


ALB Routing Rules

The ALB listener routes traffic based on host header and path pattern. Multiple hostnames share the same target groups.

chat-staging.thephenom.app

Priority Host Header Path Pattern Target Group Port
230 chat-staging.thephenom.app /_matrix/*, /_synapse/* phenom-dev-syn-tg 8008
240 chat-staging.thephenom.app /chat-admin/* phenom-dev-syn-admin-tg 80
250 chat-staging.thephenom.app /mcp/* phenom-dev-mcp-tg 3001
260 chat-staging.thephenom.app /.well-known/* phenom-dev-syn-tg 8008

chat.thephenom.app (production, prod ALB)

chat.thephenom.app is served by the prod ALB (phenom-prod-alb-1196696419), not the dev ALB. Module-internal rules in modules/chat-synapse/ and modules/chat-mcp-server/ are scoped by host_header = "chat.thephenom.app" automatically; only the Hasura /v1/* rule lives at the env level (environments/production/chat.tf).

Priority Host Header Path Pattern Target Group Port
195 chat.thephenom.app /v1/* (Hasura) prod graphql TG (env-level rule)
module-internal chat.thephenom.app /_matrix/*, /_synapse/* prod Synapse TG 8008
module-internal chat.thephenom.app /chat-admin/* prod Synapse-admin TG 80
module-internal chat.thephenom.app /mcp/* prod MCP TG 3001

These rules are defined in:

  • modules/chat-synapse/alb-rules.tf (Synapse and Admin UI rules)
  • modules/chat-mcp-server/alb-rules.tf (MCP rules)

Cognito Client Setup

The chat-shared module creates two Cognito User Pool clients:

OIDC Client (for Synapse SSO)

resource "aws_cognito_user_pool_client" "synapse_oidc" {
  name                    = "${var.project_name}-synapse-oidc"
  generate_secret         = true
  allowed_oauth_flows     = ["code"]
  allowed_oauth_scopes    = ["openid", "email", "profile"]
  callback_urls           = [
    "https://${var.synapse_server_name}/_synapse/client/oidc/callback"
  ]
  supported_identity_providers = ["COGNITO"]
}

This client uses the authorization code flow. Synapse’s OIDC provider configuration references this client’s ID and secret.

Agent Client (for MCP Server)

resource "aws_cognito_user_pool_client" "agent" {
  name            = "${var.project_name}-chat-agent"
  generate_secret = true
  explicit_auth_flows = [
    "ALLOW_USER_PASSWORD_AUTH",
    "ALLOW_REFRESH_TOKEN_AUTH",
  ]
}

This client uses USER_PASSWORD_AUTH for machine-to-machine authentication. The MCP server authenticates using AGENT_USERNAME and AGENT_PASSWORD environment variables.

Both client credentials are stored in Secrets Manager under phenom-dev-development-chat-clients.


Post-Deployment Steps

After terraform apply completes successfully:

1. Run Database Migration

Execute the SQL migration ECS task (see Database Migrations above) to create the chat tables and seed the three rooms.

2. Configure Hasura Metadata

Track the chat tables in Hasura and set up relationships and permissions:

# Run the metadata setup script as an ECS task (uses curl image)
# The script calls Hasura metadata API endpoints to:
# - Track all 5 chat tables
# - Create foreign key relationships
# - Set up role-based permissions (user, support, admin)
# - Configure event triggers for link preview resolution

3. Create Synapse Admin Bot Account

Register an admin account on Synapse using the registration shared secret:

# Get the registration shared secret from Secrets Manager
aws secretsmanager get-secret-value \
  --secret-id phenom-dev-synapse-homeserver-config \
  --query 'SecretString' \
  --output text | grep registration_shared_secret

# Register admin user via Synapse admin API
curl -X POST "https://chat-staging.thephenom.app/_synapse/admin/v1/register" \
  -H "Content-Type: application/json" \
  -d '{
    "nonce": "<get from GET /_synapse/admin/v1/register>",
    "username": "admin",
    "password": "<strong-password>",
    "admin": true
  }'

4. Create Matrix Rooms

Create the three chat rooms on Synapse:

# Using the admin access token from step 3
curl -X POST "https://chat-staging.thephenom.app/_matrix/client/v3/createRoom" \
  -H "Authorization: Bearer <admin-access-token>" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Phenom Internal",
    "topic": "Internal team chat",
    "visibility": "private",
    "preset": "private_chat"
  }'

# Repeat for Phenom Partners and Phenom Community

5. Verify Services

# Check all ECS services are running
aws ecs describe-services --region us-east-1 \
  --cluster phenom-dev-cluster \
  --services phenom-dev-synapse phenom-dev-synapse-admin phenom-dev-chat-mcp \
  --query 'services[*].{Name:serviceName,Running:runningCount,Status:status}' \
  --output table

# Verify health endpoints
curl -s https://chat-staging.thephenom.app/mcp/health | jq .
curl -s https://chat-staging.thephenom.app/_matrix/client/v3/login | jq .

Environment Wiring

All chat modules are wired together in environments/development/main.tf. The key variable connections:

module "chat_shared" {
  source               = "../../modules/chat-shared"
  project_name         = local.project_name
  environment          = "development"
  cognito_user_pool_id = module.cognito.user_pool_id
  cognito_domain       = module.cognito.domain
  synapse_server_name  = "chat-staging.thephenom.app"
}

module "chat_synapse" {
  source                     = "../../modules/chat-synapse"
  project_name               = local.project_name
  environment                = "development"
  vpc_id                     = module.networking.vpc_id
  private_subnet_ids         = module.networking.private_subnet_ids
  ecs_cluster_id             = module.ecs.cluster_id
  alb_listener_arn           = module.alb.https_listener_arn
  alb_security_group_id      = module.alb.security_group_id
  ecs_tasks_security_group_id = module.networking.ecs_tasks_sg_id
  server_name                = "chat-staging.thephenom.app"
  rds_host                   = module.rds.endpoint
  rds_port                   = module.rds.port
  rds_master_username        = module.rds.master_username
  rds_master_password        = module.rds.master_password
  cognito_user_pool_id       = module.cognito.user_pool_id
  cognito_oidc_client_id     = module.chat_shared.synapse_oidc_client_id
  cognito_oidc_client_secret = module.chat_shared.synapse_oidc_client_secret
  dockerhub_credentials_arn  = var.dockerhub_credentials_arn
}

module "chat_mcp_server" {
  source                     = "../../modules/chat-mcp-server"
  project_name               = local.project_name
  vpc_id                     = module.networking.vpc_id
  private_subnet_ids         = module.networking.private_subnet_ids
  ecs_cluster_id             = module.ecs.cluster_id
  alb_listener_arn           = module.alb.https_listener_arn
  ecs_tasks_security_group_id = module.networking.ecs_tasks_sg_id
  server_name                = "chat-staging.thephenom.app"
  hasura_endpoint            = "http://${module.ecs.graphql_service_name}:8080/v1/graphql"
  chat_secrets_arn           = module.secrets.chat_secrets_arn
  backend_type               = "hasura"
  dockerhub_credentials_arn  = var.dockerhub_credentials_arn
}

Deployment Pipeline

The standard deployment flow for chat infrastructure changes:

flowchart TD A["Code change in phenom-infra"] --> B["terraform plan"] B --> C{"Review plan output"} C -->|"Approved"| D["terraform apply"] C -->|"Rejected"| A D --> E{"Docker image changes?"} E -->|"Yes"| F["Push image to ECR phenom-synapse
Pin new digest in Terraform"] E -->|"No"| G["Skip image build"] F --> H["Terraform apply rolls the ECS service"] G --> H H --> I["Verify health endpoints"] I --> J{"Schema changes?"} J -->|"Yes"| K["Run migration ECS task
Update Hasura metadata"] J -->|"No"| L["Deployment complete"] K --> L style A fill:#1a1a1a,color:#fff,rx:30 style L fill:#121010,color:#a5e3e8,rx:30