Deployment Guide
Reconciling to the canonical page. The single source of truth for chat services is Chat Services (validated 2026-05-27). Parts of this page may still describe the retired dual-implementation design. Chat is now Matrix/Synapse only behind an mTLS-locked origin. Where this page disagrees with the canonical page, the canonical page wins.
This guide covers deploying and updating the Phenom Chat infrastructure. It assumes familiarity with Terraform, AWS CLI, and Docker.
Prerequisites
| Tool | Version | Purpose |
|---|---|---|
| Terraform | >= 1.0 | Infrastructure as code |
| AWS CLI | v2 | AWS resource management |
| Docker | Latest | Image builds (on ai.matthewstevens.org only) |
| SSH access | ai alias configured |
Build server for Docker images |
Important
Never rundocker build on the NAS. All Docker image builds happen on ai.matthewstevens.org. The NAS is a deployment target only – it pulls pre-built images.
Terraform Modules Overview
The chat system is composed of five Terraform modules, all under modules/ in the phenom-infra repository:
| Module | Purpose | Key Resources |
|---|---|---|
chat-shared |
Shared Cognito clients and secrets | Cognito OIDC client (for Synapse), Cognito agent client (for MCP), Secrets Manager secret |
chat-link-preview |
Link preview resolution | Lambda function, IAM role, CloudWatch log group |
chat-hasura-lite |
Chat schema and Hasura metadata | SQL migration script, metadata setup script (run as ECS tasks) |
chat-synapse |
Synapse homeserver and Admin UI | ECS services (Synapse + Admin UI), ALB rules, security groups, RDS schema, Synapse config in Secrets Manager |
chat-mcp-server |
MCP tool server | ECS service, ALB rules, target group |
Module Dependency Graph
Cognito clients
Secrets Manager"] LP["chat-link-preview
Lambda function"] HL["chat-hasura-lite
SQL migration
Hasura metadata"] SYN["chat-synapse
Synapse + Admin UI
ECS services"] MCP["chat-mcp-server
MCP server
ECS service"] SHARED --> SYN SHARED --> MCP LP --> HL HL --> MCP style SHARED fill:#151515,color:#e0e0e0,rx:30 style MCP fill:#121010,color:#a5e3e8,rx:30
Database Migrations
Why ECS tasks?
RDS is in a private VPC subnet unreachable from the Terraform runner (your laptop or CI server). Database migrations and Hasura metadata configuration must run as one-shot ECS Fargate tasks inside the VPC.SQL Migration (Hasura Lite Schema)
The schema migration file is at modules/chat-hasura-lite/sql/001-chat-schema.sql. It creates all five chat tables, indexes, triggers, and seeds the three rooms.
To run the migration as an ECS task:
# 1. Register a one-shot task definition using postgres:16-alpine
aws ecs register-task-definition \
--family phenom-dev-chat-migration \
--network-mode awsvpc \
--requires-compatibilities FARGATE \
--cpu 256 --memory 512 \
--execution-role-arn <ecs-task-execution-role-arn> \
--container-definitions '[{
"name": "migration",
"image": "postgres:16-alpine",
"command": ["psql", "-h", "<rds-host>", "-U", "<db-user>", "-d", "<db-name>", "-f", "/sql/001-chat-schema.sql"],
"essential": true,
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/ecs/phenom-dev-chat-migration",
"awslogs-region": "us-east-1",
"awslogs-stream-prefix": "migration"
}
}
}]'
# 2. Run the task
aws ecs run-task \
--cluster phenom-dev-cluster \
--task-definition phenom-dev-chat-migration \
--launch-type FARGATE \
--network-configuration '{
"awsvpcConfiguration": {
"subnets": ["<private-subnet-1>", "<private-subnet-2>"],
"securityGroups": ["<ecs-tasks-sg>"],
"assignPublicIp": "DISABLED"
}
}'
Monitor the task in CloudWatch logs to confirm successful completion.
Docker Image Builds
Registry: ECR, pinned by digest (both environments)
Staging and production Synapse run the same image from the in-account ECR repository phenom-synapse, pinned by digest in Terraform:
- production:
environments/production/main.tf, modulechat_synapse,synapse_image - staging:
environments/development/main.tf, modulechat_synapse_staging,synapse_image
ECS pulls ECR images through the task execution role (AmazonECSTaskExecutionRolePolicy), so neither environment sets a registry credential (dockerhub_credentials_arn = ""). The admin UI uses the public awesometechnologies/synapse-admin image and needs no credential either.
Admin UI per environment. Production serves the Synapse Admin UI at chat-admin.thephenom.app, behind Cloudflare Access. Staging does not run it (enable_admin_ui = false, phenom-infra#327): since the #94 lockdown nothing forwards to its target group there, so ECS rejected every update to the service.
Until 2026-09-25, staging pulled applepublicdotcom/phenom-synapse:testing from a private Docker Hub repo with phenom-dev-dockerhub-credentials. Those credentials went bad, and staging Synapse and its admin UI both stopped with CannotPullContainerError ... 401 Unauthorized. Moving staging to prod’s ECR image removed the Docker Hub dependency entirely (phenom-infra#325).
Synapse wrapper image
The custom Synapse image includes a startup wrapper that injects homeserver.yaml from the SYNAPSE_CONFIG_YAML environment variable (sourced from Secrets Manager). Its source is modules/chat-synapse/docker.
To ship a new build:
-
Build for
linux/amd64and push to ECR:aws ecr get-login-password --region us-east-1 \ | docker login --username AWS --password-stdin 657033058608.dkr.ecr.us-east-1.amazonaws.com docker buildx build --platform linux/amd64 --push \ -t 657033058608.dkr.ecr.us-east-1.amazonaws.com/phenom-synapse:<tag> \ modules/chat-synapse/docker -
Get the pushed digest from
aws ecr describe-images --repository-name phenom-synapse. -
Set that digest as
synapse_imagein the staging module first. Merge, verify staging, then do the same for production.
Changing the digest replaces the task definition. The pipeline counts that as a destroy (it deregisters the old revision), so the dev and prod auto-applies skip it and it ships through a confirmed workflow_dispatch apply.
MCP server (decommissioned)
The chat MCP server was decommissioned in phenom-infra#260 in both environments. Its source still lives at modules/chat-mcp-server/app, but no service runs it and CI no longer builds or pushes its image.
DNS Configuration
Chat uses Cloudflare CNAME records pointing to the ALB:
| Record | Type | Value | Proxy |
|---|---|---|---|
chat-staging.thephenom.app |
CNAME | phenom-dev-alb-c4a9c9XX.us-east-1.elb.amazonaws.com |
DNS only (grey cloud) |
chat.thephenom.app |
CNAME | phenom-prod-alb-1196696419.us-east-1.elb.amazonaws.com |
DNS only (grey cloud) |
chat.thephenom.app is production. It is served by the prod cluster on the prod ALB. Mobile clients should target it. chat-staging is the official staging tier (isolated ECS / Cognito OIDC / Synapse logical DB) for pre-prod work.
ALB Routing Rules
The ALB listener routes traffic based on host header and path pattern. Multiple hostnames share the same target groups.
chat-staging.thephenom.app
| Priority | Host Header | Path Pattern | Target Group | Port |
|---|---|---|---|---|
| 230 | chat-staging.thephenom.app |
/_matrix/*, /_synapse/* |
phenom-dev-syn-tg |
8008 |
| 240 | chat-staging.thephenom.app |
/chat-admin/* |
phenom-dev-syn-admin-tg |
80 |
| 250 | chat-staging.thephenom.app |
/mcp/* |
phenom-dev-mcp-tg |
3001 |
| 260 | chat-staging.thephenom.app |
/.well-known/* |
phenom-dev-syn-tg |
8008 |
chat.thephenom.app (production, prod ALB)
chat.thephenom.app is served by the prod ALB (phenom-prod-alb-1196696419), not the dev ALB. Module-internal rules in modules/chat-synapse/ and modules/chat-mcp-server/ are scoped by host_header = "chat.thephenom.app" automatically; only the Hasura /v1/* rule lives at the env level (environments/production/chat.tf).
| Priority | Host Header | Path Pattern | Target Group | Port |
|---|---|---|---|---|
| 195 | chat.thephenom.app |
/v1/* (Hasura) |
prod graphql TG |
(env-level rule) |
| module-internal | chat.thephenom.app |
/_matrix/*, /_synapse/* |
prod Synapse TG | 8008 |
| module-internal | chat.thephenom.app |
/chat-admin/* |
prod Synapse-admin TG | 80 |
| module-internal | chat.thephenom.app |
/mcp/* |
prod MCP TG | 3001 |
Note
The prod cluster serveschat.thephenom.app directly. Mobile clients should target chat.thephenom.app.
These rules are defined in:
modules/chat-synapse/alb-rules.tf(Synapse and Admin UI rules)modules/chat-mcp-server/alb-rules.tf(MCP rules)
Cognito Client Setup
The chat-shared module creates two Cognito User Pool clients:
OIDC Client (for Synapse SSO)
resource "aws_cognito_user_pool_client" "synapse_oidc" {
name = "${var.project_name}-synapse-oidc"
generate_secret = true
allowed_oauth_flows = ["code"]
allowed_oauth_scopes = ["openid", "email", "profile"]
callback_urls = [
"https://${var.synapse_server_name}/_synapse/client/oidc/callback"
]
supported_identity_providers = ["COGNITO"]
}
This client uses the authorization code flow. Synapse’s OIDC provider configuration references this client’s ID and secret.
Agent Client (for MCP Server)
resource "aws_cognito_user_pool_client" "agent" {
name = "${var.project_name}-chat-agent"
generate_secret = true
explicit_auth_flows = [
"ALLOW_USER_PASSWORD_AUTH",
"ALLOW_REFRESH_TOKEN_AUTH",
]
}
This client uses USER_PASSWORD_AUTH for machine-to-machine authentication. The MCP server authenticates using AGENT_USERNAME and AGENT_PASSWORD environment variables.
Both client credentials are stored in Secrets Manager under phenom-dev-development-chat-clients.
Post-Deployment Steps
After terraform apply completes successfully:
1. Run Database Migration
Execute the SQL migration ECS task (see Database Migrations above) to create the chat tables and seed the three rooms.
2. Configure Hasura Metadata
Track the chat tables in Hasura and set up relationships and permissions:
# Run the metadata setup script as an ECS task (uses curl image)
# The script calls Hasura metadata API endpoints to:
# - Track all 5 chat tables
# - Create foreign key relationships
# - Set up role-based permissions (user, support, admin)
# - Configure event triggers for link preview resolution
3. Create Synapse Admin Bot Account
Register an admin account on Synapse using the registration shared secret:
# Get the registration shared secret from Secrets Manager
aws secretsmanager get-secret-value \
--secret-id phenom-dev-synapse-homeserver-config \
--query 'SecretString' \
--output text | grep registration_shared_secret
# Register admin user via Synapse admin API
curl -X POST "https://chat-staging.thephenom.app/_synapse/admin/v1/register" \
-H "Content-Type: application/json" \
-d '{
"nonce": "<get from GET /_synapse/admin/v1/register>",
"username": "admin",
"password": "<strong-password>",
"admin": true
}'
4. Create Matrix Rooms
Create the three chat rooms on Synapse:
# Using the admin access token from step 3
curl -X POST "https://chat-staging.thephenom.app/_matrix/client/v3/createRoom" \
-H "Authorization: Bearer <admin-access-token>" \
-H "Content-Type: application/json" \
-d '{
"name": "Phenom Internal",
"topic": "Internal team chat",
"visibility": "private",
"preset": "private_chat"
}'
# Repeat for Phenom Partners and Phenom Community
5. Verify Services
# Check all ECS services are running
aws ecs describe-services --region us-east-1 \
--cluster phenom-dev-cluster \
--services phenom-dev-synapse phenom-dev-synapse-admin phenom-dev-chat-mcp \
--query 'services[*].{Name:serviceName,Running:runningCount,Status:status}' \
--output table
# Verify health endpoints
curl -s https://chat-staging.thephenom.app/mcp/health | jq .
curl -s https://chat-staging.thephenom.app/_matrix/client/v3/login | jq .
Environment Wiring
All chat modules are wired together in environments/development/main.tf. The key variable connections:
module "chat_shared" {
source = "../../modules/chat-shared"
project_name = local.project_name
environment = "development"
cognito_user_pool_id = module.cognito.user_pool_id
cognito_domain = module.cognito.domain
synapse_server_name = "chat-staging.thephenom.app"
}
module "chat_synapse" {
source = "../../modules/chat-synapse"
project_name = local.project_name
environment = "development"
vpc_id = module.networking.vpc_id
private_subnet_ids = module.networking.private_subnet_ids
ecs_cluster_id = module.ecs.cluster_id
alb_listener_arn = module.alb.https_listener_arn
alb_security_group_id = module.alb.security_group_id
ecs_tasks_security_group_id = module.networking.ecs_tasks_sg_id
server_name = "chat-staging.thephenom.app"
rds_host = module.rds.endpoint
rds_port = module.rds.port
rds_master_username = module.rds.master_username
rds_master_password = module.rds.master_password
cognito_user_pool_id = module.cognito.user_pool_id
cognito_oidc_client_id = module.chat_shared.synapse_oidc_client_id
cognito_oidc_client_secret = module.chat_shared.synapse_oidc_client_secret
dockerhub_credentials_arn = var.dockerhub_credentials_arn
}
module "chat_mcp_server" {
source = "../../modules/chat-mcp-server"
project_name = local.project_name
vpc_id = module.networking.vpc_id
private_subnet_ids = module.networking.private_subnet_ids
ecs_cluster_id = module.ecs.cluster_id
alb_listener_arn = module.alb.https_listener_arn
ecs_tasks_security_group_id = module.networking.ecs_tasks_sg_id
server_name = "chat-staging.thephenom.app"
hasura_endpoint = "http://${module.ecs.graphql_service_name}:8080/v1/graphql"
chat_secrets_arn = module.secrets.chat_secrets_arn
backend_type = "hasura"
dockerhub_credentials_arn = var.dockerhub_credentials_arn
}
Deployment Pipeline
The standard deployment flow for chat infrastructure changes:
Pin new digest in Terraform"] E -->|"No"| G["Skip image build"] F --> H["Terraform apply rolls the ECS service"] G --> H H --> I["Verify health endpoints"] I --> J{"Schema changes?"} J -->|"Yes"| K["Run migration ECS task
Update Hasura metadata"] J -->|"No"| L["Deployment complete"] K --> L style A fill:#1a1a1a,color:#fff,rx:30 style L fill:#121010,color:#a5e3e8,rx:30
Production Deployment
Production deployments follow the standard pipeline: build, push:testing, deploy to testing server, verify, run full test suite, retag as :production, then deploy to production. No image receives a :production tag without all tests passing.
Related Documentation
- Architecture – system design and component details
- Admin & Operations – post-deployment operations
- API Reference – verifying API functionality after deployment
Feedback
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.