Chat (Synapse) Runbook
The Phenom Matrix Synapse homeserver providing federated chat for the platform.
Source:
asset-registry.yaml; curl https://chat.thephenom.app → HTTP 200; Matrix Synapse homeserver
C2PA signed · SanMarcSoft AI content credential
What it is
chat.thephenom.app is the Matrix Synapse homeserver for the Phenom platform. It provides real-time federated chat, room management, and messaging infrastructure for Phenom users and internal operations teams. It is also the backend for any Matrix bridge integrations (e.g., the Haus operations room used by M for directives). Downtime disrupts team communication and any Matrix-dependent automation.
Deployment chain
| Layer | Value |
|---|---|
| URL | https://chat.thephenom.app |
| DNS | Cloudflare (proxied, orange-cloud) |
| Matrix server name | thephenom.app |
| Server software | Matrix Synapse |
| Hosting | HTTP service (AWS ECS Fargate or EC2 – verify in AWS console) |
| Region | us-east-1 |
| AWS profile | phenom |
| Data store | PostgreSQL (phenom-prod-postgres RDS, synapse database) |
| Media store | AWS S3 (synapse media bucket) |
The federation endpoint is https://chat.thephenom.app on port 443 (or via .well-known/matrix/server delegation).
Common operations
Check Synapse health
curl -sf https://chat.thephenom.app/_matrix/client/versions | python3 -m json.tool
# Expected: JSON listing supported Matrix spec versions
Restart Synapse (ECS Fargate)
aws ecs update-service \
--cluster phenom-prod-cluster \
--service synapse \
--force-new-deployment \
--profile phenom \
--region us-east-1
aws ecs wait services-stable \
--cluster phenom-prod-cluster \
--services synapse \
--profile phenom \
--region us-east-1
View Synapse logs
aws logs tail /ecs/synapse \
--follow \
--profile phenom \
--region us-east-1
Run Synapse admin API command
# Get an admin access token first (from a Synapse admin user)
# Then use the admin API:
curl -H "Authorization: Bearer <ADMIN_TOKEN>" \
https://chat.thephenom.app/_synapse/admin/v1/server_version
Purge old room events (maintenance)
# Use the Synapse admin API to purge history older than 90 days:
curl -X POST \
-H "Authorization: Bearer <ADMIN_TOKEN>" \
-H "Content-Type: application/json" \
-d '{"delete_local_events": false, "purge_up_to_ms": '"$(($(date +%s%3N) - 90*24*3600*1000))"'}' \
"https://chat.thephenom.app/_synapse/admin/v1/purge_history/<room_id>"
Verify it is working
curl -sf https://chat.thephenom.app/_matrix/client/versions | grep -o '"versions"'
# Expected: outputs "versions" (key present in JSON response)
curl -si https://chat.thephenom.app/_matrix/client/versions | grep "^HTTP"
# Expected: HTTP/2 200
Common failure modes
| Symptom | Likely cause | Remediation |
|---|---|---|
| 502 from Cloudflare | Synapse process crashed or ECS task stopped | Check ECS service status; force new deployment |
| Federation broken | .well-known/matrix/server misconfigured or port 8448 blocked |
Verify https://thephenom.app/.well-known/matrix/server returns correct delegation |
| Media upload fails | S3 media bucket permissions error or bucket full | Check ECS task IAM role; check S3 bucket storage |
| Database connection errors in logs | RDS postgres unreachable or connection pool exhausted | Check RDS status (see RDS prod runbook); restart Synapse |
| Slow performance | Too many rooms or events; Synapse needs background job | Run Synapse maintenance jobs; check CPU/memory on ECS task |
Related
Feedback
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.