Admin & Operations
This page is for the Phenom support and infrastructure team. It covers moderation workflows, monitoring, and emergency procedures for both chat implementations.
Quick Reference
Admin URLs
| Service | URL | Purpose |
|---|---|---|
| Synapse Admin UI | https://chat-testing.thephenom.app/chat-admin | Web dashboard for Synapse user/room management |
| MCP Server Health | https://chat-testing.thephenom.app/mcp/health | Health check endpoint (returns JSON status) |
| Synapse Health | https://chat-testing.thephenom.app/_matrix/client/v3/login | Verify Synapse is responding |
| Hasura Console | Contact engineering for URL | GraphQL console for Hasura Lite queries |
AWS Console Resources
| Resource | How to Find It |
|---|---|
| CloudWatch Logs | AWS Console > CloudWatch > Log groups > search for phenom-dev |
| ECS Services | AWS Console > ECS > Clusters > phenom-dev-cluster |
| Cognito User Pool | AWS Console > Cognito > User pools > search for phenom |
| Lambda Functions | AWS Console > Lambda > search for phenom-dev-link-preview or phenom-dev-chat-user-provisioner |
ECS Service Names
| Service | ECS Service Name | Log Group |
|---|---|---|
| Synapse | phenom-dev-synapse | /ecs/phenom-dev-synapse |
| Synapse Admin UI | phenom-dev-synapse-admin | /ecs/phenom-dev-synapse-admin |
| Chat MCP Server | phenom-dev-chat-mcp | /ecs/phenom-dev-chat-mcp |
| Link Preview Lambda | – | /aws/lambda/phenom-dev-link-preview |
| User Provisioner Lambda | – | /aws/lambda/phenom-dev-chat-user-provisioner |
Moderation via GraphQL (Hasura Lite)
Open the Hasura Console, go to the API tab, and run these queries/mutations.
View Recent Messages
query RecentMessages {
chat_messages(
order_by: { created_at: desc }
limit: 50
where: { is_deleted: { _eq: false } }
) {
id
content
message_type
created_at
user { username }
room { name }
}
}
Search messages from a specific user:
query MessagesByUser($username: String!) {
chat_messages(
order_by: { created_at: desc }
limit: 50
where: {
is_deleted: { _eq: false }
user: { username: { _eq: $username } }
}
) {
id
content
created_at
user { username }
}
}
Delete a Message (Soft Delete)
This marks the message as deleted so it no longer appears in the app. Replace YOUR_ADMIN_UUID with your admin user UUID.
mutation DeleteMessage($id: uuid!) {
update_chat_messages_by_pk(
pk_columns: { id: $id }
_set: { is_deleted: true, deleted_by: "YOUR_ADMIN_UUID" }
) {
id
}
}
Variables:
{
"id": "paste-the-message-uuid-here"
}
Mute a User
Prevents the user from sending messages until the specified time.
mutation MuteUser($roomId: uuid!, $userId: uuid!) {
update_chat_members(
where: {
room_id: { _eq: $roomId }
user_id: { _eq: $userId }
}
_set: {
is_muted: true,
muted_until: "2026-03-23T00:00:00Z"
}
) {
affected_rows
}
}
Unmute a User
mutation UnmuteUser($roomId: uuid!, $userId: uuid!) {
update_chat_members(
where: {
room_id: { _eq: $roomId }
user_id: { _eq: $userId }
}
_set: {
is_muted: false,
muted_until: null
}
) {
affected_rows
}
}
Ban a User
mutation BanUser($roomId: uuid!, $userId: uuid!, $reason: String) {
insert_chat_bans_one(object: {
room_id: $roomId
user_id: $userId
banned_by: "YOUR_ADMIN_UUID"
reason: $reason
}) {
id
}
}
Variables:
{
"roomId": "room-uuid",
"userId": "user-uuid",
"reason": "Violation of community guidelines"
}
Unban a User
mutation UnbanUser($roomId: uuid!, $userId: uuid!) {
delete_chat_bans(
where: {
room_id: { _eq: $roomId }
user_id: { _eq: $userId }
}
) {
affected_rows
}
}
Moderation via Synapse Admin UI
Logging In
- Go to
https://chat-testing.thephenom.app/chat-admin. - The Homeserver URL should be pre-configured to
https://chat-testing.thephenom.app. - Log in with an admin account.
Users Tab
- Search users: Type a username or Matrix ID in the search bar.
- View user details: Click a user to see profile, devices, and room memberships.
- Deactivate a user: Open user detail, click “Deactivate”. This is permanent – use with caution.
Rooms Tab
- Search rooms: Find “Phenom Internal”, “Phenom Partners”, or “Phenom Community” by name or room ID.
- View room: Click to see members, recent messages, and room state.
- Purge messages: Select messages to permanently delete from the Synapse database.
Server Notices
Use the Admin UI to send server-wide notices (e.g., planned maintenance announcements).
User Management
Assign Roles (Hasura Lite)
Via Hasura Console:
- Go to the Data tab.
- Click on
chat_members. - Find the user row (filter by
user_id). - Change the
rolefield touser,support, oradmin. - Click Save.
Via GraphQL:
mutation SetRole($roomId: uuid!, $userId: uuid!, $role: String!) {
update_chat_members(
where: {
room_id: { _eq: $roomId }
user_id: { _eq: $userId }
}
_set: { role: $role }
) {
affected_rows
}
}
Provision AI Agent Accounts
AI agent accounts use the phenom-dev-chat-agent Cognito client (machine-to-machine, USER_PASSWORD_AUTH flow).
- Go to AWS Console > Cognito > User pools > Phenom user pool.
- Create a new user with the agent’s username and a strong password.
- Confirm the user (no email verification needed for agents).
- Agent client credentials are in AWS Secrets Manager under
phenom-dev-development-chat-clients.
Check Membership Status
query CheckMembership($userId: uuid!) {
chat_members(where: { user_id: { _eq: $userId } }) {
room { name }
role
is_muted
muted_until
joined_at
}
chat_bans(where: { user_id: { _eq: $userId } }) {
room { name }
reason
banned_at
expires_at
}
}
If a user has no rows in chat_members, they have not been provisioned. Check the User Provisioner Lambda logs.
Monitoring
Quick Health Checks
# MCP Server -- should return {"status":"ok","backend":"hasura",...}
curl https://chat-testing.thephenom.app/mcp/health
# Synapse -- should return available login flows
curl https://chat-testing.thephenom.app/_matrix/client/v3/login
ECS Service Status
aws ecs describe-services --region us-east-1 \
--cluster phenom-dev-cluster \
--services phenom-dev-synapse phenom-dev-synapse-admin phenom-dev-chat-mcp \
--query 'services[*].{Name:serviceName,Running:runningCount,Desired:desiredCount,Status:status}' \
--output table
All three services should show Running: 1, Desired: 1, Status: ACTIVE.
CloudWatch Logs
# Synapse logs (last 1 hour)
aws logs tail /ecs/phenom-dev-synapse --follow --since 1h --region us-east-1
# MCP Server logs
aws logs tail /ecs/phenom-dev-chat-mcp --follow --since 1h --region us-east-1
# Link Preview Lambda logs
aws logs tail /aws/lambda/phenom-dev-link-preview --follow --since 1h --region us-east-1
# User Provisioner Lambda logs
aws logs tail /aws/lambda/phenom-dev-chat-user-provisioner --follow --since 1h --region us-east-1
Key Metrics
| Metric | Where to Check | Expected Value |
|---|---|---|
| ECS task count | ECS Console or describe-services | 1 running task per service |
| ALB target group health | EC2 > Target Groups | All targets healthy |
| Lambda invocation errors | CloudWatch > Lambda > Monitoring | 0 errors |
| RDS active connections | RDS Console > Monitoring | Stable, no sudden spikes |
Synapse /health | Browser or curl | HTTP 200 |
MCP /mcp/health | Browser or curl | HTTP 200 with {"status":"ok"} |
Recommended CloudWatch Alarms
Ask engineering to configure alarms for:
- ECS task count dropping to 0 for any chat service
- Lambda error rate exceeding 5% for link preview or user provisioner
- ALB 5xx error rate exceeding 1%
- Synapse health check failures
Troubleshooting
| Symptom | Likely Cause | How to Investigate | Resolution |
|---|---|---|---|
| Users cannot send messages | User is muted or banned | Query chat_members for is_muted = true; check chat_bans | Unmute or unban using mutations above |
| Messages not appearing in real time | WebSocket/subscription disconnected | Ask user to close and reopen chat screen | Client-side reconnection; no server action needed |
| Link previews not showing | Link Preview Lambda error | Check /aws/lambda/phenom-dev-link-preview logs | Review Lambda error, escalate to engineering |
| Synapse Admin UI blank page | REACT_APP_SERVER env var wrong | Check ECS task definition for phenom-dev-synapse-admin | Engineering must update env var and redeploy |
| MCP tools not working | MCP server cannot reach Hasura | Check /ecs/phenom-dev-chat-mcp logs for connection errors | Verify HASURA_GRAPHQL_URL env var; restart service |
| New users cannot access chat | User not provisioned into chat_members | Check chat_members for the user; check Cognito trigger Lambda logs | Run provisioner manually or add via Hasura Console |
| Service shows 0 running tasks | ECS task crashed or failed to start | Check ECS service events and CloudWatch logs | Restart service (see below) |
| 502 Bad Gateway on chat URLs | ECS task not running or unhealthy | Check target group health in EC2 Console | Restart service; check logs for crash reason |
| Cognito SSO loop / “invalid redirect” | Callback URL mismatch | Check Cognito client callback URLs | Update callback URL in Cognito client settings |
| Database connection errors | RDS at connection limit | Check RDS monitoring for active connections | Restart affected service to release connections |
Emergency Procedures
Kill Switch – Disable Chat Entirely
If chat needs to be shut down immediately (spam attack, critical bug):
- Set
chat_enabled = falsein the Hasura config table (or remote config endpoint). - The mobile app will display “Chat is temporarily unavailable” to all users.
- Notify engineering immediately.
Restart a Service
Force a new deployment to pull the latest task definition and start a fresh container:
# Restart Synapse
aws ecs update-service --region us-east-1 \
--cluster phenom-dev-cluster \
--service phenom-dev-synapse \
--force-new-deployment
# Restart Synapse Admin UI
aws ecs update-service --region us-east-1 \
--cluster phenom-dev-cluster \
--service phenom-dev-synapse-admin \
--force-new-deployment
# Restart MCP Server
aws ecs update-service --region us-east-1 \
--cluster phenom-dev-cluster \
--service phenom-dev-chat-mcp \
--force-new-deployment
New tasks typically take 1-2 minutes to become healthy. Watch ECS service events and CloudWatch logs to confirm.
Scale Down Chat Services (Emergency)
Take all chat offline at the infrastructure level:
# Scale all chat services to 0
aws ecs update-service --region us-east-1 \
--cluster phenom-dev-cluster \
--service phenom-dev-synapse \
--desired-count 0
aws ecs update-service --region us-east-1 \
--cluster phenom-dev-cluster \
--service phenom-dev-synapse-admin \
--desired-count 0
aws ecs update-service --region us-east-1 \
--cluster phenom-dev-cluster \
--service phenom-dev-chat-mcp \
--desired-count 0
To restore services, run the same commands with --desired-count 1.
Test Accounts
| Account | Password | Role | |
|---|---|---|---|
| Test User | test-user@thephenom.app | PhenomTest2026! | user |
| Test Admin | test-admin@thephenom.app | PhenomAdmin2026! | admin |
Creating New Test Accounts
- Go to AWS Console > Cognito > User pools > Phenom user pool.
- Click Create user.
- Set email and temporary password.
- Confirm the user (mark as verified).
- The User Provisioner Lambda should automatically add the user to
chat_memberson first login. - To grant admin or support role, use the
SetRolemutation or edit thechat_membersrow in Hasura Console.
Escalation Contacts
If you cannot resolve an issue using this guide:
- Collect CloudWatch logs (screenshot or copy the error messages).
- Note which service is affected and the symptoms.
- Escalate to the engineering team with this information.
Glossary
| Term | Definition |
|---|---|
| ECS | Elastic Container Service – runs chat containers on AWS |
| Fargate | Serverless compute for ECS – no servers to manage |
| ALB | Application Load Balancer – routes web traffic to the correct service |
| Synapse | Matrix homeserver (Implementation A chat backend) |
| Hasura | GraphQL engine on top of PostgreSQL (Implementation B) |
| MCP Server | Chat tool server for AI agent interactions |
| Lambda | Serverless functions (link preview resolver, user provisioner) |
| Cognito | AWS user authentication service |
| Soft delete | Marking a message as deleted without removing from the database |
| Target group | ALB concept – containers that receive traffic for a URL path |
Feedback
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.