Admin & Operations
Reconciling to the canonical page. The single source of truth for chat services is Chat Services (validated 2026-05-27). The “Moderation via GraphQL (Hasura Lite)” workflow below is from the retired Implementation B and is excised from production; moderate via Synapse. Chat is now Matrix/Synapse only. Where this page disagrees with the canonical page, the canonical page wins.
This page is for the Phenom support and infrastructure team. It covers moderation workflows, monitoring, and emergency procedures for both chat implementations.
Quick Reference
Admin URLs
| Service | URL | Purpose |
|---|---|---|
| Synapse Admin UI | https://chat-staging.thephenom.app/chat-admin |
Web dashboard for Synapse user/room management |
| MCP Server Health | https://chat-staging.thephenom.app/mcp/health |
Health check endpoint (returns JSON status) |
| Synapse Health | https://chat-staging.thephenom.app/_matrix/client/v3/login |
Verify Synapse is responding |
| Hasura Console | Contact engineering for URL | GraphQL console for Hasura Lite queries |
AWS Console Resources
| Resource | How to Find It |
|---|---|
| CloudWatch Logs | AWS Console > CloudWatch > Log groups > search for phenom-dev |
| ECS Services | AWS Console > ECS > Clusters > phenom-dev-cluster |
| Cognito User Pool | AWS Console > Cognito > User pools > search for phenom |
| Lambda Functions | AWS Console > Lambda > search for phenom-dev-link-preview or phenom-dev-chat-user-provisioner |
ECS Service Names
| Service | ECS Service Name | Log Group |
|---|---|---|
| Synapse | phenom-dev-synapse |
/ecs/phenom-dev-synapse |
| Synapse Admin UI | phenom-dev-synapse-admin |
/ecs/phenom-dev-synapse-admin |
| Chat MCP Server | phenom-dev-chat-mcp |
/ecs/phenom-dev-chat-mcp |
| Link Preview Lambda | – | /aws/lambda/phenom-dev-link-preview |
| User Provisioner Lambda | – | /aws/lambda/phenom-dev-chat-user-provisioner |
Moderation via GraphQL (Hasura Lite)
Open the Hasura Console, go to the API tab, and run these queries/mutations.
View Recent Messages
query RecentMessages {
chat_messages(
order_by: { created_at: desc }
limit: 50
where: { is_deleted: { _eq: false } }
) {
id
content
message_type
created_at
user { username }
room { name }
}
}
Search messages from a specific user:
query MessagesByUser($username: String!) {
chat_messages(
order_by: { created_at: desc }
limit: 50
where: {
is_deleted: { _eq: false }
user: { username: { _eq: $username } }
}
) {
id
content
created_at
user { username }
}
}
Delete a Message (Soft Delete)
This marks the message as deleted so it no longer appears in the app. Replace YOUR_ADMIN_UUID with your admin user UUID.
mutation DeleteMessage($id: uuid!) {
update_chat_messages_by_pk(
pk_columns: { id: $id }
_set: { is_deleted: true, deleted_by: "YOUR_ADMIN_UUID" }
) {
id
}
}
Variables:
{
"id": "paste-the-message-uuid-here"
}
Mute a User
Prevents the user from sending messages until the specified time.
mutation MuteUser($roomId: uuid!, $userId: uuid!) {
update_chat_members(
where: {
room_id: { _eq: $roomId }
user_id: { _eq: $userId }
}
_set: {
is_muted: true,
muted_until: "2026-03-23T00:00:00Z"
}
) {
affected_rows
}
}
Unmute a User
mutation UnmuteUser($roomId: uuid!, $userId: uuid!) {
update_chat_members(
where: {
room_id: { _eq: $roomId }
user_id: { _eq: $userId }
}
_set: {
is_muted: false,
muted_until: null
}
) {
affected_rows
}
}
Ban a User
mutation BanUser($roomId: uuid!, $userId: uuid!, $reason: String) {
insert_chat_bans_one(object: {
room_id: $roomId
user_id: $userId
banned_by: "YOUR_ADMIN_UUID"
reason: $reason
}) {
id
}
}
Variables:
{
"roomId": "room-uuid",
"userId": "user-uuid",
"reason": "Violation of community guidelines"
}
Unban a User
mutation UnbanUser($roomId: uuid!, $userId: uuid!) {
delete_chat_bans(
where: {
room_id: { _eq: $roomId }
user_id: { _eq: $userId }
}
) {
affected_rows
}
}
Moderation via Synapse Admin UI
Logging In
- Go to
https://chat-staging.thephenom.app/chat-admin. - The Homeserver URL should be pre-configured to
https://chat-staging.thephenom.app. - Log in with an admin account.
Users Tab
- Search users: Type a username or Matrix ID in the search bar.
- View user details: Click a user to see profile, devices, and room memberships.
- Deactivate a user: Open user detail, click “Deactivate”. This is permanent – use with caution.
Rooms Tab
- Search rooms: Find “Phenom Internal”, “Phenom Partners”, or “Phenom Community” by name or room ID.
- View room: Click to see members, recent messages, and room state.
- Purge messages: Select messages to permanently delete from the Synapse database.
Server Notices
Use the Admin UI to send server-wide notices (e.g., planned maintenance announcements).
User Management
Assign Roles (Hasura Lite)
Via Hasura Console:
- Go to the Data tab.
- Click on
chat_members. - Find the user row (filter by
user_id). - Change the
rolefield touser,support, oradmin. - Click Save.
Via GraphQL:
mutation SetRole($roomId: uuid!, $userId: uuid!, $role: String!) {
update_chat_members(
where: {
room_id: { _eq: $roomId }
user_id: { _eq: $userId }
}
_set: { role: $role }
) {
affected_rows
}
}
Provision AI Agent Accounts
AI agent accounts use the phenom-dev-chat-agent Cognito client (machine-to-machine, USER_PASSWORD_AUTH flow).
- Go to AWS Console > Cognito > User pools > Phenom user pool.
- Create a new user with the agent’s username and a strong password.
- Confirm the user (no email verification needed for agents).
- Agent client credentials are in AWS Secrets Manager under
phenom-dev-development-chat-clients.
Check Membership Status
query CheckMembership($userId: uuid!) {
chat_members(where: { user_id: { _eq: $userId } }) {
room { name }
role
is_muted
muted_until
joined_at
}
chat_bans(where: { user_id: { _eq: $userId } }) {
room { name }
reason
banned_at
expires_at
}
}
If a user has no rows in chat_members, they have not been provisioned. Check the User Provisioner Lambda logs.
Monitoring
Quick Health Checks
# MCP Server -- should return {"status":"ok","backend":"hasura",...}
curl https://chat-staging.thephenom.app/mcp/health
# Synapse -- should return available login flows
curl https://chat-staging.thephenom.app/_matrix/client/v3/login
ECS Service Status
aws ecs describe-services --region us-east-1 \
--cluster phenom-dev-cluster \
--services phenom-dev-synapse phenom-dev-synapse-admin phenom-dev-chat-mcp \
--query 'services[*].{Name:serviceName,Running:runningCount,Desired:desiredCount,Status:status}' \
--output table
All three services should show Running: 1, Desired: 1, Status: ACTIVE.
CloudWatch Logs
# Synapse logs (last 1 hour)
aws logs tail /ecs/phenom-dev-synapse --follow --since 1h --region us-east-1
# MCP Server logs
aws logs tail /ecs/phenom-dev-chat-mcp --follow --since 1h --region us-east-1
# Link Preview Lambda logs
aws logs tail /aws/lambda/phenom-dev-link-preview --follow --since 1h --region us-east-1
# User Provisioner Lambda logs
aws logs tail /aws/lambda/phenom-dev-chat-user-provisioner --follow --since 1h --region us-east-1
Key Metrics
| Metric | Where to Check | Expected Value |
|---|---|---|
| ECS task count | ECS Console or describe-services |
1 running task per service |
| ALB target group health | EC2 > Target Groups | All targets healthy |
| Lambda invocation errors | CloudWatch > Lambda > Monitoring | 0 errors |
| RDS active connections | RDS Console > Monitoring | Stable, no sudden spikes |
Synapse /health |
Browser or curl | HTTP 200 |
MCP /mcp/health |
Browser or curl | HTTP 200 with {"status":"ok"} |
Recommended CloudWatch Alarms
Ask engineering to configure alarms for:
- ECS task count dropping to 0 for any chat service
- Lambda error rate exceeding 5% for link preview or user provisioner
- ALB 5xx error rate exceeding 1%
- Synapse health check failures
Troubleshooting
| Symptom | Likely Cause | How to Investigate | Resolution |
|---|---|---|---|
| Users cannot send messages | User is muted or banned | Query chat_members for is_muted = true; check chat_bans |
Unmute or unban using mutations above |
| Messages not appearing in real time | WebSocket/subscription disconnected | Ask user to close and reopen chat screen | Client-side reconnection; no server action needed |
| Link previews not showing | Link Preview Lambda error | Check /aws/lambda/phenom-dev-link-preview logs |
Review Lambda error, escalate to engineering |
| Synapse Admin UI blank page | REACT_APP_SERVER env var wrong |
Check ECS task definition for phenom-dev-synapse-admin |
Engineering must update env var and redeploy |
| MCP tools not working | MCP server cannot reach Hasura | Check /ecs/phenom-dev-chat-mcp logs for connection errors |
Verify HASURA_GRAPHQL_URL env var; restart service |
| New users cannot access chat | User not provisioned into chat_members |
Check chat_members for the user; check Cognito trigger Lambda logs |
Run provisioner manually or add via Hasura Console |
| Service shows 0 running tasks | ECS task crashed or failed to start | Check ECS service events and CloudWatch logs | Restart service (see below) |
| 502 Bad Gateway on chat URLs | ECS task not running or unhealthy | Check target group health in EC2 Console | Restart service; check logs for crash reason |
| Cognito SSO loop / “invalid redirect” | Callback URL mismatch | Check Cognito client callback URLs | Update callback URL in Cognito client settings |
| Database connection errors | RDS at connection limit | Check RDS monitoring for active connections | Restart affected service to release connections |
Emergency Procedures
Kill Switch – Disable Chat Entirely
If chat needs to be shut down immediately (spam attack, critical bug):
- Set
chat_enabled = falsein the Hasura config table (or remote config endpoint). - The mobile app will display “Chat is temporarily unavailable” to all users.
- Notify engineering immediately.
Restart a Service
Force a new deployment to pull the latest task definition and start a fresh container:
# Restart Synapse
aws ecs update-service --region us-east-1 \
--cluster phenom-dev-cluster \
--service phenom-dev-synapse \
--force-new-deployment
# Restart Synapse Admin UI
aws ecs update-service --region us-east-1 \
--cluster phenom-dev-cluster \
--service phenom-dev-synapse-admin \
--force-new-deployment
# Restart MCP Server
aws ecs update-service --region us-east-1 \
--cluster phenom-dev-cluster \
--service phenom-dev-chat-mcp \
--force-new-deployment
New tasks typically take 1-2 minutes to become healthy. Watch ECS service events and CloudWatch logs to confirm.
Scale Down Chat Services (Emergency)
Take all chat offline at the infrastructure level:
# Scale all chat services to 0
aws ecs update-service --region us-east-1 \
--cluster phenom-dev-cluster \
--service phenom-dev-synapse \
--desired-count 0
aws ecs update-service --region us-east-1 \
--cluster phenom-dev-cluster \
--service phenom-dev-synapse-admin \
--desired-count 0
aws ecs update-service --region us-east-1 \
--cluster phenom-dev-cluster \
--service phenom-dev-chat-mcp \
--desired-count 0
To restore services, run the same commands with --desired-count 1.
Test Accounts
| Account | Password | Role | |
|---|---|---|---|
| Test User | test-user@thephenom.app |
PhenomTest2026! |
user |
| Test Admin | test-admin@thephenom.app |
PhenomAdmin2026! |
admin |
Creating New Test Accounts
- Go to AWS Console > Cognito > User pools > Phenom user pool.
- Click Create user.
- Set email and temporary password.
- Confirm the user (mark as verified).
- The User Provisioner Lambda should automatically add the user to
chat_memberson first login. - To grant admin or support role, use the
SetRolemutation or edit thechat_membersrow in Hasura Console.
Escalation Contacts
If you cannot resolve an issue using this guide:
- Collect CloudWatch logs (screenshot or copy the error messages).
- Note which service is affected and the symptoms.
- Escalate to the engineering team with this information.
Glossary
| Term | Definition |
|---|---|
| ECS | Elastic Container Service – runs chat containers on AWS |
| Fargate | Serverless compute for ECS – no servers to manage |
| ALB | Application Load Balancer – routes web traffic to the correct service |
| Synapse | Matrix homeserver (Implementation A chat backend) |
| Hasura | GraphQL engine on top of PostgreSQL (Implementation B) |
| MCP Server | Chat tool server for AI agent interactions |
| Lambda | Serverless functions (link preview resolver, user provisioner) |
| Cognito | AWS user authentication service |
| Soft delete | Marking a message as deleted without removing from the database |
| Target group | ALB concept – containers that receive traffic for a URL path |
Feedback
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.