Admin & Operations

Day-to-day administration, moderation, monitoring, troubleshooting, and emergency procedures for the Phenom Chat system.

This page is for the Phenom support and infrastructure team. It covers moderation workflows, monitoring, and emergency procedures for both chat implementations.

Quick Reference

Admin URLs

ServiceURLPurpose
Synapse Admin UIhttps://chat-testing.thephenom.app/chat-adminWeb dashboard for Synapse user/room management
MCP Server Healthhttps://chat-testing.thephenom.app/mcp/healthHealth check endpoint (returns JSON status)
Synapse Healthhttps://chat-testing.thephenom.app/_matrix/client/v3/loginVerify Synapse is responding
Hasura ConsoleContact engineering for URLGraphQL console for Hasura Lite queries

AWS Console Resources

ResourceHow to Find It
CloudWatch LogsAWS Console > CloudWatch > Log groups > search for phenom-dev
ECS ServicesAWS Console > ECS > Clusters > phenom-dev-cluster
Cognito User PoolAWS Console > Cognito > User pools > search for phenom
Lambda FunctionsAWS Console > Lambda > search for phenom-dev-link-preview or phenom-dev-chat-user-provisioner

ECS Service Names

ServiceECS Service NameLog Group
Synapsephenom-dev-synapse/ecs/phenom-dev-synapse
Synapse Admin UIphenom-dev-synapse-admin/ecs/phenom-dev-synapse-admin
Chat MCP Serverphenom-dev-chat-mcp/ecs/phenom-dev-chat-mcp
Link Preview Lambda/aws/lambda/phenom-dev-link-preview
User Provisioner Lambda/aws/lambda/phenom-dev-chat-user-provisioner

Moderation via GraphQL (Hasura Lite)

Open the Hasura Console, go to the API tab, and run these queries/mutations.

View Recent Messages

query RecentMessages {
  chat_messages(
    order_by: { created_at: desc }
    limit: 50
    where: { is_deleted: { _eq: false } }
  ) {
    id
    content
    message_type
    created_at
    user { username }
    room { name }
  }
}

Search messages from a specific user:

query MessagesByUser($username: String!) {
  chat_messages(
    order_by: { created_at: desc }
    limit: 50
    where: {
      is_deleted: { _eq: false }
      user: { username: { _eq: $username } }
    }
  ) {
    id
    content
    created_at
    user { username }
  }
}

Delete a Message (Soft Delete)

This marks the message as deleted so it no longer appears in the app. Replace YOUR_ADMIN_UUID with your admin user UUID.

mutation DeleteMessage($id: uuid!) {
  update_chat_messages_by_pk(
    pk_columns: { id: $id }
    _set: { is_deleted: true, deleted_by: "YOUR_ADMIN_UUID" }
  ) {
    id
  }
}

Variables:

{
  "id": "paste-the-message-uuid-here"
}

Mute a User

Prevents the user from sending messages until the specified time.

mutation MuteUser($roomId: uuid!, $userId: uuid!) {
  update_chat_members(
    where: {
      room_id: { _eq: $roomId }
      user_id: { _eq: $userId }
    }
    _set: {
      is_muted: true,
      muted_until: "2026-03-23T00:00:00Z"
    }
  ) {
    affected_rows
  }
}

Unmute a User

mutation UnmuteUser($roomId: uuid!, $userId: uuid!) {
  update_chat_members(
    where: {
      room_id: { _eq: $roomId }
      user_id: { _eq: $userId }
    }
    _set: {
      is_muted: false,
      muted_until: null
    }
  ) {
    affected_rows
  }
}

Ban a User

mutation BanUser($roomId: uuid!, $userId: uuid!, $reason: String) {
  insert_chat_bans_one(object: {
    room_id: $roomId
    user_id: $userId
    banned_by: "YOUR_ADMIN_UUID"
    reason: $reason
  }) {
    id
  }
}

Variables:

{
  "roomId": "room-uuid",
  "userId": "user-uuid",
  "reason": "Violation of community guidelines"
}

Unban a User

mutation UnbanUser($roomId: uuid!, $userId: uuid!) {
  delete_chat_bans(
    where: {
      room_id: { _eq: $roomId }
      user_id: { _eq: $userId }
    }
  ) {
    affected_rows
  }
}

Moderation via Synapse Admin UI

Logging In

  1. Go to https://chat-testing.thephenom.app/chat-admin.
  2. The Homeserver URL should be pre-configured to https://chat-testing.thephenom.app.
  3. Log in with an admin account.

Users Tab

  • Search users: Type a username or Matrix ID in the search bar.
  • View user details: Click a user to see profile, devices, and room memberships.
  • Deactivate a user: Open user detail, click “Deactivate”. This is permanent – use with caution.

Rooms Tab

  • Search rooms: Find “Phenom Internal”, “Phenom Partners”, or “Phenom Community” by name or room ID.
  • View room: Click to see members, recent messages, and room state.
  • Purge messages: Select messages to permanently delete from the Synapse database.

Server Notices

Use the Admin UI to send server-wide notices (e.g., planned maintenance announcements).


User Management

Assign Roles (Hasura Lite)

Via Hasura Console:

  1. Go to the Data tab.
  2. Click on chat_members.
  3. Find the user row (filter by user_id).
  4. Change the role field to user, support, or admin.
  5. Click Save.

Via GraphQL:

mutation SetRole($roomId: uuid!, $userId: uuid!, $role: String!) {
  update_chat_members(
    where: {
      room_id: { _eq: $roomId }
      user_id: { _eq: $userId }
    }
    _set: { role: $role }
  ) {
    affected_rows
  }
}

Provision AI Agent Accounts

AI agent accounts use the phenom-dev-chat-agent Cognito client (machine-to-machine, USER_PASSWORD_AUTH flow).

  1. Go to AWS Console > Cognito > User pools > Phenom user pool.
  2. Create a new user with the agent’s username and a strong password.
  3. Confirm the user (no email verification needed for agents).
  4. Agent client credentials are in AWS Secrets Manager under phenom-dev-development-chat-clients.

Check Membership Status

query CheckMembership($userId: uuid!) {
  chat_members(where: { user_id: { _eq: $userId } }) {
    room { name }
    role
    is_muted
    muted_until
    joined_at
  }
  chat_bans(where: { user_id: { _eq: $userId } }) {
    room { name }
    reason
    banned_at
    expires_at
  }
}

If a user has no rows in chat_members, they have not been provisioned. Check the User Provisioner Lambda logs.


Monitoring

Quick Health Checks

# MCP Server -- should return {"status":"ok","backend":"hasura",...}
curl https://chat-testing.thephenom.app/mcp/health

# Synapse -- should return available login flows
curl https://chat-testing.thephenom.app/_matrix/client/v3/login

ECS Service Status

aws ecs describe-services --region us-east-1 \
  --cluster phenom-dev-cluster \
  --services phenom-dev-synapse phenom-dev-synapse-admin phenom-dev-chat-mcp \
  --query 'services[*].{Name:serviceName,Running:runningCount,Desired:desiredCount,Status:status}' \
  --output table

All three services should show Running: 1, Desired: 1, Status: ACTIVE.

CloudWatch Logs

# Synapse logs (last 1 hour)
aws logs tail /ecs/phenom-dev-synapse --follow --since 1h --region us-east-1

# MCP Server logs
aws logs tail /ecs/phenom-dev-chat-mcp --follow --since 1h --region us-east-1

# Link Preview Lambda logs
aws logs tail /aws/lambda/phenom-dev-link-preview --follow --since 1h --region us-east-1

# User Provisioner Lambda logs
aws logs tail /aws/lambda/phenom-dev-chat-user-provisioner --follow --since 1h --region us-east-1

Key Metrics

MetricWhere to CheckExpected Value
ECS task countECS Console or describe-services1 running task per service
ALB target group healthEC2 > Target GroupsAll targets healthy
Lambda invocation errorsCloudWatch > Lambda > Monitoring0 errors
RDS active connectionsRDS Console > MonitoringStable, no sudden spikes
Synapse /healthBrowser or curlHTTP 200
MCP /mcp/healthBrowser or curlHTTP 200 with {"status":"ok"}

Ask engineering to configure alarms for:

  • ECS task count dropping to 0 for any chat service
  • Lambda error rate exceeding 5% for link preview or user provisioner
  • ALB 5xx error rate exceeding 1%
  • Synapse health check failures

Troubleshooting

SymptomLikely CauseHow to InvestigateResolution
Users cannot send messagesUser is muted or bannedQuery chat_members for is_muted = true; check chat_bansUnmute or unban using mutations above
Messages not appearing in real timeWebSocket/subscription disconnectedAsk user to close and reopen chat screenClient-side reconnection; no server action needed
Link previews not showingLink Preview Lambda errorCheck /aws/lambda/phenom-dev-link-preview logsReview Lambda error, escalate to engineering
Synapse Admin UI blank pageREACT_APP_SERVER env var wrongCheck ECS task definition for phenom-dev-synapse-adminEngineering must update env var and redeploy
MCP tools not workingMCP server cannot reach HasuraCheck /ecs/phenom-dev-chat-mcp logs for connection errorsVerify HASURA_GRAPHQL_URL env var; restart service
New users cannot access chatUser not provisioned into chat_membersCheck chat_members for the user; check Cognito trigger Lambda logsRun provisioner manually or add via Hasura Console
Service shows 0 running tasksECS task crashed or failed to startCheck ECS service events and CloudWatch logsRestart service (see below)
502 Bad Gateway on chat URLsECS task not running or unhealthyCheck target group health in EC2 ConsoleRestart service; check logs for crash reason
Cognito SSO loop / “invalid redirect”Callback URL mismatchCheck Cognito client callback URLsUpdate callback URL in Cognito client settings
Database connection errorsRDS at connection limitCheck RDS monitoring for active connectionsRestart affected service to release connections

Emergency Procedures

Kill Switch – Disable Chat Entirely

If chat needs to be shut down immediately (spam attack, critical bug):

  1. Set chat_enabled = false in the Hasura config table (or remote config endpoint).
  2. The mobile app will display “Chat is temporarily unavailable” to all users.
  3. Notify engineering immediately.

Restart a Service

Force a new deployment to pull the latest task definition and start a fresh container:

# Restart Synapse
aws ecs update-service --region us-east-1 \
  --cluster phenom-dev-cluster \
  --service phenom-dev-synapse \
  --force-new-deployment

# Restart Synapse Admin UI
aws ecs update-service --region us-east-1 \
  --cluster phenom-dev-cluster \
  --service phenom-dev-synapse-admin \
  --force-new-deployment

# Restart MCP Server
aws ecs update-service --region us-east-1 \
  --cluster phenom-dev-cluster \
  --service phenom-dev-chat-mcp \
  --force-new-deployment

New tasks typically take 1-2 minutes to become healthy. Watch ECS service events and CloudWatch logs to confirm.

Scale Down Chat Services (Emergency)

Take all chat offline at the infrastructure level:

# Scale all chat services to 0
aws ecs update-service --region us-east-1 \
  --cluster phenom-dev-cluster \
  --service phenom-dev-synapse \
  --desired-count 0

aws ecs update-service --region us-east-1 \
  --cluster phenom-dev-cluster \
  --service phenom-dev-synapse-admin \
  --desired-count 0

aws ecs update-service --region us-east-1 \
  --cluster phenom-dev-cluster \
  --service phenom-dev-chat-mcp \
  --desired-count 0

To restore services, run the same commands with --desired-count 1.


Test Accounts

AccountEmailPasswordRole
Test Usertest-user@thephenom.appPhenomTest2026!user
Test Admintest-admin@thephenom.appPhenomAdmin2026!admin

Creating New Test Accounts

  1. Go to AWS Console > Cognito > User pools > Phenom user pool.
  2. Click Create user.
  3. Set email and temporary password.
  4. Confirm the user (mark as verified).
  5. The User Provisioner Lambda should automatically add the user to chat_members on first login.
  6. To grant admin or support role, use the SetRole mutation or edit the chat_members row in Hasura Console.

Escalation Contacts

If you cannot resolve an issue using this guide:

  1. Collect CloudWatch logs (screenshot or copy the error messages).
  2. Note which service is affected and the symptoms.
  3. Escalate to the engineering team with this information.

Glossary

TermDefinition
ECSElastic Container Service – runs chat containers on AWS
FargateServerless compute for ECS – no servers to manage
ALBApplication Load Balancer – routes web traffic to the correct service
SynapseMatrix homeserver (Implementation A chat backend)
HasuraGraphQL engine on top of PostgreSQL (Implementation B)
MCP ServerChat tool server for AI agent interactions
LambdaServerless functions (link preview resolver, user provisioner)
CognitoAWS user authentication service
Soft deleteMarking a message as deleted without removing from the database
Target groupALB concept – containers that receive traffic for a URL path