Admin & Operations

Day-to-day administration, moderation, monitoring, troubleshooting, and emergency procedures for the Phenom Chat system.

Reconciling to the canonical page. The single source of truth for chat services is Chat Services (validated 2026-05-27). The “Moderation via GraphQL (Hasura Lite)” workflow below is from the retired Implementation B and is excised from production; moderate via Synapse. Chat is now Matrix/Synapse only. Where this page disagrees with the canonical page, the canonical page wins.

This page is for the Phenom support and infrastructure team. It covers moderation workflows, monitoring, and emergency procedures for both chat implementations.

Quick Reference

Admin URLs

Service URL Purpose
Synapse Admin UI https://chat-staging.thephenom.app/chat-admin Web dashboard for Synapse user/room management
MCP Server Health https://chat-staging.thephenom.app/mcp/health Health check endpoint (returns JSON status)
Synapse Health https://chat-staging.thephenom.app/_matrix/client/v3/login Verify Synapse is responding
Hasura Console Contact engineering for URL GraphQL console for Hasura Lite queries

AWS Console Resources

Resource How to Find It
CloudWatch Logs AWS Console > CloudWatch > Log groups > search for phenom-dev
ECS Services AWS Console > ECS > Clusters > phenom-dev-cluster
Cognito User Pool AWS Console > Cognito > User pools > search for phenom
Lambda Functions AWS Console > Lambda > search for phenom-dev-link-preview or phenom-dev-chat-user-provisioner

ECS Service Names

Service ECS Service Name Log Group
Synapse phenom-dev-synapse /ecs/phenom-dev-synapse
Synapse Admin UI phenom-dev-synapse-admin /ecs/phenom-dev-synapse-admin
Chat MCP Server phenom-dev-chat-mcp /ecs/phenom-dev-chat-mcp
Link Preview Lambda /aws/lambda/phenom-dev-link-preview
User Provisioner Lambda /aws/lambda/phenom-dev-chat-user-provisioner

Moderation via GraphQL (Hasura Lite)

Open the Hasura Console, go to the API tab, and run these queries/mutations.

View Recent Messages

query RecentMessages {
  chat_messages(
    order_by: { created_at: desc }
    limit: 50
    where: { is_deleted: { _eq: false } }
  ) {
    id
    content
    message_type
    created_at
    user { username }
    room { name }
  }
}

Search messages from a specific user:

query MessagesByUser($username: String!) {
  chat_messages(
    order_by: { created_at: desc }
    limit: 50
    where: {
      is_deleted: { _eq: false }
      user: { username: { _eq: $username } }
    }
  ) {
    id
    content
    created_at
    user { username }
  }
}

Delete a Message (Soft Delete)

This marks the message as deleted so it no longer appears in the app. Replace YOUR_ADMIN_UUID with your admin user UUID.

mutation DeleteMessage($id: uuid!) {
  update_chat_messages_by_pk(
    pk_columns: { id: $id }
    _set: { is_deleted: true, deleted_by: "YOUR_ADMIN_UUID" }
  ) {
    id
  }
}

Variables:

{
  "id": "paste-the-message-uuid-here"
}

Mute a User

Prevents the user from sending messages until the specified time.

mutation MuteUser($roomId: uuid!, $userId: uuid!) {
  update_chat_members(
    where: {
      room_id: { _eq: $roomId }
      user_id: { _eq: $userId }
    }
    _set: {
      is_muted: true,
      muted_until: "2026-03-23T00:00:00Z"
    }
  ) {
    affected_rows
  }
}

Unmute a User

mutation UnmuteUser($roomId: uuid!, $userId: uuid!) {
  update_chat_members(
    where: {
      room_id: { _eq: $roomId }
      user_id: { _eq: $userId }
    }
    _set: {
      is_muted: false,
      muted_until: null
    }
  ) {
    affected_rows
  }
}

Ban a User

mutation BanUser($roomId: uuid!, $userId: uuid!, $reason: String) {
  insert_chat_bans_one(object: {
    room_id: $roomId
    user_id: $userId
    banned_by: "YOUR_ADMIN_UUID"
    reason: $reason
  }) {
    id
  }
}

Variables:

{
  "roomId": "room-uuid",
  "userId": "user-uuid",
  "reason": "Violation of community guidelines"
}

Unban a User

mutation UnbanUser($roomId: uuid!, $userId: uuid!) {
  delete_chat_bans(
    where: {
      room_id: { _eq: $roomId }
      user_id: { _eq: $userId }
    }
  ) {
    affected_rows
  }
}

Moderation via Synapse Admin UI

Logging In

  1. Go to https://chat-staging.thephenom.app/chat-admin.
  2. The Homeserver URL should be pre-configured to https://chat-staging.thephenom.app.
  3. Log in with an admin account.

Users Tab

  • Search users: Type a username or Matrix ID in the search bar.
  • View user details: Click a user to see profile, devices, and room memberships.
  • Deactivate a user: Open user detail, click “Deactivate”. This is permanent – use with caution.

Rooms Tab

  • Search rooms: Find “Phenom Internal”, “Phenom Partners”, or “Phenom Community” by name or room ID.
  • View room: Click to see members, recent messages, and room state.
  • Purge messages: Select messages to permanently delete from the Synapse database.

Server Notices

Use the Admin UI to send server-wide notices (e.g., planned maintenance announcements).


User Management

Assign Roles (Hasura Lite)

Via Hasura Console:

  1. Go to the Data tab.
  2. Click on chat_members.
  3. Find the user row (filter by user_id).
  4. Change the role field to user, support, or admin.
  5. Click Save.

Via GraphQL:

mutation SetRole($roomId: uuid!, $userId: uuid!, $role: String!) {
  update_chat_members(
    where: {
      room_id: { _eq: $roomId }
      user_id: { _eq: $userId }
    }
    _set: { role: $role }
  ) {
    affected_rows
  }
}

Provision AI Agent Accounts

AI agent accounts use the phenom-dev-chat-agent Cognito client (machine-to-machine, USER_PASSWORD_AUTH flow).

  1. Go to AWS Console > Cognito > User pools > Phenom user pool.
  2. Create a new user with the agent’s username and a strong password.
  3. Confirm the user (no email verification needed for agents).
  4. Agent client credentials are in AWS Secrets Manager under phenom-dev-development-chat-clients.

Check Membership Status

query CheckMembership($userId: uuid!) {
  chat_members(where: { user_id: { _eq: $userId } }) {
    room { name }
    role
    is_muted
    muted_until
    joined_at
  }
  chat_bans(where: { user_id: { _eq: $userId } }) {
    room { name }
    reason
    banned_at
    expires_at
  }
}

If a user has no rows in chat_members, they have not been provisioned. Check the User Provisioner Lambda logs.


Monitoring

Quick Health Checks

# MCP Server -- should return {"status":"ok","backend":"hasura",...}
curl https://chat-staging.thephenom.app/mcp/health

# Synapse -- should return available login flows
curl https://chat-staging.thephenom.app/_matrix/client/v3/login

ECS Service Status

aws ecs describe-services --region us-east-1 \
  --cluster phenom-dev-cluster \
  --services phenom-dev-synapse phenom-dev-synapse-admin phenom-dev-chat-mcp \
  --query 'services[*].{Name:serviceName,Running:runningCount,Desired:desiredCount,Status:status}' \
  --output table

All three services should show Running: 1, Desired: 1, Status: ACTIVE.

CloudWatch Logs

# Synapse logs (last 1 hour)
aws logs tail /ecs/phenom-dev-synapse --follow --since 1h --region us-east-1

# MCP Server logs
aws logs tail /ecs/phenom-dev-chat-mcp --follow --since 1h --region us-east-1

# Link Preview Lambda logs
aws logs tail /aws/lambda/phenom-dev-link-preview --follow --since 1h --region us-east-1

# User Provisioner Lambda logs
aws logs tail /aws/lambda/phenom-dev-chat-user-provisioner --follow --since 1h --region us-east-1

Key Metrics

Metric Where to Check Expected Value
ECS task count ECS Console or describe-services 1 running task per service
ALB target group health EC2 > Target Groups All targets healthy
Lambda invocation errors CloudWatch > Lambda > Monitoring 0 errors
RDS active connections RDS Console > Monitoring Stable, no sudden spikes
Synapse /health Browser or curl HTTP 200
MCP /mcp/health Browser or curl HTTP 200 with {"status":"ok"}

Ask engineering to configure alarms for:

  • ECS task count dropping to 0 for any chat service
  • Lambda error rate exceeding 5% for link preview or user provisioner
  • ALB 5xx error rate exceeding 1%
  • Synapse health check failures

Troubleshooting

Symptom Likely Cause How to Investigate Resolution
Users cannot send messages User is muted or banned Query chat_members for is_muted = true; check chat_bans Unmute or unban using mutations above
Messages not appearing in real time WebSocket/subscription disconnected Ask user to close and reopen chat screen Client-side reconnection; no server action needed
Link previews not showing Link Preview Lambda error Check /aws/lambda/phenom-dev-link-preview logs Review Lambda error, escalate to engineering
Synapse Admin UI blank page REACT_APP_SERVER env var wrong Check ECS task definition for phenom-dev-synapse-admin Engineering must update env var and redeploy
MCP tools not working MCP server cannot reach Hasura Check /ecs/phenom-dev-chat-mcp logs for connection errors Verify HASURA_GRAPHQL_URL env var; restart service
New users cannot access chat User not provisioned into chat_members Check chat_members for the user; check Cognito trigger Lambda logs Run provisioner manually or add via Hasura Console
Service shows 0 running tasks ECS task crashed or failed to start Check ECS service events and CloudWatch logs Restart service (see below)
502 Bad Gateway on chat URLs ECS task not running or unhealthy Check target group health in EC2 Console Restart service; check logs for crash reason
Cognito SSO loop / “invalid redirect” Callback URL mismatch Check Cognito client callback URLs Update callback URL in Cognito client settings
Database connection errors RDS at connection limit Check RDS monitoring for active connections Restart affected service to release connections

Emergency Procedures

Kill Switch – Disable Chat Entirely

If chat needs to be shut down immediately (spam attack, critical bug):

  1. Set chat_enabled = false in the Hasura config table (or remote config endpoint).
  2. The mobile app will display “Chat is temporarily unavailable” to all users.
  3. Notify engineering immediately.

Restart a Service

Force a new deployment to pull the latest task definition and start a fresh container:

# Restart Synapse
aws ecs update-service --region us-east-1 \
  --cluster phenom-dev-cluster \
  --service phenom-dev-synapse \
  --force-new-deployment

# Restart Synapse Admin UI
aws ecs update-service --region us-east-1 \
  --cluster phenom-dev-cluster \
  --service phenom-dev-synapse-admin \
  --force-new-deployment

# Restart MCP Server
aws ecs update-service --region us-east-1 \
  --cluster phenom-dev-cluster \
  --service phenom-dev-chat-mcp \
  --force-new-deployment

New tasks typically take 1-2 minutes to become healthy. Watch ECS service events and CloudWatch logs to confirm.

Scale Down Chat Services (Emergency)

Take all chat offline at the infrastructure level:

# Scale all chat services to 0
aws ecs update-service --region us-east-1 \
  --cluster phenom-dev-cluster \
  --service phenom-dev-synapse \
  --desired-count 0

aws ecs update-service --region us-east-1 \
  --cluster phenom-dev-cluster \
  --service phenom-dev-synapse-admin \
  --desired-count 0

aws ecs update-service --region us-east-1 \
  --cluster phenom-dev-cluster \
  --service phenom-dev-chat-mcp \
  --desired-count 0

To restore services, run the same commands with --desired-count 1.


Test Accounts

Account Email Password Role
Test User test-user@thephenom.app PhenomTest2026! user
Test Admin test-admin@thephenom.app PhenomAdmin2026! admin

Creating New Test Accounts

  1. Go to AWS Console > Cognito > User pools > Phenom user pool.
  2. Click Create user.
  3. Set email and temporary password.
  4. Confirm the user (mark as verified).
  5. The User Provisioner Lambda should automatically add the user to chat_members on first login.
  6. To grant admin or support role, use the SetRole mutation or edit the chat_members row in Hasura Console.

Escalation Contacts

If you cannot resolve an issue using this guide:

  1. Collect CloudWatch logs (screenshot or copy the error messages).
  2. Note which service is affected and the symptoms.
  3. Escalate to the engineering team with this information.

Glossary

Term Definition
ECS Elastic Container Service – runs chat containers on AWS
Fargate Serverless compute for ECS – no servers to manage
ALB Application Load Balancer – routes web traffic to the correct service
Synapse Matrix homeserver (Implementation A chat backend)
Hasura GraphQL engine on top of PostgreSQL (Implementation B)
MCP Server Chat tool server for AI agent interactions
Lambda Serverless functions (link preview resolver, user provisioner)
Cognito AWS user authentication service
Soft delete Marking a message as deleted without removing from the database
Target group ALB concept – containers that receive traffic for a URL path