GlitchTip Runbook
Source:
phenom-infra/environments/production/glitchtip.tf; https://glitchtip.thephenom.app/_health/ returns HTTP 200 ok
Signed stamp image pending re-generation
What it is
glitchtip.thephenom.app is the self-hosted error and crash reporting service
for the Phenom mobile apps. It replaced the hosted free tier (app.glitchtip.com),
which was quota-throttled (dropping 100% of events). Downtime reduces visibility
into app errors but does not block the apps (the client SDK queues events).
Deployment chain
| Layer | Value |
|---|---|
| URL | https://glitchtip.thephenom.app |
| Health | /_health/ (expects HTTP 200, body ok) |
| DNS | Cloudflare (proxied) to a dedicated ALB |
| Hosting | AWS ECS Fargate, cluster phenom-prod-cluster (services phenom-prod-glitchtip-web, phenom-prod-glitchtip-worker) |
| Region | us-east-1 (account 657033058608) |
| Database | RDS Postgres phenom-prod-glitchtip |
| Cache/Queue | ElastiCache Redis phenom-prod-glitchtip |
| Image | glitchtip/glitchtip:6.2.0 |
| Terraform | phenom-infra/environments/production/glitchtip.tf |
Monitoring and alerts
Probed by nest-ops (Ghost Mode asset_monitor.py, key GlitchTip, topic
phenom-glitchtip) every 60s against /_health/. Two consecutive failures
(~120s) page ntfy (alerts.thephenom.app, mirrored to ghostmode-alerts and
universal-exports); recovery pages immediately; re-alert every 30 min while down.
Registered in the canonical asset-registry.yaml under phenom:.
Common operations
Check service:
aws ecs describe-services --cluster phenom-prod-cluster \
--services phenom-prod-glitchtip-web --profile phenom --region us-east-1 \
--query 'services[0].{Status:status,Running:runningCount,Desired:desiredCount}'
Restart (force new deployment):
aws ecs update-service --cluster phenom-prod-cluster \
--service phenom-prod-glitchtip-web --force-new-deployment \
--profile phenom --region us-east-1
Logs:
aws logs tail /ecs/phenom-prod-glitchtip --follow --profile phenom --region us-east-1
Run a DB migration (after a version bump):
SG=$(aws ec2 describe-security-groups --filters Name=group-name,Values=phenom-prod-glitchtip-ecs --query 'SecurityGroups[0].GroupId' --output text)
SUBNETS=$(aws ecs describe-services --cluster phenom-prod-cluster --services phenom-prod-glitchtip-web --query 'services[0].networkConfiguration.awsvpcConfiguration.subnets' --output text | tr '\t' ',')
aws ecs run-task --cluster phenom-prod-cluster --task-definition phenom-prod-glitchtip-migrate --launch-type FARGATE \
--network-configuration "awsvpcConfiguration={subnets=[$SUBNETS],securityGroups=[$SG],assignPublicIp=DISABLED}"
Verify it is working
curl -sI https://glitchtip.thephenom.app/_health/ | grep -E "^HTTP" # 200
curl -s https://glitchtip.thephenom.app/_health/ # ok
Common failure modes
| Symptom | Likely cause | Remediation |
|---|---|---|
| 502/503 from Cloudflare | ECS task or ALB target unhealthy | check ECS service + ALB target health |
/_health/ 5xx |
app container crashed | check CloudWatch logs; force new deployment |
| task keeps stopping at start | DB/Redis unreachable, or secret value missing | verify RDS + Redis SGs allow the ECS task; confirm phenom-prod-glitchtip-secrets has a value |
| dashboard slow/empty | RDS pressure | check RDS metrics; scale instance |
Related
- NEST Ops Runbook
- Deploy detail:
phenom-infra/docs/runbooks/glitchtip-self-host.md
Feedback
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.