Drop Runbook

What it is

drop.thephenom.app is the Phenom media upload and drop service. It provides an endpoint for users and automated pipelines to upload media files (images, video, documents) to the Phenom platform. It is a core dependency for any content-ingestion or media-provenance workflow. Downtime affects content submission and C2PA signing pipelines.

Deployment chain

Note: Confirm the exact hosting mechanism (ECS Fargate service name or App Runner service ARN) via the AWS console under ECS or App Runner in us-east-1.

Common operations

Check service health (App Runner)

aws apprunner list-services \
  --profile phenom \
  --region us-east-1 \
  --query 'ServiceSummaryList[?contains(ServiceName,`drop`)].{Name:ServiceName,Status:Status,URL:ServiceUrl}'

Restart / redeploy (App Runner)

# Get the service ARN first
SERVICE_ARN=$(aws apprunner list-services \
  --profile phenom --region us-east-1 \
  --query 'ServiceSummaryList[?contains(ServiceName,`drop`)].ServiceArn' \
  --output text)

# Trigger a new deployment
aws apprunner start-deployment \
  --service-arn "$SERVICE_ARN" \
  --profile phenom \
  --region us-east-1

Restart / redeploy (ECS Fargate, if applicable)

aws ecs update-service \
  --cluster phenom-dev-cluster \
  --service phenom-drop \
  --force-new-deployment \
  --profile phenom \
  --region us-east-1

Check S3 upload bucket

# List recent uploads to verify ingestion is working
aws s3 ls s3://phenom-media-uploads/ \
  --recursive \
  --human-readable \
  --summarize \
  --profile phenom \
  --region us-east-1 | tail -20

View logs

aws logs tail /aws/apprunner/drop-thephenom-app \
  --follow \
  --profile phenom \
  --region us-east-1

Verify it is working

curl -si https://drop.thephenom.app/ | grep -E "^HTTP|^server"
# Expected: HTTP/2 200, server: envoy (no cf-ray — drop is NOT behind CF proxy)

curl -si https://drop.thephenom.app/health | grep -E "^HTTP"
# Expected: HTTP/2 200 (note: /health returns the app HTML, not a JSON payload)

Common failure modes

Secrets the drop service reads

AppRunnerInstanceRole’s SecretsAccess policy grants GetSecretValue on phenom-drop/*, phenom-drop-staging/* and phenom-dev-app-secrets-* only. Every ARN referenced from env_secrets must fall inside those prefixes.

aws apprunner describe-service --service-arn "$SERVICE_ARN" \
  --query 'Service.SourceConfiguration.ImageRepository.ImageConfiguration'

HASURA_ADMIN_SECRET is a deliberate copy

Production drop reads HASURA_ADMIN_SECRET from the phenom-drop/env bag, not from phenom-prod-app-secrets where the value originates.

phenom-prod-app-secrets sits outside the instance role’s allowed prefixes, so pointing at it directly triggers the silent rejection above. The alternative fix, adding phenom-prod-app-secrets-* to the role, would also grant drop read access to database_url, jwt_secret, the Apple signing key and push_shared_secret, because Secrets Manager cannot scope a grant to a single JSON field. Drop is internet-facing, so the copy was chosen to keep its reach to the one value it actually needs.

When the Hasura admin secret rotates, update both places. random_password.admin_secret regenerates only when its inputs change, so this is rare, but a stale copy means drop silently stops recording drops. Copy the value across, then force a deployment so the container picks it up:

aws apprunner start-deployment --service-arn "$SERVICE_ARN"

Staging is unaffected: it reads its admin secret from phenom-dev-app-secrets, which the role already covers.

Related

• API runbook
• NEST runbook
• SES mail runbook