CI role permissions and the admin bootstrap

Why the Terraform CI role cannot grant itself permissions, how a new grant lands anyway, and the known gaps as of 2026-09-25.

The role is IAM-blind on purpose

Every infra workflow (dev, prod, chat) authenticates via OIDC as phenom-dev-github-actions. Its permissions live in phenom-infra/modules/ci-cd/main.tf, in two inline policies:

Inline policy Terraform resource Holds
phenom-dev-github-actions-policy aws_iam_role_policy.github_actions read for plans, scoped writes
phenom-deploy-create-only aws_iam_role_policy.github_actions_create_only create verbs, no deletes

The role deliberately has no iam:PutRolePolicy, so a compromised workflow can’t widen its own access. The cost is that a PR which adds a permission can’t be applied by CI. The apply fails 403 on the very policy it’s trying to change.

How a new grant lands

  1. Open the PR that edits modules/ci-cd/main.tf. Plans pass. ci_cd is only instantiated in environments/development.

  2. Merge it.

  3. An administrator applies only that policy from a clean checkout of main:

    cd environments/development
    terraform plan  -target=module.ci_cd.aws_iam_role_policy.github_actions -out=ci-role.plan
    terraform apply ci-role.plan
    

    Swap in github_actions_create_only if that’s the policy that changed. Check the plan touches exactly one resource before applying.

  4. Read it back with aws iam get-role-policy --role-name phenom-dev-github-actions --policy-name <name>.

  5. Re-run the failed prod or dev workflow. Terraform now sees no diff for the policy, and the resource that needed the permission applies.

The same pattern covers other IAM writes CI can’t make. For example, the PreSignUp linker’s prod role policy was written by an admin with aws iam put-role-policy, using exactly the document Terraform renders (phenom-infra#321).

Size limit

AWS caps all inline policies on one role combined at 10,240 characters (minified). As of 2026-09-25 the two policies total 8,183. Measure before adding large statements:

for p in phenom-deploy-create-only phenom-dev-github-actions-policy; do
  aws iam get-role-policy --role-name phenom-dev-github-actions --policy-name $p \
    --query PolicyDocument --output json | jq -c . | tr -d ' ' | wc -c
done

Known gaps (2026-09-25)

Gap Symptom on prod apply Fix
apigateway:PATCH on /account updating API Gateway Account 403; prod API Gateway logs still write under the dev CloudWatch role phenom-infra#310 / PR #311
ecr:SetRepositoryPolicy on the file-validator repos putting ECR Repository Policy (phenom-prod-file-validator) 403; prod keeps the broader console-created repo policy phenom-infra#323