Root Causes: The Nine Epics

142 of the 184 findings trace back to nine recurring habits. Fix a habit and you close a block of findings and stop the next one being written. Fix findings one at a time and you will be back here next quarter.

Treat each root cause as one GitHub epic. The defect counts below are your epic sizing.

The table

Two things to read out of that table. First, RC-3 owns both Criticals and a quarter of the Highs, so deny-by-default is where the risk is, not where the count is. Second, RC-9 has the second-highest High count from only 13 findings, which makes it the best risk-per-ticket in the set.

The 42 unexplained findings are mostly not architectural. 27 of them are Low-severity implementation defects in two services (ghostmode-thephenom-app and sablier-weblogon) that the architecture argument was never about. Treat them as a separate cleanup track.

RC-1 No shared security component

32 findings. 1 High, 12 Medium, 17 Low, 2 informational.

The same security decision is written from scratch in every service that needs it. There are nine separate token verifiers in the estate. There is no shared session component, so POST /auth/session accepts a cross-site request with no Origin, Referer, CSRF-token or Content-Type check in four places; three places have no __Host- cookie prefix; three have no logout, or a logout defeated by a page reload; four have no revocation; and four client stores keep authenticated state alive after logout.

Why it keeps happening: writing a verifier is easy, and nobody owns the one everyone should import.

What closing it means: build one session component with an origin assertion, a __Host--prefixed opaque session id, a revocation handle and a logout purge. That single component closes or bounds all of the above. Effort is large. It is the biggest single win in the set and the one most likely to be underestimated.

Good news: you already have the reference implementation. nest-api’s Cognito verifier is sound, verified at line, and is what to standardise on.

RC-3 Deny-by-default was never adopted

29 findings. Both Criticals, 7 High, 10 Medium, 9 Low, 1 informational.

Where a system has a default, the default is permissive. A load balancer listener whose default action forwards. A Worker classifier whose unmatched arm falls through to allow. A Hasura permission block whose row filter is {}. A verifier that skips its binding check when an environment variable is empty.

The headline number: 47 of 112 Hasura permission blocks carry filter: {}. 41 of 63 select blocks are unscoped, and 0 of 22 insert blocks are.

Why it keeps happening: nothing requires a predicate, and the engine’s default is {}. The 47 will regenerate as tables are added.

What closing it means: land the guardrail before the edits. A CI assertion that fails any select permission with an empty filter on a non-public table, over hasura/metadata/databases/**/tables/*.yaml, is worth more than the 47 individual edits, because it prevents the 48th. Then write down row ownership, then replace the filters. See the ordering constraint in MGR-H11: the mobile client depends on the over-fetch and must ship first.

RC-4 Boundaries are values inside a request

20 findings. 3 High, 12 Medium, 5 Low.

Which environment you are in, which tenant you are, which role you have: these are computed from something in the request rather than from where the code is deployed. nest-api defines PROD_SURFACE_HOSTS as a set containing one hostname, and on a match swaps in the production Cognito pool, the production client-id allowlist, the production Hasura URL, the production Hasura admin secret and the production S3 bucket. One Worker script, both environments’ secrets, and a string comparison deciding which you get.

Why it keeps happening: it is genuinely convenient, and it looks like configuration.

What closing it means: deploy separately. Two Worker scripts with two secret bindings means a non-production deployment cannot hold a production secret at all. That removes the comparison rather than hardening it. The same principle applies to the CI OIDC trust policy, which currently trusts any ref in the repository.

RC-7 No enforcement point between a change and production

14 findings. 2 High, 10 Medium, 2 Low.

There are written rules and there is no place they are checked. The CI job named “Security Gate” scans modules/ and does not read environments/, which is where production lives, so 42 files sit outside the gate that is named for them.

Why it keeps happening: a rule in a README costs nothing to write and nothing to enforce.

What closing it means: change directory: modules/ to cover environments/ in all three workflows, expect a large first-run failure list, and land it behind an expanded .checkov.baseline so new findings on the same checks still fail the build. That is the adoption pattern the estate already uses correctly elsewhere. Pin the action to a commit SHA in the same change.

The cheapest version of this epic: dot-github and dot-github-private each contain exactly one file. All 121 first-party action references across 9 repositories are mutable tags, including the scanner that gates PhenomApp pull requests and the checkov action that gates the production apply. Filling the org repository is the cheapest way to make one change reach 32 repositories, and it touches no product code.

RC-9 Credential material has no lifecycle owner

13 findings. 5 High, 2 Medium, 6 Low. Best risk-per-ticket in the set.

Credentials are generated ad hoc, distributed by copying, bounded by a filename convention, and have no issuance record, no expiry, no rotation procedure and no revocation path.

The proof that this is a lifecycle problem and not a missing-gate problem: PhenomApp has a TruffleHog CI job and has .gitignore rules for *.key and signing.key, added under a comment naming a previous incident. It did not hold, because .claude/settings.local.json matches no rule and is tracked at HEAD carrying a live service-token pair. The gate existed. What was missing was any notion that a credential is a thing with a lifecycle, so the response to one leaking was to blocklist its filename.

Roughly a dozen findings across the whole set end with “rotate the credential”, and there is no SECURITY.md in any first-party repository, no disclosure policy, no compromise runbook and no on-call definition.

What closing it means: replace filename blocklists with a content allowlist that rejects any staged file containing a private key body or a high-entropy literal regardless of its name. Write the rotation procedure. Replace file-based credentials with workload identity federation so there is no key file to commit. Then work the backlog of rotations that are currently blocked on there being no procedure.

RC-6 A declared control does not bind

13 findings. 6 Medium, 5 Low, 2 informational.

A control is declared and does not actually apply. A header-presence check that does not verify anything. A WAF rule whose ordering means it never matches. A TLS mode set on the wrong listener. An audience allowlist that exists only as a comment.

What closing it means: for each declared control, write the test that fails when the control is removed. If you cannot write that test, the control is a convention rather than a mechanism.

RC-5 No artifact reconciles what exists

11 findings. 1 High, 3 Medium, 6 Low, 1 informational.

Nobody can produce a list of what is deployed. 4 DNS records are declared in code against 32 resolving hostnames. 5 S3 buckets are declared against 9 referenced. There are 3 provisioning systems with no cross-reference between them. A production service reads and writes a bucket that is declared as a resource nowhere in the infrastructure repository, and whose name identifies it as a development-tier resource.

What closing it means: generate one register from the accounts, diff it against the infrastructure code in the build, and fail on drift. Until that exists, a resource can be load-bearing for production and invisible to every control you believe you have. Also replace the two “instructions to a human” in environments/development/outputs.tf with real cloudflare_dns_record resources, so a new hostname cannot enter the estate ungoverned.

RC-2 The Hasura admin secret is the only service identity

8 findings. 3 High, 4 Medium, 1 Low.

Every service that talks to the database talks to it as the database superuser, using one shared static secret. The MCP tool server, two Cloudflare Workers and several Lambda functions all present the same credential. No row permission is consulted for anything they do, and no access log can attribute a query to a caller, because there is only one caller.

What closing it means: define named per-service Hasura roles and take the admin secret off the Workers. Size this correctly. This is not “narrow an existing admin role”. On the default source there is no admin permission block to narrow, so the work is creating roles that do not exist, and the estate’s own convention rejects defining one as inconsistent. Weeks, not days.

Sequencing note: this blocks RC-8. You cannot alert on unusual use of the credential every service uses for everything, and a static shared secret carries no caller identity, so even a complete access log cannot attribute a request.

RC-8 Nothing observes a security decision

2 findings. Both Medium. Structural in effect.

The count is small and misleading. The measured state: zero across eight detection resource classes, zero ALB access logging, zero Cognito threat protection, eight alarms delivering into three SNS topics with zero declared subscriptions, and 18 of 22 log-group resources resolving to 7-day retention with none at 90, against a README that states “90 days minimum” four times and marks it Implemented.

What closing it means, and the order is not optional: do RC-2’s named-service-principal work and the credential work first, because detection on a shared superuser secret cannot attribute anything. Then turn on WAF logging, because the WAF’s own promotion criterion is an observation window with no data source. Then subscribe the three topics, since eight alarms including the content-safety alarm currently deliver into nothing that any file declares.

Where to go next

• Priority fixes sequences the work across these nine epics.
• Architectural weaknesses gives the ten concrete structural problems, each tagged with its root cause.
• Filing guide turns this page into GitHub epics.