feat(security): an origin gate for the app container, shipped off

[MarkusNeusinger]

Summary

• The second door has a gate now, and it ships switched off. api/origin_gate.py closed the API service's door and wrote the app service's door down as the thing it could not close from its own side: anyplot-app stands with ingress=all, serves the whole site from https://anyplot-app-r3tvmejsmq-ez.a.run.app with no bot challenge, no WAF and no rate limit, and relays any crawler user agent through @seo_proxy to https://api.anyplot.ai — where the edge stamps the API's secret legitimately, so the API gate cannot tell. app/origin-gate.conf.template is the nginx half of the same mechanism: same secret, same five verdicts, ORIGIN_GATE unset = off.
• Nothing is armed by merging this. The image defaults ORIGIN_GATE=off, the service declares no environment variables at all, and /_health already reports X-Origin-Gate — so every route into the container can be measured before anything is switched on. The runbook is in infra/cloudflare/README.md § "The site's own origin", summarised at the bottom here.
• Three callers reach that origin without the edge and each now stamps its own header, plus one that turned out to be a live production path nobody had connected to the gate: the apex Worker's /api/event.

The mechanism, in three sentences

app/origin-gate.conf.template renders into /etc/nginx/conf.d/00-origin-gate.conf at container start, via the entrypoint script 20-envsubst-on-templates.sh that nginxinc/nginx-unprivileged:alpine already ships — no start script of ours, and NGINX_ENVSUBST_FILTER=^ORIGIN_ keeps envsubst away from $host and $uri. Seven map blocks turn the presented header into $origin_gate_status (off · off-seen · ok · missing · mismatch, the API's own vocabulary) and $origin_gate_deny; every server block in app/nginx.conf carries if ($origin_gate_deny) { return 421; } at server level, which runs in the server-rewrite phase and therefore covers every location the block has and every location added later. The secret is written in exactly one map key, app/nginx.conf never names $http_x_origin_secret at all, and every proxy_pass clears the header so no upstream is ever handed it — three rules with a test each, plus a CI smoke that greps the refusal page and the container log for the value.

The finding the brief asked for: Host would have worked, and still cannot be the mechanism

api/origin_gate.py guessed that $host at this origin might be the run.app name for all traffic, which would have settled the question. It is not:

gcloud beta run domain-mappings list --region=europe-west4 --project=anyplot	anyplot.ai → anyplot-app, www.anyplot.ai → anyplot-app, api.anyplot.ai → anyplot-api

They are Cloud Run domain mappings, so Cloudflare forwards the original Host and $host really does distinguish the edge from the raw URL — and the value cannot be spoofed, because Google's frontend answers a foreign Host on a run.app address with its own 404 before the container is reached (measured in #11208).

A Host rule still cannot be the gate. bot-serving-check.yml probes this exact origin with crawler user agents because Cloudflare 403s GitHub-runner IPs even for a UA-spoofed Googlebot; it cannot spoof the Host either, and any exception keyed on something it could present instead — a header it invents, a user agent — is public with this repository. The exception has to be the shared secret. Once the workflow carries the secret, the Host rule buys nothing the header does not, so it is not built.

Every hostname this container serves

The Transform Rule has to cover all of them, or arming locks out the visitors it protects.

Hostname	Reaches the container via	Transform Rule
anyplot.ai	Cloud Run domain mapping, proxied	required
www.anyplot.ai	Cloud Run domain mapping, proxied — it serves the site, it does not redirect (curl -sI https://www.anyplot.ai/ → 200, no Location)	required
anyplot.ai/api/event	the apex Worker, fetch(request) back to this origin	none — the Worker stamps it (see below)
python.anyplot.ai	nothing today: it is a server_name in app/nginx.conf with no DNS record and no domain mapping (curl → Could not resolve host, checked 2026-09-04)	add it the day the DNS record is added; the block is gated already
anyplot-app-r3tvmejsmq-ez.a.run.app	direct	none — this is the door being closed
anyplot-app-239660669828.europe-west4.run.app	direct, the service's second URL	none, same door
candidate---anyplot-app-r3tvmejsmq-ez.a.run.app	direct, the pre-traffic tag URL	none — the smoke sends the header itself

The rule must be a Set, not an Add: a caller supplying its own X-Origin-Secret has to have it replaced.

Every legitimate direct caller, and what each now sends

Caller	Now sends	Notes
app/cloudbuild.yaml pre-traffic smoke	X-Origin-Secret on every probe, read from Secret Manager inside the step	Not availableSecrets, which resolves at build start and would fail every build until the secret exists — the same reasoning api/cloudbuild.yaml already carries. It also asks /_health for the verdict before any content probe, so a wired-up-wrong secret is reported as itself instead of as a mystifying 403 on the home page.
.github/workflows/bot-serving-check.yml	the header from the ORIGIN_SECRET repository secret, on all ~36 probes	Reads /_health first: missing and mismatch are hard failures with a message naming the secret; off is a warning; an absent header is a warning naming deploy lag. Without that, an armed gate plus a missing secret would open an incident saying "every crawler page is broken".
the apex Worker, /api/event	stamps from its own ORIGIN_SECRET binding	The one that would have broken production. anyplot.ai/api/event is the only path under the Worker's route that goes to the site's origin instead of the API host, and a Worker subrequest to a host in the same zone skips that zone's Transform Rules — the exact finding infra/cloudflare/README.md exists for, biting a second time. Arming without this answers every Plausible pageview on the site with a 403, quietly.
Cloud Run startup probe	nothing, and needs nothing	gcloud run services describe → startupProbe: tcpSocket: port 8080, failureThreshold 1. Not an HTTP probe, so no exemption.
IndexNow (indexnow-submit.yml) + Bing's key verification	nothing, and needs nothing	Both fetch https://anyplot.ai/<key>.txt, i.e. through the edge.
PageSpeed Insights (audit harness)	nothing, and needs nothing	Audits https://anyplot.ai/..., through the edge.

Also checked and empty: gcloud monitoring uptime list-configs --project=anyplot → Listed 0 items. No Lighthouse CI workflow exists. The ORIGIN_SECRET repository secret already exists (created 2026-09-03 for the API rollout, used by sync-postgres.yml), and the build service account 239660669828-compute@developer.gserviceaccount.com — the account both triggers run as — already holds roles/secretmanager.secretAccessor on it. So no grant and no new secret are needed; the runbook only confirms them.

Correcting a claim from #11220

That PR's description recorded, as the reason a Cloud Run env-var switch was not built:

/etc/nginx is read-only in nginxinc/nginx-unprivileged (checked: no chown/chmod g+w on it, final USER 101), so the image's own NGINX_ENVSUBST_OUTPUT_DIR mechanism bails out with "not writable" and renders nothing.

That reading came from the wrong file in the image's repository. nginxinc/nginx-unprivileged:alpine is FROM nginxinc/nginx-unprivileged:1.31.5-alpine-slim, and mainline/alpine/Dockerfile — the variant that adds the modules — genuinely contains no such lines, because they are in the base it inherits from. mainline/alpine-slim/Dockerfile:

# nginx user must own the cache and etc directory to write cache and tweak the nginx config
    && chown -R $UID:0 /var/cache/nginx \
    && chmod -R g+w /var/cache/nginx \
    && chown -R $UID:0 /etc/nginx \
    && chmod -R g+w /etc/nginx

/etc/nginx/conf.d is owned by uid 101, which is the uid the container runs as. Now confirmed live rather than read: the new CI job's log carries 20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/00-origin-gate.conf.template to /etc/nginx/conf.d/00-origin-gate.conf. #11220 was right that nothing could exercise it before it mattered — which is what that job is for, and it earned its keep on the first run (below).

Verification

CI builds the app image and runs the gate against it. app/Dockerfile was hadolinted but never built before Cloud Build, i.e. after the merge — and what it produces is not a program that fails to import but an nginx whose config is rendered at container start. The new app-image job in ci-image.yml runs the real image three ways:

gate off	/ 200; /_health → X-Origin-Gate: off; with any header → off-seen
armed	/ 403 bare, 403 with a wrong secret, 200 with the right one; /_health still 200 (exempt) and reporting missing / mismatch / ok
armed, no secret	403 with no header and with an empty one — the tagged map keys, proven
always	the 403 is the gate's own page; the secret appears in neither that page nor docker logs; nginx -t validates the rendered config (nginx -T would print the secret into the CI log, so it is never run)

It failed on its first run, with a defect nothing else here could have found. With the gate armed:

nginx: [emerg] could not build map_hash, you should increase map_hash_bucket_size: 64

nginx cannot hash a map key longer than one bucket, and the default bucket is the processor's cache line — 64 bytes. The tagged key is presented: plus the whole secret, so a 32-byte secret written as hex is 74 characters. With the gate off the container starts perfectly, because the key is short then; the failure would have appeared at the exact moment of arming and nowhere earlier. Cloud Run would have kept the previous revision serving, so it would have been a safe failure rather than an outage — but it would have been a failure in the middle of the one procedure this whole PR exists to make undramatic. The template now sets map_hash_bucket_size 512, the smoke uses a production-length 64-character secret so the ceiling stays exercised, and a test pins the directive.

The rendered config, parsed by nginx's own grammar. No Docker in this environment, so crossplane (nginx's own config parser) was run over the template rendered three ways and assembled into the same http {} context the container has:

gate on,  secret set    → status: ok    undefined variables: []
gate off, secret empty  → status: ok    undefined variables: []   ← the shipping state
gate on,  secret empty  → status: ok    undefined variables: []   ← fail-closed

Worth noting for calibration: crossplane parses, it does not build hash tables, so it passed the map_hash defect cleanly. A text check cannot replace a running container, which is the argument for the CI job.

Static guards, tests/unit/api/test_app_origin_gate.py (14 cases), next door to the API gate's own tests: every server block gated and each with its own error_page/@origin_denied; the gate before the trailing-slash rewrite; app/nginx.conf never naming $http_x_origin_secret; every $origin_gate_* it reads defined by the template; the secret written once and tagged; the hash bucket raised; no proxy_pass without clearing the header; the exemption exactly /_health; both /_health blocks reporting the verdict and re-including the header snippet; the Worker stamping and deleting-before-stamping on /api/event; the smoke and the monitor carrying the header; the Dockerfile's three lines.

Local gates: uv run pytest tests/unit tests/integration — 1988 passed. ruff check ., ruff format --check ., mypy api core clean. uv run python -m tools.changelog check --base origin/main — 7 fragments well-formed.

No /verify-frontend run, and the reason is worth stating rather than skipping: the changed behaviour is nginx's, there is no SPA change in the diff (no file under app/src, no TypeScript at all), and the flow cannot be driven from a browser without the container. The container smoke above is that loop, and the deploy's pre-traffic smoke is the second one.

Review round

Four findings, three of them defects, all applied; threads answered and resolved.

• The gate header was forwarded to every upstream. The sharpest one. nginx passes incoming request headers to a proxied server by default, so once the Transform Rule stamps this host, /js/script.js and /api/event would have handed the shared secret to plausible.io — and with it the API service's key, since both take the same value. Fixed wider than reported: the rule is now that the header is consumed by this server and never forwarded, so every proxy_pass in both blocks clears it, and a test refuses a location that proxies without doing so.
• Arming was written as two flags. This service pins traffic to a named revision (app/cloudbuild.yaml promotes with --to-revisions=<name>=100), so gcloud run services update alone creates an armed revision that serves nothing — and step (e) would have read off on a path that carries the header. The runbook is now the API's own block, copied rather than paraphrased: refuse while a Cloud Build is in flight, pin the serving image rather than the latest template, --revision-suffix, then update-traffic --to-revisions=…=100. Rolling back is its own block that looks nothing up.
• ORIGIN_SECRET:latest instead of a pinned version. Cloud Run resolves a secret-backed variable when each instance starts, so a rotation reaches new instances while older ones keep the old value — intermittent 403s inside one revision. The block resolves the newest ENABLED version number and refuses if there is none.
• ORIGIN_GATE=ON also arms, because a plain map key is matched without regard to case. Kept, and the documentation corrected instead: nobody sets that variable to ON without meaning to arm, and the other failure direction — an operator who armed the gate, was told nothing, and still has an open origin — is the one worth avoiding.

One more, from CI rather than review: hadolint DL3064 reads the ENV variable name and warns that a secret may be baked into the image. The value is the empty string, and it is declared for the opposite reason — so a service supplying no secret renders a config that refuses everyone rather than one nginx cannot parse. Renaming would silence the rule and break a four-place contract (Secret Manager, the API service, the Worker binding, the repository secret), so the exception sits on an ENV instruction of its own with its reason beside it, per the repository's own hadolint convention, and ci-image.yml's claim that app/Dockerfile needs no exceptions is updated.

Two decisions worth reviewing

• return 421, not return 403. An error_page 403 sends the refusal back through the server-rewrite phase, where it hits the same if again and loses the custom page. A distinct internal code also keeps this trick apart from the crawler's 418. nginx generates a 421 of its own only for a coalesced HTTP/2 connection with a mismatched authority, which cannot happen here — Cloud Run speaks HTTP/1.1 to this container (ports: name: http1).
• A custom 403 page instead of nginx's stock one. The stock body prints the exact nginx version to anyone knocking on the raw origin. The refusal is a named location, entered without re-running the server-rewrite phase, and it carries X-Origin-Gate so a half-applied rotation says mismatch instead of just failing.
• The map comparison is case-insensitive and not constant-time, unlike api/secret_compare.py. nginx lowercases a map source before hashing and offers no constant-time primitive. Both are written down at the top of the template; against a random 256-bit secret the answer to both is the entropy, not the comparison.

Rollout — for the main session

Steps (a) and (b) are safe on their own and can sit for days. The arm and the rollback are full blocks, not one-liners — they live in infra/cloudflare/README.md § "Arming, in full" and § "Rolling back", and mirror the API's.

# (a) merge + deploy. Nothing is armed.
curl -sI https://anyplot.ai/_health | grep -i x-origin-gate            # expect: off

# (b) widen the Transform Rule to anyplot.ai AND www.anyplot.ai (dashboard →
#     Rules → Transform Rules → Modify Request Header, "Set"), and redeploy the
#     Worker so its /api/event branch stamps too. Then measure EVERY path —
#     each must read off-seen before anything is armed:
curl -sI https://anyplot.ai/_health                          | grep -i x-origin-gate
curl -sI https://www.anyplot.ai/_health                      | grep -i x-origin-gate
curl -si -X POST -A 'Googlebot' https://anyplot.ai/api/event -d '{}' | grep -i -e '^HTTP' -e x-origin-gate
curl -sI https://anyplot-app-r3tvmejsmq-ez.a.run.app/_health | grep -i x-origin-gate   # stays off

# (c) confirm the two callers (both already exist — nothing to create):
gh secret list --repo MarkusNeusinger/anyplot | grep ORIGIN_SECRET
gcloud secrets get-iam-policy ORIGIN_SECRET --project=anyplot   # the compute SA has secretAccessor
gh workflow run bot-serving-check.yml --repo MarkusNeusinger/anyplot   # expect "origin gate: off-seen"

# (d) arm — infra/cloudflare/README.md § "Arming, in full". In outline:
#       refuse while a Cloud Build is in flight; resolve the SERVING revision's
#       image; resolve the newest ENABLED secret VERSION (never :latest);
#       services update --image=… --update-secrets=ORIGIN_SECRET=ORIGIN_SECRET:$VERSION
#         --update-env-vars=ORIGIN_GATE=on --revision-suffix="arm-<stamp>";
#       services update-traffic --to-revisions="anyplot-app-arm-<stamp>=100".

# (e) verify:
curl -sI https://anyplot.ai/_health | grep -i x-origin-gate                            # ok
curl -s -o /dev/null -w '%{http_code}\n' https://anyplot.ai/                           # 200
curl -s -o /dev/null -w '%{http_code}\n' https://anyplot-app-r3tvmejsmq-ez.a.run.app/  # 403
curl -s -A 'Mozilla/5.0 (compatible; Googlebot/2.1)' https://anyplot.ai/scatter-basic | grep canonical
curl -si -X POST -A 'Googlebot' https://anyplot.ai/api/event -d '{}' | head -1         # 202, not 403
gh workflow run bot-serving-check.yml --repo MarkusNeusinger/anyplot                   # green
gcloud run services describe anyplot-app --project=anyplot --region=europe-west4 --format="value(status.traffic)"

# (f) rollback — infra/cloudflare/README.md § "Rolling back". Same shape, no
#     lookups: pin the serving image, --remove-env-vars=ORIGIN_GATE,
#     --revision-suffix="disarm-<stamp>", promote that revision by name.

Rotation now touches five copies of one value: Secret Manager, the API service, the app service, the Worker binding and the GitHub repository secret. Roll back, rotate, arm again; the gate is off in between, which is the documented safe state. Note also the secret's length ceiling: the map key is presented: plus the secret and the bucket is 512 bytes, so a secret past roughly 500 characters would keep the armed revision from ever becoming ready.

Plan

N/A — audit item A30, decided directly.

Test plan

• CI app-image: the real image passes the gate matrix — off / armed / armed-with-no-secret, the exempt path, each verdict, the refusal page, and the secret in neither the page nor the logs
• crossplane parses the rendered config in all three states inside the container's own http {}; no undefined variables
• uv run pytest tests/unit tests/integration — 1988 passed, 14 of them new in tests/unit/api/test_app_origin_gate.py
• uv run ruff check . + ruff format --check . + mypy api core — clean
• uv run python -m tools.changelog check --base origin/main — fragment well-formed
• the base image's entrypoint, template directory and /etc/nginx ownership verified against nginxinc/docker-nginx-unprivileged mainline/alpine-slim/Dockerfile, then confirmed live in the CI job's entrypoint log
• gcloud run services describe anyplot-app — no env vars today, startupProbe is tcpSocket (no HTTP probe to exempt)
• After merge, before arming: every path in step (b) of the rollout reads off-seen
• After arming: step (e), including anyplot.ai/api/event answering 202 rather than 403

🤖 Generated with Claude Code

https://claude.ai/code/session_01UEScQMZFvxxNNyNJYryfa3

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[Copilot]

🔵 Needs a closer look

A critical secret-disclosure issue and multiple rollout-safety issues remain unresolved.

Pull request overview

Adds a disabled-by-default shared-secret origin gate protecting the app’s direct Cloud Run endpoints.

Changes:

• Adds nginx gate enforcement and health verdicts.
• Authenticates legitimate direct callers.
• Adds container tests, monitoring, documentation, and rollout guidance.

File summaries

File	Change	Review
tests/unit/api/test_app_origin_gate.py	Adds configuration guards.	Nit (line 28, 1 vote): Correct the inconsistent file counts.
infra/cloudflare/README.md	Documents configuration and rollout.	Nit (line 299, 1 vote): Convert commented rollout actions into numbered Markdown steps. Moderate (lines 327, 340, 1 vote): Explicitly promote the newly armed revision. Moderate (line 328, 1 vote): Pin a numeric secret version instead of latest.
infra/cloudflare/anyplot-api-proxy.js	Stamps Worker analytics requests.	No unresolved finding.
docs/reference/seo.md	Documents protected crawler routing.	No unresolved finding.
docs/reference/api.md	Updates origin-gate documentation.	No unresolved finding.
changelog.d/app-origin-gate.md	Records the security feature.	No unresolved finding.
app/origin-gate.conf.template	Defines gate verdict and denial maps.	Moderate (lines 85, 97, 1 vote): Match on case-sensitively to prevent accidental activation.
app/nginx.conf	Enforces the gate and exposes health status.	Critical (lines 168, 470, 1 vote): Clear X-Origin-Secret before proxying requests to Plausible and add an egress guard.
app/Dockerfile	Renders the gate template at startup.	No unresolved finding.
app/cloudbuild.yaml	Authenticates candidate-revision probes.	No unresolved finding.
.github/workflows/ci-image.yml	Tests the built container’s gate states.	No unresolved finding.
.github/workflows/bot-serving-check.yml	Authenticates direct crawler probes.	No unresolved finding.

Review details

Suppressed comments (5)

app/nginx.conf:472

• This second server block has the same secret-egress problem: accepted edge requests retain X-Origin-Secret, and both Plausible proxy locations forward request headers by default. Clear the header before proxying to plausible.io; otherwise enabling this hostname later exposes the shared gate secret.

    if ($origin_gate_deny) {
        return 421;
    }

app/origin-gate.conf.template:97

• This is not an exact case-sensitive exemption: nginx map literal keys ignore case. With the gate armed, an unauthenticated request such as /_HEALTH is therefore admitted; because the nginx location is case-sensitive, it falls through to the normal SPA/bot routing rather than the health handler, bypassing the origin gate. Make this a case-sensitive anchored regex and update the static/runtime tests to cover a case variant.

    "/_health"  1;

infra/cloudflare/README.md:304

• This rollout is a user procedure, but its actions are embedded as lettered comments inside a shell block. The repository documentation contract requires numbered Markdown steps with one action per step and says manual actions must not be hidden in code-block comments; move steps (a)–(f), especially the dashboard and Worker deployment actions, into a numbered list with command blocks under the relevant steps.

### Rollout

Ordered so that nothing is armed before it has been measured. Steps (a) and (b)
are safe on their own and can sit for days.

```bash

infra/cloudflare/README.md:341

• This primary rollback also creates an unserved revision because traffic is pinned to the currently named revision. During an incident, removing ORIGIN_GATE without an explicit revision suffix and update-traffic leaves the armed revision at 100%, so the rollback does not take effect. Use the same serving-image and explicit-promotion sequence as the API rollback.

gcloud run services update anyplot-app --region=europe-west4 --project=anyplot \
  --remove-env-vars=ORIGIN_GATE

tests/unit/api/test_app_origin_gate.py:28

• The introduction counts six files (item 4 names two), so this sentence's “five” and “four” are inconsistent with the test module's own description.

None of the five can see the other four, and every one of them fails silently:

• Files reviewed: 12/12 changed files
• Comments generated: 4
• Review effort level: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.