Plan agent connectivity to the backend API from outside the cluster

[vredchenko]

Still undecided, and the option space below still stands. Two things have changed since this was
written: a default has emerged by accident, and that default is currently broken for the exact
workload it needs to carry.

Current state (2026-09)

Staging and production now have an ingress.yaml, but it defines only the frontend route:

# The SPA pod's nginx reverse-proxies /api/ to smartem-http-api-service
# internally, so a single route covering / is sufficient.
- host: smartem.diamond.ac.uk
  http:
    paths:
    - path: /
      backend:
        service:
          name: smartem-frontend-service

There is no backend ingress. However, an agent could reach the backend at
smartem.diamond.ac.uk/api/..., because the SPA pod's nginx proxies /api/ onward. In other
words Option B(i) is now the de facto answer, arrived at as a side effect of the frontend
deployment rather than by decision - and B(i) is the option this issue argues hardest against,
on the grounds that it couples machine traffic to a pod sized for browser navigations.

The current route does not support SSE

The /api/ block in apps/smartem/nginx.conf (smartem-frontend) contains no streaming
configuration:

location /api/ {
    set $backend_upstream "${BACKEND_HOST}";
    rewrite ^/api/(.*) /$1 break;
    proxy_pass http://$backend_upstream;
    proxy_set_header Host $host;
    ...
}

There is no proxy_buffering off, no extended proxy_read_timeout, and no
proxy_http_version 1.1. Under nginx defaults that means responses are buffered, so events
do not stream as they are produced, and the read timeout is 60 seconds, so any stream idle
for a minute is terminated.

Since long-lived SSE is the primary reason the agent needs this route at all, the accidental
default fails at precisely the thing it exists for.

This affects the front end too, not only the agent. Frontend SSE consumption
(smartem-frontend#75) will meet the same configuration. It will appear to work in local
development, where Vite proxies directly to the backend and never traverses this nginx, and then
fail in staging and production. That is a latent defect waiting for the next piece of frontend
work rather than a hypothetical.

Note also that ingress controllers apply their own buffering and timeout defaults, so fixing the
pod's nginx is necessary but may not be sufficient - whichever route is chosen needs
SSE-appropriate settings at every hop.

What this changes about the decision

The choice is now between ratifying B(i) and configuring it properly, or moving to a
separate backend route (Option A) and leaving the SPA proxy for browser traffic only.

Arguments that have strengthened since the original write-up:

• The SSE defect is real and must be fixed under any option, because the front end needs
  streaming through whichever path it uses. It is therefore not a differentiator between the
  options, but it does mean "do nothing" is no longer a neutral position.
• Separating machine traffic from browser traffic (Option A) means SSE tuning, timeouts and
  sizing can be set for streaming without compromising the SPA's settings, and the two failure
  domains stay independent. That was already the argument for A; the discovery that the shared
  path needs materially different tuning makes it more concrete.

Related, updated

• smartem-devtools#181 - closed as completed. The proxy-pass architecture it introduced is
  what created the accidental default described above.
• smartem-devtools#179 - closed as superseded by the wider deployment rework. This issue is
  now where the connectivity question actually survives.
• smartem-devtools#205 - closed; added the frontend ingress and explicitly deferred this.
• smartem-frontend#75 - frontend SSE consumption; will be affected by whatever is decided here,
  and by the buffering defect regardless.

Out of scope

Implementing the choice, as before. Note however that the nginx SSE configuration is a defect
rather than part of this decision, and does not need to wait for it.