Skip to content

fix: keep the runtime alive when the agent fetch leaks a rejection - #67

Open
hdimer wants to merge 1 commit into
CopilotKit:mainfrom
hdimer:fix/unhandled-rejection-backstop
Open

fix: keep the runtime alive when the agent fetch leaks a rejection#67
hdimer wants to merge 1 commit into
CopilotKit:mainfrom
hdimer:fix/unhandled-rejection-backstop

Conversation

@hdimer

@hdimer hdimer commented Aug 27, 2026

Copy link
Copy Markdown

Fixes #33 (the UND_ERR_SOCKET half; see Scope below).

The problem

app/channel.tsx and app/commands/index.ts both already wrap await thread.runAgent(...) in runAgentSafely, and that try/catch works fine. The process still dies.

One failed fetch to the agent produces two rejections: the one the caller awaits, and a second unawaited copy of the identical error. Nothing is attached to the second one, so Node's default handler terminates the process. No application-level try/catch can reach it, because it is not on the promise chain the application awaits.

Reproduced against the real SanitizingHttpAgent on current main (v0.4.1), pointed at a local server that writes a partial SSE response then destroys the socket, which is what an agent restart mid-turn looks like from the runtime's side:

AWAIT_CAUGHT_CLEANLY: terminated          <- runAgentSafely did its job
node:internal/process/promises:394
    triggerUncaughtException(err, true /* fromPromise */);
TypeError: terminated
  [cause]: SocketError: other side closed
    code: 'UND_ERR_SOCKET'

Exit code 1.

Worth spelling out why this is worse than "the container restarts": .railway/railway.ts sets restartPolicyType: "ON_FAILURE" with restartPolicyMaxRetries: 5. An agent that flaps a handful of times doesn't just bounce the runtime, it exhausts the restart budget and the service stays down until a human notices.

The fix

installUnhandledRejectionBackstop(), called from server.ts's isMain block. This is the issue author's own suggested fix #2. Same repro, with it in place:

AWAIT_CAUGHT_CLEANLY: terminated
[opentag] unhandled rejection; runtime kept alive TypeError: terminated
RUNTIME_SURVIVED                          <- exit 0

This is not a substitute for fixing the leak upstream. The duplicate rejection originates in @ag-ui/client / @copilotkit/channels internals, which this repo can't patch, so the right long-term fix is in HttpAgent.runAgent(). This is a one-line stop-the-bleeding measure that stays correct afterwards: it does not swallow or alter the awaited rejection, so every call site still receives its error and still posts the user-facing message it posts today. Happy to close this if you'd rather fix it at the source.

Three deliberate non-decisions:

  • Not installed on import. Only isMain calls it, so importing server.ts doesn't install a process-wide handler on a consumer. It sits next to the 30-minute undici bodyTimeout dispatcher, which is production-only hardening for the same failure mode.
  • No process.exitCode. Given ON_FAILURE with 5 retries, a runtime that survives a hiccup at 09:00 and then takes a clean SIGTERM at 18:00 would exit non-zero and spend a restart on a deploy that worked. The two existing exitCode = 1 sites in this file are genuinely terminal paths; a backstop isn't.
  • No cause.code === "UND_ERR_SOCKET" filter. That shape is undocumented internals two packages deep, behind a caret dep. If it drifts by one field the crash returns with a fully green suite.

I also considered reportRecoverableError from app/channel-helpers.ts and left it: it's [channel]-tagged app-layer, it requires an operation/recovery pair that a process-boundary handler can't honestly supply, and server.ts uses bare console.error throughout.

Scope

This is the UND_ERR_SOCKET half of #33. The UND_ERR_BODY_TIMEOUT half was already largely mitigated by the 30-minute bodyTimeout in 4b336d7; the backstop catches whatever still gets through.

Verification

pnpm check-types                            clean
pnpm test                                   23 files, 233 tests
(cd agent && uv run pytest)                 180 tests
pnpm --dir deployment/aws test              11 tests
node node_modules/railway/dist/iac/bin.js   no diagnostics

The two tests in app/server.test.ts use the injectable-target idiom already there for signalTarget. I mutation-tested them: swapping on for once, changing the default target away from process, guarding on reason instanceof Error, and emptying the handler body each turn one red. The second test exists specifically because the default target is the only thing standing between this fix and doing nothing.

One thing I can't cover from vitest: the isMain block itself never executes under test, so nothing would catch that call being deleted. The end-to-end repro above was run both with and without the backstop to confirm the exit code flips from 1 to 0.

Disclosure: I used an AI assistant while working on this. The repro, the diagnosis, and the verification runs above are mine and I stand behind them.

The transport under thread.runAgent() rejects the awaited promise AND
leaks a second, unawaited copy of the same error. runAgentSafely catches
the first; nothing can reach the second, so Node terminates the process
on any dropped socket or body timeout between the runtime and the agent.

With ON_FAILURE and restartPolicyMaxRetries: 5 in .railway/railway.ts, a
flapping agent exhausts the restart budget and the service stays down.

Install a process-level unhandledRejection backstop in the isMain block,
next to the undici dispatcher that hardens the same call path. It logs
and keeps running; the awaited rejection still reaches every call site
unchanged, so this stays correct once the leak is fixed upstream in
@ag-ui/client.
@hdimer
hdimer marked this pull request as ready for review August 27, 2026 19:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

runtime: process exits on any failed agent fetch (UND_ERR_SOCKET / UND_ERR_BODY_TIMEOUT)

1 participant