fix: keep the runtime alive when the agent fetch leaks a rejection - #67
Open
hdimer wants to merge 1 commit into
Open
fix: keep the runtime alive when the agent fetch leaks a rejection#67hdimer wants to merge 1 commit into
hdimer wants to merge 1 commit into
Conversation
The transport under thread.runAgent() rejects the awaited promise AND leaks a second, unawaited copy of the same error. runAgentSafely catches the first; nothing can reach the second, so Node terminates the process on any dropped socket or body timeout between the runtime and the agent. With ON_FAILURE and restartPolicyMaxRetries: 5 in .railway/railway.ts, a flapping agent exhausts the restart budget and the service stays down. Install a process-level unhandledRejection backstop in the isMain block, next to the undici dispatcher that hardens the same call path. It logs and keeps running; the awaited rejection still reaches every call site unchanged, so this stays correct once the leak is fixed upstream in @ag-ui/client.
hdimer
marked this pull request as ready for review
August 27, 2026 19:30
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #33 (the
UND_ERR_SOCKEThalf; see Scope below).The problem
app/channel.tsxandapp/commands/index.tsboth already wrapawait thread.runAgent(...)inrunAgentSafely, and that try/catch works fine. The process still dies.One failed fetch to the agent produces two rejections: the one the caller awaits, and a second unawaited copy of the identical error. Nothing is attached to the second one, so Node's default handler terminates the process. No application-level try/catch can reach it, because it is not on the promise chain the application awaits.
Reproduced against the real
SanitizingHttpAgenton currentmain(v0.4.1), pointed at a local server that writes a partial SSE response then destroys the socket, which is what an agent restart mid-turn looks like from the runtime's side:Exit code 1.
Worth spelling out why this is worse than "the container restarts":
.railway/railway.tssetsrestartPolicyType: "ON_FAILURE"withrestartPolicyMaxRetries: 5. An agent that flaps a handful of times doesn't just bounce the runtime, it exhausts the restart budget and the service stays down until a human notices.The fix
installUnhandledRejectionBackstop(), called fromserver.ts'sisMainblock. This is the issue author's own suggested fix #2. Same repro, with it in place:This is not a substitute for fixing the leak upstream. The duplicate rejection originates in
@ag-ui/client/@copilotkit/channelsinternals, which this repo can't patch, so the right long-term fix is inHttpAgent.runAgent(). This is a one-line stop-the-bleeding measure that stays correct afterwards: it does not swallow or alter the awaited rejection, so every call site still receives its error and still posts the user-facing message it posts today. Happy to close this if you'd rather fix it at the source.Three deliberate non-decisions:
isMaincalls it, so importingserver.tsdoesn't install a process-wide handler on a consumer. It sits next to the 30-minute undicibodyTimeoutdispatcher, which is production-only hardening for the same failure mode.process.exitCode. GivenON_FAILUREwith 5 retries, a runtime that survives a hiccup at 09:00 and then takes a clean SIGTERM at 18:00 would exit non-zero and spend a restart on a deploy that worked. The two existingexitCode = 1sites in this file are genuinely terminal paths; a backstop isn't.cause.code === "UND_ERR_SOCKET"filter. That shape is undocumented internals two packages deep, behind a caret dep. If it drifts by one field the crash returns with a fully green suite.I also considered
reportRecoverableErrorfromapp/channel-helpers.tsand left it: it's[channel]-tagged app-layer, it requires anoperation/recoverypair that a process-boundary handler can't honestly supply, andserver.tsuses bareconsole.errorthroughout.Scope
This is the
UND_ERR_SOCKEThalf of #33. TheUND_ERR_BODY_TIMEOUThalf was already largely mitigated by the 30-minutebodyTimeoutin 4b336d7; the backstop catches whatever still gets through.Verification
The two tests in
app/server.test.tsuse the injectable-target idiom already there forsignalTarget. I mutation-tested them: swappingonforonce, changing the default target away fromprocess, guarding onreason instanceof Error, and emptying the handler body each turn one red. The second test exists specifically because the default target is the only thing standing between this fix and doing nothing.One thing I can't cover from vitest: the
isMainblock itself never executes under test, so nothing would catch that call being deleted. The end-to-end repro above was run both with and without the backstop to confirm the exit code flips from 1 to 0.Disclosure: I used an AI assistant while working on this. The repro, the diagnosis, and the verification runs above are mine and I stand behind them.