fix: bound GitHub installation token refresh with a request timeout - #4742
Conversation
Signed-off-by: Gašper Grom <gasper.grom@gmail.com>
PR SummaryMedium Risk Overview Without it, a stuck GitHub HTTP call never completes, so the module-level Reviewed by Cursor Bugbot for commit 11eecc8. Bugbot is set up for automated code reviews on this repo. Configure here. |
|
Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability. Example:
Projects:
Please add a Jira issue key to your PR title. |
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
Address the unsupported external API comment before approval.
Review effort: Balanced
Findings: None
What changed in this PR
Adds a 15-second timeout to prevent GitHub installation-token refreshes from hanging indefinitely.
Changes:
- Bounds the Axios token request.
- Preserves existing promise cleanup and retry behavior.
| File | Description |
|---|---|
services/libs/common_services/src/services/github.integration.service.ts |
Adds the timeout; the Axios explanatory comment needs a documentation link or removal (nit). |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
themarolt
left a comment
There was a problem hiding this comment.
lgtm - one question on the timeout semantics
| }, | ||
| // axios has no timeout by default; without one, a hung request never settles | ||
| // refreshPromise below, wedging every concurrent caller sharing it indefinitely. | ||
| timeout: 15_000, |
There was a problem hiding this comment.
AFAIK axios in node applies timeout via req.setTimeout, which is a socket idle timeout - it only gets armed once the socket connects and resets on every chunk. so DNS/connect stalls fall back to the OS connect timeout, and a slow-dripping response can run past 15s. should we add signal: AbortSignal.timeout(15_000) next to it for a hard wall-clock bound? that's what discovery/github.ts and discovery/http.ts already do (or am I missing something?)
| // axios has no timeout by default; without one, a hung request never settles | ||
| // refreshPromise below, wedging every concurrent caller sharing it indefinitely. | ||
| timeout: 15_000, |

Summary
The actual root cause of the production docs-readiness sweep's complete freezes (found while diagnosing the same live incident that led to #4737 — that fix was real but not sufficient, since this second stall happened well past its 25-minute ceiling).
getGithubInstallationToken()(@crowd/common_services) refreshes via a plainaxios.post()to GitHub's API with no timeout configured — axios defaults to no timeout unless one is set. Worse, the in-flight refresh is cached in a module-level singleton (refreshPromise) shared across every concurrent caller in the whole worker process. If that one HTTP call hangs,refreshPromisenever settles, so it never clears via its own.finally()— every subsequent call togetGithubInstallationToken(), from any concurrent activity needing a GitHub token, joins the same permanently-stuck promise. Confirmed live: pulled the stalled workflow's event history again and found it completely frozen since the last completion, well past #4737's 25-minute scoring bound, meaning the freeze had moved to discovery, not scoring.Two consumers (
docs_readiness_worker,star_snapshot_worker), both call it for the same GitHub App installation token; both strictly benefit from a bounded timeout — nothing depends on this call being allowed to hang forever.Fix: add
timeout: 15_000to the axios call. Once it fails cleanly instead of hanging, the existing.finally(() => refreshPromise = undefined)correctly clears the shared singleton, so the next caller gets a fresh retry instead of joining a dead promise forever.Test plan
npx oxlint services/libs/common_services— cleannpx oxfmt --check services/libs/common_services— cleanpnpm tsc-check(repo-wide project-reference build, confirms both consumer workers still typecheck) — cleannpx vitest runacrosscommon_services,docs_readiness_worker,star_snapshot_worker— 235/235 passing (no existing test file for this specific service to extend)