Skip to content

fix: bound GitHub installation token refresh with a request timeout - #4742

Merged
gaspergrom merged 2 commits into
mainfrom
fix/github-installation-token-timeout
Sep 23, 2026
Merged

gaspergrom merged 2 commits into
mainfrom
fix/github-installation-token-timeout

Conversation

@gaspergrom

Copy link
Copy Markdown
Contributor

Summary

The actual root cause of the production docs-readiness sweep's complete freezes (found while diagnosing the same live incident that led to #4737 — that fix was real but not sufficient, since this second stall happened well past its 25-minute ceiling).

getGithubInstallationToken() (@crowd/common_services) refreshes via a plain axios.post() to GitHub's API with no timeout configured — axios defaults to no timeout unless one is set. Worse, the in-flight refresh is cached in a module-level singleton (refreshPromise) shared across every concurrent caller in the whole worker process. If that one HTTP call hangs, refreshPromise never settles, so it never clears via its own .finally() — every subsequent call to getGithubInstallationToken(), from any concurrent activity needing a GitHub token, joins the same permanently-stuck promise. Confirmed live: pulled the stalled workflow's event history again and found it completely frozen since the last completion, well past #4737's 25-minute scoring bound, meaning the freeze had moved to discovery, not scoring.

Two consumers (docs_readiness_worker, star_snapshot_worker), both call it for the same GitHub App installation token; both strictly benefit from a bounded timeout — nothing depends on this call being allowed to hang forever.

Fix: add timeout: 15_000 to the axios call. Once it fails cleanly instead of hanging, the existing .finally(() => refreshPromise = undefined) correctly clears the shared singleton, so the next caller gets a fresh retry instead of joining a dead promise forever.

Test plan

  • npx oxlint services/libs/common_services — clean
  • npx oxfmt --check services/libs/common_services — clean
  • pnpm tsc-check (repo-wide project-reference build, confirms both consumer workers still typecheck) — clean
  • npx vitest run across common_services, docs_readiness_worker, star_snapshot_worker — 235/235 passing (no existing test file for this specific service to extend)

Signed-off-by: Gašper Grom <gasper.grom@gmail.com>
@gaspergrom
gaspergrom requested review from epipav and themarolt and a balanced review from Copilot September 23, 2026 18:05
@cursor

cursor Bot commented Sep 23, 2026

Copy link
Copy Markdown

PR Summary

Medium Risk
Touches shared GitHub App installation token refresh used across workers; bounded timeout changes failure mode from indefinite hang to retryable errors after 15s.

Overview
Adds a 15 second timeout on the GitHub App installation access-token axios.post inside refreshGithubInstallationToken() in @crowd/common_services.

Without it, a stuck GitHub HTTP call never completes, so the module-level refreshPromise singleton never settles and its .finally() never runs—every later getGithubInstallationToken() caller in the process joins the same hung refresh (including docs-readiness discovery and other workers). After a timeout, the request fails, refreshPromise clears, and the next caller can retry instead of freezing indefinitely.

Reviewed by Cursor Bugbot for commit 11eecc8. Bugbot is set up for automated code reviews on this repo. Configure here.

@github-actions

Copy link
Copy Markdown
Contributor

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

  • feat: add user authentication (CM-123)
  • feat: add user authentication (IN-123)

Projects:

  • CM: Community Data Platform
  • IN: Insights

Please add a Jira issue key to your PR title.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Address the unsupported external API comment before approval.

Review effort: Balanced
Findings: None

What changed in this PR

Adds a 15-second timeout to prevent GitHub installation-token refreshes from hanging indefinitely.

Changes:

  • Bounds the Axios token request.
  • Preserves existing promise cleanup and retry behavior.
File Description
services/​libs/​common_services/​src/​services/​github.integration.service.ts Adds the timeout; the Axios explanatory comment needs a documentation link or removal (nit).

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@themarolt themarolt left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm - one question on the timeout semantics

},
// axios has no timeout by default; without one, a hung request never settles
// refreshPromise below, wedging every concurrent caller sharing it indefinitely.
timeout: 15_000,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AFAIK axios in node applies timeout via req.setTimeout, which is a socket idle timeout - it only gets armed once the socket connects and resets on every chunk. so DNS/connect stalls fall back to the OS connect timeout, and a slow-dripping response can run past 15s. should we add signal: AbortSignal.timeout(15_000) next to it for a hard wall-clock bound? that's what discovery/github.ts and discovery/http.ts already do (or am I missing something?)

Copilot AI review requested due to automatic review settings September 23, 2026 19:02
@gaspergrom
gaspergrom merged commit 5a87168 into main Sep 23, 2026
6 checks passed
@gaspergrom
gaspergrom deleted the fix/github-installation-token-timeout branch September 23, 2026 19:02
Copilot stopped reviewing on behalf of gaspergrom due to an error September 23, 2026 19:02

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Copilot was unable to run its full agentic suite in this review.

Copilot review overview

Review effort: Lite
Findings: 1 Medium severity

Open (1)

Comment on lines +56 to +58
// axios has no timeout by default; without one, a hung request never settles
// refreshPromise below, wedging every concurrent caller sharing it indefinitely.
timeout: 15_000,
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants