fix: retry rejected capture batches and self-heal newly-gapped repos (CM-1441)

[themarolt]

Summary

• captureStarSnapshots: retries batches still rejected after activity/rate-limit handling, once, after a 30s cooldown - most transient GitHub/DB blips no longer leave a permanent snapshot gap for that day.
• selfHealStarBackfill: now also sweeps completed-but-newly-gapped repos (via findReposNeedingGapHeal) once per full run, alongside the existing never-completed page scan, so a dropped batch self-heals without manual intervention.
• star-snapshot-backfill bin script: new --gapped-only flag targets exactly the repos with a current snapshot gap instead of a full sweep, for fixing a specific gap on demand.
• starSnapshotHealthReporting job: the reported gap count now matches the day-list it displays (drops repos whose gap closed between the two underlying queries).

Root cause: a capture batch on 2026-09-22 was dropped after exhausting retries, leaving ~1884 repos missing one day - not a date-boundary bug in the health report.

Test plan

• pnpm tsc-check passes
• Next 08:00 UTC capture run: verify no repos come up short after a simulated/real batch failure
• Next 09:00 UTC self-heal run: verify newly-gapped completed repos get backfilled
• Manual: --gapped-only run against a known-gapped repo set

[cursor]

PR Summary

Medium Risk
Changes scheduled Temporal capture and self-heal workflows (including deploy-time patched() behavior) and expands automated backfill dispatch for completed repos; mistakes could affect daily snapshot completeness or GitHub API load.

Overview
Addresses dropped daily star-snapshot capture batches and repos that become gapped after backfill is marked complete (CM-1441).

captureStarSnapshots refactors batch handling so rejected batches are collected instead of immediately counted as failures, then—behind patched('CM-1441-retry-rejected-batches')—runs one extra pass after a 30s sleep before repos are marked failed.

selfHealStarBackfill adds a second paginated scan of completed, non-dead-lettered repos (findReposNeedingGapHeal / findCompletedReposEligibleForGapHeal + gap check) alongside the existing incomplete-backfill scan, with separate cursors and gap-heal-namespaced child workflow IDs (patched for safe mid-deploy replay).

Data layer centralizes batched gap detection in findAllRepoIdsWithStarSnapshotGaps; the backfill CLI gains --gapped-only (separate checkpoint/completed files) to target only repos with a current gap. The health reporting cron uses the shared gap helper and reports gap counts only for repos with missingDays > 0, avoiding over-count when a gap closes between queries.

^{Reviewed by Cursor Bugbot for commit 7dfa1f9. Bugbot is set up for automated code reviews on this repo. Configure here.}

[Copilot]

Copilot review overview

🟡 Changes recommended

Critical scalability and checkpoint issues, plus a retry logic defect, must be addressed before approval.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 3 · 1

Open (4)

• Bound unbounded history scan to avoid activity timeout · New
• Bound gap-heal sweep and continue across workflow runs · New
• Index and paginate the gap-heal candidate scan · New
• Retry when all batches fail, including single-batch pages · New

What changed in this PR

Improves star-snapshot recovery through capture retries, automated gap healing, targeted backfills, and accurate health reporting.

Changes:

• Retries rejected capture batches after a cooldown.
• Adds scheduled and --gapped-only gap healing.
• Aligns reported gap counts with displayed results.

File	Description
services/​libs/​data-access-layer/​src/​repositoryStarSnapshots/​index.ts	Adds completed-repository lookup and gap detection.
services/​apps/​star_snapshot_worker/​src/​workflows/​selfHealStarBackfill.ts	Dispatches gap-healing child workflows.
services/​apps/​star_snapshot_worker/​src/​workflows/​captureStarSnapshots.ts	Adds rejected-batch retry handling.
services/​apps/​star_snapshot_worker/​src/​bin/​star-snapshot-backfill.ts	Adds targeted gapped-only backfills.
services/​apps/​star_snapshot_worker/​src/​activities/​index.ts	Implements gap-healing discovery.
services/​apps/​star_snapshot_worker/​src/​activities.ts	Exports the new activity.
services/​apps/​cron_service/​src/​jobs/​starSnapshotHealthReporting.job.ts	Reconciles gap counts with displayed results.

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Copilot review overview

🔵 Needs a closer look

Moderate retry correctness, pagination, validation, indexing, timeout, and payload-limit concerns remain unresolved.

Review effort: Balanced
Findings: 3 · 1

Open (4)

• Index and paginate the gap-heal candidate scan
• Bound gap-heal sweep and continue across workflow runs
• Bound unbounded history scan to avoid activity timeout
• Retry when all batches fail, including single-batch pages

[Copilot]

Copilot review overview

🔵 Needs a closer look

Unbounded gap scans, quadratic backfill persistence, and a retry guard bug must be addressed before approval.

Review effort: Balanced
Findings: 3 · 1

Open (4)

• Index and paginate the gap-heal candidate scan
• Bound gap-heal sweep and continue across workflow runs
• Bound unbounded history scan to avoid activity timeout
• Retry when all batches fail, including single-batch pages

[Copilot]

Copilot review overview

🟡 Changes recommended

Temporal compatibility, overlapping scans, and gapped-only scalability and checkpoint consistency must be addressed.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 2

Open (2)

• Version durable sleep to preserve Temporal replay behavior · New
• Version workflow command changes for existing executions · New

Resolved since last review (4)

• Index and paginate the gap-heal candidate scan
• Bound gap-heal sweep and continue across workflow runs
• Bound unbounded history scan to avoid activity timeout
• Retry when all batches fail, including single-batch pages

[Copilot]

Copilot review overview

🟡 Changes recommended

The resume correctness, unbounded scanning, and pre-patch failure-accounting issues must be addressed.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1 · 2

Open (3)

• Count rejected batches for pre-patch workflows · New
• Condense retry comment to the Temporal invariant · New
• Condense comment to the Temporal replay invariant · New

Resolved since last review (2)

• Version workflow command changes for existing executions
• Version durable sleep to preserve Temporal replay behavior

Previously missed (3)

In code that hasn't changed since last review

Reduce comment to the repository's two-line limit

services/​apps/​cron_service/​src/​jobs/​starSnapshotHealthReporting.job.ts:68

The race explanation is useful, but this comment exceeds the repository's two-line maximum (CLAUDE.md:74-86). It can be reduced without losing the invariant.

Shorten function comment to the non-obvious constraint

services/​apps/​star_snapshot_worker/​src/​activities/​index.ts:408

This three-line function narration exceeds the repository's two-line limit (CLAUDE.md:74-86). Keep only the non-obvious performance constraint.

Shorten comment to the pagination invariant

services/​apps/​star_snapshot_worker/​src/​workflows/​selfHealStarBackfill.ts:84

This three-line narration exceeds the repository's two-line maximum for allowed comments (CLAUDE.md:74-86). Keep only the non-obvious pagination invariant.

[Copilot]

Copilot review overview

🔵 Needs a closer look

Retry compatibility, child-workflow reuse, and gapped-only scalability issues remain unresolved.

Review effort: Balanced
Findings: 1

Open (1)

• Count rejected batches for pre-patch workflows

Resolved since last review (2)

• Condense comment to the Temporal replay invariant
• Condense retry comment to the Temporal invariant

Previously missed (3)

In code that hasn't changed since last review

Condense race explanation comment to protected invariant

services/​apps/​cron_service/​src/​jobs/​starSnapshotHealthReporting.job.ts:68

The race explanation is useful, but this comment exceeds the two-line limit. Condense it to the invariant the filter protects.

Shorten header to the timeout pagination constraint

services/​apps/​star_snapshot_worker/​src/​activities/​index.ts:408

This three-line function header mostly restates the function name and implementation. Keep only the timeout-related pagination constraint in a concise form.

Shorten comment to cursor and page-bound rationale

services/​apps/​star_snapshot_worker/​src/​workflows/​selfHealStarBackfill.ts:84

This three-line comment narrates the newly added control flow. Keep only the non-obvious reason for separate cursors and the page bound, within two lines.

[Copilot]

Copilot review overview

🟡 Changes recommended

The workflow can fail during replay when the new patch is disabled, and the identified comment violations remain unresolved.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1

Open (1)

• Preserve legacy continuation when the patch is disabled · New

Resolved since last review (3)

• Count rejected batches for pre-patch workflows
• Condense comment to the Temporal replay invariant
• Condense retry comment to the Temporal invariant

Previously missed (2)

In code that hasn't changed since last review

Condense race explanation comment to two lines

services/​apps/​cron_service/​src/​jobs/​starSnapshotHealthReporting.job.ts:68

The race explanation is useful, but this comment exceeds the repository's two-line maximum (CLAUDE.md:76-86). Condense it while retaining the concurrency invariant.

Condense comment to separation and pagination invariants

services/​apps/​star_snapshot_worker/​src/​activities/​index.ts:408

This new comment exceeds the repository's two-line limit (CLAUDE.md:76-86) and mostly narrates the function. Keep only the non-obvious separation and bounded-pagination invariant.

[Copilot]

Copilot review overview

🔵 Needs a closer look

The critical resume correctness issue and moderate unbounded-scan issue must be addressed.

Review effort: Balanced
Findings: None

Resolved since last review (1)

• Preserve deployed gap-heal version marker for replay

[Copilot]

Copilot review overview

🔵 Needs a closer look

The --gapped-only path must use a bounded, paginated candidate scan to avoid estate-wide memory and database load.

Review effort: Balanced
Findings: None

[Copilot]

Copilot review overview

🟡 Changes recommended

The critical child-workflow ID collision can prevent detected snapshot gaps from being healed.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1

Open (1)

• Use unique child IDs for gap-heal batches · New

[Copilot]

Copilot review overview

🟡 Changes recommended

Critical gap-detection coverage and moderate resume-safety issues remain unresolved.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1 · 2

Open (3)

• Gap detection misses completed repositories without snapshot rows · New
• Collision explanation comment exceeds the two-line limit · New
• Replay-invariant comment exceeds the two-line limit · New

Resolved since last review (1)

• Use unique child IDs for gap-heal batches

[gaspergrom]

Traced the retry-once/cooldown logic in captureStarSnapshots and the dual-cursor continueAsNew handling in selfHealStarBackfill against the patch-ID replay scenarios Copilot and Cursor flagged earlier in this thread; both hold up. The health report fix reconciles the two underlying queries and closes the race that caused the mismatch. Good to merge.

[Copilot]

Copilot review overview

🔵 Needs a closer look

The batch-ID collision risk and full-population --gapped-only scan must be addressed before approval.

Review effort: Balanced
Findings: None

Resolved since last review (3)

• Gap detection misses completed repositories without snapshot rows
• Replay-invariant comment exceeds the two-line limit
• Collision explanation comment exceeds the two-line limit

[Copilot]

Copilot review overview

🟡 Changes recommended

The --gapped-only implementation must avoid its unbounded double scan before approval.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: None