fix(#4060): add bounded jittered backoff to StartableToolSet retry path

[aheritier]

🤖 Automated implementer agent — this comment was posted by the implementer bot from Docker Agentic Platform, not by a human developer

Refs #4060 (partial — MCP/LSP startup errors and RAG 5xx/408 pacing deferred to follow-up)

What

Adds bounded, cancellable, jittered exponential backoff to StartableToolSet's non-blocking start path. A rate-limited embedding-provider 429 that previously triggered a full concurrent re-index on every agent turn is now paced to at most one retry per 15 s – 5 min window.

Design

Gate location — tryStartLocked (TryStart/TryStartWithTimeout only). Blocking Start() bypasses the gate so mcpcatalog enable and skill sub-session startup always get a fresh attempt.

Classifier — startBackoffRetryable. Requires a *modelerrors.StatusError in the error chain via errors.As. Port numbers and chunk-progress counters in plain error strings cannot arm the gate.

Bounds — base 15 s, cap 5 min, additive jitter [d, 1.2d]. The additive floor guarantees the full nominal wait; jitter de-synchronises concurrent toolset sources.

Retry-After. When the 429 response carries a Retry-After header, that hint overrides the computed delay (capped at 5 min, with the same additive jitter applied to avoid re-synchronisation).

Embedding-provider gap fixed. openai/client.go and dmr/embed.go now call oaistream.WrapOpenAIError on their embedding errors. WrapOpenAIError wraps *openaisdk.Error in *modelerrors.StatusError carrying the HTTP status code, so a 429 from the embedding provider reaches the gate as *StatusError and arms it.

Known limitation. Code-mode composites (codemode.Wrap) return PartialStartError on a partial failure; the partial-start branch resets the gate and leaves the failed-subset retry unpaced. Tracked in #4067.

Changed files

File	What changed
pkg/modelerrors/modelerrors.go	New exported RetryableHTTPStatus(err) classifier
pkg/modelerrors/modelerrors_test.go	Tests for the classifier (incl. regex false-positive cases)
pkg/tools/startable_backoff.go	New: constants, computeStartBackoff, startBackoffRetryable, additiveJitter, retryAfterHint
pkg/tools/startable_backoff_test.go	Unit tests: gate fires on 429/408/5xx, not on plain text; Retry-After honoured/ignored/capped; jitter spread
pkg/tools/startable_backoff_regression_test.go	Consumer-shaped regression suite (RAG/MCP/LSP shapes, no-leak, latch, blocking Start ungated)
pkg/tools/startable.go	tryStartLocked, StartableOption, WithStartRetryJitter, WithStartRetryClock, variadic NewStartable
pkg/tools/export_test.go	Exported helpers for deterministic test control
pkg/tools/builtin/rag/rag_backoff_test.go	Real-toolset integration test via rag.New + fake clock
pkg/model/provider/openai/client.go	Wrap batch-embedding error in oaistream.WrapOpenAIError
pkg/model/provider/openai/embed_test.go	httptest 429 → proves *StatusError chain; 5xx equivalent
pkg/model/provider/dmr/embed.go	Same — wrap DMR embedding error in oaistream.WrapOpenAIError
pkg/model/provider/dmr/embed_test.go	httptest 429 → proves *StatusError chain
pkg/agent/agent.go	Update stale log messages
pkg/tools/mcp/mcp.go	Update stale log message
docs/tools/rag/index.md	New section: trigger table (429-only), retry schedule, Retry-After note, troubleshooting
docs/tools/mcp/index.md	Lifecycle note: local startup failures are not paced by the gate
docs/tools/lsp/index.md	Same note for LSP
docs/community/troubleshooting/index.md	Cross-reference to RAG backoff section

[aheritier]

🤖 Automated implementer agent — this comment was posted by the implementer bot from Docker Agentic Platform, not by a human developer

Re-reviewed at head e4ae8fd7 (identical content to the previously-reviewed c82351f6, force-pushed as a rebase onto a fresher main — same 18 files, same 1690/13 insertion/deletion counts, commit message unchanged; verified the rebase was mechanically conflict-free since the only intervening main commit, #4071, touched only CHANGELOG.md). CI is green: 16 checks pass, 3 skipping (post-merge publish jobs), 0 failing. mergeable=MERGEABLE; mergeStateStatus=BLOCKED solely on the missing required approving review.

This addresses my previous review (fe986e7, #pullrequestreview-5052930334). Going through each finding against the current head:

[blocking] Retry-After override branch — now covered

Four dedicated tests exercise every sub-case in pkg/tools/startable_backoff_test.go:

• TestStartableToolSet_RetryAfterHonoured (line 613): hint > computed delay → window extends to the hint exactly.
• TestStartableToolSet_RetryAfterBelowComputed (line 696): hint (5s) < computed delay (30s at attempt 2) → hint ignored, computed window holds.
• TestStartableToolSet_RetryAfterCapAtMax (line 730): a 24h hint is capped at startBackoffMax (5min).
• TestStartableToolSet_RetryAfterJittered (line 656): 60 independent gates armed with the same hint, probed at the midpoint of [hint, 1.2×hint] — confirms windows are spread across the jitter band, not all set to the exact hint.

go tool cover on ./pkg/tools/ (86.4%) now shows setStartBackoff (pkg/tools/startable.go:466) at 100% and no zero-coverage lines remain in startable.go or startable_backoff.go — the previously-unexecuted override branch (was lines 476-477, 0 executions) is exercised. All four tests pass:

--- PASS: TestStartableToolSet_RetryAfterHonoured
--- PASS: TestStartableToolSet_RetryAfterBelowComputed
--- PASS: TestStartableToolSet_RetryAfterCapAtMax
--- PASS: TestStartableToolSet_RetryAfterJittered

[should-fix] Retry-After jitter bypass — fixed

setStartBackoff (pkg/tools/startable.go:476-481) now applies the same jitter function to the Retry-After branch as the computed-delay path:

if hint := retryAfterHint(err); hint > delay {
    if s.startJitter != nil {
        delay = s.startJitter(min(hint, startBackoffMax))
    } else {
        delay = additiveJitter(min(hint, startBackoffMax))
    }
}

TestStartableToolSet_RetryAfterJittered statistically confirms the spread (not all 60 samples land on the same side of the midpoint).

[should-fix] Retry-After undocumented — fixed

docs/tools/rag/index.md:212 now has a "Retry-After override" bullet under "Retry policy and parameters", stating the hint overrides the computed delay, is capped at the 5-minute maximum, and gets the same additive jitter. This also incidentally closes the optional finding below about the 5min→6min ceiling — the same section's "Maximum delay" line now reads "up to ~6 minutes (5-minute cap plus up to 20% additive jitter)".

[should-fix] Closes #4060 → fixed

The commit message and PR description both now read Refs #4060 (partial — ...). Squash-merging this PR will no longer auto-close #4060.

[should-fix] PR body "Changed files" table incomplete — fixed

The table now lists all 18 files in the diff (verified against git show --stat on the current head), including the previously-omitted docs/tools/rag/index.md, docs/tools/mcp/index.md, docs/tools/lsp/index.md, docs/community/troubleshooting/index.md, pkg/tools/startable_backoff_regression_test.go, pkg/tools/builtin/rag/rag_backoff_test.go, and pkg/model/provider/dmr/embed_test.go. The Design section's embedding-wrap description is also now precise: it names oaistream.WrapOpenAIError (not the imprecise WrapHTTPError) and explains that it wraps *openaisdk.Error specifically.

[optional] "5 minutes" → "up to 6 minutes" — fixed (see above)

[optional] Only DMR embed site tested — fixed

pkg/model/provider/openai/embed_test.go now exists with TestCreateBatchEmbedding_429SurfacesAsStatusError and a 5xx equivalent, mirroring the DMR test. Both embedding-wrap sites are now pinned.

Re-validation on the current head

go build ./...                                                          # clean
go test ./pkg/tools/... ./pkg/modelerrors/... ./pkg/model/provider/...   # all green
golangci-lint run ./pkg/tools/ ./pkg/tools/mcp/ ./pkg/modelerrors/       # 0 issues

(-race isn't available in this environment — no gcc for cgo — so I could not re-run the race-detector pass from my previous review; the non-race suite is green and nothing in this delta touches concurrency-sensitive code paths beyond what was already reviewed.)

All previously-blocking and should-fix findings are resolved. Filing as a comment rather than a formal approval only because I authored the PR and GitHub blocks self-approval, same as every prior round.