Skip to content

v0.8.15: ci improvements, trigger machine resizing, connectors fixes - #7217

Merged
waleedlatif1 merged 6 commits into
mainfrom
staging
Aug 28, 2026
Merged

v0.8.15: ci improvements, trigger machine resizing, connectors fixes#7217
waleedlatif1 merged 6 commits into
mainfrom
staging

Conversation

@waleedlatif1

Copy link
Copy Markdown
Collaborator

icecrasher321 and others added 6 commits August 28, 2026 00:57
* fix(api): retire route contracts that no route serves

The staging integ alarm fired because the session knowledge-document
inline-create check got a 405. #7179 moved tool operations in process and
deleted the routes that only existed to serve them, but
`createKnowledgeDocumentsContract` kept declaring
`POST /api/knowledge/[id]/documents` — a path whose surviving GET/PATCH make
Next.js answer POST with 405 rather than an honest 404. Nothing in the repo
called it: the KB UI creates documents through the presigned upload flow, and
the capability itself is unaffected because `knowledge_create_document` reaches
the same use case in process.

Audited all 1125 contracts for the same drift. It was the only one whose path
resolves to a live route missing the declared method; 259 others declare paths
of routes that were deleted outright, which 404 honestly and are left alone.

- Drop the create-documents route contract for plain `params`/`body` schemas
  plus a named response schema, so nothing declares an endpoint we do not
  serve. The schemas stay in the contracts tree next to the siblings they share
  (`documentDataSchema` is used by the v2 contracts, and
  `createKnowledgeDocumentsBodySchema` already backed the in-process operation).
- Delete four contracts with no consumer at all — both TTS contracts, docusign,
  and mistral. Their handlers own better schemas: TTS dispatches by `toolId`
  with eight per-provider schemas instead of one passthrough superset, and
  mistral bounds `pages` by the OCR request policy. crowdstrike and windchill
  look similar but are load-bearing (schema and derived types are imported by
  live code), so they stay.
- Add `check:api-contract-routes`, picked up automatically by `run-audits`.
  `check:route-verbs` scans routes to contracts, so a contract whose route
  method was deleted is invisible to it — verified it passes clean against the
  exact regression this catches.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(api): read contracts by import, and retire two more stale declarations

Greptile flagged the audit's brace counter as blind to braces inside strings,
template literals, regexes and comments. It was, but the bigger problem was that
a text scan can only see contracts whose `method`/`path` are inline literals —
the 70-plus built through `definePostSelector(path, …)` and friends were never
checked at all. Comparing raw `defineRouteContract(` occurrences against parsed
ones showed the scanner silently skipping declarations.

Read the contracts by importing each contract module and inspecting its exported
objects instead, the way `check-route-verbs.ts` already resolves the contract
behind a route. Contract modules are pure Zod so importing them is safe; route
files stay a static scan because importing one drags in `@sim/db`, auth and
`next/server`. Barrels re-export the same object, so entries are keyed by
identity. Coverage goes from 1125 contracts to 1283.

That immediately surfaced two more instances of exactly what this PR retires.
`/api/tools/confluence/page` kept its `PUT` and `DELETE` contracts after #7179
reduced the route to the selector `POST`, so both declared verbs the live route
answers with 405. Neither is fetched — `lib/internal/confluence/execute-tool.ts`
is the only consumer — so they become plain schemas like the knowledge one, and
`executeOperation` now delegates to a schema form rather than growing a second
pattern beside it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…#7210)

The Blacksmith sticky disks backing our docker layer caches had no eviction
policy. setup-docker-builder skips pruning entirely unless max-cache-size-mb
is set, and BuildKit's own GC is time-based only (8 days unused), so on a repo
that builds this often nothing ever aged out: app.Dockerfile/linux-amd64 hit
351 GB within a day of being created, and realtime — an image under 300 MB —
sat at 249 GB. Layer caches alone were 920 GB across ten disks.

Cap them per image via a cache_mb matrix field, alongside the bs_runner field
that already encodes per-image sizing. The app image keeps 100 GB (several
generations over its working set of ~34 layers plus monorepo apt/bun cache
mounts); everything else takes the 25 GB default, which is still 4x the
tightest working set in the matrix (pii, whose spaCy models are ~2.2 GB).
The fallback lives in the composite action rather than an input default,
because an unset matrix key arrives as the empty string and would bypass a
default — silently restoring unbounded growth on any row that forgot the field.

Runner sizes follow measured CPU and memory percentiles over 30 days:

- CodeQL splits per language. javascript-typescript peaks at 19.5 GB so it
  stays on 8 vCPU; actions peaks at 1.3 GB and averages 22% CPU over a 39s
  median run, and drops to 4 vCPU.
- The pii and realtime image builds drop to 2 vCPU. Both already ran on 8 vCPU
  earlier in the window, so the 8->4 step is measured, not modelled: realtime
  went 52s -> 51s and pii 24s -> 27s.
- Five desktop release jobs drop to 2 vCPU. They peak under 0.4 GB and finish
  in 4-13s.

None of these sit on their group's critical path — each has 190-275s of slack
behind an app build that dominates it — so wall-clock is unchanged.

Those five desktop jobs also hardcoded a Blacksmith label with no CI_PROVIDER
fallback, against the invariant stated at the top of ci.yml. In GitHub
break-glass mode they would have sat in `queued` forever; they now fall back
like every other job.

Left alone deliberately: the 16 vCPU app builds (memory-bound, and 16 vCPU
measured 2.1x faster and 6% cheaper than 8 vCPU), Lint and Test (CPU-bound,
62% of the run above 80%), and the push-path Build App (build-amd64 has no
needs:, so it is what stops a migration applying for a build that cannot ship).
…7212)

Both knowledge tasks reserved machine presets well above their measured
ceilings. Sized each from production telemetry on both memory and CPU:

- knowledge-connector-sync: large-2x -> large-1x. Peak sampled RSS 2.6 GB
  and peak 1.4 vCPU, so 8 GB/4 vCPU keeps ~3x memory and ~2.8x CPU headroom
  against a preset that reserved 16 GB.
- knowledge-process-document: large-1x -> medium-2x. Peak sampled RSS 902 MB
  and peak 1.2 vCPU, with no document exceeding 2 GB, so 4 GB/2 vCPU keeps
  ~4x memory and ~1.7x CPU headroom.

CPU figures are core-normalized (OTel process.cpu.utilization divides by
cores available), so neither task loses headroom it was actually using and
neither can be throttled by the smaller preset.

No retry or concurrency semantics change.
…ain daily (#7213)

Feature PRs land on staging and are ~90% of PR scan volume (90 of the last 100
PRs target staging, 4 target main). Every one of them is scanned again — against
the exact tree being promoted — when the staging->main PR opens, so restricting
PR scans to main defers the signal to the promotion boundary rather than
dropping it. No ruleset or branch protection requires a CodeQL check, and the
alert view is fed by the push-to-main and scheduled analyses, not by PR runs.

Deliberately a branch cut rather than an activity-type cut. Dropping
`synchronize` would have cut a similar share of runs, but it scans a PR's first
commit and never its final state — backwards, since review fixups land in later
pushes.

The scheduled scan moves from weekly to daily. Pushes to main are rare, so with
PR scans limited to main the default-branch alert view leans on the cron more
than it used to, and a week is too long to leave it stale. It also reseeds the
overlay-base database that PR runs restore from: that cache key embeds the
CodeQL bundle version, so a bundle bump invalidates it, and an unused Actions
cache is evicted after 7 days.

Also records, in codeql-config.yml, why the obvious speed-up is a trap: adding
`queries:`/`packs:`/`query-filters:` trips OverlayDisabledReason.NonDefaultQueries
and permanently disables overlay analysis, trading a documented up-to-10x win on
the extraction phase (~53% of a run) for a few percent off the query phase.
Turborepo's local cache eviction is opt-in until 3.0, so without cacheMaxAge and
cacheMaxSize the filesystem cache grows forever. In CI each cache dir lives on a
Blacksmith sticky disk that is mounted many times a day, so it never idles long
enough for Blacksmith's own 7-day inactivity purge to fire, and one cache-missing
app build writes a ~400 MB artifact. The build cache disk reached 206 GB over 43
days — roughly 4.8 GB/day of sediment — at ~$0.51/GB-month.

Size is the real bound; age is hygiene. A cache hit only happens when a task's
input hash is unchanged, which recurs within hours, not weeks, so nothing written
days ago can ever be read again — the sibling PR-keyed disk does the same job in
30 GB. The cache still earns its keep: the app build hits ~17% of the time and a
hit saves ~7 minutes, so this is a ceiling, not a removal. That distinguishes it
from the Turbopack persistent cache, which was removed because it measured 3.2x
SLOWER; this one is measurably faster, just unbounded.

Set in turbo.json rather than per-step env vars so there is one source of truth
and turbo validates it. Verified against the installed 2.9.14: both keys are in
its schema, turbo parses them, and the task hash is byte-identical with and
without them, so enabling eviction does not invalidate the existing cache.
* fix(connectors): honor provider retry deadlines

* fix(connectors): validate retry response lifecycles
@waleedlatif1
waleedlatif1 requested a review from a team as a code owner August 28, 2026 09:41
@vercel

vercel Bot commented Aug 28, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
docs Skipped Skipped Aug 28, 2026 9:41am

Request Review

@greptile-apps

greptile-apps Bot commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This release improves CI cache bounds and runner sizing, resizes two knowledge background tasks, narrows and refreshes CodeQL scanning, removes contracts for retired routes, hardens Atlassian discovery validation, and propagates provider retry deadlines into connector scheduling.

  • Adds bounded Docker and Turborepo cache configuration and right-sizes CI runners.
  • Replaces retired HTTP route contracts with direct in-process schemas and adds a route-contract audit command.
  • Consolidates retryable HTTP error construction and carries provider retry timing into connector failure scheduling.
  • Tightens Atlassian accessible-resource payload validation.

Confidence Score: 4/5

The PR appears safe to merge, with a non-blocking cleanup needed in the new route-audit script's logging and error handling.

The changed runtime paths preserve internal schema validation and bounded connector backoff behavior; the only accepted issue is that the new developer audit command bypasses repository-standard diagnostics utilities.

Files Needing Attention: scripts/check-api-contract-routes.ts

Important Files Changed

Filename Overview
apps/sim/lib/knowledge/documents/utils.ts Centralizes bounded retryable HTTP errors, response cleanup, and retrieval of validated retry delays through error cause chains.
apps/sim/lib/knowledge/connectors/sync-engine.ts Propagates provider retry deadlines into connector scheduling without weakening the existing failure backoff or one-day ceiling.
apps/sim/lib/internal/tool-operations/parse-contract-input.ts Generalizes internal request parsing so retired HTTP operations can validate against bare schemas while preserving contract callers.
scripts/check-api-contract-routes.ts Adds a route/contract consistency audit, but its diagnostics bypass required shared logging and error-normalization utilities.
.github/workflows/ci.yml Bounds Docker caches and adjusts runner sizes across development, release, and desktop jobs.
.github/workflows/codeql.yml Moves PR scanning to the main promotion boundary, refreshes main daily, and sizes runners by analyzed language.
apps/sim/lib/atlassian/discovery.ts Validates accessible-resource response shape and distinguishes malformed responses from credentials with no site grants.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Provider[Connector provider] --> Fetch[Retrying fetch layer]
  Fetch --> Error[Retryable HTTP error]
  Error --> Delay[Validated retryAfterMs]
  Delay --> Sync[Connector sync failure handler]
  Sync --> Backoff[Max of failure ladder and provider delay]
  Backoff --> Schedule[nextSyncAt, capped at one day]
Loading

Reviews (1): Last reviewed commit: "fix(connectors): honor provider retry de..." | Re-trigger Greptile

Comment thread scripts/check-api-contract-routes.ts
@waleedlatif1
waleedlatif1 merged commit 336ec07 into main Aug 28, 2026
47 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants