Skip to content

fix(ci): actually prune the BuildKit layer cache - #7249

Merged
waleedlatif1 merged 1 commit into
stagingfrom
ci/prune-buildkit-layer-cache-explicitly
Aug 29, 2026
Merged

fix(ci): actually prune the BuildKit layer cache#7249
waleedlatif1 merged 1 commit into
stagingfrom
ci/prune-buildkit-layer-cache-explicitly

Conversation

@waleedlatif1

@waleedlatif1 waleedlatif1 commented Aug 29, 2026

Copy link
Copy Markdown
Collaborator

Summary

The max-cache-size-mb input I added in 8fa7f0cc4e never did anything. Every Docker build since has logged:

##[warning]Unexpected input(s) 'max-cache-size-mb', valid inputs are ['cache-key',
'buildx-version', 'platforms', 'nofallback', 'github-token', 'buildkit-version',
'skip-integrity-check', 'driver-opts', 'max-parallelism']

setup-docker-builder v1 accepted that input and pruned in its own post step. The v2 rewrite dropped it, and we pin v2.1.0. GitHub only warns on an unknown composite input, so it failed silently — the app disk sat above 200 GB for a full day while the config looked correct.

Scanned every tag rather than infer:

Tag has max-cache-size-mb
v2.1.0 (pinned), v2.0.1, v2.0.0 no
v1.12.0, v1.11.0, v1.10.0, v1.9.0, v1.8.0 yes

The fix

Rather than downgrade a builder rewrite to reach a config knob, run the prune ourselves — v1's command verbatim (from its dist/index.js pruneBuildkitCache), against the fixed address v2 itself uses for buildctl du and debug workers:

sudo buildctl --addr tcp://127.0.0.1:1234 prune --all --keep-storage <MB>

Both flags verified against BuildKit master, not assumed

  • buildctl's --all is not docker buildx prune --all. It means "Include internal/frontend references". cache/manager.go shows the only records skipped without it:
    if !opt.all {
        if recordType == UsageRecordTypeInternal || recordType == UsageRecordTypeFrontend || shared {
            continue
        }
    }
    It does not wipe the cache. (I initially justified --all as being required to reclaim RUN --mount=type=cache dirs — that was wrong. Those are typed exec.cachemount and are reclaimed either way. --all is kept because it's what v1 used and it prunes strictly more.)
  • --keep-storage is not deprecated. In cmd/buildctl/prune.go it maps straight onto the modern MaxUsedSpace field — it is buildctl's spelling of --max-used-space, and the only size flag buildctl exposes.

Why a foreground command

Turborepo's equivalent eviction runs on a detached thread that is never joined, and on a 206 GB cache it makes no measurable progress inside a 150s job (measured: the disk moved 206.2 → 206.7 GB across 17h of builds). A blocking buildctl prune either completes or reports why not.

Warns rather than fails — an oversized cache isn't worth failing a deploy over — but prints buildctl du either side, because a silent no-op is precisely the failure mode that hid this for a day.

Verification

  • Extracted v1's and v2's bundled dist/index.js; confirmed v1's exact command and that v2 hardcodes tcp://127.0.0.1:1234 and already shells out to buildctl, so binary and address are both present at that point in the job.
  • Flag semantics read from BuildKit master source (cmd/buildctl/prune.go, client/prune.go, cache/manager.go, client/diskusage.go).
  • actionlint output identical to the staging baseline; shellcheck clean on the new run block.
  • bun run lint and all 39 check:audits pass.

Effect is not yet measurable — it applies from the first build after merge. I'll report the real du deltas from build logs rather than assume.

Type of Change

  • Bug fix

Testing

As above. The real verification is the before prune -> / after prune -> lines this adds to every Docker build log.

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

@vercel

vercel Bot commented Aug 29, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
docs Ready Ready Preview Aug 29, 2026 2:32am

Request Review

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 1 file

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Re-trigger cubic

@greptile-apps

greptile-apps Bot commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR replaces an unsupported setup-docker-builder cache-size input with an explicit, foreground BuildKit cache prune for Blacksmith builds.

  • Removes the ineffective max-cache-size-mb argument from builder setup.
  • Runs buildctl prune --all with configured retention after the image build.
  • Logs cache usage before and after pruning while treating cleanup failures as warnings.

Confidence Score: 5/5

The PR appears safe to merge, with no concrete blocking or independently actionable non-blocking issues identified.

Current callers provide valid retention values, the fallback handles omitted matrix fields, and diagnostic or prune failures remain non-fatal as intended.

Important Files Changed

Filename Overview
.github/actions/docker-build/action.yml Replaces the ignored builder input with a guarded BuildKit prune step and preserves the intended matrix-aware retention fallback.

Reviews (1): Last reviewed commit: "fix(ci): actually prune the BuildKit lay..." | Re-trigger Greptile

The max-cache-size-mb input added in 8fa7f0c never did anything.
setup-docker-builder v1 accepted it and pruned in its own post step, but the v2
rewrite dropped the input, and the repo pins v2.1.0. GitHub only WARNS on an
unknown composite input, so every build since has logged

  Unexpected input(s) 'max-cache-size-mb', valid inputs are ['cache-key', ...]

and pruned nothing. Scanned every tag to confirm: the input exists in v1.8.0
through v1.12.0 and in none of v2.0.0, v2.0.1, v2.1.0.

Rather than downgrade a builder rewrite to reach a config knob, run the prune
ourselves — v1's command verbatim, against the fixed address v2 itself uses for
`buildctl du` and `debug workers`:

  sudo buildctl --addr tcp://127.0.0.1:1234 prune --all --keep-storage <MB>

Both flags read from BuildKit master rather than assumed. buildctl's --all is not
`docker buildx prune --all`: it means "include internal/frontend references", and
cache/manager.go shows the only records skipped without it are those typed
internal or frontend plus refs shared with an external source. It does not wipe
the cache. --keep-storage maps onto the modern MaxUsedSpace field, so it is
buildctl's spelling of --max-used-space, not a deprecated alias, and it is the
only size flag buildctl exposes.

Two guards, both for failure modes that are silent and expensive:

Reject a non-positive-integer budget. buildctl parses --keep-storage as a float,
and BuildKit treats keepBytes==0 as "no cap" (`gcMode := opt.keepBytes != 0`),
pruning everything eligible instead of trimming. A typo like '40GB' — valid in
turbo.json, but this flag is a bare MB number — would empty the cache and make
every later build cold, costing far more than the storage saved.

Wait for `du` to settle after pruning. buildctl prune returns before buildkitd
has finished deleting (moby/buildkit#1198), and the builder's post step SIGTERMs
buildkitd then SIGKILLs it after 30s (shutdownBuildkitd: `const a=3e4`); on
SIGKILL it sets sigkillUsed and skips the sticky disk commit outright, discarding
the run's cache and risking a corrupt bbolt metadata DB. The wait is bounded, and
a steady-state trim settles almost immediately — it is the first catch-up prune
against a 200 GB backlog that would otherwise run into that window.

Warn rather than fail throughout, since an oversized cache is not worth failing a
deploy over — but print `buildctl du` either side, because a silent no-op is
exactly the failure mode that hid this regression for a day.
@waleedlatif1
waleedlatif1 force-pushed the ci/prune-buildkit-layer-cache-explicitly branch from de7134d to 753e3b0 Compare August 29, 2026 02:30
@waleedlatif1
waleedlatif1 merged commit 4f65103 into staging Aug 29, 2026
26 checks passed
@waleedlatif1
waleedlatif1 deleted the ci/prune-buildkit-layer-cache-explicitly branch August 29, 2026 07:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant