feat(fetch-content): merge concurrently extracted fetches in a fixed order - #1047
Merged
Merged
Conversation
…order Firefox is moving its test package downloads from mozharness to fetches (bug 2073846). A test task will then fetch several archives into one destination with overlapping contents, such as the test harness binaries that both `common` and `trainhop` ship under `bin`. fetch-content extracts every fetch in parallel, directly into its destination, so two threads write the same tree at once and the outcome depends on thread timing. A path present in two archives ends up as whichever copy was extracted last, and on Windows the race can fail outright when `tarfile` finds a parent directory another thread just created. Each fetch now downloads and extracts into its own staging directory, and the staging directories are moved into the destination one at a time in list order. Downloads and extraction remain parallel and the merge consists of renames. When two fetches provide the same path, the later one wins, directory modes included. The Windows failure alone could be patched around in the `tarfile` subclass, but the ordering would remain undefined. The overlapping files are identical today, so that would go unnoticed until they differ and could produce an intermittent failure instead of a reproducible one.
ahochheiden
force-pushed
the
fetch-content-staged-extract
branch
from
September 21, 2026 15:54
08d7bb5 to
b41c515
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Firefox is moving its test package downloads from
mozharnessto fetches (bug 2073846). A test task will then fetch several archives into one destination with overlapping contents, such as the test harness binaries that bothcommonandtrainhopship underbin. fetch-content extracts every fetch in parallel, directly into its destination, so two threads write the same tree at once and the outcome depends on thread timing. A path present in two archives ends up as whichever copy was extracted last, and on Windows the race can fail outright whentarfilefinds a parent directory another thread just created.Each fetch now downloads and extracts into its own staging directory, and the staging directories are moved into the destination one at a time in list order. Downloads and extraction remain parallel and the merge consists of renames. When two fetches provide the same path, the later one wins, directory modes included.
The Windows failure alone could be patched around in the
tarfilesubclass, but the ordering would remain undefined. The overlapping files are identical today, so that would go unnoticed until they differ and could produce an intermittent failure instead of a reproducible one.