feat(fetch): add fetch-mode to work around HuggingFace clone failures (bug 2047876)

[JohanLorenzo]

I propose to add fetch-mode option to git fetch tasks to workaround the case of HuggingFace. Their backend compresses the request once the uncompressed body exceeds 1024 bytes. Which request? A clone cannot name the references it needs (like branches and tags), which git calls it a want so all of them are requested.

As of this writing, Mozilla/smart-tab-topic has 34 refs (5 branches + 29 tags) which means 1880B.

Why not --depth?

The workaround proposed in the bug comments is git clone -n --depth 100000 which is more a band-aid. Fetching the single pinned revision is 174 bytes regardless of how many branches, tags or PR refs the repo accumulates over time.

approach	wants	uncompressed	result	verdict
fetch origin <revision>	1	174 B	PASS	this PR — constant size, full ancestry kept
--depth 100000	1	200 B	PASS	implies --single-branch, silently narrowing to ancestors of the default-branch tip. The depth number was never the operative part.
--depth 1	1	200 B	PASS	pinned revision absent — only fetches the tip
--single-branch	1	200 B	PASS	same narrowing; 4a34f12 (tip of add-weights) becomes unreachable
--no-tags	6	424 B	PASS	works, and keeps named branches, but the want count still tracks ref count — ~11 more branches and the bug returns. A reasonable future fetch-mode if a repo needs the clone semantics.
--filter=blob:none / tree:0	35	1880 B	FAIL	still a full-ref request
protocol.version=0 / 1	—	—	FAIL	as reported in the bug
disable request gzip via config	—	—	n/a	no such git option exists

Example usage

mozilla-smart-tab-topic:
    description: Smart Tab Topic Model
    fetch:
        repo: https://huggingface.co/Mozilla/smart-tab-topic
        revision: b0fd1b23dcb06bb68b5d60d18c7b1524c65a94af
        # [...]
        fetch-mode: init_and_fetch

[jcristau]

Their backend compresses the request once the uncompressed body exceeds 1024 bytes

What does this have to do with compression?

[JohanLorenzo]

Past 1024B, the git client requests gzip. E.g.:

GIT_CURL_VERBOSE=1 git clone -n https://huggingface.co/Mozilla/smart-tab-topic 2>&1  | grep -E "Send header: Content-Encoding|Send header: Content-Length|fatal"
17:19:48.893354 http.c:848              => Send header: Content-Length: 182
17:19:49.023396 http.c:848              => Send header: Content-Encoding: gzip
17:19:49.023398 http.c:848              => Send header: Content-Length: 879
fatal: expected 'packfile'

GIT_CURL_VERBOSE=1 git clone --single-branch -n https://huggingface.co/Mozilla/smart-tab-topic 2>&1 | grep -E "Send header: Content-Encoding|Send header: Content-Length|fatal"
17:43:17.201862 http.c:848              => Send header: Content-Length: 182
17:43:17.343137 http.c:848              => Send header: Content-Length: 239

[JohanLorenzo]

To be more precise, git clone -n -vv https://huggingface.co/Mozilla/smart-tab-topic returns:

Cloning into 'smart-tab-topic'...
POST git-upload-pack (182 bytes)
want 91a6ebc0318de964b0e318c89ba01514746aadb8 (HEAD)
want 4a34f12d3775c21a2fa02e0324e1fdb87139a83c (refs/heads/add-weights)
[...]
POST git-upload-pack (gzip 1859 to 866 bytes)
fatal: expected 'packfile'

Then, to identify the threshold:

cd "$(mktemp -d)"
R=https://huggingface.co/Mozilla/smart-tab-topic
git ls-remote "$R" | awk '{print $1}' | sort -u > oids
for n in 17 18 19 20; do
  git init -q "p$n" && git -C "p$n" remote add origin "$R"
  printf '%2d wants: ' "$n"
  GIT_LFS_SKIP_SMUDGE=1 git -C "p$n" fetch -vv --no-tags origin \
    $(head -"$n" oids | tr '\n' ' ') 2>&1 | grep -E 'POST|fatal' | tail -2
done

17 wants: POST git-upload-pack (974 bytes)
18 wants: POST git-upload-pack (1024 bytes)
19 wants: POST git-upload-pack (gzip 1074 to 598 bytes)
fatal: expected 'packfile'
20 wants: POST git-upload-pack (gzip 1124 to 623 bytes)
fatal: expected 'packfile'

So, the HuggingFace git backend doesn't support gzip correctly on git-upload-pack which is used during a clone.