Skip to content

blob/fileblob: don't end a list page before a key that sorts earlier - #3791

Merged
vangent merged 2 commits into
google:masterfrom
rootkiller6788:fix-fileblob-listpaged-key-drop
Sep 13, 2026
Merged

vangent merged 2 commits into
google:masterfrom
rootkiller6788:fix-fileblob-listpaged-key-drop

Conversation

@rootkiller6788

Copy link
Copy Markdown
Contributor

Found this while poking at fileblob's listing with a delimiter set.

Write "dir/a", "dir/b", "dir-file" and then page through with pageSize=1: you get "dir/" and then an empty page. "dir-file" never shows up at all. A plain List on the same bucket returns "dir-file", "dir/", which is what I'd expect.

The cause is in ListPaged's early bail-out. It walks the tree with WalkDir, which visits the directory "dir" before the sibling file "dir-file", so the collapsed "dir/" key lands in the page first. When "dir-file" comes along the page is already full and it isn't a directory, so the old code stopped the walk there and set the page token to "dir/". The next page skips anything <= that token, and "dir-file" < "dir/" because '-' sorts before '/', so it gets skipped. Anything that sorts before a directory key it was walked after is affected - "foo-baz" next to "foo/", "n+a" next to "n/", and so on.

The fix just narrows when we're allowed to stop: only if the key we're looking at really does sort after the last key in the page. If it doesn't, we keep walking and let the existing trim at the end of the walk sort out the page. That's the same thing the code already does when the entry is a directory.

Test is in fileblob_test.go: it walks a few layouts with the page size from 1 up and checks the paged result matches an unpaged list. It fails on master for all three layouts, passes with the change.

One thing I left alone: an unpaged List can still return keys slightly out of order for exotic names (the one-slot swap a bit below only fixes an adjacent inversion). I didn't want to rewrite that sorting here, but happy to look at it separately if you want.

ListPaged walks the directory tree and stops as soon as it has a full
page, unless the current entry is a "directory", since a file under a
directory can sort before the "directory" key we generate for the
delimiter. But a plain file can sort before it too: "dir-file" sorts
before "dir/" ("-" < "/"), and the walk visits "dir/" first.

When that happened we ended the page right there and made the file the
start of the next page, but the next page drops anything <= the page
token, so the file was never returned at all. Listing the same bucket
without paging still showed it.

Only stop walking when the current key really does sort after the last
key in the page.
…tory

Walks a few layouts with a page size from 1 upwards and checks the keys
match an unpaged list, which is how I noticed the missing key.
@google-cla

google-cla Bot commented Sep 12, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@vangent vangent changed the title fileblob: don't end a list page before a key that sorts earlier ListPaged walks the directory tree and stops as soon as it has a full page, unless the current entry is a "directory", since a file under a directory can sort before the "directory" key we generate for the delimiter. But a plain file can sort before it too: "dir-file" sorts before "dir/" ("-" < "/"), and the walk visits "dir/" first. When that happened we ended the page right there and made the file the start of the next page, but the next page drops anything <= the page token, so the file was never returned at all. Listing the same bucket without paging still showed it. Only stop walking when the current key really does sort after the last key in the page. fileblob: don't end a list page before a key that sorts earlier Sep 12, 2026
@vangent vangent changed the title fileblob: don't end a list page before a key that sorts earlier blob/fileblob: don't end a list page before a key that sorts earlier Sep 12, 2026
@codecov

codecov Bot commented Sep 12, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.98%. Comparing base (d381d80) to head (1f5ab6c).
⚠️ Report is 4 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #3791      +/-   ##
==========================================
- Coverage   80.07%   79.98%   -0.09%     
==========================================
  Files         104      104              
  Lines       12293    12303      +10     
==========================================
- Hits         9844     9841       -3     
- Misses       2448     2461      +13     
  Partials        1        1              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@vangent

vangent commented Sep 12, 2026

Copy link
Copy Markdown
Collaborator

Nice catch, and thanks for adding the test. If you sign the CLA I can merge.

@rootkiller6788
rootkiller6788 force-pushed the fix-fileblob-listpaged-key-drop branch from f2fb340 to 1f5ab6c Compare September 13, 2026 05:10
@vangent
vangent merged commit 25fdfb8 into google:master Sep 13, 2026
15 of 16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants