blob/fileblob: don't end a list page before a key that sorts earlier - #3791
Merged
vangent merged 2 commits intoSep 13, 2026
Merged
Conversation
ListPaged walks the directory tree and stops as soon as it has a full
page, unless the current entry is a "directory", since a file under a
directory can sort before the "directory" key we generate for the
delimiter. But a plain file can sort before it too: "dir-file" sorts
before "dir/" ("-" < "/"), and the walk visits "dir/" first.
When that happened we ended the page right there and made the file the
start of the next page, but the next page drops anything <= the page
token, so the file was never returned at all. Listing the same bucket
without paging still showed it.
Only stop walking when the current key really does sort after the last
key in the page.
…tory Walks a few layouts with a page size from 1 upwards and checks the keys match an unpaged list, which is how I noticed the missing key.
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #3791 +/- ##
==========================================
- Coverage 80.07% 79.98% -0.09%
==========================================
Files 104 104
Lines 12293 12303 +10
==========================================
- Hits 9844 9841 -3
- Misses 2448 2461 +13
Partials 1 1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
vangent
approved these changes
Sep 12, 2026
Collaborator
|
Nice catch, and thanks for adding the test. If you sign the CLA I can merge. |
rootkiller6788
force-pushed
the
fix-fileblob-listpaged-key-drop
branch
from
September 13, 2026 05:10
f2fb340 to
1f5ab6c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found this while poking at fileblob's listing with a delimiter set.
Write "dir/a", "dir/b", "dir-file" and then page through with pageSize=1: you get "dir/" and then an empty page. "dir-file" never shows up at all. A plain List on the same bucket returns "dir-file", "dir/", which is what I'd expect.
The cause is in ListPaged's early bail-out. It walks the tree with WalkDir, which visits the directory "dir" before the sibling file "dir-file", so the collapsed "dir/" key lands in the page first. When "dir-file" comes along the page is already full and it isn't a directory, so the old code stopped the walk there and set the page token to "dir/". The next page skips anything <= that token, and "dir-file" < "dir/" because '-' sorts before '/', so it gets skipped. Anything that sorts before a directory key it was walked after is affected - "foo-baz" next to "foo/", "n+a" next to "n/", and so on.
The fix just narrows when we're allowed to stop: only if the key we're looking at really does sort after the last key in the page. If it doesn't, we keep walking and let the existing trim at the end of the walk sort out the page. That's the same thing the code already does when the entry is a directory.
Test is in fileblob_test.go: it walks a few layouts with the page size from 1 up and checks the paged result matches an unpaged list. It fails on master for all three layouts, passes with the change.
One thing I left alone: an unpaged List can still return keys slightly out of order for exotic names (the one-slot swap a bit below only fixes an adjacent inversion). I didn't want to rewrite that sorting here, but happy to look at it separately if you want.