CLDSRV-961 Tests for lifecycle listings over PHD master keys

[nicolas2bert]

Problem

Lifecycle listings can get stuck forever on PHD master keys.

A PHD (placeholder) master is written when the current version of a key is deleted. A repair job later promotes the newest surviving version back to master. Until the repair runs, the master key holds a PHD value. If the repair never runs, or if no version survives, the PHD stays.

DelimiterVersions accepted a PHD master and skipped it without touching the resume marker. That is fine for a plain version listing, which has no scan budget. It is not fine for the two lifecycle listings, DelimiterNonCurrent and DelimiterOrphanDeleteMarker. Both count every scanned entry against maxScannedLifecycleListingEntries and truncate when the budget runs out.

So a run of PHD masters longer than the scan limit burns the whole budget without moving the marker. The listing returns IsTruncated: true with no NextKeyMarker. The next listing starts at the same place and returns the same thing. Lifecycle never gets past that point, and nothing beyond it is ever expired.

Example

Keyspace (v0), scan limit 3:

phd-1 .. phd-6 dangling PHD masters
zebra\0v1 current
zebra\0v2 noncurrent -> must be expired

Before:

page 1: scans phd-1, phd-2, phd-3 -> truncated, no NextKeyMarker
page 2: identical request -> identical result
...forever. zebra\0v2 is never expired.

After:

page 1: phd-1, phd-2, phd-3 -> truncated, NextKeyMarker=phd-2
page 2: phd-3, phd-4, phd-5 -> truncated, NextKeyMarker=phd-4
page 3: phd-5, phd-6, zebra\0v1, zebra\0v2 -> zebra\0v2 listed

What changed

DelimiterVersions — new handlePHDMaster hook

Both PHD branches (keyHandler_NotSkippingV0 and keyHandler_NotSkippingV1) now call handlePHDMaster(key, versionId, value) instead of returning FILTER_ACCEPT directly.

The default implementation returns FILTER_ACCEPT and changes no state. Plain version listings behave exactly as before.

DelimiterOrphanDeleteMarker — treat a PHD as a key transition

The override runs the same new-key branch as addVersion():

• It advances prevKeyName / keyName, so truncation has a resume position.
• It emits the held delete-marker candidate first.
• It sets value = null, so a PHD is never listed as an orphan delete marker.

The second point matters. A delete marker is held in memory until an entry of another key proves it is an orphan. A PHD master is such an entry. If the marker moved past the candidate without emitting it, the candidate would never be scanned again, and the orphan delete marker would never expire.

scan limit 3, keyspace: banana\0v1 (delete marker), phd-1, phd-2, ...
banana\0v1 -> held as candidate
phd-1 -> new key: banana is proven orphan -> emitted, then marker moves

DelimiterNonCurrent — keep the marker one PHD key behind

The override sets nextKeyMarker to the previous PHD key, never to the key being scanned, and clears nextVersionIdMarker. The marker only moves forward.

A marker on the key being scanned would give the next listing a key-marker with no version-id-marker. S3 reads that as "start after every version of this key". The versions of a PHD master that still has some would be skipped, and they would never be expired. One key behind costs one re-scanned entry per truncation, and skips nothing.

The override leaves prevKey and staleDate untouched. This is deliberate. The next version key scanned is the newest surviving version under the PHD — the one the repair promotes back to master. It must stay classified as current:

apple (PHD) -> marker left behind apple, prevKey untouched
apple\0v1 -> first version seen for apple -> current, protected
apple\0v2 -> noncurrent -> expirable, staleDate = v1's date

Setting prevKey = 'apple' would make apple\0v1 look noncurrent. Noncurrent version expiration would then delete the very version the repair needs to promote. That is data loss.

One fallback remains: if the scan limit lands on the first PHD of the listing, there is no previous PHD and no marker yet. The marker then points at that key, and that one key loses its versions for this pass. Progress matters more than one missed pass, and the alternative is the infinite loop described above.

Tests

• delimiterVersions.spec.js — guard: a PHD master is still not listed and still does not advance the marker in a plain version listing.
• delimiterNonCurrent.spec.js — marker set inside a PHD run; the newest surviving version under a PHD stays protected; crawl across a PHD run longer than the scan limit with marker feedback; scan limit landing exactly on a PHD.
• delimiterOrphanDeleteMarker.spec.js — NextMarker set inside a PHD run; a dangling PHD is never listed as an orphan; crawl across a PHD run; a held candidate is emitted when the PHD run begins.

All cases run against both v0 and v1 bucket formats.

[bert-e]

Hello nicolas2bert,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
/after_pull_request	Wait for the given pull request id to be merged before continuing with the current one.
/bypass_author_approval	Bypass the pull request author's approval	⭐
/bypass_build_status	Bypass the build and test status	⭐
/bypass_commit_size	Bypass the check on the size of the changeset TBA	⭐
/bypass_incompatible_branch	Bypass the check on the source branch prefix	⭐
/bypass_jira_check	Bypass the Jira issue check	⭐
/bypass_peer_approval	Bypass the pull request peers' approval	⭐
/bypass_leader_approval	Bypass the pull request leaders' approval	⭐
/approve	Instruct Bert-E that the author has approved the pull request.		✍️
/create_pull_requests	Allow the creation of integration pull requests.
/create_integration_branches	Allow the creation of integration branches.
/no_octopus	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
/unanimity	Change review acceptance criteria from one reviewer at least to all reviewers
/wait	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
/help	Print Bert-E's manual in the pull request.
/status	Print Bert-E's current status in the pull request.
/clear	Remove all comments from Bert-E from the history TBA
/retry	Re-start a fresh build TBA
/build	Re-start a fresh build TBA
/force_reset	Delete integration branches & pull requests, and restart merge process from the beginning.
/reset	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

[bert-e]

Incorrect fix version

The Fix Version/s in issue CLDSRV-961 contains:

• None

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

• 9.4.2

Please check the Fix Version/s of CLDSRV-961, or the target
branch of this pull request.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.46%. Comparing base (691163d) to head (329f1f0).
✅ All tests successful. No failed tests found.

Additional details and impacted files

see 1 file with indirect coverage changes

@@                 Coverage Diff                 @@
##           development/9.4    #6250      +/-   ##
===================================================
- Coverage            86.50%   86.46%   -0.05%     
===================================================
  Files                  212      212              
  Lines                14581    14581              
===================================================
- Hits                 12614    12607       -7     
- Misses                1967     1974       +7     

Flag	Coverage Δ
checksums-disabled-tests	35.31% <ø> (ø)
file-ft-tests	70.00% <ø> (ø)
file-ft-tests-null-compat	70.48% <ø> (ø)
kmip-ft-tests	28.07% <ø> (-0.03%)	⬇️
mongo-v0-ft-tests	71.12% <ø> (ø)
mongo-v1-ft-tests	71.10% <ø> (-0.05%)	⬇️
multiple-backend	36.08% <ø> (ø)
sur-tests	36.59% <ø> (ø)
sur-tests-inflights	39.42% <ø> (-0.03%)	⬇️
unit	74.21% <ø> (ø)
utapi-v2-tests	35.29% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[bert-e]

Incorrect fix version

The Fix Version/s in issue CLDSRV-961 contains:

• None

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

• 9.4.4

• 9.5.0

Please check the Fix Version/s of CLDSRV-961, or the target
branch of this pull request.