Disk-copy publication is not atomic with its ownership check

[anoop-narang]

Follow-up to #49, which narrowed this window but deliberately did not close it. Recording it so it is tracked rather than rediscovered.

What is still open

write_batch_to_disk validates ownership and publishes the disk copy as two separate steps:

// src/core/src/cache/core.rs, ~1157
if is_rewrite
    && self.index.get_with_identity(&entry_id)
        .is_some_and(|(current, _)| current != identity)
{
    // stale: remove our own object, release the reservation, return
}
let previous = self.disk_copies.lock().unwrap().insert(entry_id, DiskCopy { .. });
if let Some(previous) = previous {
    if previous.identity != identity {
        self.store.remove(&entry_id_to_key(&entry_id, previous.identity)).await?;
    }
    self.budget.release_disk(previous.bytes);
}

Two interleavings remain:

1. Takeover between validation and publication. The index read and the disk_copies swap are not one operation. If identity B takes the key over in between, stale identity A still replaces B's record and deletes B's object. B's next disk read then finds the object gone. There is no .await between the two, so a single task cannot be preempted there, but the cache runs on a multi-threaded runtime, so another thread can.

2. Entry removed during the write. is_some_and treats a missing index entry as "not stale", so a rewrite whose entry was evicted mid-write still publishes a DiskCopy. The record and object are then reachable only through disk_copies, with nothing in the index naming them. They are reclaimed when the key is next taken over or by reset, so this is untidy accounting rather than an unbounded leak — but nothing reclaims it proactively.

Why it is worth doing

The precondition is not exotic. Since ids are leased and recycled, a recycled id landing on a still-resident entry is a routine event by design — identity_mismatches counts it. So the window is narrow in time but sampled often.

The consequence is a panic on a subsequent disk read, not incorrect data: the identity check introduced in #48 means a mismatched entry reads as a miss, so a collision can no longer serve the wrong bytes.

Suggested direction

Make validation and publication one synchronized operation, so that once B owns the key, A cannot replace B's record or delete B's object. That needs lock ordering against the ART index to be worked out, which is why #49 did not attempt it.

For the second interleaving, a rewrite that finds no current owner should clean up its object and reservation instead of publishing an orphan.

Testing

#49 could not test its change: reproducing the interleaving needs the write paused mid-await, which the suite cannot do. Two attempts were discarded — one never reached the new code, the other asserted an unreachable state.

Shuttle is already wired up here and already covers the file-id pool, so model-checking this check-then-act pair — and the others in this file — looks like the right tool, rather than adding instance-specific tests.