Put CentralFreeList and miss counters on separate cachelines in TransferCache. - #994
Draft
copybara-service[bot] wants to merge 1 commit into
Draft
copybara-service[bot] wants to merge 1 commit into
copybara-service[bot] wants to merge 1 commit into
Conversation
…ferCache.
TransferCache embeds its CentralFreeList at offset 0x30, so CentralFreeList::lock_
shares the first cacheline with the transfer cache's lock_, slot_info_, hit
counters and slots_. Cores hitting in the transfer cache (lock CAS + slot_info_
RMW) and cores holding the CentralFreeList lock on a miss ping-pong that line.
The four miss counters (0x258-0x288) likewise share a line with the tail of
CentralFreeList::nonempty_, which is mutated under the CentralFreeList lock.
GWP: TransferCache::RemoveRange 597k / InsertRange 264k self.
New layout:
line 0: lock_, low_water_mark_, hit counters, slot_info_, slots_, owner_,
max_capacity_ (60 B; owner_/max_capacity_ are immutable)
line 1: insert/remove miss counters (48 B)
line 2: CentralFreeList, cacheline aligned (its lock_ is at offset 0)
A pointer-sized FreeList (BackingTransferCache in the sharded cache) has no
lock and packs into line 1. CentralFreeList's tail padding absorbs the shift:
sizeof(TransferCache<CentralFreeList, ...>) stays 704 B, the sharded
TransferCache stays 128 B and TransferCacheManager stays 181,632 B (8K pages).
static_asserts in transfer_cache_test pin the layout.
Also record misses after, rather than before, the freelist call in InsertRange
and RemoveRange. The counter RMW usually misses (the line is written by every
missing core); issued ahead of the CentralFreeList lock acquisition, its store
had to drain before the lock cmpxchg (a full barrier on x86) could complete,
serializing two cross-core transfers on the miss path. Issued afterwards it
drains asynchronously from the store buffer. Counts are identical.
RemoveRange miss path (-c opt, x86-64) now ends in:
lea 0x80(%rbx),%rdi ; call CentralFreeList::RemoveRange
incq 0x48(%rbx) ; addq %r14,0x60(%rbx) ; ret
PiperOrigin-RevId: 983583281
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Put CentralFreeList and miss counters on separate cachelines in TransferCache.
TransferCache embeds its CentralFreeList at offset 0x30, so CentralFreeList::lock_
shares the first cacheline with the transfer cache's lock_, slot_info_, hit
counters and slots_. Cores hitting in the transfer cache (lock CAS + slot_info_
RMW) and cores holding the CentralFreeList lock on a miss ping-pong that line.
The four miss counters (0x258-0x288) likewise share a line with the tail of
CentralFreeList::nonempty_, which is mutated under the CentralFreeList lock.
GWP: TransferCache::RemoveRange 597k / InsertRange 264k self.
New layout:
line 0: lock_, low_water_mark_, hit counters, slot_info_, slots_, owner_,
max_capacity_ (60 B; owner_/max_capacity_ are immutable)
line 1: insert/remove miss counters (48 B)
line 2: CentralFreeList, cacheline aligned (its lock_ is at offset 0)
A pointer-sized FreeList (BackingTransferCache in the sharded cache) has no
lock and packs into line 1. CentralFreeList's tail padding absorbs the shift:
sizeof(TransferCache<CentralFreeList, ...>) stays 704 B, the sharded
TransferCache stays 128 B and TransferCacheManager stays 181,632 B (8K pages).
static_asserts in transfer_cache_test pin the layout.
Also record misses after, rather than before, the freelist call in InsertRange
and RemoveRange. The counter RMW usually misses (the line is written by every
missing core); issued ahead of the CentralFreeList lock acquisition, its store
had to drain before the lock cmpxchg (a full barrier on x86) could complete,
serializing two cross-core transfers on the miss path. Issued afterwards it
drains asynchronously from the store buffer. Counts are identical.
RemoveRange miss path (-c opt, x86-64) now ends in:
lea 0x80(%rbx),%rdi ; call CentralFreeList::RemoveRange
incq 0x48(%rbx) ; addq %r14,0x60(%rbx) ; ret