Skip to content

Put CentralFreeList and miss counters on separate cachelines in TransferCache. - #994

Draft
copybara-service[bot] wants to merge 1 commit into
masterfrom
test_983583281
Draft

copybara-service[bot] wants to merge 1 commit into
masterfrom
test_983583281

Conversation

@copybara-service

Copy link
Copy Markdown

Put CentralFreeList and miss counters on separate cachelines in TransferCache.

TransferCache embeds its CentralFreeList at offset 0x30, so CentralFreeList::lock_
shares the first cacheline with the transfer cache's lock_, slot_info_, hit
counters and slots_. Cores hitting in the transfer cache (lock CAS + slot_info_
RMW) and cores holding the CentralFreeList lock on a miss ping-pong that line.
The four miss counters (0x258-0x288) likewise share a line with the tail of
CentralFreeList::nonempty_, which is mutated under the CentralFreeList lock.
GWP: TransferCache::RemoveRange 597k / InsertRange 264k self.

New layout:
line 0: lock_, low_water_mark_, hit counters, slot_info_, slots_, owner_,
max_capacity_ (60 B; owner_/max_capacity_ are immutable)
line 1: insert/remove miss counters (48 B)
line 2: CentralFreeList, cacheline aligned (its lock_ is at offset 0)

A pointer-sized FreeList (BackingTransferCache in the sharded cache) has no
lock and packs into line 1. CentralFreeList's tail padding absorbs the shift:
sizeof(TransferCache<CentralFreeList, ...>) stays 704 B, the sharded
TransferCache stays 128 B and TransferCacheManager stays 181,632 B (8K pages).
static_asserts in transfer_cache_test pin the layout.

Also record misses after, rather than before, the freelist call in InsertRange
and RemoveRange. The counter RMW usually misses (the line is written by every
missing core); issued ahead of the CentralFreeList lock acquisition, its store
had to drain before the lock cmpxchg (a full barrier on x86) could complete,
serializing two cross-core transfers on the miss path. Issued afterwards it
drains asynchronously from the store buffer. Counts are identical.

RemoveRange miss path (-c opt, x86-64) now ends in:
lea 0x80(%rbx),%rdi ; call CentralFreeList::RemoveRange
incq 0x48(%rbx) ; addq %r14,0x60(%rbx) ; ret

…ferCache.

TransferCache embeds its CentralFreeList at offset 0x30, so CentralFreeList::lock_
shares the first cacheline with the transfer cache's lock_, slot_info_, hit
counters and slots_.  Cores hitting in the transfer cache (lock CAS + slot_info_
RMW) and cores holding the CentralFreeList lock on a miss ping-pong that line.
The four miss counters (0x258-0x288) likewise share a line with the tail of
CentralFreeList::nonempty_, which is mutated under the CentralFreeList lock.
GWP: TransferCache::RemoveRange 597k / InsertRange 264k self.

New layout:
  line 0: lock_, low_water_mark_, hit counters, slot_info_, slots_, owner_,
          max_capacity_ (60 B; owner_/max_capacity_ are immutable)
  line 1: insert/remove miss counters (48 B)
  line 2: CentralFreeList, cacheline aligned (its lock_ is at offset 0)

A pointer-sized FreeList (BackingTransferCache in the sharded cache) has no
lock and packs into line 1.  CentralFreeList's tail padding absorbs the shift:
sizeof(TransferCache<CentralFreeList, ...>) stays 704 B, the sharded
TransferCache stays 128 B and TransferCacheManager stays 181,632 B (8K pages).
static_asserts in transfer_cache_test pin the layout.

Also record misses after, rather than before, the freelist call in InsertRange
and RemoveRange.  The counter RMW usually misses (the line is written by every
missing core); issued ahead of the CentralFreeList lock acquisition, its store
had to drain before the lock cmpxchg (a full barrier on x86) could complete,
serializing two cross-core transfers on the miss path.  Issued afterwards it
drains asynchronously from the store buffer.  Counts are identical.

RemoveRange miss path (-c opt, x86-64) now ends in:
  lea 0x80(%rbx),%rdi ; call CentralFreeList::RemoveRange
  incq 0x48(%rbx) ; addq %r14,0x60(%rbx) ; ret

PiperOrigin-RevId: 983583281
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant