Unpickler memo assignment can read reallocated donor storage

[Nievesjyl]

Description

Assigning one Unpickler's memo proxy to another reads the donor's memo size, native array, and slots without locking the donor. A concurrent donor.load() can grow or reallocate that array while Unpickler_set_memo() is copying and incrementing its entries.

Observed Behavior

On a free-threaded ASan build, the reproducer crashes in _Py_XNewRef() from Unpickler_set_memo() at Modules/_pickle.c:7773. A scaled compatibility-GIL control completes 80 donor loads and 21,846 memo copies normally.

Affected Version

CPython 3.14.7 at commit 823f0323ee6ec1402088b73bce1a38473cac36dc, built with ASan and free-threading support.

Reproduction

Run:

ASAN_OPTIONS=abort_on_error=1:detect_leaks=0 PYTHON_GIL=0 CONCURDEP_BATCHES=300 CONCURDEP_WIDTH=1000 python3.14 poc/reproduce.py

PoC Source Code

poc/reproduce.py:

import io
import os
import pickle
import threading


batches = int(os.environ.get("CONCURDEP_BATCHES", "300"))
width = int(os.environ.get("CONCURDEP_WIDTH", "1000"))
workers = int(os.environ.get("CONCURDEP_WORKERS", "8"))

stream = io.BytesIO()
pickler = pickle.Pickler(stream, protocol=5)
for batch in range(batches):
    pickler.dump([[batch, index, bytearray(32)] for index in range(width)])
stream.seek(0)
donor = pickle.Unpickler(stream)
proxy = donor.memo
stop = threading.Event()
gate = threading.Barrier(workers + 1)
counts = [0] * (workers + 1)


def load():
    gate.wait()
    for _ in range(batches):
        donor.load()
        counts[0] += 1
    stop.set()


def copy(slot):
    target = pickle.Unpickler(io.BytesIO(b""))
    gate.wait()
    while not stop.is_set():
        target.memo = proxy
        counts[slot] += 1


threads = [threading.Thread(target=load)]
threads += [threading.Thread(target=copy, args=(i,)) for i in range(1, workers + 1)]
for thread in threads:
    thread.start()
for thread in threads:
    thread.join()
print("counts", counts)

[ZeroIntensity]

Refer to #157124 (comment)