Unpickler can use a freed extension-cache object during concurrent clearing

[Nievesjyl]

Description

_pickle.load_extension() retrieves a borrowed object from copyreg._extension_cache and promotes it separately when pushing it onto the unpickler stack. Concurrent clear_extension_cache() can remove the cache's last reference before that promotion.

Observed Behavior

On a free-threaded ASan build, clearing the extension cache and replacing the registered module attribute while other threads load an EXT1 payload crashes in _Py_Dealloc(). The same executable completes 884,461 loads with the compatibility GIL enabled.

Affected Version

CPython 3.14.7 at commit 823f0323ee6ec1402088b73bce1a38473cac36dc, built with ASan and free-threading support.

Reproduction

Run:

ASAN_OPTIONS=abort_on_error=1:detect_leaks=0 PYTHON_GIL=0 CONCURDEP_DURATION=20 python3.14 poc/reproduce.py

For comparison, run the same command with PYTHON_GIL=1.

PoC Source Code

poc/reproduce.py:

import copyreg
import gc
import os
import pickle
import sys
import threading
import time
import types


duration = float(os.environ.get("CONCURDEP_DURATION", "20"))
stop = threading.Event()
counts = [0] * 17
module = types.ModuleType("concurdep_pickle_mod")
sys.modules[module.__name__] = module


class Blob:
    __slots__ = ("padding", "cycle")

    def __init__(self):
        self.padding = bytearray(16384)
        self.cycle = None

    def __del__(self):
        for _ in range(16):
            bytearray(8192)
        time.sleep(0)


module.target = Blob()
copyreg.add_extension(module.__name__, "target", 1)
payload = b"\x80\x05\x82\x01."
pickle.loads(payload)


def mutate():
    iterations = 0
    while not stop.is_set():
        copyreg.clear_extension_cache()
        module.target = Blob()
        if iterations % 32 == 0:
            gc.collect()
        iterations += 1
    counts[0] = iterations


def load(slot):
    while not stop.is_set():
        try:
            pickle.loads(payload)
        except BaseException:
            pass
        counts[slot] += 1


threads = [threading.Thread(target=mutate)]
threads += [threading.Thread(target=load, args=(i,)) for i in range(1, 17)]
for thread in threads:
    thread.start()
time.sleep(duration)
stop.set()
for thread in threads:
    thread.join()
print("counts", counts)

[ZeroIntensity]

Refer to #157124 (comment)