Pickle memo setters can traverse a concurrently mutated source dictionary

[Nievesjyl]

Description

The C Pickler.memo and Unpickler.memo setters traverse a caller-supplied dictionary with PyDict_Next() without a critical section or owned snapshot. Concurrent clear or refill can invalidate traversal and reclaim borrowed keys, values, tuples, or tuple fields before conversion and insertion.

Observed Behavior

On a free-threaded build, both the Pickler and Unpickler modes terminate with SIGSEGV in about 0.21 seconds. Both compatibility-GIL controls complete normally.

Affected Version

CPython 3.14.7 at commit 823f0323ee6ec1402088b73bce1a38473cac36dc, using the free-threaded build with PYTHON_GIL=0.

Reproduction

Run both modes:

PYTHON_GIL=0 python3.14t poc/reproduce.py 10 pickler
PYTHON_GIL=0 python3.14t poc/reproduce.py 10 unpickler

PoC Source Code

poc/reproduce.py:

#!/usr/bin/env python3
"""Race Pickler/Unpickler memo assignment with source-dict mutation."""

import io
import pickle
import sys
import threading
import time


MODE = sys.argv[2] if len(sys.argv) > 2 else "pickler"
if MODE == "pickler":
    SHARED = {i: (i, object()) for i in range(4096)}
elif MODE == "unpickler":
    SHARED = {i: object() for i in range(4096)}
else:
    raise ValueError("mode must be 'pickler' or 'unpickler'")
KEYS = tuple(SHARED)
STOP = threading.Event()
FAILURES = []


def fresh_items():
    if MODE == "pickler":
        return {i: (i, object()) for i in KEYS}
    return {i: object() for i in KEYS}


def mutate():
    while not STOP.is_set():
        SHARED.clear()
        SHARED.update(fresh_items())


def set_worker(deadline):
    count = 0
    try:
        while time.monotonic() < deadline and not STOP.is_set():
            if MODE == "pickler":
                obj = pickle.Pickler(io.BytesIO())
            else:
                obj = pickle.Unpickler(io.BytesIO(b"N."))
            obj.memo = SHARED
            count += 1
    except BaseException as exc:
        FAILURES.append((type(exc).__name__, repr(exc), count))
        STOP.set()


def main():
    seconds = float(sys.argv[1]) if len(sys.argv) > 1 else 10.0
    deadline = time.monotonic() + seconds
    threads = [threading.Thread(target=mutate) for _ in range(2)]
    threads += [threading.Thread(target=set_worker, args=(deadline,)) for _ in range(8)]
    for thread in threads:
        thread.start()
    for thread in threads[2:]:
        thread.join()
    STOP.set()
    for thread in threads[:2]:
        thread.join()
    print("mode=", MODE, "gil=", sys._is_gil_enabled(), "failures=", FAILURES)
    return bool(FAILURES)


raise SystemExit(main())

[ZeroIntensity]

Refer to #157124 (comment)