json.dumps() can use freed dictionary entries during concurrent mutation

[Nievesjyl]

Description

The exact-dictionary path in _json obtains borrowed key and value references with PyDict_Next() and increments them without holding the dictionary's critical section. A concurrent removal or replacement can destroy an entry between acquisition and ownership promotion.

Observed Behavior

On a free-threaded ASan build, racing json.dumps() with mutation of the encoded dictionary produces an ASan SEGV while consuming a stale nested value from encoder_listencode_dict().

Affected Version

CPython 3.14.7 at commit 823f0323ee6ec1402088b73bce1a38473cac36dc, built with --disable-gil --with-address-sanitizer and run with PYTHON_GIL=0.

Reproduction

Run the stress reproducer, repeating it if necessary because the schedule is probabilistic:

ASAN_OPTIONS=detect_leaks=0:halt_on_error=1 PYTHON_GIL=0 python3.14 poc/reproduce.py --seconds 10

PoC Source Code

poc/reproduce.py:

#!/usr/bin/env python3
"""Race json.dumps() against mutation of the exact dictionary it traverses."""

import argparse
import json
import threading
import time


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--seconds", type=float, default=10.0)
    args = parser.parse_args()

    shared = {str(index): [index] for index in range(512)}
    stop = threading.Event()
    counts = [0, 0]

    def encode() -> None:
        while not stop.is_set():
            try:
                json.dumps(shared, check_circular=False)
            except (RuntimeError, ValueError):
                pass
            counts[0] += 1

    def mutate() -> None:
        generation = 0
        while not stop.is_set():
            key = str(generation % 512)
            shared[key] = [generation]
            if generation % 4 == 0:
                shared.pop(key, None)
                shared[key] = [generation + 1]
            generation += 1
            counts[1] += 1

    threads = [threading.Thread(target=encode), threading.Thread(target=mutate)]
    for thread in threads:
        thread.start()
    time.sleep(args.seconds)
    stop.set()
    for thread in threads:
        thread.join()
    print({"encode": counts[0], "mutate": counts[1]})


if __name__ == "__main__":
    main()

[ZeroIntensity]

Refer to #157124 (comment)