Concurrent multibyte stream-reader reads can corrupt decoder state

[Nievesjyl]

Description

Multibyte stream-reader read() and readline() operations run a stateful decode without a critical section on the reader. Concurrent calls share and modify its codec state, pending bytes, error mode, and writer transition even when the underlying stream is C-backed.

Observed Behavior

On a free-threaded ASan/debug build, eight threads calling read(31) on one Shift-JIS reader trigger a fatal allocator guard-byte corruption after about 0.7 seconds. The same executable completes normally with the compatibility GIL enabled.

Affected Version

CPython 3.14.7 at commit 823f0323ee6ec1402088b73bce1a38473cac36dc, built with ASan, the debug allocator, and free-threading support.

Reproduction

Run:

ASAN_OPTIONS=abort_on_error=1:detect_leaks=0 PYTHON_GIL=0 python3.14 poc/reproduce.py 10

For comparison, run the same command with PYTHON_GIL=1.

PoC Source Code

poc/reproduce.py:

import codecs
import io
import os
import sys
import threading
import time


duration = float(sys.argv[1]) if len(sys.argv) > 1 else 10.0
stream = io.BytesIO(os.urandom(4_000_000))
shared = codecs.getreader("shift_jis")(stream, "replace")
stop = threading.Event()


def read():
    while not stop.is_set():
        try:
            shared.read(31)
        except (UnicodeError, ValueError):
            pass


threads = [threading.Thread(target=read) for _ in range(8)]
for thread in threads:
    thread.start()
time.sleep(duration)
stop.set()
for thread in threads:
    thread.join()
print("completed")

[ZeroIntensity]

Refer to #157124 (comment)