Data race between `sys._current_exceptions()` and a concurrently attaching thread

[Naserume]

Bug description:

In the free-threaded build, sys._current_exceptions() iterates every thread state under stop-the-world + HEAD_LOCK and reads t->thread_id.

cpython/Python/pystate.c

Lines 2808 to 2817 in ecdef17

	_PyEval_StopTheWorldAll(runtime);
	HEAD_LOCK(runtime);
	PyInterpreterState *i;
	for (i = runtime->interpreters.head; i != NULL; i = i->next) {
	_Py_FOR_EACH_TSTATE_UNLOCKED(i, t) {
	_PyErr_StackItem *err_info = _PyErr_GetTopmostException(t);
	if (err_info == NULL) {
	continue;
	}
	PyObject *id = PyLong_FromUnsignedLong(t->thread_id);

But a newly started thread sets tstate->thread_id in bind_tstate() without holding HEAD_LOCK. Its PyThreadState is already on the interpreter's thread list by this point.

cpython/Python/pystate.c

Lines 164 to 178 in ecdef17

	bind_tstate(PyThreadState *tstate)
	{
	assert(tstate != NULL);
	assert(tstate_is_alive(tstate) && !tstate->_status.bound);
	assert(!tstate->_status.unbound); // just in case
	assert(!tstate->_status.bound_gilstate);
	assert(tstate != gilstate_get());
	assert(!tstate->_status.active);
	assert(tstate->thread_id == 0);
	assert(tstate->native_thread_id == 0);
	// Currently we don't necessarily store the thread state
	// in thread-local storage (e.g. per-interpreter).
	tstate->thread_id = PyThread_get_thread_ident();

A thread that is still attaching is not yet stop-the-world-stoppable. _PyEval_StopTheWorldAll() therefore does not pause it, and its unlocked write to tstate->thread_id races the stop-the-world reader in _PyThread_CurrentExceptions.

Reproducer:

import sys
import threading

stop = threading.Event()

def reader():                      # keep stop-the-world almost always active
    for _ in range(20000):
        sys._current_exceptions()

def churn():                       # continuously attach new threads (bind_tstate)
    for _ in range(20000):
        t = threading.Thread(target=lambda: None)
        t.start()
        t.join()

readers  = [threading.Thread(target=reader) for _ in range(8)]
churners = [threading.Thread(target=churn)  for _ in range(4)]
for t in readers + churners: t.start()
for t in readers + churners: t.join()

TSAN Report:

WARNING: ThreadSanitizer: data race (pid=94157)
  Read of size 8 at 0x0001081180b0 by thread T1:
    #0 _PyThread_CurrentExceptions pystate.c:2817
    #1 sys__current_exceptions sysmodule.c.h:1180
    #2 cfunction_vectorcall_NOARGS methodobject.c:508
    #3 PyObject_Vectorcall call.c:327
    #4 _Py_VectorCallInstrumentation_StackRefSteal ceval.c:768
    #5 _PyEval_EvalFrameDefault generated_cases.c.h:1906
...
  Previous write of size 8 by thread T9:
    #0 bind_tstate pystate.c:178
    #1 _PyThreadState_Bind pystate.c:2660
    #2 thread_run _threadmodule.c:384
    #3 pythread_wrapper thread_pthread.h:234

SUMMARY: ThreadSanitizer: data race pystate.c:2817 in _PyThread_CurrentExceptions

CPython versions tested on:

CPython main branch

Operating systems tested on:

macOS

Linked PRs

• gh-152741: Fix data race on PyThreadState.thread_id during thread bind #152755
• gh-152741: Publish thread states to the interpreter's thread list only after they are fully initialized #156202

[zainnadeem786]

I took a closer look at this on the latest main (564c58c718bc) using a free-threaded Windows debug build.

I couldn't independently reproduce the reported ThreadSanitizer warning because TSAN isn't available with the Windows/MSVC free-threaded build, so I focused on tracing the implementation and verifying the reported code paths.

From what I found, the synchronization concern described in the report appears consistent with the current implementation.

_PyThreadState_New() publishes the new thread state into interp->threads.head while holding HEAD_LOCK, but the child thread later calls _PyThreadState_Bind(), where bind_tstate() initializes tstate->thread_id without HEAD_LOCK.

Meanwhile, _PyThread_CurrentExceptions() performs stop-the-world, acquires HEAD_LOCK, iterates the published thread-state list, and reads t->thread_id. Since bind_tstate() runs before _PyThreadState_Attach(), my understanding is that an attaching thread is not yet fully covered by stop-the-world, so the write to thread_id may race with those readers.

While tracing this path, I also noticed that _PyThread_CurrentFrames() and PyThreadState_SetAsyncExc() use the same thread_id publication/read pattern, so the discussion may extend beyond _current_exceptions() alone.

I also considered a few possible implementation approaches (synchronizing publication under HEAD_LOCK, atomic publication/access, or excluding published-but-unbound thread states from these APIs), but each seems to have different lifecycle implications, so I don't want to guess at the intended design.

Before attempting a PR, I'd appreciate some guidance from the runtime/free-threading maintainers on the intended invariant here.

Should published-but-not-yet-bound thread states be visible to APIs such as _current_exceptions() and _current_frames(), or is the expectation that thread_id publication itself should be synchronized (for example via HEAD_LOCK or atomic access)?

If there's a preferred direction, I'd be happy to investigate further and work on a patch.

[tonghuaroot]

Confirmed on a free-threaded ThreadSanitizer build (--with-thread-sanitizer --disable-gil --with-pydebug); the reproducer fires on essentially every run:

WARNING: ThreadSanitizer: data race
  Read  of size 8 ... _PyThread_CurrentExceptions  Python/pystate.c:2817
  Write of size 8 ... bind_tstate                  Python/pystate.c:178

bind_tstate writes thread_id after the PyThreadState is on interp->threads.head but before the thread attaches, so stop-the-world can't pause it and the store races sys._current_exceptions(). _PyThread_CurrentFrames reads the same field but I couldn't make it race — its if (frame == NULL) continue; gate means the thread only reaches the read after PyEval_AcquireThread has synchronized through tstate->state.

Suggested labels: topic-free-threading, interpreter-core (on top of type-bug).

Fix is small — store thread_id/native_thread_id with a relaxed atomic in bind_tstate and read them with a relaxed atomic load in the thread-list readers, matching tstate->state. Opened #152755 (TSan-clean across 30+ runs).