Skip to content

gh-144446: Fix thread safety of gi_frame, cr_frame and ag_frame in free-threading - #156037

Merged
kumaraditya303 merged 5 commits into
python:mainfrom
kumaraditya303:gen-gi-frame-thread-safety
Sep 2, 2026
Merged

gh-144446: Fix thread safety of gi_frame, cr_frame and ag_frame in free-threading#156037
kumaraditya303 merged 5 commits into
python:mainfrom
kumaraditya303:gen-gi-frame-thread-safety

Conversation

@kumaraditya303

@kumaraditya303 kumaraditya303 commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Fixes thread safety of reading gi_frame, cr_frame and ag_frame while the generator is running or finishing in another thread. Previously the reader could create a frame object for an interpreter frame that was concurrently being cleared, or two threads could each create a frame object for the same frame.

The frame object is now created within the generator's critical section, re-checking the frame state after acquiring it so a finished generator returns None instead of a frame for a cleared iframe. Clearing the generator's frame in gen_clear_frame() and clear_gen_frame() now also holds the critical section so it cannot race with the getter. frame->frame_obj is now set with a compare exchange as the running thread can create it without holding the generator's critical section (e.g. through sys._getframe()) — the losing thread discards its frame object and uses the existing one — and it is cleared with an atomic exchange and read with an acquire load.

@bedevere-app bedevere-app Bot mentioned this pull request Aug 19, 2026
3 tasks
Comment thread Misc/NEWS.d/next/Core_and_Builtins/2026-08-19-06-43-54.gh-issue-144446.k3QzXa.rst Outdated
@kumaraditya303

Copy link
Copy Markdown
Contributor Author

ping @mpage for review

@mpage

mpage commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

This looks correct to me. I'm a little worried about the performance impact of adding a _Py_atomic_compare_exchange_ptr on the return path of every Python function, though. I kicked off a benchmarking run to see what the impact is on the benchmark suite.

@mpage

mpage commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Benchmark results look like a small regression (~1%).

I don't think we need to use an exchange in _PyFrame_ClearExceptCode, though, because access to frame->frame_obj is always mediated by the generator's critical section when _PyFrame_ClearExceptCode is executing. Assuming that's correct, I think the change to _PyFrame_ClearExceptCode can just be:

PyFrameObject *f = FT_ATOMIC_LOAD_PTR_RELAXED(frame->frame_obj);
if (f != NULL) {
    FT_ATOMIC_STORE_PTR_RELAXED(frame->frame_obj, NULL);
    if (!_PyObject_IsUniquelyReferenced((PyObject *)f)) {
        take_ownership(f, frame);
        Py_DECREF(f);
        return;
    }
    Py_DECREF(f);
}

@kumaraditya303
kumaraditya303 merged commit db92901 into python:main Sep 2, 2026
63 checks passed
@kumaraditya303
kumaraditya303 deleted the gen-gi-frame-thread-safety branch September 2, 2026 17:56
@bedevere-bot

Copy link
Copy Markdown

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 CentOS9 NoGIL 3.x (tier-1) has failed when building commit db92901.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1609/builds/6887) and take a look at the build logs.
  4. Check if the failure is related to this commit (db92901) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1609/builds/6887

Failed tests:

  • test_cppext

Failed subtests:

  • test_build - test.test_cppext.TestInteralCAPI.test_build

Summary of the results of the build (if available):

==

Click to see traceback logs
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.nogil/build/Lib/test/test_cppext/__init__.py", line 146, in test_build
    self.check_build('_testcppext_internal', **kwargs)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.nogil/build/Lib/test/test_cppext/__init__.py", line 37, in check_build
    self._check_build(extension_name, python_exe,
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                      std=std, limited=limited,
                      ^^^^^^^^^^^^^^^^^^^^^^^^^
                      extra_cflags=extra_cflags)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.nogil/build/Lib/test/test_cppext/__init__.py", line 82, in _check_build
    run_cmd('Install', cmd)
    ~~~~~~~^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.nogil/build/Lib/test/test_cppext/__init__.py", line 63, in run_cmd
    subprocess.run(cmd, check=True, env=env)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.nogil/build/Lib/subprocess.py", line 692, in run
    raise CalledProcessError(retcode, process.args,
                             output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command ['/home/buildbot/buildarea/3.x.itamaro-centos-aws.nogil/build/build/test_python_508950æ/tempcwd/env/bin/python', '-X', 'dev', '-m', 'pip', 'install', '--no-build-isolation', '/home/buildbot/buildarea/3.x.itamaro-centos-aws.nogil/build/build/test_python_508950æ/tempcwd/pkg', '-v'] returned non-zero exit status 1.


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.nogil/build/Lib/test/test_cppext/__init__.py", line 146, in test_build
    self.check_build('_testcppext_internal', **kwargs)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.nogil/build/Lib/test/test_cppext/__init__.py", line 37, in check_build
    self._check_build(extension_name, python_exe,
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                      std=std, limited=limited,
                      ^^^^^^^^^^^^^^^^^^^^^^^^^
                      extra_cflags=extra_cflags)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.nogil/build/Lib/test/test_cppext/__init__.py", line 82, in _check_build
    run_cmd('Install', cmd)
    ~~~~~~~^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.nogil/build/Lib/test/test_cppext/__init__.py", line 63, in run_cmd
    subprocess.run(cmd, check=True, env=env)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.nogil/build/Lib/subprocess.py", line 692, in run
    raise CalledProcessError(retcode, process.args,
                             output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command ['/home/buildbot/buildarea/3.x.itamaro-centos-aws.nogil/build/build/test_python_554276æ/tempcwd/env/bin/python', '-X', 'dev', '-m', 'pip', 'install', '--no-build-isolation', '/home/buildbot/buildarea/3.x.itamaro-centos-aws.nogil/build/build/test_python_554276æ/tempcwd/pkg', '-v'] returned non-zero exit status 1.

@bedevere-bot

Copy link
Copy Markdown

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot ARM64 MacOS M1 NoGIL 3.x (tier-2) has failed when building commit db92901.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1270/builds/8482) and take a look at the build logs.
  4. Check if the failure is related to this commit (db92901) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1270/builds/8482

Failed tests:

  • test_cppext

Failed subtests:

  • test_build - test.test_cppext.TestInteralCAPI.test_build

Summary of the results of the build (if available):

==

Click to see traceback logs
Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.itamaro-macos-arm64-aws.macos-with-brew.nogil/build/Lib/test/test_cppext/__init__.py", line 146, in test_build
    self.check_build('_testcppext_internal', **kwargs)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/3.x.itamaro-macos-arm64-aws.macos-with-brew.nogil/build/Lib/test/test_cppext/__init__.py", line 37, in check_build
    self._check_build(extension_name, python_exe,
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                      std=std, limited=limited,
                      ^^^^^^^^^^^^^^^^^^^^^^^^^
                      extra_cflags=extra_cflags)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/3.x.itamaro-macos-arm64-aws.macos-with-brew.nogil/build/Lib/test/test_cppext/__init__.py", line 82, in _check_build
    run_cmd('Install', cmd)
    ~~~~~~~^^^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/3.x.itamaro-macos-arm64-aws.macos-with-brew.nogil/build/Lib/test/test_cppext/__init__.py", line 63, in run_cmd
    subprocess.run(cmd, check=True, env=env)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/3.x.itamaro-macos-arm64-aws.macos-with-brew.nogil/build/Lib/subprocess.py", line 692, in run
    raise CalledProcessError(retcode, process.args,
                             output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command ['/Users/buildbot/buildarea/3.x.itamaro-macos-arm64-aws.macos-with-brew.nogil/build/build/test_python_39839æ/tempcwd/env/bin/python.exe', '-X', 'dev', '-m', 'pip', 'install', '--no-build-isolation', '/Users/buildbot/buildarea/3.x.itamaro-macos-arm64-aws.macos-with-brew.nogil/build/build/test_python_39839æ/tempcwd/pkg', '-v'] returned non-zero exit status 1.


Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.itamaro-macos-arm64-aws.macos-with-brew.nogil/build/Lib/test/test_cppext/__init__.py", line 146, in test_build
    self.check_build('_testcppext_internal', **kwargs)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/3.x.itamaro-macos-arm64-aws.macos-with-brew.nogil/build/Lib/test/test_cppext/__init__.py", line 37, in check_build
    self._check_build(extension_name, python_exe,
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                      std=std, limited=limited,
                      ^^^^^^^^^^^^^^^^^^^^^^^^^
                      extra_cflags=extra_cflags)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/3.x.itamaro-macos-arm64-aws.macos-with-brew.nogil/build/Lib/test/test_cppext/__init__.py", line 82, in _check_build
    run_cmd('Install', cmd)
    ~~~~~~~^^^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/3.x.itamaro-macos-arm64-aws.macos-with-brew.nogil/build/Lib/test/test_cppext/__init__.py", line 63, in run_cmd
    subprocess.run(cmd, check=True, env=env)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/3.x.itamaro-macos-arm64-aws.macos-with-brew.nogil/build/Lib/subprocess.py", line 692, in run
    raise CalledProcessError(retcode, process.args,
                             output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command ['/Users/buildbot/buildarea/3.x.itamaro-macos-arm64-aws.macos-with-brew.nogil/build/build/test_python_49042æ/tempcwd/env/bin/python.exe', '-X', 'dev', '-m', 'pip', 'install', '--no-build-isolation', '/Users/buildbot/buildarea/3.x.itamaro-macos-arm64-aws.macos-with-brew.nogil/build/build/test_python_49042æ/tempcwd/pkg', '-v'] returned non-zero exit status 1.

@bedevere-bot

Copy link
Copy Markdown

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Alpine Linux NoGIL 3.x (tierless) has failed when building commit db92901.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1819/builds/2983) and take a look at the build logs.
  4. Check if the failure is related to this commit (db92901) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1819/builds/2983

Summary of the results of the build (if available):

==

Click to see traceback logs
Traceback (most recent call last):
  File �[35m"/buildbot/buildarea/3.x.ware-alpine.nogil/build/Lib/test/test_cppext/__init__.py"�[0m, line �[35m146�[0m, in �[35mtest_build�[0m
    �[31mself.check_build�[0m�[1;31m('_testcppext_internal', **kwargs)�[0m
    �[31m~~~~~~~~~~~~~~~~�[0m�[1;31m^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^�[0m
  File �[35m"/buildbot/buildarea/3.x.ware-alpine.nogil/build/Lib/test/test_cppext/__init__.py"�[0m, line �[35m37�[0m, in �[35mcheck_build�[0m
    �[31mself._check_build�[0m�[1;31m(extension_name, python_exe,�[0m
    �[31m~~~~~~~~~~~~~~~~~�[0m�[1;31m^^^^^^^^^^^^^^^^^^^^^^^^^^^^�[0m
                      �[1;31mstd=std, limited=limited,�[0m
                      �[1;31m^^^^^^^^^^^^^^^^^^^^^^^^^�[0m
                      �[1;31mextra_cflags=extra_cflags)�[0m
                      �[1;31m^^^^^^^^^^^^^^^^^^^^^^^^^^�[0m
  File �[35m"/buildbot/buildarea/3.x.ware-alpine.nogil/build/Lib/test/test_cppext/__init__.py"�[0m, line �[35m82�[0m, in �[35m_check_build�[0m
    �[31mrun_cmd�[0m�[1;31m('Install', cmd)�[0m
    �[31m~~~~~~~�[0m�[1;31m^^^^^^^^^^^^^^^^�[0m
  File �[35m"/buildbot/buildarea/3.x.ware-alpine.nogil/build/Lib/test/test_cppext/__init__.py"�[0m, line �[35m63�[0m, in �[35mrun_cmd�[0m
    �[31msubprocess.run�[0m�[1;31m(cmd, check=True, env=env)�[0m
    �[31m~~~~~~~~~~~~~~�[0m�[1;31m^^^^^^^^^^^^^^^^^^^^^^^^^^�[0m
  File �[35m"/buildbot/buildarea/3.x.ware-alpine.nogil/build/Lib/subprocess.py"�[0m, line �[35m692�[0m, in �[35mrun�[0m
    raise CalledProcessError(retcode, process.args,
                             output=stdout, stderr=stderr)
�[1;35msubprocess.CalledProcessError�[0m: �[35mCommand ['/buildbot/buildarea/3.x.ware-alpine.nogil/build/build/test_python_3353076æ/tempcwd/env/bin/python', '-X', 'dev', '-m', 'pip', 'install', '--no-build-isolation', '/buildbot/buildarea/3.x.ware-alpine.nogil/build/build/test_python_3353076æ/tempcwd/pkg', '-v'] returned non-zero exit status 1.�[0m


Traceback (most recent call last):
  File �[35m"/buildbot/buildarea/3.x.ware-alpine.nogil/build/Lib/test/test_cppext/__init__.py"�[0m, line �[35m146�[0m, in �[35mtest_build�[0m
    �[31mself.check_build�[0m�[1;31m('_testcppext_internal', **kwargs)�[0m
    �[31m~~~~~~~~~~~~~~~~�[0m�[1;31m^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^�[0m
  File �[35m"/buildbot/buildarea/3.x.ware-alpine.nogil/build/Lib/test/test_cppext/__init__.py"�[0m, line �[35m37�[0m, in �[35mcheck_build�[0m
    �[31mself._check_build�[0m�[1;31m(extension_name, python_exe,�[0m
    �[31m~~~~~~~~~~~~~~~~~�[0m�[1;31m^^^^^^^^^^^^^^^^^^^^^^^^^^^^�[0m
                      �[1;31mstd=std, limited=limited,�[0m
                      �[1;31m^^^^^^^^^^^^^^^^^^^^^^^^^�[0m
                      �[1;31mextra_cflags=extra_cflags)�[0m
                      �[1;31m^^^^^^^^^^^^^^^^^^^^^^^^^^�[0m
  File �[35m"/buildbot/buildarea/3.x.ware-alpine.nogil/build/Lib/test/test_cppext/__init__.py"�[0m, line �[35m82�[0m, in �[35m_check_build�[0m
    �[31mrun_cmd�[0m�[1;31m('Install', cmd)�[0m
    �[31m~~~~~~~�[0m�[1;31m^^^^^^^^^^^^^^^^�[0m
  File �[35m"/buildbot/buildarea/3.x.ware-alpine.nogil/build/Lib/test/test_cppext/__init__.py"�[0m, line �[35m63�[0m, in �[35mrun_cmd�[0m
    �[31msubprocess.run�[0m�[1;31m(cmd, check=True, env=env)�[0m
    �[31m~~~~~~~~~~~~~~�[0m�[1;31m^^^^^^^^^^^^^^^^^^^^^^^^^^�[0m
  File �[35m"/buildbot/buildarea/3.x.ware-alpine.nogil/build/Lib/subprocess.py"�[0m, line �[35m692�[0m, in �[35mrun�[0m
    raise CalledProcessError(retcode, process.args,
                             output=stdout, stderr=stderr)
�[1;35msubprocess.CalledProcessError�[0m: �[35mCommand ['/buildbot/buildarea/3.x.ware-alpine.nogil/build/build/test_python_3389655æ/tempcwd/env/bin/python', '-X', 'dev', '-m', 'pip', 'install', '--no-build-isolation', '/buildbot/buildarea/3.x.ware-alpine.nogil/build/build/test_python_3389655æ/tempcwd/pkg', '-v'] returned non-zero exit status 1.�[0m

@encukou

encukou commented Sep 3, 2026

Copy link
Copy Markdown
Member

This fails on some buildbots with:

  /usr/bin/g++ -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -fno-strict-overflow -Wsign-compare -g -Og -Wall -fPIC -I/home/buildbot/buildarea/3.x.itamaro-centos-aws.nogil/build/build/test_python_2302880æ/tempcwd/env/include -I/home/buildbot/buildarea/3.x.itamaro-centos-aws.nogil/build/Include -I/home/buildbot/buildarea/3.x.itamaro-centos-aws.nogil/build -c extension.cpp -o build/temp.linux-x86_64-cpython-316t-pydebug/extension.o -Werror -DMODULE_NAME=_testcppext_internal -DTEST_INTERNAL_C_API=1
  In file included from /home/buildbot/buildarea/3.x.itamaro-centos-aws.nogil/build/Include/internal/pycore_object.h:13,
                   from /home/buildbot/buildarea/3.x.itamaro-centos-aws.nogil/build/Include/internal/pycore_cell.h:5,
                   from extension.cpp:29:
  /home/buildbot/buildarea/3.x.itamaro-centos-aws.nogil/build/Include/internal/pycore_interpframe.h: In function ‘PyFrameObject* _PyFrame_GetFrameObject(_PyInterpreterFrame*)’:
  /home/buildbot/buildarea/3.x.itamaro-centos-aws.nogil/build/Include/internal/pycore_pyatomic_ft_wrappers.h:33:32: error: invalid conversion from ‘void*’ to ‘PyFrameObject*’ {aka ‘_frame*’} [-fpermissive]
     33 |     _Py_atomic_load_ptr_acquire(&value)
        |     ~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~
        |                                |
        |                                void*
  /home/buildbot/buildarea/3.x.itamaro-centos-aws.nogil/build/Include/internal/pycore_interpframe.h:348:26: note: in expansion of macro ‘FT_ATOMIC_LOAD_PTR_ACQUIRE’
    348 |     PyFrameObject *res = FT_ATOMIC_LOAD_PTR_ACQUIRE(frame->frame_obj);
        |                          ^~~~~~~~~~~~~~~~~~~~~~~~~~

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants