Add support for the free-threaded build #178
Conversation
fix various errors and warnings seen on clang and gcc
…d-flag Add CT_UNDER_CONSTRUCTION to indicate that a type is being mutated
(Kumar will fix this soon!)
Move CT_CUSTOM_FIELD_POS and CT_WITH_PACKED_CHANGE to ct_flags_mut
Replace ct_is_hidden with asserts
fix headers to avoid duplicated definitions
…dsafe fix thread safety of `_realize_c_struct_or_union`
Co-authored-by: Nathan Goldbaum <[email protected]>
remove unnecessary atomics
Simplify ct_stuff handling
use _CFFI_LOAD_OP at more places
|
@mattip @nitzmahone I think this is ready from our end - @kumaraditya303 and @colesbury figured out the thread safety issue we ran into last week. I also resolved the remaining comments from the last round of review. |
OK, now it's really ready - apologies for the noise. I misread a message from Kumar and I accidentally used the wrong commits from him for the final thread safety fix. Everything should be correct now. |
|
Would it be too much to ask that you try out a few prime CFFI libraries with good test suites, like cryptography and curl_cffi to see if their tests pass? I don't know how hard it would be to switch their build systems to use this PR. |
Sure, I can try those two. We already tried argon2-cffi with an older version of this PR and saw no races running its test suite in parallel. |
|
Cool, thanks
That is good news. |
Take a look at this CI run on cryptography: https://github.com/pyca/cryptography/actions/runs/16029864342/job/45227459492 All the failures are because of the very hacky way I added this PR as a depenency to cryptography, which breaks the test runs that depend on creating a lockfile.
This is trickier - right now it's using a cibuildwheel version from March 2024 and updating the CI config looks nontrivial. I'd also need to figure out how to get libffi on the build container to allow cffi to build on the CI host---cryptography already has libffi set up so it's not an issue there. I'm also not able to run the the tests on a local build using a GIL-enabled 3.12 interpreter - not sure what I'm doing wrong.
I tried this again with the latest version of the PR. The tests all pass on the free-threaded build and the GIL-enabled build. The tests also pass if I force the GIL to be disabled. There's an explicitly multithreaded test in the argon2-cffi tests that I added earlier this year. |
Cool. So if I am reading that correctly, the new CFFI works everywhere, even on non-free-threading builds. The Python3.8 failures are not connected.
OK, thanks for trying. No need to go all out, I thought it would be easier.
Cool! |
Co-authored-by: Kumar Aditya <[email protected]>
yup! |
There was a problem hiding this comment.
From what I can gather, this seems to make sense. There might be some design choices around granularity of the locks that could impact performance on the free-threaded build, but I think those can be adjusted in subsequent PRs to address any performance impact from lock churning vs. thread suspension without changing user-facing API.
As discussed, this will be part of a 2.0 release, so the version needs updating, not necessarily in this PR.
I went ahead and updated the version numbers based on 0bf786c. |
| @@ -35,7 +35,7 @@ def test_doc_version(): | |||
| content = _read(p) | |||
| # | |||
| v = cffi.__version__ | |||
| assert ("version = '%s'\n" % v[:4]) in content | |||
| assert ("version = '%s'\n" % ".".join(v.split('.')[:2])) in content | |||
There was a problem hiding this comment.
This is needed because the code currently assumes CFFI has a minor version number greater than 10.
Fixes #126.
Adds support for the free-threaded build of Python 3.13 and 3.14 by improving the thread safety of CFFI internals.
Overview
Mutable state that is accessible to more than one thread in the CFFI backend is now mediated via atomic operations, critical sections, or mutexes, depending on the use-case:
get_primitive_type: We initialize all primitive types at module initialization, instead of doing them lazily. This is pretty inexpensive (~100 µs).file_struct: now initialized during module initialization_get_ct_int: now initialized during module initializationinit_once_cache: The cache itself is initialized under a critical section and we use APIs that return strong references instead of borrowed references.malloc_closure: use a global mutexb_complete_struct_or_union: Added critical section.pytest-run-paralleland sets uppytest-run-parallelLinux, Mac, and Windows CI as well as a run using a TSAN-instrumented build running on Python 3.14.Open questions
ctypesmodule in CPython isn't thread-safe in 3.13t. Should we worry about thectypesbackend on 3.13t?cc @kumaraditya303 @colesbury
@colesbury also wants to do another round of code review on the final version of this PR.