-
-
Notifications
You must be signed in to change notification settings - Fork 15.9k
python312Packages.granian: don't force a custom allocator #414234
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
I'm currently testing diff --git a/pkgs/development/python-modules/granian/default.nix b/pkgs/development/python-modules/granian/default.nix
index 6c9b5ceba619..77b5a22cacc4 100644
--- a/pkgs/development/python-modules/granian/default.nix
+++ b/pkgs/development/python-modules/granian/default.nix
@@ -19,6 +19,17 @@
nix-update-script,
}:
+let
+ rust-jemalloc-sys' = rust-jemalloc-sys.override (old: {
+ jemalloc = old.jemalloc.override {
+ # "libjemalloc.so.2: cannot allocate memory in static TLS block"
+ disableInitExecTls = true;
+ };
+ });
+in
+
+assert builtins.elem "--disable-initial-exec-tls" rust-jemalloc-sys'.configureFlags;
+
buildPythonPackage rec {
pname = "granian";
version = "2.3.2";
@@ -41,20 +52,11 @@ buildPythonPackage rec {
maturinBuildHook
];
- buildInputs = lib.optionals (stdenv.hostPlatform.isAarch64) [
+ buildInputs = [
# fix "Unsupported system page size" on aarch64-linux with 16k pages
# https://github.com/NixOS/nixpkgs/issues/410572
- # only enabled on aarch64 due to https://github.com/NixOS/nixpkgs/pull/410611#issuecomment-2939564567
- (rust-jemalloc-sys.overrideAttrs (
- { configureFlags, ... }:
- {
- configureFlags = configureFlags ++ [
- # otherwise import check fails with:
- # ImportError: {{storeDir}}/lib/libjemalloc.so.2: cannot allocate memory in static TLS block
- "--disable-initial-exec-tls"
- ];
- }
- ))
+ # Another alternative may be to try `mimalloc`
+ rust-jemalloc-sys'
];
dependencies = [ but this is also a sound approach |
Honestly forcing a global allocator at library level will never be a good idea, for any reason. |
Granian maintainer here. I'm not a big fan of never/always takes. |
OK, let me rephrase that: replacing the global allocator from a library is exceptionally brittle as it effectively makes the behavior dependent on the load/constructor order. I understand the performance concerns here, but also, "it works on x86-manylinux", as you can see, is clearly not enough. |
I can't ensure that on 3rd party builds. I was talking about the official wheels produced by the project.
|
Also, not sure why the patch for disabling the initial tls exec was applied on Anyways, if the build process is too painful for nix, in 2.4 I can gate also (your patch here also drop mimalloc, but unless you explicitly set the feature cfg in the build process that won't be picked up) |
The problem isn't the build process, it's that Granian simply crashes if something uses the system allocator, then gets jemalloc'd. |
when trying to apply this patch on latest nixpks unstable i get:
|
Are you maybe trying to apply it to an older nixpkgs version? |
As I said
So I think you're not looking at the root cause for this, but rather trying to solve a side effect. IMHO the change @pbsds proposed actually tackle the real issue here.
When you launch whatever application with Granian, Granian itself is the entry point, thus having whatever allocator on its side should be perfectly fine. If you just don't want to deal with it, again, the only thing I can do to simplify downstream management is to gate the jemalloc thing behind a feature in 2.4. but for 2.3, correctly configuring jemalloc should just work. |
NixOS is a dsitro, hence we try to compile everything on our own. Otherwise security updates to libraries would be even more impossible as they are right now and distro specific patches to some software would be hard to apply. Also those links do not exist on NixOS but we could make it work with patchelf.
We already do that for jemalloc
Na, so far we can do everything, we just need to figure out what and where exactly 😂 |
Fixes #414214
Things done
nix.conf
? (See Nix manual)sandbox = relaxed
sandbox = true
nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD"
. Note: all changes have to be committed, also see nixpkgs-review usage./result/bin/
)Add a 👍 reaction to pull requests you find important.