Skip to content

pip install from source fails due to internal compiler error with gcc 13.2.0 (only on x86-64) #1665

Open
@BenWibking

Description

@BenWibking

Performed steps
Building this Dockerfile (https://github.com/parthenon-hpc-lab/parthenon/blob/ebe3f9bc020073aa262045233b1c17ce04a2d759/scripts/docker/Dockerfile.nvcc) with

docker build --file Dockerfile.nvcc --tag parthenon-hpc-lab/new-ci-image --platform linux/amd64 .

led to the following output/problem during install

15.80 Building wheels for collected packages: openpmd-api, mpi4py
15.80   Building wheel for openpmd-api (pyproject.toml): started
87.63   Building wheel for openpmd-api (pyproject.toml): still running...
148.0   Building wheel for openpmd-api (pyproject.toml): still running...
208.6   Building wheel for openpmd-api (pyproject.toml): still running...
237.0   Building wheel for openpmd-api (pyproject.toml): finished with status 'error'
237.0   error: subprocess-exited-with-error
237.0
237.0   × Building wheel for openpmd-api (pyproject.toml) did not run successfully.
237.0   │ exit code: 1
237.0   ╰─> [241 lines of output]
237.0       running bdist_wheel
237.0       running build
237.0       running build_ext
237.0       <string>:28: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
237.0       -- The C compiler identification is GNU 13.2.0
237.0       -- The CXX compiler identification is GNU 13.2.0
237.0       -- Detecting C compiler ABI info
237.0       -- Detecting C compiler ABI info - done
237.0       -- Check for working C compiler: /usr/bin/cc - skipped
237.0       -- Detecting C compile features
237.0       -- Detecting C compile features - done
237.0       -- Detecting CXX compiler ABI info
237.0       -- Detecting CXX compiler ABI info - done
237.0       -- Check for working CXX compiler: /usr/bin/c++ - skipped
237.0       -- Detecting CXX compile features
237.0       -- Detecting CXX compile features - done
237.0       -- Found MPI_CXX: /opt/openmpi/lib/libmpi.so (found version "3.1")
237.0       -- Found MPI: TRUE (found version "3.1") found components: CXX
237.0       CMake Deprecation Warning at share/openPMD/thirdParty/json/CMakeLists.txt:1 (cmake_minimum_required):
237.0         Compatibility with CMake < 3.5 will be removed from a future version of
237.0         CMake.
237.0
237.0         Update the VERSION argument <min> value or use a ...<max> suffix to tell
237.0         CMake that the project does not need compatibility with older versions.
237.0
237.0
237.0       -- Using the single-header code from /tmp/pip-install-ta16014x/openpmd-api_9c05105b422c4f5fab8485e5103983fa/share/openPMD/thirdParty/json/single_include/
237.0       -- nlohmann-json: Using INTERNAL version '3.9.1'
237.0       CMake Deprecation Warning at share/openPMD/thirdParty/toml11/CMakeLists.txt:1 (cmake_minimum_required):
237.0         Compatibility with CMake < 3.5 will be removed from a future version of
237.0         CMake.
237.0
237.0         Update the VERSION argument <min> value or use a ...<max> suffix to tell
237.0         CMake that the project does not need compatibility with older versions.
237.0
237.0
237.0       -- toml11: Using INTERNAL version '3.7.1'
237.0       -- Could NOT find HDF5 (missing: HDF5_LIBRARIES HDF5_INCLUDE_DIRS C) (found version "")
237.0       CMake Deprecation Warning at share/openPMD/cmake/FindADIOS.cmake:91 (cmake_minimum_required):
237.0         Compatibility with CMake < 3.5 will be removed from a future version of
237.0         CMake.
237.0
237.0         Update the VERSION argument <min> value or use a ...<max> suffix to tell
237.0         CMake that the project does not need compatibility with older versions.
237.0       Call Stack (most recent call first):
237.0         CMakeLists.txt:384 (find_package)
237.0
237.0
237.0       -- Found MPI_C: /opt/openmpi/lib/libmpi.so (found version "3.1")
237.0       -- Found MPI: TRUE (found version "3.1")
237.0       -- Found ADIOS2: /usr/local/lib/cmake/adios2/adios2-config.cmake (found suitable version "2.10.0.393", minimum required is "2.7.0") found components: C CXX MPI
237.0       -- Found Python: /usr/bin/python3 (found version "3.12.3") found components: Interpreter Development.Module
237.0       CMake Deprecation Warning at share/openPMD/thirdParty/pybind11/CMakeLists.txt:8 (cmake_minimum_required):
237.0         Compatibility with CMake < 3.5 will be removed from a future version of
237.0         CMake.
237.0
237.0         Update the VERSION argument <min> value or use a ...<max> suffix to tell
237.0         CMake that the project does not need compatibility with older versions.
237.0
237.0
237.0       -- pybind11 v2.10.1
237.0       -- Performing Test HAS_FLTO
237.0       -- Performing Test HAS_FLTO - Success
237.0       -- pybind11: Using INTERNAL version 2.10.1
237.0       -- Python LTO/IPO: ON
237.0
237.0       openPMD build configuration:
237.0         library Version: 0.15.2
237.0         openPMD Standard: 1.1.0
237.0         C++ Compiler: GNU 13.2.0
237.0           /usr/bin/c++
237.0
237.0         Install with RPATHs: ON
237.0         Installation prefix: /usr/local
237.0               bin: bin
237.0               lib: lib
237.0           include: include
237.0             cmake: lib/cmake/openPMD
237.0            python: lib/python3.12/site-packages
237.0
237.0         Build Type: Release
237.0         Library: static
237.0         CLI Tools: OFF
237.0         Examples: OFF
237.0         Testing: OFF
237.0         Invasive Tests: OFF
237.0         Internal VERIFY: ON
237.0         Build Options:
237.0           MPI: ON
237.0           HDF5: OFF
237.0           ADIOS1: OFF
237.0           ADIOS2: ON
237.0           PYTHON: ON
237.0           CUDA_EXAMPLES: OFF
237.0
237.0       -- Configuring done (6.3s)
237.0       -- Generating done (0.0s)
237.0       -- Build files have been written to: /tmp/pip-install-ta16014x/openpmd-api_9c05105b422c4f5fab8485e5103983fa/build/temp.linux-x86_64-cpython-312
237.0       [  1%] Building CXX object CMakeFiles/openPMD.dir/src/config.cpp.o
237.0       [  2%] Building CXX object CMakeFiles/openPMD.dir/src/ChunkInfo.cpp.o
237.0       [  4%] Building CXX object CMakeFiles/openPMD.dir/src/Dataset.cpp.o
237.0       [  5%] Building CXX object CMakeFiles/openPMD.dir/src/Datatype.cpp.o
237.0       [  6%] Building CXX object CMakeFiles/openPMD.dir/src/Error.cpp.o
237.0       [  8%] Building CXX object CMakeFiles/openPMD.dir/src/Format.cpp.o
237.0       [  9%] Building CXX object CMakeFiles/openPMD.dir/src/Iteration.cpp.o
237.0       [ 10%] Building CXX object CMakeFiles/openPMD.dir/src/IterationEncoding.cpp.o
237.0       [ 12%] Building CXX object CMakeFiles/openPMD.dir/src/Mesh.cpp.o
237.0       [ 13%] Building CXX object CMakeFiles/openPMD.dir/src/ParticlePatches.cpp.o
237.0       [ 15%] Building CXX object CMakeFiles/openPMD.dir/src/ParticleSpecies.cpp.o
237.0       [ 16%] Building CXX object CMakeFiles/openPMD.dir/src/ReadIterations.cpp.o
237.0       [ 17%] Building CXX object CMakeFiles/openPMD.dir/src/Record.cpp.o
237.0       [ 19%] Building CXX object CMakeFiles/openPMD.dir/src/RecordComponent.cpp.o
237.0       [ 20%] Building CXX object CMakeFiles/openPMD.dir/src/Series.cpp.o
237.0       [ 21%] Building CXX object CMakeFiles/openPMD.dir/src/version.cpp.o
237.0       [ 23%] Building CXX object CMakeFiles/openPMD.dir/src/WriteIterations.cpp.o
237.0       [ 24%] Building CXX object CMakeFiles/openPMD.dir/src/auxiliary/Date.cpp.o
237.0       [ 26%] Building CXX object CMakeFiles/openPMD.dir/src/auxiliary/Filesystem.cpp.o
237.0       [ 27%] Building CXX object CMakeFiles/openPMD.dir/src/auxiliary/JSON.cpp.o
237.0       [ 28%] Building CXX object CMakeFiles/openPMD.dir/src/backend/Attributable.cpp.o
237.0       [ 30%] Building CXX object CMakeFiles/openPMD.dir/src/backend/BaseRecordComponent.cpp.o
237.0       [ 31%] Building CXX object CMakeFiles/openPMD.dir/src/backend/Container.cpp.o
237.0       [ 32%] Building CXX object CMakeFiles/openPMD.dir/src/backend/MeshRecordComponent.cpp.o
237.0       [ 34%] Building CXX object CMakeFiles/openPMD.dir/src/backend/PatchRecord.cpp.o
237.0       [ 35%] Building CXX object CMakeFiles/openPMD.dir/src/backend/PatchRecordComponent.cpp.o
237.0       [ 36%] Building CXX object CMakeFiles/openPMD.dir/src/backend/Writable.cpp.o
237.0       [ 38%] Building CXX object CMakeFiles/openPMD.dir/src/benchmark/mpi/OneDimensionalBlockSlicer.cpp.o
237.0       [ 39%] Building CXX object CMakeFiles/openPMD.dir/src/helper/list_series.cpp.o
237.0       [ 41%] Building CXX object CMakeFiles/openPMD.dir/src/IO/AbstractIOHandler.cpp.o
237.0       [ 42%] Building CXX object CMakeFiles/openPMD.dir/src/IO/AbstractIOHandlerImpl.cpp.o
237.0       [ 43%] Building CXX object CMakeFiles/openPMD.dir/src/IO/AbstractIOHandlerHelper.cpp.o
237.0       [ 45%] Building CXX object CMakeFiles/openPMD.dir/src/IO/DummyIOHandler.cpp.o
237.0       [ 46%] Building CXX object CMakeFiles/openPMD.dir/src/IO/IOTask.cpp.o
237.0       [ 47%] Building CXX object CMakeFiles/openPMD.dir/src/IO/FlushParams.cpp.o
237.0       [ 49%] Building CXX object CMakeFiles/openPMD.dir/src/IO/HDF5/HDF5IOHandler.cpp.o
237.0       [ 50%] Building CXX object CMakeFiles/openPMD.dir/src/IO/HDF5/ParallelHDF5IOHandler.cpp.o
237.0       [ 52%] Building CXX object CMakeFiles/openPMD.dir/src/IO/HDF5/HDF5Auxiliary.cpp.o
237.0       [ 53%] Building CXX object CMakeFiles/openPMD.dir/src/IO/JSON/JSONIOHandler.cpp.o
237.0       [ 54%] Building CXX object CMakeFiles/openPMD.dir/src/IO/JSON/JSONIOHandlerImpl.cpp.o
237.0       [ 56%] Building CXX object CMakeFiles/openPMD.dir/src/IO/JSON/JSONFilePosition.cpp.o
237.0       [ 57%] Building CXX object CMakeFiles/openPMD.dir/src/IO/ADIOS/ADIOS2IOHandler.cpp.o
237.0       [ 58%] Building CXX object CMakeFiles/openPMD.dir/src/IO/ADIOS/ADIOS2Auxiliary.cpp.o
237.0       [ 60%] Building CXX object CMakeFiles/openPMD.dir/src/IO/ADIOS/ADIOS2PreloadAttributes.cpp.o
237.0       [ 61%] Building CXX object CMakeFiles/openPMD.dir/src/IO/InvalidatableFile.cpp.o
237.0       [ 63%] Building CXX object CMakeFiles/openPMD.dir/src/IO/ADIOS/CommonADIOS1IOHandler.cpp.o
237.0       [ 64%] Building CXX object CMakeFiles/openPMD.dir/src/IO/ADIOS/ADIOS1IOHandler.cpp.o
237.0       [ 65%] Building CXX object CMakeFiles/openPMD.dir/src/IO/ADIOS/ParallelADIOS1IOHandler.cpp.o
237.0       [ 67%] Linking CXX static library lib/libopenPMD.a
237.0       [ 67%] Built target openPMD
237.0       [ 68%] Building CXX object CMakeFiles/openPMD.py.dir/src/binding/python/openPMD.cpp.o
237.0       [ 69%] Building CXX object CMakeFiles/openPMD.py.dir/src/binding/python/Access.cpp.o
237.0       [ 71%] Building CXX object CMakeFiles/openPMD.py.dir/src/binding/python/Attributable.cpp.o
237.0       [ 72%] Building CXX object CMakeFiles/openPMD.py.dir/src/binding/python/BaseRecord.cpp.o
237.0       [ 73%] Building CXX object CMakeFiles/openPMD.py.dir/src/binding/python/BaseRecordComponent.cpp.o
237.0       [ 75%] Building CXX object CMakeFiles/openPMD.py.dir/src/binding/python/ChunkInfo.cpp.o
237.0       [ 76%] Building CXX object CMakeFiles/openPMD.py.dir/src/binding/python/Container.cpp.o
237.0       [ 78%] Building CXX object CMakeFiles/openPMD.py.dir/src/binding/python/Dataset.cpp.o
237.0       [ 79%] Building CXX object CMakeFiles/openPMD.py.dir/src/binding/python/Datatype.cpp.o
237.0       [ 80%] Building CXX object CMakeFiles/openPMD.py.dir/src/binding/python/Error.cpp.o
237.0       [ 82%] Building CXX object CMakeFiles/openPMD.py.dir/src/binding/python/Helper.cpp.o
237.0       [ 83%] Building CXX object CMakeFiles/openPMD.py.dir/src/binding/python/Iteration.cpp.o
237.0       [ 84%] Building CXX object CMakeFiles/openPMD.py.dir/src/binding/python/IterationEncoding.cpp.o
237.0       [ 86%] Building CXX object CMakeFiles/openPMD.py.dir/src/binding/python/Mesh.cpp.o
237.0       [ 87%] Building CXX object CMakeFiles/openPMD.py.dir/src/binding/python/ParticlePatches.cpp.o
237.0       [ 89%] Building CXX object CMakeFiles/openPMD.py.dir/src/binding/python/ParticleSpecies.cpp.o
237.0       [ 90%] Building CXX object CMakeFiles/openPMD.py.dir/src/binding/python/PatchRecord.cpp.o
237.0       [ 91%] Building CXX object CMakeFiles/openPMD.py.dir/src/binding/python/PatchRecordComponent.cpp.o
237.0       [ 93%] Building CXX object CMakeFiles/openPMD.py.dir/src/binding/python/Record.cpp.o
237.0       [ 94%] Building CXX object CMakeFiles/openPMD.py.dir/src/binding/python/RecordComponent.cpp.o
237.0       [ 95%] Building CXX object CMakeFiles/openPMD.py.dir/src/binding/python/MeshRecordComponent.cpp.o
237.0       [ 97%] Building CXX object CMakeFiles/openPMD.py.dir/src/binding/python/Series.cpp.o
237.0       [ 98%] Building CXX object CMakeFiles/openPMD.py.dir/src/binding/python/UnitDimension.cpp.o
237.0       [100%] Linking CXX shared module /tmp/pip-install-ta16014x/openpmd-api_9c05105b422c4f5fab8485e5103983fa/build/lib.linux-x86_64-cpython-312/openpmd_api/openpmd_api_cxx.cpython-312-x86_64-linux-gnu.so
237.0       lto1: internal compiler error: in return_token, at opts-common.cc:2135
237.0       0x19bb73a internal_error(char const*, ...)
237.0           ???:0
237.0       0x683a32 fancy_abort(char const*, int, char const*)
237.0           ???:0
237.0       0x6ba314 lto_main()
237.0           ???:0
237.0       Please submit a full bug report, with preprocessed source (by using -freport-bug).
237.0       Please include the complete backtrace with any bug report.
237.0       See <file:///usr/share/doc/gcc-13/README.Bugs> for instructions.
237.0       lto-wrapper: fatal error: /usr/bin/c++ returned 1 exit status
237.0       compilation terminated.
237.0       /usr/bin/ld: error: lto-wrapper failed
237.0       collect2: error: ld returned 1 exit status
237.0       gmake[2]: *** [CMakeFiles/openPMD.py.dir/build.make:453: /tmp/pip-install-ta16014x/openpmd-api_9c05105b422c4f5fab8485e5103983fa/build/lib.linux-x86_64-cpython-312/openpmd_api/openpmd_api_cxx.cpython-312-x86_64-linux-gnu.so] Error 1
237.0       gmake[1]: *** [CMakeFiles/Makefile2:920: CMakeFiles/openPMD.py.dir/all] Error 2
237.0       gmake: *** [Makefile:146: all] Error 2
237.0       Traceback (most recent call last):
237.0         File "/usr/lib/python3/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
237.0           main()
237.0         File "/usr/lib/python3/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
237.0           json_out['return_val'] = hook(**hook_input['kwargs'])
237.0                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
237.0         File "/usr/lib/python3/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
237.0           return _build_backend().build_wheel(wheel_directory, config_settings,
237.0                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
237.0         File "/tmp/pip-build-env-al2hbpy9/overlay/local/lib/python3.12/dist-packages/setuptools/build_meta.py", line 421, in build_wheel
237.0           return self._build_with_temp_dir(
237.0                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
237.0         File "/tmp/pip-build-env-al2hbpy9/overlay/local/lib/python3.12/dist-packages/setuptools/build_meta.py", line 403, in _build_with_temp_dir
237.0           self.run_setup()
237.0         File "/tmp/pip-build-env-al2hbpy9/overlay/local/lib/python3.12/dist-packages/setuptools/build_meta.py", line 318, in run_setup
237.0           exec(code, locals())
237.0         File "<string>", line 170, in <module>
237.0         File "/tmp/pip-build-env-al2hbpy9/overlay/local/lib/python3.12/dist-packages/setuptools/__init__.py", line 117, in setup
237.0           return distutils.core.setup(**attrs)
237.0                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
237.0         File "/tmp/pip-build-env-al2hbpy9/overlay/local/lib/python3.12/dist-packages/setuptools/_distutils/core.py", line 184, in setup
237.0           return run_commands(dist)
237.0                  ^^^^^^^^^^^^^^^^^^
237.0         File "/tmp/pip-build-env-al2hbpy9/overlay/local/lib/python3.12/dist-packages/setuptools/_distutils/core.py", line 200, in run_commands
237.0           dist.run_commands()
237.0         File "/tmp/pip-build-env-al2hbpy9/overlay/local/lib/python3.12/dist-packages/setuptools/_distutils/dist.py", line 953, in run_commands
237.0           self.run_command(cmd)
237.0         File "/tmp/pip-build-env-al2hbpy9/overlay/local/lib/python3.12/dist-packages/setuptools/dist.py", line 950, in run_command
237.0           super().run_command(command)
237.0         File "/tmp/pip-build-env-al2hbpy9/overlay/local/lib/python3.12/dist-packages/setuptools/_distutils/dist.py", line 972, in run_command
237.0           cmd_obj.run()
237.0         File "/tmp/pip-build-env-al2hbpy9/overlay/local/lib/python3.12/dist-packages/setuptools/command/bdist_wheel.py", line 384, in run
237.0           self.run_command("build")
237.0         File "/tmp/pip-build-env-al2hbpy9/overlay/local/lib/python3.12/dist-packages/setuptools/_distutils/cmd.py", line 316, in run_command
237.0           self.distribution.run_command(command)
237.0         File "/tmp/pip-build-env-al2hbpy9/overlay/local/lib/python3.12/dist-packages/setuptools/dist.py", line 950, in run_command
237.0           super().run_command(command)
237.0         File "/tmp/pip-build-env-al2hbpy9/overlay/local/lib/python3.12/dist-packages/setuptools/_distutils/dist.py", line 972, in run_command
237.0           cmd_obj.run()
237.0         File "/tmp/pip-build-env-al2hbpy9/overlay/local/lib/python3.12/dist-packages/setuptools/_distutils/command/build.py", line 135, in run
237.0           self.run_command(cmd_name)
237.0         File "/tmp/pip-build-env-al2hbpy9/overlay/local/lib/python3.12/dist-packages/setuptools/_distutils/cmd.py", line 316, in run_command
237.0           self.distribution.run_command(command)
237.0         File "/tmp/pip-build-env-al2hbpy9/overlay/local/lib/python3.12/dist-packages/setuptools/dist.py", line 950, in run_command
237.0           super().run_command(command)
237.0         File "/tmp/pip-build-env-al2hbpy9/overlay/local/lib/python3.12/dist-packages/setuptools/_distutils/dist.py", line 972, in run_command
237.0           cmd_obj.run()
237.0         File "<string>", line 36, in run
237.0         File "<string>", line 115, in build_extension
237.0         File "/usr/lib/python3.12/subprocess.py", line 413, in check_call
237.0           raise CalledProcessError(retcode, cmd)
237.0       subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--config', 'Release', '--', '-j2']' returned non-zero exit status 2.
237.0       [end of output]
237.0
237.0   note: This error originates from a subprocess, and is likely not a problem with pip.
237.0   ERROR: Failed building wheel for openpmd-api

You can also add images such as screenshots :-)

System environment

  • operating system: Ubuntu 24.04 (see Dockerfile)
  • machine: Docker VM
  • name and version of Python implementation: see Dockerfile

Additional context
MPI is enabled in the Dockerfile:
https://github.com/parthenon-hpc-lab/parthenon/blob/ebe3f9bc020073aa262045233b1c17ce04a2d759/scripts/docker/Dockerfile.nvcc#L48

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions