Exawind (GPU) Build Failure on Perlmutter

[ajpowelsnl]

• A "bare metal" build on Perlmutter fails on amr-wind (below).
• Nota bene: Trilinos-13.4.0 is being used, and is specified in packages.yaml. It is possible that this version (over a year old) is the problem. The current release is 14.4.0. In the 1+ years, there have been significant changes in Kokkos. The fact that the errors are undefined host / device code, suggests the possiblity of an outmoded Trilinos.
• I will attempt a (spack-manger) build with the current Trilinos

Error

==> Installing amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx
==> No binary for amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx found: installing from source
==> No patches needed for amr-wind
==> amr-wind: Executing phase: 'cmake'
==> amr-wind: Executing phase: 'build'
==> Error: ProcessError: Command exited with status 2:
    '/pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/gmake-4.4.1-f2rv7vhqnyerq2ch37skbefh36avh7if/bin/make' '-j20'

23 errors found in build log:
     664    make[2]: Leaving directory '/pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0c
            b6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-build-ddqsi3o'
     665    [ 47%] Built target fcompare
     666    [ 47%] Building CUDA object CMakeFiles/amrwind_obj.dir/amr-wind/convection/incflo_godunov_ppm.cpp.o
     667    /opt/nvidia/hpc_sdk/Linux_x86_64/22.7/cuda/11.7/bin/nvcc -forward-unknown-to-host-compiler -ccbin=/pscratch/sd/a/ajpowel/gpu_containe
            r_spack_man/spack-manager/spack/lib/spack/env/case-insensitive/CC -DAMREX_SPACEDIM=3 -DAMR_WIND_USE_NETCDF -I/pscratch/sd/a/ajpowel/g
            pu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjama
            w3bolfshinl6olf6ghx/spack-src -I/pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1d
            cc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-build-ddqsi3o -I/pscratch/sd/a/ajpowel/gpu_containe
            r_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl
            6olf6ghx/spack-src/submods/amrex/Tools/C_scripts -isystem=/pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/snapshots/exaw
            ind/perlmutter/2023-10-09/opt/linux-sles15-zen3/gcc-11.2.0/netcdf-c-4.7.4-xaosb7jql5cpbuztkhxtfzykdfe5js3l/include -isystem=/pscratch
            /sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphas
            e-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/submods/amrex/Src/Base -isystem=/pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-man
            ager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/su
            bmods/amrex/Src/Base/Parser -isystem=/pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb
            12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/submods/amrex/Src/Boundary -isystem=/pscra
            tch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multip
            hase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/submods/amrex/Src/AmrCore -isystem=/pscratch/sd/a/ajpowel/gpu_container_spack_man/spa
            ck-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-
            src/submods/amrex/Src/LinearSolvers/MLMG -isystem=/pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-
            wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/submods/amrex/Src/LinearSolve
            rs/OpenBC -isystem=/pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b
            9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/submods/amrex/Src/Particle -isystem=/pscratch/sd/a/ajpowel/g
            pu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjama
            w3bolfshinl6olf6ghx/spack-src/submods/amrex/Src/Extern/HYPRE -isystem=/pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/st
            age/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-build-ddqsi3o
            /submods/amrex -isystem=/pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c2
            8c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/submods/AMReX-Hydro/Utils -isystem=/pscratch/sd/a/ajpow
            el/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqx
            jamaw3bolfshinl6olf6ghx/spack-src/submods/AMReX-Hydro/MOL -isystem=/pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage
            /spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/submods/AMRe
            X-Hydro/Godunov -isystem=/pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c
            28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/submods/AMReX-Hydro/BDS -isystem=/pscratch/sd/a/ajpowe
            l/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxj
            amaw3bolfshinl6olf6ghx/spack-src/submods/AMReX-Hydro/Projections -isystem=/opt/nvidia/hpc_sdk/Linux_x86_64/22.7/cuda/11.7/include -is
            ystem=/pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/snapshots/exawind/perlmutter/2023-10-09/opt/linux-sles15-zen3/gcc-
            11.2.0/hypre-git.fc49a5ec6b31763dbf0b1f25c5e559859204aa91=develop-njcszkyrclumtmknksx2zy545sfdoygt/include -O3 -DNDEBUG --generate-co
            de=arch=compute_80,code=[compute_80,sm_80] -Xcompiler=-fPIC --expt-relaxed-constexpr --expt-extended-lambda -Xcudafe --diag_suppress=
            esa_on_defaulted_function_ignored -Xcudafe --diag_suppress=implicit_return_from_non_void_function -maxrregcount=255 -Xcudafe --displa
            y_error_number --Wext-lambda-captures-this --use_fast_math --generate-line-info -std=c++17 -MD -MT CMakeFiles/amrwind_obj.dir/amr-win
            d/convection/incflo_godunov_ppm.cpp.o -MF CMakeFiles/amrwind_obj.dir/amr-wind/convection/incflo_godunov_ppm.cpp.o.d -x cu -rdc=true -
            c /pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff
            4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/amr-wind/convection/incflo_godunov_ppm.cpp -o CMakeFiles/amrwind_obj.dir/amr-
            wind/convection/incflo_godunov_ppm.cpp.o
     668    [ 47%] Building CUDA object CMakeFiles/amrwind_obj.dir/amr-wind/convection/incflo_godunov_predict.cpp.o
     669    /opt/nvidia/hpc_sdk/Linux_x86_64/22.7/cuda/11.7/bin/nvcc -forward-unknown-to-host-compiler -ccbin=/pscratch/sd/a/ajpowel/gpu_containe
            r_spack_man/spack-manager/spack/lib/spack/env/case-insensitive/CC -DAMREX_SPACEDIM=3 -DAMR_WIND_USE_NETCDF -I/pscratch/sd/a/ajpowel/g
            pu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjama
            w3bolfshinl6olf6ghx/spack-src -I/pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1d
            cc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-build-ddqsi3o -I/pscratch/sd/a/ajpowel/gpu_containe
            r_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl
            6olf6ghx/spack-src/submods/amrex/Tools/C_scripts -isystem=/pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/snapshots/exaw
            ind/perlmutter/2023-10-09/opt/linux-sles15-zen3/gcc-11.2.0/netcdf-c-4.7.4-xaosb7jql5cpbuztkhxtfzykdfe5js3l/include -isystem=/pscratch
            /sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphas
            e-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/submods/amrex/Src/Base -isystem=/pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-man
            ager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/su
            bmods/amrex/Src/Base/Parser -isystem=/pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb
            12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/submods/amrex/Src/Boundary -isystem=/pscra
            tch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multip
            hase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/submods/amrex/Src/AmrCore -isystem=/pscratch/sd/a/ajpowel/gpu_container_spack_man/spa
            ck-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-
            src/submods/amrex/Src/LinearSolvers/MLMG -isystem=/pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-
            wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/submods/amrex/Src/LinearSolve
            rs/OpenBC -isystem=/pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b
            9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/submods/amrex/Src/Particle -isystem=/pscratch/sd/a/ajpowel/g
            pu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjama
            w3bolfshinl6olf6ghx/spack-src/submods/amrex/Src/Extern/HYPRE -isystem=/pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/st
            age/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-build-ddqsi3o
            /submods/amrex -isystem=/pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c2
            8c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/submods/AMReX-Hydro/Utils -isystem=/pscratch/sd/a/ajpow
            el/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqx
            jamaw3bolfshinl6olf6ghx/spack-src/submods/AMReX-Hydro/MOL -isystem=/pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage
            /spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/submods/AMRe
            X-Hydro/Godunov -isystem=/pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c
            28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/submods/AMReX-Hydro/BDS -isystem=/pscratch/sd/a/ajpowe
            l/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxj
            amaw3bolfshinl6olf6ghx/spack-src/submods/AMReX-Hydro/Projections -isystem=/opt/nvidia/hpc_sdk/Linux_x86_64/22.7/cuda/11.7/include -is
            ystem=/pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/snapshots/exawind/perlmutter/2023-10-09/opt/linux-sles15-zen3/gcc-
            11.2.0/hypre-git.fc49a5ec6b31763dbf0b1f25c5e559859204aa91=develop-njcszkyrclumtmknksx2zy545sfdoygt/include -O3 -DNDEBUG --generate-co
            de=arch=compute_80,code=[compute_80,sm_80] -Xcompiler=-fPIC --expt-relaxed-constexpr --expt-extended-lambda -Xcudafe --diag_suppress=
            esa_on_defaulted_function_ignored -Xcudafe --diag_suppress=implicit_return_from_non_void_function -maxrregcount=255 -Xcudafe --displa
            y_error_number --Wext-lambda-captures-this --use_fast_math --generate-line-info -std=c++17 -MD -MT CMakeFiles/amrwind_obj.dir/amr-win
            d/convection/incflo_godunov_predict.cpp.o -MF CMakeFiles/amrwind_obj.dir/amr-wind/convection/incflo_godunov_predict.cpp.o.d -x cu -rd
            c=true -c /pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb0
            8cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/amr-wind/convection/incflo_godunov_predict.cpp -o CMakeFiles/amrwind_
            obj.dir/amr-wind/convection/incflo_godunov_predict.cpp.o
  >> 670    /pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=
            multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/amr-wind/overset/sharpen_nalu_data.H(48): error: calling a __host__ function("a
            mr_wind::overset::asdf(double, double)") from a __device__ function("amr_wind::overset::populate_psi( ::amrex::MultiFab &,  ::amrex::
            MultiFab &, double)::[lambda(int, int, int) (instance 1)]::operator () const") is not allowed
     671    
  >> 672    /pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=
            multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/amr-wind/overset/sharpen_nalu_data.H(48): error: identifier "amr_wind::overset:
            :asdf" is undefined in device code
     673    
  >> 674    /pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=
            multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/amr-wind/overset/sharpen_nalu_data.H(291): error: calling a __host__ function("
            amr_wind::overset::alpha_discrete_flux(int, int, int, int, double, const  ::amrex::Array4<const double>  &, const  ::amrex::Array4<co
            nst double>  &, const  ::amrex::Array4<const double>  &)") from a __device__ function("amr_wind::overset::populate_sharpen_discrete_f
            luxes( ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFa
            b &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &, double, double, double)::[lambda(int, int, int) (instance 1)]:
            :operator () const") is not allowed
     675    
  >> 676    /pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=
            multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/amr-wind/overset/sharpen_nalu_data.H(291): error: identifier "amr_wind::overset
            ::alpha_discrete_flux" is undefined in device code
     677    
  >> 678    /pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=
            multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/amr-wind/overset/sharpen_nalu_data.H(299): error: calling a __host__ function("
            amr_wind::overset::velocity_discrete_face(int, int, int, int, const  ::amrex::Array4<const double>  &, const  ::amrex::Array4<const d
            ouble>  &, double &, double &, double &)") from a __device__ function("amr_wind::overset::populate_sharpen_discrete_fluxes( ::amrex::
            MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::M
            ultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &, double, double, double)::[lambda(int, int, int) (instance 1)]::operator () con
            st") is not allowed
     679    
  >> 680    /pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=
            multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/amr-wind/overset/sharpen_nalu_data.H(299): error: identifier "amr_wind::overset
            ::velocity_discrete_face" is undefined in device code
     681    
  >> 682    /pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=
            multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/amr-wind/overset/sharpen_nalu_data.H(303): error: calling a __host__ function("
            amr_wind::overset::gp_rho_discrete_face(int, int, int, int, const  ::amrex::Array4<const double>  &, const  ::amrex::Array4<const dou
            ble>  &, const  ::amrex::Array4<const double>  &, double &, double &, double &)") from a __device__ function("amr_wind::overset::popu
            late_sharpen_discrete_fluxes( ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiF
            ab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &, double, double, double)::[lambda(int, i
            nt, int) (instance 1)]::operator () const") is not allowed
     683    
  >> 684    /pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=
            multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/amr-wind/overset/sharpen_nalu_data.H(303): error: identifier "amr_wind::overset
            ::gp_rho_discrete_face" is undefined in device code
     685    
  >> 686    /pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=
            multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/amr-wind/overset/sharpen_nalu_data.H(309): error: calling a __host__ function("
            amr_wind::overset::alpha_discrete_flux(int, int, int, int, double, const  ::amrex::Array4<const double>  &, const  ::amrex::Array4<co
            nst double>  &, const  ::amrex::Array4<const double>  &)") from a __device__ function("amr_wind::overset::populate_sharpen_discrete_f
            luxes( ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFa
            b &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &, double, double, double)::[lambda(int, int, int) (instance 2)]:
            :operator () const") is not allowed
     687    
  >> 688    /pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=
            multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/amr-wind/overset/sharpen_nalu_data.H(309): error: identifier "amr_wind::overset
            ::alpha_discrete_flux" is undefined in device code
     689    
  >> 690    /pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=
            multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/amr-wind/overset/sharpen_nalu_data.H(315): error: calling a __host__ function("
            amr_wind::overset::velocity_discrete_face(int, int, int, int, const  ::amrex::Array4<const double>  &, const  ::amrex::Array4<const d
            ouble>  &, double &, double &, double &)") from a __device__ function("amr_wind::overset::populate_sharpen_discrete_fluxes( ::amrex::
            MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::M
            ultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &, double, double, double)::[lambda(int, int, int) (instance 2)]::operator () con
            st") is not allowed
     691    
  >> 692    /pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=
            multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/amr-wind/overset/sharpen_nalu_data.H(315): error: identifier "amr_wind::overset
            ::velocity_discrete_face" is undefined in device code
     693    
  >> 694    /pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=
            multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/amr-wind/overset/sharpen_nalu_data.H(319): error: calling a __host__ function("
            amr_wind::overset::gp_rho_discrete_face(int, int, int, int, const  ::amrex::Array4<const double>  &, const  ::amrex::Array4<const dou
            ble>  &, const  ::amrex::Array4<const double>  &, double &, double &, double &)") from a __device__ function("amr_wind::overset::popu
            late_sharpen_discrete_fluxes( ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiF
            ab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &, double, double, double)::[lambda(int, i
            nt, int) (instance 2)]::operator () const") is not allowed
     695    
  >> 696    /pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=
            multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/amr-wind/overset/sharpen_nalu_data.H(319): error: identifier "amr_wind::overset
            ::gp_rho_discrete_face" is undefined in device code
     697    
  >> 698    /pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=
            multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/amr-wind/overset/sharpen_nalu_data.H(325): error: calling a __host__ function("
            amr_wind::overset::alpha_discrete_flux(int, int, int, int, double, const  ::amrex::Array4<const double>  &, const  ::amrex::Array4<co
            nst double>  &, const  ::amrex::Array4<const double>  &)") from a __device__ function("amr_wind::overset::populate_sharpen_discrete_f
            luxes( ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFa
            b &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &, double, double, double)::[lambda(int, int, int) (instance 3)]:
            :operator () const") is not allowed
     699    
  >> 700    /pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=
            multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/amr-wind/overset/sharpen_nalu_data.H(325): error: identifier "amr_wind::overset
            ::alpha_discrete_flux" is undefined in device code
     701    
  >> 702    /pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=
            multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/amr-wind/overset/sharpen_nalu_data.H(331): error: calling a __host__ function("
            amr_wind::overset::velocity_discrete_face(int, int, int, int, const  ::amrex::Array4<const double>  &, const  ::amrex::Array4<const d
            ouble>  &, double &, double &, double &)") from a __device__ function("amr_wind::overset::populate_sharpen_discrete_fluxes( ::amrex::
            MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::M
            ultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &, double, double, double)::[lambda(int, int, int) (instance 3)]::operator () con
            st") is not allowed
     703    
  >> 704    /pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=
            multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/amr-wind/overset/sharpen_nalu_data.H(331): error: identifier "amr_wind::overset
            ::velocity_discrete_face" is undefined in device code
     705    
  >> 706    /pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=
            multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/amr-wind/overset/sharpen_nalu_data.H(335): error: calling a __host__ function("
            amr_wind::overset::gp_rho_discrete_face(int, int, int, int, const  ::amrex::Array4<const double>  &, const  ::amrex::Array4<const dou
            ble>  &, const  ::amrex::Array4<const double>  &, double &, double &, double &)") from a __device__ function("amr_wind::overset::popu
            late_sharpen_discrete_fluxes( ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiF
            ab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &, double, double, double)::[lambda(int, i
            nt, int) (instance 3)]::operator () const") is not allowed
     707    
  >> 708    /pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=
            multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/amr-wind/overset/sharpen_nalu_data.H(335): error: identifier "amr_wind::overset
            ::gp_rho_discrete_face" is undefined in device code
     709    
     710    20 errors detected in the compilation of "/pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git
            .32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/amr-wind/incflo_advance.cpp".
  >> 711    make[2]: *** [CMakeFiles/amrwind_obj.dir/build.make:79: CMakeFiles/amrwind_obj.dir/amr-wind/incflo_advance.cpp.o] Error 1
     712    make[2]: *** Waiting for unfinished jobs....
     713    make[2]: Leaving directory '/pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0c
            b6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-build-ddqsi3o'
  >> 714    make[1]: *** [CMakeFiles/Makefile2:1583: CMakeFiles/amrwind_obj.dir/all] Error 2
     715    make[1]: Leaving directory '/pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0c
            b6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-build-ddqsi3o'
  >> 716    make: *** [Makefile:149: all] Error 2

See build log for details:
  /pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-build-out.txt

==> Warning: Skipping build of exawind-git.d3c1aa4656fc3c6eccaec8c684671c82a3895172=multiphase-dgi3536dajqp7rakx4qgw4mxegb62l26 since amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx failed

...

==> nalu-wind: Successfully installed nalu-wind-git.90b42cad6ab7f1cfce82ac7b04d51cec89dae42e=multiphase-6vr3qakbxhidbkh6fstefpql3tu2frnk
  Stage: 16.72s.  Cmake: 16.27s.  Build: 17m 28.03s.  Install: 9.06s.  Post-install: 1.47s.  Total: 18m 12.42s
[+] /pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/snapshots/exawind/perlmutter/2023-10-09/opt/linux-sles15-zen3/gcc-11.2.0/nalu-wind-git.90b42cad6ab7f1cfce82ac7b04d51cec89dae42e=multiphase-6vr3qakbxhidbkh6fstefpql3tu2frnk
==> Error: exawind-git.d3c1aa4656fc3c6eccaec8c684671c82a3895172=multiphase-dgi3536dajqp7rakx4qgw4mxegb62l26: Package was not installed
==> Updating view at /pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/snapshots/exawind/perlmutter/2023-10-09/.spack-env/view
==> Warning: Skipping external package: cray-mpich@8.1.25%gcc@11.2.0+wrappers build_system=generic arch=linux-sles15-zen3/sy43dsx
==> Warning: Skipping external package: cuda@11.7%gcc@11.2.0~allow-unsupported-compilers~dev build_system=generic arch=linux-sles15-zen3/r2wfbez
==> Warning: Skipping external package: cmake@3.24.3%gcc@11.2.0~doc+ncurses+ownlibs~qt build_system=generic build_type=Release arch=linux-sles15-zen3/4ukgdvk
==> Error: Installation request failed.  Refer to reported errors for failing package(s).

real	89m32.385s

Reproducer

git clone --recursive spack-manager git@github.com:sandialabs/spack-manager.git
cd spack-manager
export SPACK_MANAGER=${PWD)
source start.sh && spack-start
./scripts/create-exawind-snapshot.sh

[ajpowelsnl]

Update: the containers (#573, #575) fix this problem by either 1) applying patches to pinned Spack version (HEAD detached at ee68baf254) or 2) making failing (TPL dependency) packages part of the base container.

Using a more recent version of Spack would likely fix most / all of the failures reported here.

[psakievich]

Thanks @ajpowelsnl. We'll try to update spack soon. It is old. We also need to update trilinos and cut a release of the stack. Hoping to get spack and a release done before the winter shutdown. Updating trilinos is something I'm not sure about. I will be working on that in the coming weeks though.