Skip to content

Online graph partitioning using Scotch #1339

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 5 commits into
base: develop
Choose a base branch
from
Open

Online graph partitioning using Scotch #1339

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
3998949
Working with upto 24km global mesh
abishekg7 Apr 8, 2025
4e5564b
Now checking for zero connectivities. Works with limited area grid
abishekg7 Apr 9, 2025
0d626db
Modifying the logic for when Scotch partitioning is active
abishekg7 Jun 24, 2025
e19d74d
WIP: Makefile changes to enable SCOTCH
abishekg7 Jun 24, 2025
e75ea06
Adding compile-time link test for the Scotch library
abishekg7 Jun 24, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 44 additions & 0 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -759,6 +759,15 @@ endif
LIBS += $(NCLIB)
endif

ifneq "$(SCOTCH)" ""
SCOTCH_FCINCLUDES += -I$(SCOTCH)/src/include
SCOTCH_LIBS += -L$(SCOTCH)/lib -lscotch -lscotcherr
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It may depend on the version on Scotch, but when I installed v7.0.7, the header files are in .../include (rather than in .../src/include, and the libraries are in .../lib64 rather than in .../lib.

SCOTCH_FFLAGS = -DMPAS_SCOTCH

FCINCLUDES += $(SCOTCH_FCINCLUDES)
LIBS += $(SCOTCH_LIBS)
override CPPFLAGS += $(SCOTCH_FFLAGS)
endif

ifneq "$(PNETCDF)" ""
ifneq ($(wildcard $(PNETCDF)/lib/libpnetcdf.*), )
Expand Down Expand Up @@ -1415,6 +1424,33 @@ musica_fortran_test:
$(eval MUSICA_FORTRAN_VERSION := $(shell pkg-config --modversion musica-fortran))
$(if $(findstring 1,$(MUSICA_FORTRAN_TEST)), $(info Built a simple test program with MUSICA-Fortran version $(MUSICA_FORTRAN_VERSION)), )

scotch_fortran_test:
@#
@# Create a Fortran test program that will link against the SCOTCH library
@#
$(info Checking for a working MUSICA-Fortran library...)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should "MUSICA-Fortran" be "SCOTCH"?

$(eval SCOTCH_FORTRAN_TEST := $(shell $\
printf "program test_scotch_fortran\n$\
& include \"scotchf.h\"\n$\
& doubleprecision :: scotchgraph (scotch_graphdim)\n$\
& integer :: ierr\n$\
& ierr = 0\n$\
& call scotchfgraphinit(scotchgraph (1), ierr)\n$\
& call scotchfgraphexit(scotchgraph(1))\n$\
end program test_scotch_fortran\n" | sed 's/&/ /' > test_scotch_fortran.f90; $\
$\
$(FC) $(SCOTCH_FCINCLUDES) $(SCOTCH_FFLAGS) test_scotch_fortran.f90 -o test_scotch_fortran.x $(SCOTCH_LIBS) > /dev/null 2>&1; $\
scotch_fortran_status=$$?; $\
rm -f test_scotch_fortran.f90 test_scotch_fortran.x; $\
if [ $$scotch_fortran_status -eq 0 ]; then $\
printf "1"; $\
else $\
printf "0"; $\
fi $\
))
$(if $(findstring 0,$(SCOTCH_FORTRAN_TEST)), $(error Could not build a simple test program with Scotch))
$(if $(findstring 1,$(SCOTCH_FORTRAN_TEST)), $(info Built a simple test program with Scotch ))

pnetcdf_test:
@#
@# Create test C programs that look for PNetCDF header file and some symbols in it
Expand Down Expand Up @@ -1471,6 +1507,13 @@ else
MUSICA_MESSAGE = "MPAS was not linked with the MUSICA-Fortran library."
endif

ifneq "$(SCOTCH_FFLAGS)" ""
MAIN_DEPS += scotch_fortran_test
SCOTCH_MESSAGE = "MPAS has been linked with the Scotch graph partitioning library."
else
SCOTCH_MESSAGE = "MPAS was NOT linked with the Scotch graph partitioning library."
endif

mpas_main: $(MAIN_DEPS)
cd src; $(MAKE) FC="$(FC)" \
CC="$(CC)" \
Expand Down Expand Up @@ -1508,6 +1551,7 @@ mpas_main: $(MAIN_DEPS)
@echo $(OPENMP_OFFLOAD_MESSAGE)
@echo $(OPENACC_MESSAGE)
@echo $(MUSICA_MESSAGE)
@echo $(SCOTCH_MESSAGE)
@echo $(SHAREDLIB_MESSAGE)
ifeq "$(AUTOCLEAN)" "true"
@echo $(AUTOCLEAN_MESSAGE)
Expand Down
2 changes: 1 addition & 1 deletion src/core_atmosphere/Registry.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -319,7 +319,7 @@
</nml_record>

<nml_record name="decomposition" in_defaults="true">
<nml_option name="config_block_decomp_file_prefix" type="character" default_value="x1.40962.graph.info.part."
<nml_option name="config_block_decomp_file_prefix" type="character" default_value=""
units="-"
description="Prefix of graph decomposition file, to be suffixed with the MPI task count"
possible_values="Any valid filename"/>
Expand Down
Loading