Installation of SAMRAI

Installation of SAMRAI

Structured Adaptive Mesh Refinement Application Infrastructure (SAMRAI) is an Adaptive Mesh Refinement (AMR) library from the Lawrence Livermore National Laboratory (LLNL). HAMeRS mainly uses the SAMRAI library for the dynamic load balancing of cells at different refinement levels, communication of data through MPI, and storage of restart data and visualization data in the .samrai format. The visualization data can be opened by either the distributed, parallel visualization tool VisIt by LLNL or the Flow Physics and Aeroacoustics Toolbox with Python (FloATPy) of Flow Physics and Aeroacoustics Laboratory (FPAL) at the Department of Aeronautics and Astronautics of Stanford University.

To compile SAMRAI, Boost library has to be installed first. Also, to output the restart and visualization files, HDF5 library is required.

Compilation of SAMRAI

To compile SAMRAI on a machine/cluster, we first need to make sure that Boost and HDF5 libraries are already installed on the machine/cluster. Then, we need to set the environment paths for the Boost and HDF5 libraries:

export BOOST_ROOT=<path to the directory of Boost>
export HDF5_ROOT=<path to the directory of HDF5>

After that, we have to download the SAMRAI library and uncompress it at a temporary location. For example, if the SAMRAI version 3.11.2 is chosen:

wget https://computation.llnl.gov/projects/samrai/download/SAMRAI-v3.11.2.tar.gz
gunzip SAMRAI-v3.11.2.tar.gz
mkdir SAMRAI-v3.11.2
cd SAMRAI-v3.11.2
tar xvf ../SAMRAI-v3.11.2.tar
mkdir objs
cd objs

Before installing the library, we also have to set up the location to install. For example, if we want to install the library at Codes/SAMRAI under the home directory:

export SAMRAI_ROOT=${HOME}/Codes/SAMRAI

To compile with MPI compilers and HDF5 support, we need the following configurations:

sh ../SAMRAI/configure --prefix=$SAMRAI_ROOT --enable-opt --with-CXX=mpicxx --with-CC=mpicc --with-F77=mpif77 \
  --with-boost=$BOOST_ROOT --with-hdf5=$HDF5_ROOT

Finally, to build and install the library, we can follow the steps below:

gmake library
gmake install

Compilation of SAMRAI on Knights Landing (KNL) Clusters

To compile SAMRAI on the KNL cluster, it's better to also set the architecture flags. What we only need to do is to change the configuration step with additional flags:

sh ../SAMRAI/configure --prefix=$SAMRAI_ROOT --enable-opt --with-CXX=mpiicpc --with-CC=mpiicc --with-F77=mpiifort \
  --with-boost=$BOOST_ROOT --with-hdf5=$HDF5_ROOT CFLAGS='-xCORE-AVX2 -axCORE-AVX512,MIC-AVX512' \
  CPPFLAGS='-xCORE-AVX2 -axCORE-AVX512,MIC-AVX512'

For version of SAMRAI that uses CMake, please consider to use the follow command:

cmake .. -DCMAKE_INSTALL_PREFIX=$SAMRAI_ROOT -DCMAKE_CXX_COMPILER=mpiicpc -DCMAKE_C_COMPILER=mpiicc \
  -DCMAKE_Fortran_COMPILER=mpiifort -DCMAKE_C_FLAGS='-xCORE-AVX2 -axCORE-AVX512,MIC-AVX512' \
  -DCMAKE_CXX_FLAGS='-xCORE-AVX2 -axCORE-AVX512,MIC-AVX512'