A guide for using the MASDOC node on the CoW can be found here.

Building DUNE on Tinis

Building DUNE on the Tinis cluster can be challenging due to a compiler bug listed here. This bug breaks the build scripts for DUNE. The OpenMPI module is also unstable, so the correct modules to load (correct as of July 2017) are:

---

module load CMake

module load intel impi

---

To get around the compiler bug, place the following scripts in your working directory on Tinis, eg. /home/maths/[USERCODE]/bin, and make sure they are executable (directories correct July 2017):

---

Filename: g++

---

#!/bin/bash
# sleep after compile workaround for
# GCC 4.9.3 racetime condition bug

GCCPATH=/csc/tinis/software/Core/GCC/4.9.3-binutils-2.25/bin

set -e # return exit code from g++
$GCCPATH/g++ "$@"
sleep 5

---

---

Filename: gcc

---

#!/bin/bash
# sleep after compile workaround for
# GCC 4.9.3 racetime condition bug

GCCPATH=/csc/tinis/software/Core/GCC/4.9.3-binutils-2.25/bin

set -e # return exit code from gcc
$GCCPATH/gcc "$@"
sleep 5

---

---

Filename: gfortran

---

#!/bin/bash
# sleep after compile workaround for
# GCC 4.9.3 racetime condition bug

GCCPATH=/csc/tinis/software/Core/GCC/4.9.3-binutils-2.25/bin

set -e # return exit code from gfortran
$GCCPATH/gfortran "$@"
sleep 5

---

---

Filename: mpicc_sleep

---

#!/bin/bash
# sleep after compile workaround for
# GCC 4.9.3 racetime condition bug

MPIPATH=/csc/tinis/software/Compiler/intel/2015.5.223-GNU-4.9.3-2.25/impi/5.0.3.049/bin64

set -e # return exit code from mpicc
$MPIPATH/mpicc "$@"
sleep 5

---

---

Filename: mpicxx_sleep

---

#!/bin/bash
# sleep after compile workaround for
# GCC 4.9.3 racetime condition bug

MPIPATH=/csc/tinis/software/Compiler/intel/2015.5.223-GNU-4.9.3-2.25/impi/5.0.3.049/bin64

set -e # return exit code from mpicxx
$MPIPATH/mpicxx "$@"
sleep 5

---

Filename: mpif90_sleep

---

#!/bin/bash
# sleep after compile workaround for
# GCC 4.9.3 racetime condition bug

MPIPATH=/csc/tinis/software/Compiler/intel/2015.5.223-GNU-4.9.3-2.25/impi/5.0.3.049/bin64

set -e # return exit code from mpif90
$MPIPATH/mpif90 "$@"
sleep 5

---

Now you must point towards these scripts to be used as your compilers which can be done in the config.opts file. If you are using PETSc then you will also need to point to these scripts as your compilers when you are buidling PETSc. You may need to add the directory where you put these scripts to PATH:

---

export PATH=/home/maths/[USERCODE]/bin:$PATH

---

and use the configure options:

---

./configure --with-cc=/home/maths/[USERCODE]/bin/mpicc_sleep --with-cxx=/home/maths/[USERCODE]/bin/mpicxx_sleep --with-fc=/home/maths/[USERCODE]/bin/mpif90_sleep --prefix=/home/maths/[USERCODE]/petsc/petscBUILD --with-debugging=no --CFLAGS="-O3 -DNDEBUG -ffast-math" --CXXFLAGS="-O3 -DNDEBUG -ffast-math" --LDFLAGS="-L/usr/lib/ -L/usr/lib64/" --with-parmetis=1 --download-parmetis=yes --with-hypre=1 --download-hypre=yes --with-superlu=1 --download-superlu=1 --with-superlu_dist=1 --download-superlu_dist=yes --with-mumps=1 --download-mumps=yes --with-ml=1 --download-ml=yes --with-metis=1 --download-metis=yes --download-fblaslapack --download-scalapack=yes --download-blacs=yes --with-suitesparse=1 --download-suitesparse=yes

---

If you recieve an error relating to 64 bit libraries you may need to add them to LD_LIBRARY_PATH:

---

export LD_LIBRARY_PATH=/usr/lib:/usr/lib64:$LD_LIBRARY_PATH

---

Note that this fix pauses after each call to the compiler which can make the build process take several hours.