Building MFEM

A simple tutorial on how to build and run the serial and parallel versions of MFEM together with GLVis. For more details, see the INSTALL file and make help.

In addition to the native build system described below, MFEM packages are also available in the following package managers:

• Homebrew
• Spack
• OpenHPC

MFEM can also be installed as part of

• xSDK
• E4S
• FASTMath
• RADIUSS
• CEED

A pre-built version of MFEM is also available in a container form, see our AWS tutorial and the mfem/containers repo.

Instructions

Download MFEM and GLVis

• https://mfem.org
• https://glvis.org

Below we assume that we are working with versions mfem-4.5 and glvis-4.2.

Serial version of MFEM and GLVis

Put everything in the same directory:

~> ls
glvis-4.2.tgz   mfem-4.5.tgz

Build the serial version of MFEM:

~> tar -zxvf mfem-4.5.tgz
~> cd mfem-4.5
~/mfem-4.5> make serial -j
~/mfem-4.5> cd ..

Build GLVis:

~> tar -zxvf glvis-4.2.tgz
~> cd glvis-4.2
~/glvis-4.2> make MFEM_DIR=../mfem-4.5 -j
~/glvis-4.2> cd ..

That's it! The MFEM library can be found in mfem-4.5/libmfem.a, while the glvis executable will be in the glvis-4.2 directory.

Note: as of version 4.0, GLVis has additional dependencies that need to be installed first, see its building documentation.

To start a GLVis server, open a new terminal and type

~> cd glvis-4.2
~/glvis-4.2> ./glvis

The serial examples can be built with:

~> cd mfem-4.5/examples
~/mfem-4.5/examples> make -j

All serial examples and miniapps can be built with:

~> cd mfem-4.5
~/mfem-4.5> make all -j

Parallel MPI version of MFEM

Download hypre and METIS from

• https://github.com/hypre-space/hypre/tags
• https://github.com/mfem/tpls

Note: We recommend MFEM's mirror of metis-4.0.3 and metis-5.1.0 above because the METIS webpage, is often down and we don't support yet the new GitHub repo.

Below we assume that we are working with hypre-2.26.0 and metis-4.0.3 (see below for METIS version 5 and later). We also assume that the serial version of MFEM and GLVis have been built as described above.

Put everything in the same directory:

~> ls
glvis-4.2/  hypre-2.26.0.tar.gz   metis-4.0.3.tar.gz   mfem-4.5/

Build hypre:

~> tar -zxvf hypre-2.26.0.tar.gz
~> cd hypre-2.26.0/src/
~/hypre-2.26.0/src> ./configure --disable-fortran
~/hypre-2.26.0/src> make -j
~/hypre-2.26.0/src> cd ../..
~> ln -s hypre-2.26.0 hypre

Build METIS:

~> tar -zxvf metis-4.0.3.tar.gz
~> cd metis-4.0.3
~/metis-4.0.3> make OPTFLAGS=-Wno-error=implicit-function-declaration
~/metis-4.0.3> cd ..
~> ln -s metis-4.0.3 metis-4.0

(If you are using METIS 5, see the instructions below.)

Build the parallel version of MFEM:

~> cd mfem-4.5
~/mfem-4.5> make parallel -j
~/mfem-4.5> cd ..

Note that if hypre or METIS are in different locations, or you have different versions of these libraries, you will need to update the corresponding paths in the config/defaults.mk file, or create you own config/user.mk, as described in the INSTALL file.

The parallel examples can be built with:

~> cd mfem-4.5/examples
~/mfem-4.5/examples> make -j

The serial examples can also be built with the parallel version of the library, e.g.

~/mfem-4.5/examples> make ex1 ex2

All parallel examples and miniapps can be built with:

~> cd mfem-4.5
~/mfem-4.5> make all -j

One can also use the parallel library to optionally (re-)build GLVis:

~> cd glvis-4.2
~/glvis-4.2> make clean
~/glvis-4.2> make MFEM_DIR=../mfem-4.5 -j

This, however, is generally not recommended, since the additional MPI thread can interfere with the other GLVis threads.

Parallel build using METIS 5

Build METIS 5:

~> tar zvxf metis-5.1.0.tar.gz
~> cd metis-5.1.0
~/metis-5.1.0> make BUILDDIR=lib config
~/metis-5.1.0> make BUILDDIR=lib
~/metis-5.1.0> cp lib/libmetis/libmetis.a lib

Build the parallel version of MFEM, setting the options MFEM_USE_METIS_5 and METIS_DIR, e.g.:

~> cd mfem-4.5
~/mfem-4.5> make parallel -j MFEM_USE_METIS_5=YES METIS_DIR=@MFEM_DIR@/../metis-5.1.0

CUDA version of MFEM

To build the CUDA version of MFEM, one needs to specify the CUDA compute capability, with the CUDA_ARCH flag. In the examples below we use CUDA_ARCH=sm_70 to build the MFEM serial and parallel versions for compute capability 7.0 (V100).

Build the serial CUDA version of MFEM:

~/mfem> make cuda CUDA_ARCH=sm_70 -j

Build the parallel CUDA version of MFEM:

~/mfem> make pcuda CUDA_ARCH=sm_70 -j

To use hypre with CUDA support in MFEM, follow the instructions above but configure it with the following command, specifying the CUDA compute capability:

~/hypre-2.26.0/src> ./configure --with-cuda --with-gpu-arch="70" --disable-fortran

HIP version of MFEM

To build the HIP version of MFEM, one needs to specify the HIP architecture, with the HIP_ARCH flag. In the examples below we use HIP_ARCH=gfx908 to build the MFEM serial and parallel versions for gfx908 (MI100).

Build the serial HIP version of MFEM:

~/mfem> make hip HIP_ARCH=gfx908 -j

Build the parallel HIP version of MFEM:

~/mfem> make phip HIP_ARCH=gfx908 -j

To use hypre with HIP support in MFEM, follow the instructions above but configure it with the following command, specifying the HIP architecture:

~/hypre-2.26.0/src> ./configure --with-hip --with-gpu-arch="gfx908" --disable-fortran

Installing MFEM with Spack

If Spack is already available on your system and is visible in your PATH, you can install the MFEM software simply with:

spack install mfem

To enable package testing during the build process, use instead:

spack install -v --test=all mfem

If you don't have Spack, you can download it and install MFEM with the following commands:

git clone https://github.com/spack/spack.git
cd spack
./bin/spack install -v mfem

Building MFEM with CMake

To build a serial version of MFEM with CMake first create a build directory. For example, using a build directory named build inside the MFEM source directory:

mkdir build
cd build

Run the CMake configuration on the MFEM source directory.

cmake ..

Run the build command associated with the CMake configuration, specifying the number of parallel build tasks with the -j flag (4 tasks in this case).

cmake --build . -j 4

Parallel build using CMake

To build a parallel version of MFEM with CMake first build METIS and Hypre as described above. From the MFEM source directory, create a build directory. For example, using a build directory named build inside the MFEM source directory:

mkdir build
cd build

Run the CMake configuration on the MFEM source directory using the MFEM_USE_MPI CMake variable to enable MPI. This will automatically search for the system MPI implementation, the METIS installation (in <mfem-source-dir>/../metis-4.0), and Hypre installation (in <mfem-source-dir/../hypre).

cmake .. -DMFEM_USE_MPI=YES

Alternatively, run the CMake configuration also using the MFEM_FETCH_TPLS CMake variable to enable fetching of Hypre and METIS. This will automatically download, configure, and build Hypre and METIS alongside MFEM (note that this option is not currently supported for GPU builds).

cmake .. -DMFEM_USE_MPI=YES -DMFEM_FETCH_TPLS=YES

For either CMake configuration approach, now run the build command associated with the configuration, specifying the number of parallel build tasks with the -j flag (4 tasks in this case).

cmake --build . -j 4

Advanced configuration steps

To build with CUDA:

cmake .. -DMFEM_USE_CUDA=YES

To specify what CUDA architecture to target:

cmake .. -DCUDA_ARCH="sm_70"

The CUDA architecture is formatted as sm_{CC}, or just {CC}, where CC is the GPU compute capability of the target GPU without the decimal point. A list of NVIDIA GPU compute capabilities can be found in the NVIDIA developers documentation. Multiple CUDA architectures can be targeted with a comma or semicolon separated list.

cmake .. -DCUDA_ARCH="{ARCH1},{ARCH2},{ARCH3}"

or

cmake .. -DCUDA_ARCH="{ARCH1};{ARCH2};{ARCH3}"

Other accepted architecture identifies are "all" which targets all CUDA architectures, "all-major" which targets all major versions sm_{*0}, and "native" which targets the visible GPUs on the system.

To build with METIS 5, after following the instructions to build METIS 5 above:

cmake .. -DMFEM_USE_MPI=YES -DMFEM_USE_METIS_5=YES -DMETIS_DIR=../../metis-5.1.0

To build with HIP:

cmake .. -DMFEM_USE_HIP=YES

To specify what HIP architecture(s) to target:

cmake .. -DCMAKE_HIP_ARCHITECTURES="gfx942;gfx90a"

Multiple architectures can be targeted using a semi-colon separated list. The HIP architecture for different GPU models can be found in the LLVM documentation

When building for GPUs, it is recommended to enable Umpire with -DMFEM_USE_UMPIRE=ON; if it is not automatically found by CMake the installation directory can be specified with -DUMPIRE_DIR=<path-to-umpire-installation-dir>. Umpire can be downloaded from

• https://github.com/LLNL/Umpire/tags

Advanced build steps

Different targets can be built with the --target flag in the build step

cmake --build . -j 4 --target <target-name>

To build the examples use the examples target (the executables will be in the build/examples directory).

cmake --build . -j 4 --target examples

To quickly check if the code is successfully built using example 1/1p use the check target.

cmake --build . -j 4 --target check

To build the miniapps use the miniapps target

cmake --build . -j 4 --target miniapps

To build everything use the exec target

cmake --build . -j 4 --target exec