HowTo: Use partial assembly and matrix-free assembly
MFEM provides different levels of assembly for
mfem::NonlinearForm based on the operator decomposition:
These different levels of assembly are:
- LEGACY, in the case of a
mfem::BilinearFormLEGACY corresponds to a fully assembled form, i.e. a global sparse matrix in MFEM, Hypre or PETSC format. In the case of a
mfem::NonlinearFormLEGACY corresponds to an operator that is fully evaluated on the fly. The LEGACY assembly level is ALWAYS performed on the host.
- FULL, fully assembled form, i.e. a global sparse matrix in MFEM format. This assembly is compatible with device execution, and therefore the sparse matrix is assembled on device if available. This corresponds to storing the whole A = GT BT D B G operator as a sparse matrix.
- ELEMENT, Form assembled at element level, which computes and stores dense element matrices. This corresponds to storing the element-local dense matrices AE = BT D B. This format allows to have some access to the matrix entries, while also providing a data format that is more friendly with GPU architectures.
- PARTIAL, Partially-assembled form, which computes and stores data only at quadrature points. This corresponds to storing only quadrature points values D, this format results in significantly faster computations and less storage usage compared to format storing matrices. Only the diagonal entries of the operator are accessible.
- NONE, "Matrix-free" form that computes all of its action on-the-fly without any substantial storage. In this case D is computed on the fly, this format is also significantly faster than the matrix formats, but is currently slower than partial assembly due to the increased number of computations. However, in the case of operators that need to be reassembled frequently this assembly level might be faster than partial assembly by skipping any reassembly steps.
The different assembly levels are accessed through the following unified interface:
AssemblyLevel assembly_level = ...; a->SetAssemblyLevel(assembly_level);
a is either an
Assembly levels and backend device configuration
MFEM integrates three backends that interact with the assembly levels, namely the RAJA backend, the OCCA backend, and the libCEED backend.
Backends are accessible by configuring the
||Default CPU backend: sequential execution on each MPI rank.|
||OpenMP backend. Enabled when MFEM_USE_OPENMP = YES.|
||CUDA backend. Enabled when MFEM_USE_CUDA = YES.|
||HIP backend. Enabled when MFEM_USE_HIP = YES.|
||RAJA CPU backend: sequential execution on each MPI rank. Enabled when MFEM_USE_RAJA = YES.|
||RAJA OpenMP backend. Enabled when MFEM_USE_RAJA = YES and MFEM_USE_OPENMP = YES.|
||RAJA CUDA backend. Enabled when MFEM_USE_RAJA = YES and MFEM_USE_CUDA = YES.|
||RAJA HIP backend. Enabled when MFEM_USE_RAJA = YES and MFEM_USE_HIP = YES.|
||OCCA CPU backend: sequential execution on each MPI rank. Enabled when MFEM_USE_OCCA = YES.|
||OCCA OpenMP backend. Enabled when MFEM_USE_OCCA = YES.|
||OCCA CUDA backend. Enabled when MFEM_USE_OCCA = YES and MFEM_USE_CUDA = YES.|
||CEED CPU backend. GPU backends can still be used, but with expensive memory transfers. Enabled when MFEM_USE_CEED = YES.|
||CEED CUDA backend working together with the CUDA backend. Enabled when MFEM_USE_CEED = YES and MFEM_USE_CUDA = YES. NOTE: The current default libCEED CUDA backend is non-deterministic!|
||CEED HIP backend working together with the HIP backend. Enabled when MFEM_USE_CEED = YES and MFEM_USE_HIP = YES.|
||Debug backend: host memory is READ/WRITE protected while a device is in use. It allows to test the "device" code-path (using separate host/device memory pools and host <-> device transfers) without any GPU hardware. As 'DEBUG' is sometimes used as a macro,
It is also possible to request the backend of a backend, for instance if we want to use the
/gpu/cuda/shared backend of libCEED one can specify this with the following syntax:
The native MFEM backend and the RAJA backend support the same features and Integrators. However, the OCCA backend, and the libCEED backend each offer different features, and support different Integrators with different performance characteristics.
|Supported Integrators||native MFEM||OCCA backend||libCEED backend|
|Vector Mass Integrator||✅||❌||✅|
|Vector FE Mass Integrator||✅||❌||❌|
|Non-linear Convection Integrator||✅||❌||✅|
|Vector Diffusion Integrator||✅||❌||✅|
|Mixed Vector Gradient Integrator||✅||❌||❌|
|Mixed Vector Curl Integrator||✅||❌||❌|
|Mixed Vector Weak Curl Integrator||✅||❌||❌|
|Vector Divergence Integrator||✅||❌||❌|
|Vector FE Divergence Integrator||✅||❌||❌|
|Curl Curl Integrator||✅||❌||❌|
|Div Div Integrator||✅||❌||❌|
|Features||native MFEM||OCCA backend||libCEED backend|
|Tensor elements support||✅||✅||✅|
|Mixed elements support||❌||❌||❌|