James Hawkes, postgraduate research student at the University of Southampton, presented his research on the scalability of parallel Computational Fluid Dynamics (CFD) codes. His project on Massively-Parallel Computational Fluid Dynamics is sponsored by the Dutch maritime research institute MARIN. Within this project ReFRESCO, a CFD code for fluid problems around ships and offshore structures, is used.
James started by presenting MARIN, which is a non-profit organisation based in Wageningen, Netherlands. It was founded in 1932 and performs research and development for maritime activities. Amongst their facilities is an ocean basin, where waves, currents, wind and rough sea conditions can be simulated. Its capacity allows for tests on submarine models of 3 to 5 metres. Impacts on ship movements can be tested with various wave generators. Furthermore there are five towing tanks.
Around 85 percent of all experiments are backed up with CFD simulations, for which two clusters are available. The CFD code ReFRESCO is implemented in Fortran and C and parallelised using MPI.
After this introduction of MARIN, James went on with a general overview of the recent developments in High Performance Computing (HPC). Over the last years, growth in computing power has been exponential. Nowadays the main issues to deal with concern power consumption rather than hardware speed. Further growth of HPC architectures is mostly happening at an intra-nodal level.
James then presented results of his scalability studies for the ReFRESCO code. As ReFRESCO uses the SIMPLE algorithm, over 90 percent of runtime is used for the linear solver, for which a good speed-up is difficult to achieve. As can be seen from the scalability plot, total scalability is only slightly better than the scalability of the solver.
The most commonly used parallel linear solvers are Krylov subspace methods such as GMRES. One approach to avoid reductions is to use pipelined GMRES. Another alternative would be to go back to simple stationary methods like the Jacobi solver. With this approach scalability is quite good. The limiting factor is global synchronisation.
A promising approach is to use so-called Chaotic Iterative Methods. The idea is to let each process access local memory whenever required. A drawback of these methods is their worse convergence compared to Krylov methods (up to 1000 times more iterations). The disadvantages of the chaotic iterative methods can be compensated by their better memory management. This is particularly true when using multigrid methods. ReFRESCO uses the PETSc library, which includes chaotic iterative solvers.
Finally chaotic iterative solvers have the advantage that they can be easily adapted to make use of Xeon Phi accelerators or GPUs.
The project aims at improving the scalability of ReFRESCO. As the PETSc library is used in many other CFD codes with implicit solvers, the results will hopefully have a direct impact on CFD software in all kinds of industries, beyond the maritime sector. More generally speaking, the methods can also be applied to other libraries and to any kind of model that uses parallel linear solvers.
Posted by Thorsten Wittemeier