-
Efficient and scalable hybrid fluid-particle simulations with geometrically resolved particles on heterogeneous CPU-GPU architectures
Authors:
Samuel Kemmler,
Christoph Rettinger,
Harald Köstler
Abstract:
In recent years, it has become increasingly popular to accelerate numerical simulations using Graphics Processing Unit (GPU)s. In multiphysics simulations, the various combined methodologies may have distinctly different computational characteristics. Therefore, the best-suited hardware architecture can differ between the simulation components. Furthermore, not all coupled software frameworks may…
▽ More
In recent years, it has become increasingly popular to accelerate numerical simulations using Graphics Processing Unit (GPU)s. In multiphysics simulations, the various combined methodologies may have distinctly different computational characteristics. Therefore, the best-suited hardware architecture can differ between the simulation components. Furthermore, not all coupled software frameworks may support all hardware. These issues predestinate or even force hybrid implementations, i.e., different simulation components running on different hardware. We introduce a hybrid coupled fluid-particle implementation with geometrically resolved particles. The simulation utilizes GPUs for the fluid dynamics, whereas the particle simulation runs on Central Processing Unit (CPU)s. We examine the performance of two contrasting cases of a fluidized bed simulation on a heterogeneous supercomputer. The hybrid overhead (i.e., the CPU-GPU communication) is negligible. The fluid simulation shows good performance utilizing nearly the entire memory bandwidth. Still, the GPU run time accounts for most of the total time. The parallel efficiency in a weak scaling benchmark for 1024 A100 GPUs is up to 71%. Frequent CPU-CPU communications occurring in the particle simulation are the leading cause of the decrease in parallel efficiency. The results show that hybrid implementations are promising for large-scale multiphysics simulations on heterogeneous supercomputers.
△ Less
Submitted 21 March, 2023;
originally announced March 2023.
-
waLBerla: A block-structured high-performance framework for multiphysics simulations
Authors:
Martin Bauer,
Sebastian Eibl,
Christian Godenschwager,
Nils Kohl,
Michael Kuron,
Christoph Rettinger,
Florian Schornbaum,
Christoph Schwarzmeier,
Dominik Thönnes,
Harald Köstler,
Ulrich Rüde
Abstract:
Programming current supercomputers efficiently is a challenging task. Multiple levels of parallelism on the core, on the compute node, and between nodes need to be exploited to make full use of the system. Heterogeneous hardware architectures with accelerators further complicate the development process. waLBerla addresses these challenges by providing the user with highly efficient building blocks…
▽ More
Programming current supercomputers efficiently is a challenging task. Multiple levels of parallelism on the core, on the compute node, and between nodes need to be exploited to make full use of the system. Heterogeneous hardware architectures with accelerators further complicate the development process. waLBerla addresses these challenges by providing the user with highly efficient building blocks for develo** simulations on block-structured grids. The block-structured domain partitioning is flexible enough to handle complex geometries, while the structured grid within each block allows for highly efficient implementations of stencil-based algorithms. We present several example applications realized with waLBerla, ranging from lattice Boltzmann methods to rigid particle simulations. Most importantly, these methods can be coupled together, enabling multiphysics simulations. The framework uses meta-programming techniques to generate highly efficient code for CPUs and GPUs from a symbolic method formulation. To ensure software quality and performance portability, a continuous integration toolchain automatically runs an extensive test suite encompassing multiple compilers, hardware architectures, and software configurations.
△ Less
Submitted 30 September, 2019;
originally announced September 2019.
-
Dynamic Load Balancing Techniques for Particulate Flow Simulations
Authors:
Christoph Rettinger,
Ulrich Rüde
Abstract:
Parallel multiphysics simulations often suffer from load imbalances originating from the applied coupling of algorithms with spatially and temporally varying workloads. It is thus desirable to minimize these imbalances to reduce the time to solution and to better utilize the available hardware resources. Taking particulate flows as an illustrating example application, we present and evaluate load…
▽ More
Parallel multiphysics simulations often suffer from load imbalances originating from the applied coupling of algorithms with spatially and temporally varying workloads. It is thus desirable to minimize these imbalances to reduce the time to solution and to better utilize the available hardware resources. Taking particulate flows as an illustrating example application, we present and evaluate load balancing techniques that tackle this challenging task. This involves a load estimation step in which the currently generated workload is predicted. We describe in detail how such a workload estimator can be developed. In a second step, load distribution strategies like space-filling curves or graph partitioning are applied to dynamically distribute the load among the available processes. To compare and analyze their performance, we employ these techniques to a benchmark scenario and observe a reduction of the load imbalances by almost a factor of four. This results in a decrease of the overall runtime by 14% for space-filling curves.
△ Less
Submitted 30 November, 2018;
originally announced November 2018.
-
A Coupled Lattice Boltzmann Method and Discrete Element Method for Discrete Particle Simulations of Particulate Flows
Authors:
Christoph Rettinger,
Ulrich Rüde
Abstract:
Discrete particle simulations are widely used to study large-scale particulate flows in complex geometries where particle-particle and particle-fluid interactions require an adequate representation but the computational cost has to be kept low. In this work, we present a novel coupling approach for such simulations. A lattice Boltzmann formulation of the generalized Navier-Stokes equations is used…
▽ More
Discrete particle simulations are widely used to study large-scale particulate flows in complex geometries where particle-particle and particle-fluid interactions require an adequate representation but the computational cost has to be kept low. In this work, we present a novel coupling approach for such simulations. A lattice Boltzmann formulation of the generalized Navier-Stokes equations is used to describe the fluid motion. This promises efficient simulations suitable for high performance computing and, since volume displacement effects by the solid phase are considered, our approach is also applicable to non-dilute particulate systems. The discrete element method is combined with an explicit evaluation of interparticle lubrication forces to simulate the motion of individual submerged particles. Drag, pressure and added mass forces determine the momentum transfer by fluid-particle interactions. A stable coupling algorithm is presented and discussed in detail. We demonstrate the validity of our approach for dilute as well as dense systems by predicting the settling velocity of spheres over a broad range of solid volume fractions in good agreement with semi-empirical correlations. Additionally, the accuracy of particle-wall interactions in a viscous fluid is thoroughly tested and established. Our approach can thus be readily used for various particulate systems and can be extended straightforward to e.g. non-spherical particles.
△ Less
Submitted 1 November, 2017;
originally announced November 2017.
-
A comparative study of fluid-particle coupling methods for fully resolved lattice Boltzmann simulations
Authors:
Christoph Rettinger,
Ulrich Rüde
Abstract:
The direct numerical simulation of particulate systems offers a unique approach to study the dynamics of fluid-solid suspensions by fully resolving the submerged particles and without introducing empirical models. For the lattice Boltzmann method, different variants exist to incorporate the fluid-particle interaction into the simulation. This paper provides a detailed and systematic comparison of…
▽ More
The direct numerical simulation of particulate systems offers a unique approach to study the dynamics of fluid-solid suspensions by fully resolving the submerged particles and without introducing empirical models. For the lattice Boltzmann method, different variants exist to incorporate the fluid-particle interaction into the simulation. This paper provides a detailed and systematic comparison of two different methods, namely the momentum exchange method and the partially saturated cells method by Noble and Torczynski. Three subvariants of each method are used in the benchmark scenario of a single heavy sphere settling in ambient fluid to study their characteristics and accuracy for particle Reynolds numbers from 185 up to 365. The sphere must be resolved with at least 24 computational cells per diameter to achieve velocity errors below 5%. The momentum exchange method is found to be more accurate in predicting the streamwise velocity component whereas the partially saturated cells method is more accurate in the spanwise components. The study reveals that the resolution should be chosen with respect to the coupling dynamics, and not only based on the flow properties, to avoid large errors in the fluid-particle interaction.
△ Less
Submitted 16 February, 2017;
originally announced February 2017.
