-
Portable, Massively Parallel Implementation of a Material Point Method for Compressible Flows
Authors:
Paolo Joseph Baioni,
Tommaso Benacchio,
Luigi Capone,
Carlo de Falco
Abstract:
The recent evolution of software and hardware technologies is leading to a renewed computational interest in Particle-In-Cell (PIC) methods such as the Material Point Method (MPM). Indeed, provided some critical aspects are properly handled, PIC methods can be cast in formulations suitable to the requirements of data locality and fine-grained parallelism of modern hardware accelerators as Graphics…
▽ More
The recent evolution of software and hardware technologies is leading to a renewed computational interest in Particle-In-Cell (PIC) methods such as the Material Point Method (MPM). Indeed, provided some critical aspects are properly handled, PIC methods can be cast in formulations suitable to the requirements of data locality and fine-grained parallelism of modern hardware accelerators as Graphics Processing Units (GPUs). Such a rapid and continuous technological development increases also the importance of generic and portable implementations. While continuum mechanics simulations have already shown the capabilities of MPM on a wide range of phenomena, the use of the method in compressible fluid dynamics is less frequent. In this paper we present a portable, highly parallel, GPU based MPM solver for compressible gas dynamics. The implementation aims to reach a good compromise between portability and efficiency and to give a first assessment of the potential of this approach in reproducing strongly compressible gas flows, also taking into account solid obstacles. The proposed model constitutes a first step towards the development of a monolithic MPM solver for Fluid-Structure Interaction (FSI) problems at all Mach numbers up to the supersonic regime.
△ Less
Submitted 2 July, 2024; v1 submitted 25 April, 2024;
originally announced April 2024.
-
Impact of curved elements for flows over orography with a Discontinuous Galerkin scheme
Authors:
Giuseppe Orlando,
Tommaso Benacchio,
Luca Bonaventura
Abstract:
We present a quantitative assessment of the impact of high-order map**s on the simulation of flows over complex orography. Curved boundaries were not used in earlier numerical methods, whereas they are employed nowadays to an increasing extent in combination with high-order methods, such as the Finite Element Method (FEM) and the Spectral Element Method (SEM). We consider here a specific Discont…
▽ More
We present a quantitative assessment of the impact of high-order map**s on the simulation of flows over complex orography. Curved boundaries were not used in earlier numerical methods, whereas they are employed nowadays to an increasing extent in combination with high-order methods, such as the Finite Element Method (FEM) and the Spectral Element Method (SEM). We consider here a specific Discontinuous Galerkin (DG) method implemented in the framework of the deal.II library, which natively supports high-order map**s. A number of numerical experiments based on classical benchmarks over idealized orographic profiles demonstrate the positive impact of curved boundaries on the accuracy of the results. These findings are also supported by results of the application of this approach to non-smooth and realistic orographic profiles.
△ Less
Submitted 14 April, 2024;
originally announced April 2024.
-
A mixed finite-element, finite-volume, semi-implicit discretisation for atmospheric dynamics: Spherical geometry
Authors:
Thomas Melvin,
Ben Shipway,
Nigel Wood,
Tommaso Benacchio,
Thomas Bendall,
Ian Boutle,
Alex Brown,
Christine Johnson,
James Kent,
Stephen Pring,
Chris Smith,
Mohamed Zerroukat,
Colin Cotter,
John Thuburn
Abstract:
The reformulation of the Met Office's dynamical core for weather and climate prediction previously described by the authors is extended to spherical domains using a cubed-sphere mesh. This paper updates the semi-implicit mixed finite-element formulation to be suitable for spherical domains. In particular the finite-volume transport scheme is extended to take account of non-uniform, non-orthogonal…
▽ More
The reformulation of the Met Office's dynamical core for weather and climate prediction previously described by the authors is extended to spherical domains using a cubed-sphere mesh. This paper updates the semi-implicit mixed finite-element formulation to be suitable for spherical domains. In particular the finite-volume transport scheme is extended to take account of non-uniform, non-orthogonal meshes and uses an advective-then-flux formulation so that increment from the transport scheme is linear in the divergence. The resulting model is then applied to a standard set of dry dynamical core tests and compared to the existing semi-implicit semi-Lagrangian dynamical core currently used in the Met Office's operational model.
△ Less
Submitted 21 February, 2024;
originally announced February 2024.
-
Robust and accurate simulations of flows over orography using non-conforming meshes
Authors:
Giuseppe Orlando,
Tommaso Benacchio,
Luca Bonaventura
Abstract:
We systematically validate the static local mesh refinement capabilities of a recently proposed IMEX-DG scheme implemented in the framework of the deal.II library. Non-conforming meshes are employed in atmospheric flow simulations to increase the resolution around complex orography. A number of numerical experiments based on classical benchmarks with idealized as well as real orography profiles de…
▽ More
We systematically validate the static local mesh refinement capabilities of a recently proposed IMEX-DG scheme implemented in the framework of the deal.II library. Non-conforming meshes are employed in atmospheric flow simulations to increase the resolution around complex orography. A number of numerical experiments based on classical benchmarks with idealized as well as real orography profiles demonstrate that simulations with the refined mesh are stable for long lead times and no spurious effects arise at the interfaces of mesh regions with different resolutions. Moreover, correct values of the momentum flux are retrieved and the correct large-scale orographic response is established. Hence, large-scale orography-driven flow features can be simulated without loss of accuracy using a much lower total amount of degrees of freedom. In a context of spatial resolutions approaching the hectometric scale in numerical weather prediction models, these results support the use of locally refined, non-conforming meshes as a reliable and effective tool to greatly reduce the dependence of atmospheric models on orographic wave drag parametrizations.
△ Less
Submitted 8 July, 2024; v1 submitted 12 February, 2024;
originally announced February 2024.
-
An IMEX-DG solver for atmospheric dynamics simulations with adaptive mesh refinement
Authors:
Giuseppe Orlando,
Tommaso Benacchio,
Luca Bonaventura
Abstract:
We present an accurate and efficient solver for atmospheric dynamics simulations that allows for non-conforming mesh refinement. The model equations are the conservative Euler equations for compressible flows. The numerical method is based on an $h-$adaptive Discontinuous Galerkin spatial discretization and on a second order Additive Runge Kutta IMEX method for time discretization, especially desi…
▽ More
We present an accurate and efficient solver for atmospheric dynamics simulations that allows for non-conforming mesh refinement. The model equations are the conservative Euler equations for compressible flows. The numerical method is based on an $h-$adaptive Discontinuous Galerkin spatial discretization and on a second order Additive Runge Kutta IMEX method for time discretization, especially designed for low Mach regimes. The solver is implemented in the framework of the $deal.II$ library, whose mesh refinement capabilities are employed to enhance efficiency. A number of numerical experiments based on classical benchmarks for atmosphere dynamics demonstrate the properties and advantages of the proposed method.
△ Less
Submitted 17 May, 2023; v1 submitted 14 October, 2022;
originally announced October 2022.
-
Efficient hyperbolic-parabolic models on multi-dimensional unbounded domains using an extended DG approach
Authors:
Federico Vismara,
Tommaso Benacchio
Abstract:
We introduce an extended discontinuous Galerkin discretization of hyperbolic-parabolic problems on multidimensional semi-infinite domains. Building on previous work on the one-dimensional case, we split the strip-shaped computational domain into a bounded region, discretized by means of discontinuous finite elements using Legendre basis functions, and an unbounded subdomain, where scaled Laguerre…
▽ More
We introduce an extended discontinuous Galerkin discretization of hyperbolic-parabolic problems on multidimensional semi-infinite domains. Building on previous work on the one-dimensional case, we split the strip-shaped computational domain into a bounded region, discretized by means of discontinuous finite elements using Legendre basis functions, and an unbounded subdomain, where scaled Laguerre functions are used as a basis. Numerical fluxes at the interface allow for a seamless coupling of the two regions. The resulting coupling strategy is shown to produce accurate numerical solutions in tests on both linear and non-linear scalar and vectorial model problems. In addition, an efficient absorbing layer can be simulated in the semi-infinite part of the domain in order to damp outgoing signals with negligible spurious reflections at the interface. By tuning the scaling parameter of the Laguerre basis functions, the extended DG scheme simulates transient dynamics over large spatial scales with a substantial reduction in computational cost at a given accuracy level compared to standard single-domain discontinuous finite element techniques.
△ Less
Submitted 30 August, 2023; v1 submitted 30 September, 2022;
originally announced September 2022.
-
Next-generation HPC models for future rotorcraft applications
Authors:
Nicoletta Sanguini,
Tommaso Benacchio,
Daniele Malacrida,
Federico Cipolletta,
Francesco Rondina,
Antonio Sciarappa,
Luigi Capone
Abstract:
Rotorcraft technologies pose great scientific and industrial challenges for numerical computing. As available computational resources approach the exascale, finer scales and therefore more accurate simulations of engineering test cases become accessible. However, shifting legacy workflows and optimizing parallel efficiency and scalability of existing software on new hardware is often demanding. Th…
▽ More
Rotorcraft technologies pose great scientific and industrial challenges for numerical computing. As available computational resources approach the exascale, finer scales and therefore more accurate simulations of engineering test cases become accessible. However, shifting legacy workflows and optimizing parallel efficiency and scalability of existing software on new hardware is often demanding. This paper reports preliminary results in CFD and structural dynamics simulations using the T106A Low Pressure Turbine (LPT) blade geometry on Leonardo S.p.A.'s davinci-1 high-performance computing (HPC) facility. Time to solution and scalability are assessed for commercial packages Ansys Fluent, STAR-CCM+, and ABAQUS, and the open-source scientific computing framework PyFR. In direct numerical simulations of compressible fluid flow, normalized time to solution values obtained using PyFR are found to be up to 8 times smaller than those obtained using Fluent and STAR-CCM+. The findings extend to the incompressible case. All models offer weak and strong scaling in tests performed on up to 48 compute nodes, each with 4 Nvidia A100 GPUs. In linear elasticity simulations with ABAQUS, both the iterative solver and the direct solver provide speedup in preliminary scaling tests, with the iterative solver outperforming the direct solver in terms of time-to-solution and memory usage. The results provide a first indication of the potential of HPC architectures in scaling engineering applications towards certification by simulation, and the first step for the Company towards the use of cutting-edge HPC toolkits in the field of Rotorcraft technologies.
△ Less
Submitted 5 July, 2022;
originally announced July 2022.
-
A one-step blended soundproof-compressible model with balanced data assimilation: theory and idealised tests
Authors:
Ray Chew,
Tommaso Benacchio,
Gottfried Hastermann,
Rupert Klein
Abstract:
A challenge arising from the local Bayesian assimilation of data in an atmospheric flow simulation is the imbalances it may introduce. Acoustic fast-mode imbalances of the order of the slower dynamics can be negated by employing a blended numerical model with seamless access to the compressible and the soundproof pseudo-incompressible dynamics. Here, the blended modelling strategy by Benacchio et…
▽ More
A challenge arising from the local Bayesian assimilation of data in an atmospheric flow simulation is the imbalances it may introduce. Acoustic fast-mode imbalances of the order of the slower dynamics can be negated by employing a blended numerical model with seamless access to the compressible and the soundproof pseudo-incompressible dynamics. Here, the blended modelling strategy by Benacchio et al., MWR, vol. 142 (2014) is upgraded in an advanced numerical framework and extended with a Bayesian local ensemble data assimilation method. Upon assimilation of data, the model configuration is switched to the pseudo-incompressible regime for one time-step. After that, the model configuration is switched back to the compressible model for the duration of the assimilation window. The switching between model regimes is repeated for each subsequent assimilation window. An improved blending strategy for the numerical model ensures that a single time-step in the pseudo-incompressible regime is sufficient to suppress imbalances coming from the initialisation and data assimilation. This improvement is based on three innovations: (i) the association of pressure fields computed at different stages of the numerical integration with actual time levels; (ii) a conversion of pressure-related variables between the model regimes derived from low Mach number asymptotics; and (iii) a judicious selection of the pressure variables used in converting numerical model states when a switch of models occurs. Idealised two-dimensional travelling vortex and buoyancy-driven bubble convection experiments show that acoustic imbalances arising from data assimilation can be eliminated by using this blended model, thereby achieving balanced analysis fields.
△ Less
Submitted 17 July, 2021; v1 submitted 22 March, 2021;
originally announced March 2021.
-
FT-GCR: a fault-tolerant generalized conjugate residual elliptic solver
Authors:
Mike Gillard,
Tommaso Benacchio
Abstract:
With the steady advance of high performance computing systems featuring smaller and smaller hardware components, the systems and algorithms used for numerical simulations increasingly contend with disruptions caused by hardware failures and bit-levels misrepresentations of computing data. In numerical frameworks exploiting massive processing power, the solution of linear systems often represents t…
▽ More
With the steady advance of high performance computing systems featuring smaller and smaller hardware components, the systems and algorithms used for numerical simulations increasingly contend with disruptions caused by hardware failures and bit-levels misrepresentations of computing data. In numerical frameworks exploiting massive processing power, the solution of linear systems often represents the most computationally intensive component. Given the large amount of repeated operations involved, iterative solvers are particularly vulnerable to bit-flips.
A new method named FT-GCR is proposed here that supplies the preconditioned Generalized Conjugate Residual Krylov solver with detection of, and recovery from, soft faults. The algorithm tests on the monotonic decrease of the residual norm and, upon failure, restarts the iteration within the local Krylov space. Numerical experiments on the solution of an elliptic problem arising from a stationary flow over an isolated hill on the sphere show the skill of the method in addressing bit-flips on a range of grid sizes and data loss scenarios, with best returns and detection rates obtained for larger corruption events. The simplicity of the method makes it easily extendable to other solvers and an ideal candidate for algorithmic fault tolerance within integrated model resilience strategies.
△ Less
Submitted 21 March, 2021; v1 submitted 12 March, 2021;
originally announced March 2021.
-
A seamless, extended DG approach for advection-diffusion problems on unbounded domains
Authors:
Federico Vismara,
Tommaso Benacchio,
Luca Bonaventura
Abstract:
We propose and analyze a seamless extended Discontinuous Galerkin (DG) discretization of advection-diffusion equations on semi-infinite domains. The semi-infinite half line is split into a finite subdomain where the model uses a standard polynomial basis, and a semi-unbounded subdomain where scaled Laguerre functions are employed as basis and test functions. Numerical fluxes enable the coupling at…
▽ More
We propose and analyze a seamless extended Discontinuous Galerkin (DG) discretization of advection-diffusion equations on semi-infinite domains. The semi-infinite half line is split into a finite subdomain where the model uses a standard polynomial basis, and a semi-unbounded subdomain where scaled Laguerre functions are employed as basis and test functions. Numerical fluxes enable the coupling at the interface between the two subdomains in the same way as standard single domain DG interelement fluxes. A novel linear analysis on the extended DG model yields unconditional stability with respect to the Péclet number. Errors due to the use of different sets of basis functions on different portions of the domain are negligible, as highlighted in numerical experiments with the linear advection-diffusion and viscous Burgers' equations. With an added dam** term on the semi-infinite subdomain, the extended framework is able to efficiently simulate absorbing boundary conditions without additional conditions at the interface. A few modes in the semi-infinite subdomain are found to suffice to deal with outgoing single wave and wave train signals more accurately than standard approaches at a given computational cost, thus providing an appealing model for fluid flow simulations in unbounded regions.
△ Less
Submitted 22 July, 2021; v1 submitted 10 December, 2020;
originally announced December 2020.
-
Resiliency in Numerical Algorithm Design for Extreme Scale Simulations
Authors:
Emmanuel Agullo,
Mirco Altenbernd,
Hartwig Anzt,
Leonardo Bautista-Gomez,
Tommaso Benacchio,
Luca Bonaventura,
Hans-Joachim Bungartz,
Sanjay Chatterjee,
Florina M. Ciorba,
Nathan DeBardeleben,
Daniel Drzisga,
Sebastian Eibl,
Christian Engelmann,
Wilfried N. Gansterer,
Luc Giraud,
Dominik Goeddeke,
Marco Heisig,
Fabienne Jezequel,
Nils Kohl,
Xiaoye Sherry Li,
Romain Lion,
Miriam Mehl,
Paul Mycek,
Michael Obersteiner,
Enrique S. Quintana-Orti
, et al. (11 additional authors not shown)
Abstract:
This work is based on the seminar titled ``Resiliency in Numerical Algorithm Design for Extreme Scale Simulations'' held March 1-6, 2020 at Schloss Dagstuhl, that was attended by all the authors.
Naive versions of conventional resilience techniques will not scale to the exascale regime: with a main memory footprint of tens of Petabytes, synchronously writing checkpoint data all the way to backgr…
▽ More
This work is based on the seminar titled ``Resiliency in Numerical Algorithm Design for Extreme Scale Simulations'' held March 1-6, 2020 at Schloss Dagstuhl, that was attended by all the authors.
Naive versions of conventional resilience techniques will not scale to the exascale regime: with a main memory footprint of tens of Petabytes, synchronously writing checkpoint data all the way to background storage at frequent intervals will create intolerable overheads in runtime and energy consumption. Forecasts show that the mean time between failures could be lower than the time to recover from such a checkpoint, so that large calculations at scale might not make any progress if robust alternatives are not investigated.
More advanced resilience techniques must be devised. The key may lie in exploiting both advanced system features as well as specific application knowledge. Research will face two essential questions: (1) what are the reliability requirements for a particular computation and (2) how do we best design the algorithms and software to meet these requirements? One avenue would be to refine and improve on system- or application-level checkpointing and rollback strategies in the case an error is detected. Developers might use fault notification interfaces and flexible runtime systems to respond to node failures in an application-dependent fashion. Novel numerical algorithms or more stochastic computational approaches may be required to meet accuracy requirements in the face of undetectable soft errors.
The goal of this Dagstuhl Seminar was to bring together a diverse group of scientists with expertise in exascale computing to discuss novel ways to make applications resilient against detected and undetected faults. In particular, participants explored the role that algorithms and applications play in the holistic approach needed to tackle this challenge.
△ Less
Submitted 26 October, 2020;
originally announced October 2020.
-
A semi-implicit compressible model for atmospheric flows with seamless access to soundproof and hydrostatic dynamics
Authors:
Tommaso Benacchio,
Rupert Klein
Abstract:
We introduce a second-order numerical scheme for compressible atmospheric motions at small to planetary scales. The collocated finite volume method treats the advection of mass, momentum, and mass-weighted potential temperature in conservation form while relying on Exner pressure for the pressure gradient term. It discretises the rotating compressible equations by evolving full variables rather th…
▽ More
We introduce a second-order numerical scheme for compressible atmospheric motions at small to planetary scales. The collocated finite volume method treats the advection of mass, momentum, and mass-weighted potential temperature in conservation form while relying on Exner pressure for the pressure gradient term. It discretises the rotating compressible equations by evolving full variables rather than perturbations around a background state, and operates with time steps constrained by the advection speed only. Perturbation variables are only used as auxiliary quantities in the formulation of the elliptic problem. Borrowing ideas on forward-in-time differencing, the algorithm reframes the authors' previously proposed schemes into a sequence of implicit midpoint, advection, and implicit trapezoidal steps that allows for a time integration unconstrained by the internal gravity wave speed. Compared with existing approaches, results on a range of benchmarks of nonhydrostatic- and hydrostatic-scale dynamics are competitive. The test suite includes a new planetary-scale inertia-gravity wave test highlighting the properties of the scheme and its large time step capabilities. In the hydrostatic-scale cases the model is run in pseudo-incompressible and hydrostatic mode with simple switching within a uniform discretization framework. The differences with the compressible runs return expected relative magnitudes. By providing seamless access to soundproof and hydrostatic dynamics, the developments represent a necessary step towards an all-scale blended multimodel solver.
△ Less
Submitted 15 March, 2019;
originally announced March 2019.
-
A seamless extension of DG methods for hyperbolic problems to unbounded domains
Authors:
T. Benacchio,
L. Bonaventura
Abstract:
We consider spectral discretizations of hyperbolic problems on unbounded domains using Laguerre basis functions. Taking as model problem the scalar advection equation, we perform a comprehensive stability analysis that includes strong collocation formulations, nodal and modal weak formulations, with either inflow or outflow boundary conditions, using either Gauss - Laguerre or Gauss - Laguerre - R…
▽ More
We consider spectral discretizations of hyperbolic problems on unbounded domains using Laguerre basis functions. Taking as model problem the scalar advection equation, we perform a comprehensive stability analysis that includes strong collocation formulations, nodal and modal weak formulations, with either inflow or outflow boundary conditions, using either Gauss - Laguerre or Gauss - Laguerre - Radau quadrature nodes and based on either scaled Laguerre functions or scaled Laguerre polynomials. We show that some of these combinations give rise to intrinsically unstable schemes, while the combination of scaled Laguerre functions with Gauss - Laguerre - Radau nodes appears to be stable for both strong and weak formulations. We then show how a modal discretization approach for hyperbolic systems on an unbounded domain can be naturally and seamlessly coupled to a discontinuous finite element discretization on a finite domain. Examples of one dimensional hyperbolic systems are solved with the proposed domain decomposition technique. The errors obtained with the proposed approach are found to be small, enabling the use of the coupled scheme for the simulation of Rayleigh dam** layers in the semi-infinite part. Energy errors and reflection ratios of the scheme in absorbing wavetrains and single Gaussian signals show that a small number of nodes in the semi-infinite domain are sufficient to damp the waves. The theoretical insight and numerical results corroborate previous findings by the authors and establish the scaled Laguerre functions-based discretization as a flexible and efficient tool for absorbing layers as well as for the accurate simulation of waves in unbounded regions.
△ Less
Submitted 29 March, 2018;
originally announced March 2018.
