-
A directive based hybrid Met Office NERC Cloud model
Authors:
Nick Brown,
Angus Lepper,
Michèle Weiland,
Adrian Hill,
Ben Shipway,
Chris Maynard
Abstract:
Large Eddy Simulation is a critical modelling tool for the investigation of atmospheric flows, turbulence and cloud microphysics. The models used by the UK atmospheric research community are homogeneous and the latest model, MONC, is designed to run on substantial HPC systems with very high CPU core counts. In order to future proof these codes it is worth investigating other technologies and archi…
▽ More
Large Eddy Simulation is a critical modelling tool for the investigation of atmospheric flows, turbulence and cloud microphysics. The models used by the UK atmospheric research community are homogeneous and the latest model, MONC, is designed to run on substantial HPC systems with very high CPU core counts. In order to future proof these codes it is worth investigating other technologies and architectures which might support the communities running their codes at the exa-scale.
In this paper we present a hybrid version of MONC, where the most computationally intensive aspect is offloaded to the GPU while the rest of the functionality runs concurrently on the CPU. Developed using the directive driven OpenACC, we consider the suitability and maturity of this technology to modern Fortran scientific codes as well general software engineering techniques which aid this type of porting work. The performance of our hybrid model at scale is compared against the CPU version before considering other tuning options and making a comparison between the energy usage of the homo- and hetero-geneous versions. The result of this work is a promising hybrid model that shows performance benefits of our approach when the GPU has a significant computational workload which can not only be applied to the MONC model but also other weather and climate simulations in use by the community.
△ Less
Submitted 27 September, 2020;
originally announced September 2020.
-
A highly scalable Met Office NERC Cloud model
Authors:
Nick Brown,
Michèle Weiland,
Adrian Hill,
Ben Shipway,
Chris Maynard,
Thomas Allen,
Mike Rezny
Abstract:
Large Eddy Simulation is a critical modelling tool for scientists investigating atmospheric flows, turbulence and cloud microphysics. Within the UK, the principal LES model used by the atmospheric research community is the Met Office Large Eddy Model (LEM). The LEM was originally developed in the late 1980s using computational techniques and assumptions of the time, which means that the it does no…
▽ More
Large Eddy Simulation is a critical modelling tool for scientists investigating atmospheric flows, turbulence and cloud microphysics. Within the UK, the principal LES model used by the atmospheric research community is the Met Office Large Eddy Model (LEM). The LEM was originally developed in the late 1980s using computational techniques and assumptions of the time, which means that the it does not scale beyond 512 cores. In this paper we present the Met Office NERC Cloud model, MONC, which is a re-write of the existing LEM. We discuss the software engineering and architectural decisions made in order to develop a flexible, extensible model which the community can easily customise for their own needs. The scalability of MONC is evaluated, along with numerous additional customisations made to further improve performance at large core counts. The result of this work is a model which delivers to the community significant new scientific modelling capability that takes advantage of the current and future generation HPC machines.
△ Less
Submitted 27 September, 2020;
originally announced September 2020.
-
Multigrid preconditioners for the mixed finite element dynamical core of the LFRic atmospheric model
Authors:
Christopher Maynard,
Thomas Melvin,
Eike Hermann Müller
Abstract:
Due to the wide separation of time scales in geophysical fluid dynamics, semi-implicit time integrators are commonly used in operational atmospheric forecast models. They guarantee the stable treatment of fast (acoustic and gravity) waves, while not suffering from severe restrictions on the timestep size. To propagate the state of the atmosphere forward in time, a non-linear equation for the progn…
▽ More
Due to the wide separation of time scales in geophysical fluid dynamics, semi-implicit time integrators are commonly used in operational atmospheric forecast models. They guarantee the stable treatment of fast (acoustic and gravity) waves, while not suffering from severe restrictions on the timestep size. To propagate the state of the atmosphere forward in time, a non-linear equation for the prognostic variables has to be solved at every timestep. Since the nonlinearity is typically weak, this is done with a small number of Newton- or Picard- iterations, which in turn require the efficient solution of a large system on linear equations with $\mathcal{O}(10^6-10^9)$ unknowns. This linear solve is often the computationally most costly part of the model. In this paper an efficient linear solver for the LFRic next-generation model, currently developed by the Met Office, is described. The model uses an advanced mimetic finite element discretisation which makes the construction of efficient solvers challenging compared to models using standard finite-difference and finite-volume methods. The linear solver hinges on a bespoke multigrid preconditioner of the Schur-complement system for the pressure correction. By comparing to Krylov-subspace methods, the superior performance and robustness of the multigrid algorithm is demonstrated for standard test cases and realistic model setups. In production mode, the model will have to run in parallel on 100,000s of processing elements. As confirmed by numerical experiments, one particular advantage of the multigrid solver is its excellent parallel scalability due to avoiding expensive global reduction operations.
△ Less
Submitted 21 July, 2020; v1 submitted 31 January, 2020;
originally announced February 2020.
-
Precision of the ENDGame: Mixed-precision arithmetic in the iterative solver of the Unified Model
Authors:
Christopher M Maynard,
David N Walters
Abstract:
The Met Office's weather and climate simulation code the Unified Model is used for both operational Numerical Weather Prediction and Climate modelling. The computational performance of the model running on parallel supercomputers is a key consideration. A Krylov sub-space solver is employed to solve the equations of the dynamical core of the model, known as ENDGame. These describe the evolution of…
▽ More
The Met Office's weather and climate simulation code the Unified Model is used for both operational Numerical Weather Prediction and Climate modelling. The computational performance of the model running on parallel supercomputers is a key consideration. A Krylov sub-space solver is employed to solve the equations of the dynamical core of the model, known as ENDGame. These describe the evolution of the Earth's atmosphere. Typically, 64-bit precision is used throughout weather and climate applications. This work presents a mixed-precision implementation of the solver, the beneficial effect on run-time and the impact on solver convergence. The complex interplay of errors arising from accumulated round-off in floating-point arithmetic and other numerical effects is discussed. A careful analysis is required, however, the mixed-precision solver is now employed in the operational forecast to satisfy run-time constraints without compromising the accuracy of the solution.
△ Less
Submitted 9 November, 2018;
originally announced November 2018.
-
LFRic: Meeting the challenges of scalability and performance portability in Weather and Climate models
Authors:
S. V. Adams,
R. W. Ford,
M. Hambley,
J. M. Hobson,
I. Kavcic,
C. M. Maynard,
T. Melvin,
E. H Mueller,
S. Mullerworth,
A. R. Porter,
M. Rezny,
B. J. Shipway,
R. Wong
Abstract:
This paper describes LFRic: the new weather and climate modelling system being developed by the UK Met Office to replace the existing Unified Model in preparation for exascale computing in the 2020s. LFRic uses the GungHo dynamical core and runs on a semi-structured cubed-sphere mesh. The design of the supporting infrastructure follows object orientated principles to facilitate modularity and the…
▽ More
This paper describes LFRic: the new weather and climate modelling system being developed by the UK Met Office to replace the existing Unified Model in preparation for exascale computing in the 2020s. LFRic uses the GungHo dynamical core and runs on a semi-structured cubed-sphere mesh. The design of the supporting infrastructure follows object orientated principles to facilitate modularity and the use of external libraries where possible. In particular, a `separation of concerns' between the science code and parallel code is imposed to promote performance portability. An application called PSyclone, developed at the STFC Hartree centre, can generate the parallel code enabling deployment of a single source science code onto different machine architectures. This paper provides an overview of the scientific requirement, the design of the software infrastructure, and examples of PSyclone usage. Preliminary performance results show strong scaling and an indication that hybrid MPI/OpenMP performs better than pure MPI.
△ Less
Submitted 12 July, 2019; v1 submitted 19 September, 2018;
originally announced September 2018.
-
International Lattice Data Grid: Turn on, plug in,and download
Authors:
C. M. Maynard
Abstract:
In the beginning there was the internet, then came the world wide web, and now there is the grid. In the future perhaps there will be the cloud. In the age of persistent, pervasive, and pandemic networks I review how the lattice QCD community embraced the open source paradigm for both code and data whilst adopting the emerging grid technologies, and why having your data persistently accessible v…
▽ More
In the beginning there was the internet, then came the world wide web, and now there is the grid. In the future perhaps there will be the cloud. In the age of persistent, pervasive, and pandemic networks I review how the lattice QCD community embraced the open source paradigm for both code and data whilst adopting the emerging grid technologies, and why having your data persistently accessible via standardized protocols and services might be a good idea.
△ Less
Submitted 28 January, 2010;
originally announced January 2010.
