-
The PETSc Community Is the Infrastructure
Authors:
Mark Adams,
Satish Balay,
Oana Marin,
Lois Curfman McInnes,
Richard Tran Mills,
Todd Munson,
Hong Zhang,
Junchao Zhang,
Jed Brown,
Victor Eijkhout,
Jacob Faibussowitsch,
Matthew Knepley,
Fande Kong,
Scott Kruger,
Patrick Sanan,
Barry F. Smith,
Hong Zhang
Abstract:
The communities who develop and support open source scientific software packages are crucial to the utility and success of such packages. Moreover, these communities form an important part of the human infrastructure that enables scientific progress. This paper discusses aspects of the PETSc (Portable Extensible Toolkit for Scientific Computation) community, its organization, and technical approac…
▽ More
The communities who develop and support open source scientific software packages are crucial to the utility and success of such packages. Moreover, these communities form an important part of the human infrastructure that enables scientific progress. This paper discusses aspects of the PETSc (Portable Extensible Toolkit for Scientific Computation) community, its organization, and technical approaches that enable community members to help each other efficiently.
△ Less
Submitted 3 January, 2022;
originally announced January 2022.
-
The PetscSF Scalable Communication Layer
Authors:
Junchao Zhang,
Jed Brown,
Satish Balay,
Jacob Faibussowitsch,
Matthew Knepley,
Oana Marin,
Richard Tran Mills,
Todd Munson,
Barry F. Smith,
Stefano Zampini
Abstract:
PetscSF, the communication component of the Portable, Extensible Toolkit for Scientific Computation (PETSc), is designed to provide PETSc's communication infrastructure suitable for exascale computers that utilize GPUs and other accelerators. PetscSF provides a simple application programming interface (API) for managing common communication patterns in scientific computations by using a star-fores…
▽ More
PetscSF, the communication component of the Portable, Extensible Toolkit for Scientific Computation (PETSc), is designed to provide PETSc's communication infrastructure suitable for exascale computers that utilize GPUs and other accelerators. PetscSF provides a simple application programming interface (API) for managing common communication patterns in scientific computations by using a star-forest graph representation. PetscSF supports several implementations based on MPI and NVSHMEM, whose selection is based on the characteristics of the application or the target architecture. An efficient and portable model for network and intra-node communication is essential for implementing large-scale applications. The Message Passing Interface, which has been the de facto standard for distributed memory systems, has developed into a large complex API that does not yet provide high performance on the emerging heterogeneous CPU-GPU-based exascale systems. In this paper, we discuss the design of PetscSF, how it can overcome some difficulties of working directly with MPI on GPUs, and we demonstrate its performance, scalability, and novel features.
△ Less
Submitted 21 May, 2021; v1 submitted 25 February, 2021;
originally announced February 2021.
-
Toward Performance-Portable PETSc for GPU-based Exascale Systems
Authors:
Richard Tran Mills,
Mark F. Adams,
Satish Balay,
Jed Brown,
Alp Dener,
Matthew Knepley,
Scott E. Kruger,
Hannah Morgan,
Todd Munson,
Karl Rupp,
Barry F. Smith,
Stefano Zampini,
Hong Zhang,
Junchao Zhang
Abstract:
The Portable Extensible Toolkit for Scientific computation (PETSc) library delivers scalable solvers for nonlinear time-dependent differential and algebraic equations and for numerical optimization.The PETSc design for performance portability addresses fundamental GPU accelerator challenges and stresses flexibility and extensibility by separating the programming model used by the application from…
▽ More
The Portable Extensible Toolkit for Scientific computation (PETSc) library delivers scalable solvers for nonlinear time-dependent differential and algebraic equations and for numerical optimization.The PETSc design for performance portability addresses fundamental GPU accelerator challenges and stresses flexibility and extensibility by separating the programming model used by the application from that used by the library, and it enables application developers to use their preferred programming model, such as Kokkos, RAJA, SYCL, HIP, CUDA, or OpenCL, on upcoming exascale systems. A blueprint for using GPUs from PETSc-based codes is provided, and case studies emphasize the flexibility and high performance achieved on current GPU-based systems.
△ Less
Submitted 29 September, 2021; v1 submitted 1 November, 2020;
originally announced November 2020.
-
PETSc TSAdjoint: a discrete adjoint ODE solver for first-order and second-order sensitivity analysis
Authors:
Hong Zhang,
Emil M. Constantinescu,
Barry F. Smith
Abstract:
We present a new software system PETSc TSAdjoint for first-order and second-order adjoint sensitivity analysis of time-dependent nonlinear differential equations. The derivative calculation in PETSc TSAdjoint is essentially a high-level algorithmic differentiation process. The adjoint models are derived by differentiating the timestep** algorithms and implemented based on the parallel infrastruc…
▽ More
We present a new software system PETSc TSAdjoint for first-order and second-order adjoint sensitivity analysis of time-dependent nonlinear differential equations. The derivative calculation in PETSc TSAdjoint is essentially a high-level algorithmic differentiation process. The adjoint models are derived by differentiating the timestep** algorithms and implemented based on the parallel infrastructure in PETSc. Full differentiation of the library code including MPI routines thus is avoided, and users do not need to derive their own adjoint models for their specific applications. PETSc TSAdjoint can compute the first-order derivative, that is, the gradient of a scalar functional, and the Hessian-vector product that carries second-order derivative information, while requiring minimal input (a few callbacks) from the users. Optimal checkpointing schemes are employed by the adjoint model in a manner that is transparent to users. Usability, efficiency, and scalability are demonstrated through examples from a variety of applications.
△ Less
Submitted 26 October, 2021; v1 submitted 16 December, 2019;
originally announced December 2019.
-
PETSc/TS: A Modern Scalable ODE/DAE Solver Library
Authors:
Shrirang Abhyankar,
Jed Brown,
Emil M. Constantinescu,
Debojyoti Ghosh,
Barry F. Smith,
Hong Zhang
Abstract:
High-quality ordinary differential equation (ODE) solver libraries have a long history, going back to the 1970s. Over the past several years we have implemented, on top of the PETSc linear and nonlinear solver package, a new general-purpose, extensive, extensible library for solving ODEs and differential algebraic equations (DAEs). Package includes support for both forward and adjoint sensitivitie…
▽ More
High-quality ordinary differential equation (ODE) solver libraries have a long history, going back to the 1970s. Over the past several years we have implemented, on top of the PETSc linear and nonlinear solver package, a new general-purpose, extensive, extensible library for solving ODEs and differential algebraic equations (DAEs). Package includes support for both forward and adjoint sensitivities that can be easily utilized by the TAO optimization package, which is also part of PETSc. The ODE/DAE integrator library strives to be highly scalable but also to deliver high efficiency for modest-sized problems. The library includes explicit solvers, implicit solvers, and a collection of implicit-explicit solvers, all with a common user interface and runtime selection of solver types, adaptive error control, and monitoring of solution progress. The library also offers enormous flexibility in selection of nonlinear and linear solvers, including the entire suite of PETSc iterative solvers, as well as several parallel direct solvers.
△ Less
Submitted 4 June, 2018;
originally announced June 2018.
-
Composing Scalable Nonlinear Algebraic Solvers
Authors:
Peter R. Brune,
Matthew G. Knepley,
Barry F. Smith,
Xuemin Tu
Abstract:
Most efficient linear solvers use composable algorithmic components, with the most common model being the combination of a Krylov accelerator and one or more preconditioners. A similar set of concepts may be used for nonlinear algebraic systems, where nonlinear composition of different nonlinear solvers may significantly improve the time to solution. We describe the basic concepts of nonlinear com…
▽ More
Most efficient linear solvers use composable algorithmic components, with the most common model being the combination of a Krylov accelerator and one or more preconditioners. A similar set of concepts may be used for nonlinear algebraic systems, where nonlinear composition of different nonlinear solvers may significantly improve the time to solution. We describe the basic concepts of nonlinear composition and preconditioning and present a number of solvers applicable to nonlinear partial differential equations. We have developed a software framework in order to easily explore the possible combinations of solvers. We show that the performance gains from using composed solvers can be substantial compared with gains from standard Newton-Krylov methods.
△ Less
Submitted 14 July, 2016;
originally announced July 2016.
-
Extreme-scale Multigrid Components within PETSc
Authors:
Dave A. May,
Patrick Sanan,
Karl Rupp,
Matthew G. Knepley,
Barry F. Smith
Abstract:
Elliptic partial differential equations (PDEs) frequently arise in continuum descriptions of physical processes relevant to science and engineering. Multilevel preconditioners represent a family of scalable techniques for solving discrete PDEs of this type and thus are the method of choice for high-resolution simulations. The scalability and time-to-solution of massively parallel multilevel precon…
▽ More
Elliptic partial differential equations (PDEs) frequently arise in continuum descriptions of physical processes relevant to science and engineering. Multilevel preconditioners represent a family of scalable techniques for solving discrete PDEs of this type and thus are the method of choice for high-resolution simulations. The scalability and time-to-solution of massively parallel multilevel preconditioners can be adversely effected by using a coarse-level solver with sub-optimal algorithmic complexity. To maintain scalability, agglomeration techniques applied to the coarse level have been shown to be necessary.
In this work, we present a new software component introduced within the Portable Extensible Toolkit for Scientific computation (PETSc) which permits agglomeration. We provide an overview of the design and implementation of this functionality, together with several use cases highlighting the benefits of agglomeration. Lastly, we demonstrate via numerical experiments employing geometric multigrid with structured meshes, the flexibility and performance gains possible using our MPI-rank agglomeration implementation.
△ Less
Submitted 25 April, 2016;
originally announced April 2016.
-
Run-time extensibility and librarization of simulation software
Authors:
Jed Brown,
Matthew G. Knepley,
Barry F. Smith
Abstract:
Build-time configuration and environment assumptions are hampering progress and usability in scientific software. That which would be utterly unacceptable in non-scientific software somehow passes for the norm in scientific packages. The community needs reusable software packages that are easy use and flexible enough to accommodate next-generation simulation and analysis demands.
Build-time configuration and environment assumptions are hampering progress and usability in scientific software. That which would be utterly unacceptable in non-scientific software somehow passes for the norm in scientific packages. The community needs reusable software packages that are easy use and flexible enough to accommodate next-generation simulation and analysis demands.
△ Less
Submitted 10 July, 2014;
originally announced July 2014.
-
Achieving High Performance with Unified Residual Evaluation
Authors:
Matthew G. Knepley,
Jed Brown,
Karl Rupp,
Barry F. Smith
Abstract:
We examine residual evaluation, perhaps the most basic operation in numerical simulation. By raising the level of abstraction in this operation, we can eliminate specialized code, enable optimization, and greatly increase the extensibility of existing code.
We examine residual evaluation, perhaps the most basic operation in numerical simulation. By raising the level of abstraction in this operation, we can eliminate specialized code, enable optimization, and greatly increase the extensibility of existing code.
△ Less
Submitted 6 September, 2013; v1 submitted 4 September, 2013;
originally announced September 2013.
