-
SciAI4Industry -- Solving PDEs for industry-scale problems with deep learning
Authors:
Philipp A. Witte,
Russell J. Hewett,
Kumar Saurabh,
AmirHossein Sojoodi,
Ranveer Chandra
Abstract:
Solving partial differential equations with deep learning makes it possible to reduce simulation times by multiple orders of magnitude and unlock scientific methods that typically rely on large numbers of sequential simulations, such as optimization and uncertainty quantification. Two of the largest challenges of adopting scientific AI for industrial problem settings is that training datasets must…
▽ More
Solving partial differential equations with deep learning makes it possible to reduce simulation times by multiple orders of magnitude and unlock scientific methods that typically rely on large numbers of sequential simulations, such as optimization and uncertainty quantification. Two of the largest challenges of adopting scientific AI for industrial problem settings is that training datasets must be simulated in advance and that neural networks for solving large-scale PDEs exceed the memory capabilities of current GPUs. We introduce a distributed programming API in the Julia language for simulating training data in parallel on the cloud and without requiring users to manage the underlying HPC infrastructure. In addition, we show that model-parallel deep learning based on domain decomposition allows us to scale neural networks for solving PDEs to commercial-scale problem settings and achieve above 90% parallel efficiency. Combining our cloud API for training data generation and model-parallel deep learning, we train large-scale neural networks for solving the 3D Navier-Stokes equation and simulating 3D CO2 flow in porous media. For the CO2 example, we simulate a training dataset based on a commercial carbon capture and storage (CCS) project and train a neural network for CO2 flow simulation on a 3D grid with over 2 million cells that is 5 orders of magnitudes faster than a conventional numerical simulator and 3,200 times cheaper.
△ Less
Submitted 23 November, 2022;
originally announced November 2022.
-
Breakup dynamics in primary jet atomization using mesh- and interface- refined Cahn-Hilliard Navier-Stokes
Authors:
Makrand A. Khanwale,
Kumar Saurabh,
Masado Ishii,
Hari Sundar,
Baskar Ganapathysubramanian
Abstract:
We present a technique to perform interface-resolved simulations of complex breakup dynamics in two-phase flows using the Cahn-Hilliard Navier-Stokes equations. The method dynamically decreases the interface thickness parameter in relevant regions and simultaneously increases local mesh resolution, preventing numerical artifacts. We perform a detailed numerical simulation of pulsed jet atomization…
▽ More
We present a technique to perform interface-resolved simulations of complex breakup dynamics in two-phase flows using the Cahn-Hilliard Navier-Stokes equations. The method dynamically decreases the interface thickness parameter in relevant regions and simultaneously increases local mesh resolution, preventing numerical artifacts. We perform a detailed numerical simulation of pulsed jet atomization that shows a complex cascade of break-up mechanisms involving sheet rupture and filament formation. To understand the effect of refinement on the breakup, we analyze the droplet size distribution. The proposed approach opens up resolved simulations for various multiphase flow phenomena.
△ Less
Submitted 27 September, 2022;
originally announced September 2022.
-
CyRSoXS: A GPU-accelerated virtual instrument for Polarized Resonant Soft X-ray Scattering (P-RSoXS)
Authors:
Kumar Saurabh,
Peter J. Dudenas,
Eliot Gann,
Veronica G. Reynolds,
Subhrangsu Mukherjee,
Daniel Sunday,
Tyler B. Martin,
Peter A. Beaucage,
Michael L. Chabinyc,
Dean M. DeLongchamp,
Adarsh Krishnamurthy,
Baskar Ganapathysubramanian
Abstract:
Polarized Resonant Soft X-ray scattering (P-RSoXS) has emerged as a powerful synchrotron-based tool that combines principles of X-ray scattering and X-ray spectroscopy. P-RSoXS provides unique sensitivity to molecular orientation and chemical heterogeneity in soft materials such as polymers and biomaterials. Quantitative extraction of orientation information from P-RSoXS pattern data is challengin…
▽ More
Polarized Resonant Soft X-ray scattering (P-RSoXS) has emerged as a powerful synchrotron-based tool that combines principles of X-ray scattering and X-ray spectroscopy. P-RSoXS provides unique sensitivity to molecular orientation and chemical heterogeneity in soft materials such as polymers and biomaterials. Quantitative extraction of orientation information from P-RSoXS pattern data is challenging because the scattering processes originate from sample properties that must be represented as energy-dependent three-dimensional tensors with heterogeneities at nanometer to sub-nanometer length scales. We overcome this challenge by develo** an open-source virtual instrument that uses GPUs to simulate P-RSoXS patterns from real-space material representations with nanoscale resolution. Our computational framework CyRSoXS (https://github.com/usnistgov/cyrsoxs) is designed to maximize GPU performance. We demonstrate the accuracy and robustness of our approach by validating against an extensive set of test cases, which include both analytical solutions and numerical comparisons, demonstrating a speedup of over three orders relative to the current state-of-the-art simulation software. Such fast simulations open up a variety of applications that were previously computationally infeasible, including (a) pattern fitting, (b) co-simulation with the physical instrument for operando analytics, data exploration, and decision support, (c) data creation and integration into machine learning workflows, and (d) utilization in multi-modal data assimilation approaches. Finally, we abstract away the complexity of the computational framework from the end-user by exposing CyRSoXS to Python using Pybind. This eliminates I/O requirements for large-scale parameter exploration and inverse design, and democratizes usage by enabling seamless integration with a Python ecosystem (https://github.com/usnistgov/nrss).
△ Less
Submitted 26 September, 2022;
originally announced September 2022.
-
Scalable adaptive algorithms for next-generation multiphase flow simulations
Authors:
Kumar Saurabh,
Masado Ishii,
Makrand A. Khanwale,
Hari Sundar,
Baskar Ganapathysubramanian
Abstract:
High-fidelity flow simulations are indispensable when analyzing systems exhibiting multiphase flow phenomena. The accuracy of multiphase flow simulations is strongly contingent upon the finest mesh resolution used to represent the fluid-fluid interfaces. However, the increased resolution comes at a higher computational cost. In this work, we propose algorithmic advances that aim to reduce the comp…
▽ More
High-fidelity flow simulations are indispensable when analyzing systems exhibiting multiphase flow phenomena. The accuracy of multiphase flow simulations is strongly contingent upon the finest mesh resolution used to represent the fluid-fluid interfaces. However, the increased resolution comes at a higher computational cost. In this work, we propose algorithmic advances that aim to reduce the computational cost without compromising on the physics by selectively detecting key regions of interest (droplets/filaments) that require significantly higher resolution. The framework uses an adaptive octree-based meshing framework that is integrated with PETSc's linear algebra solvers. We demonstrate scaling of the framework up to 114,688 processes on TACC's Frontera. Finally, we deploy the framework to simulate one of the most resolved simulations of primary jet atomization. This simulation -- equivalent to 35 trillion grid points on a uniform grid -- is 64 times larger than the current state-of-the-art simulations and provides unprecedented insights into an important flow physics problem with a diverse array of engineering applications.
△ Less
Submitted 3 April, 2023; v1 submitted 24 September, 2022;
originally announced September 2022.
-
A projection-based, semi-implicit time-step** approach for the Cahn-Hilliard Navier-Stokes equations on adaptive octree meshes
Authors:
Makrand A. Khanwale,
Kumar Saurabh,
Masado Ishii,
Hari Sundar,
James A. Rossmanith,
Baskar Ganapathysubramanian
Abstract:
The Cahn-Hilliard Navier-Stokes (CHNS) system provides a computationally tractable model that can be used to effectively capture interfacial dynamics in two-phase fluid flows. In this work, we present a semi-implicit, projection-based finite element framework for solving the CHNS system. We use a projection-based semi-implicit time discretization for the Navier-Stokes equation and a fully-implicit…
▽ More
The Cahn-Hilliard Navier-Stokes (CHNS) system provides a computationally tractable model that can be used to effectively capture interfacial dynamics in two-phase fluid flows. In this work, we present a semi-implicit, projection-based finite element framework for solving the CHNS system. We use a projection-based semi-implicit time discretization for the Navier-Stokes equation and a fully-implicit time discretization for the Cahn-Hilliard equation. We use a conforming continuous Galerkin (cG) finite element method in space equipped with a residual-based variational multiscale (RBVMS) formulation. Pressure is decoupled using a projection step, which results in two linear positive semi-definite systems for velocity and pressure, instead of the saddle point system of a pressure-stabilized method. All the linear systems are solved using an efficient and scalable algebraic multigrid (AMG) method. We deploy this approach on a massively parallel numerical implementation using parallel octree-based adaptive meshes. The overall approach allows the use of relatively large time steps with much faster time-to-solve than similar fully-implicit methods. We present comprehensive numerical experiments showing detailed comparisons with results from the literature for canonical cases, including the single bubble rise and Rayleigh-Taylor instability.
△ Less
Submitted 23 December, 2022; v1 submitted 11 July, 2021;
originally announced July 2021.
-
A fully-coupled framework for solving Cahn-Hilliard Navier-Stokes equations: Second-order, energy-stable numerical methods on adaptive octree based meshes
Authors:
Makrand A Khanwale,
Kumar Saurabh,
Milinda Fernando,
Victor M. Calo,
James A. Rossmanith,
Hari Sundar,
Baskar Ganapathysubramanian
Abstract:
We present a fully-coupled, implicit-in-time framework for solving a thermodynamically-consistent Cahn-Hilliard Navier-Stokes system that models two-phase flows. In this work, we extend the block iterative method presented in Khanwale et al. [\textit{Simulating two-phase flows with thermodynamically consistent energy stable Cahn-Hilliard Navier-Stokes equations on parallel adaptive octree based me…
▽ More
We present a fully-coupled, implicit-in-time framework for solving a thermodynamically-consistent Cahn-Hilliard Navier-Stokes system that models two-phase flows. In this work, we extend the block iterative method presented in Khanwale et al. [\textit{Simulating two-phase flows with thermodynamically consistent energy stable Cahn-Hilliard Navier-Stokes equations on parallel adaptive octree based meshes}, J. Comput. Phys. (2020)], to a fully-coupled, provably second-order accurate scheme in time, while maintaining energy-stability. The new method requires fewer matrix assemblies in each Newton iteration resulting in faster solution time. The method is based on a fully-implicit Crank-Nicolson scheme in time and a pressure stabilization for an equal order Galerkin formulation. That is, we use a conforming continuous Galerkin (cG) finite element method in space equipped with a residual-based variational multiscale (RBVMS) procedure to stabilize the pressure. We deploy this approach on a massively parallel numerical implementation using parallel octree-based adaptive meshes. We present comprehensive numerical experiments showing detailed comparisons with results from the literature for canonical cases, including the single bubble rise, Rayleigh-Taylor instability, and lid-driven cavity flow problems. We analyze in detail the scaling of our numerical implementation.
△ Less
Submitted 13 August, 2022; v1 submitted 13 September, 2020;
originally announced September 2020.
-
Industrial scale large eddy simulations (LES) with adaptive octree meshes using immersogeometric analysis
Authors:
Kumar Saurabh,
Boshun Gao,
Milinda Fernando,
Songzhe Xu,
Makrand A. Khanwale,
Biswajit Khara,
Ming-Chen Hsu,
Adarsh Krishnamurthy,
Hari Sundar,
Baskar Ganapathysubramanian
Abstract:
We present a variant of the immersed boundary method integrated with octree meshes for highly efficient and accurate Large-Eddy Simulations (LES) of flows around complex geometries. We demonstrate the scalability of the proposed method up to $\mathcal{O}(32K)$ processors. This is achieved by (a) rapid in-out tests; (b) adaptive quadrature for an accurate evaluation of forces; (c) tensorized evalua…
▽ More
We present a variant of the immersed boundary method integrated with octree meshes for highly efficient and accurate Large-Eddy Simulations (LES) of flows around complex geometries. We demonstrate the scalability of the proposed method up to $\mathcal{O}(32K)$ processors. This is achieved by (a) rapid in-out tests; (b) adaptive quadrature for an accurate evaluation of forces; (c) tensorized evaluation during matrix assembly. We showcase this method on two non-trivial applications: accurately computing the drag coefficient of a sphere across Reynolds numbers $1-10^6$ encompassing the drag crisis regime; simulating flow features across a semi-truck for investigating the effect of platooning on efficiency.
△ Less
Submitted 9 February, 2021; v1 submitted 28 August, 2020;
originally announced September 2020.
