-
A 3D super-resolution of wind fields via physics-informed pixel-wise self-attention generative adversarial network
Authors:
Takuya Kurihana,
Kyongmin Yeo,
Daniela Szwarcman,
Bruce Elmegreen,
Karthik Mukkavilli,
Johannes Schmude,
Levente Klein
Abstract:
To mitigate global warming, greenhouse gas sources need to be resolved at a high spatial resolution and monitored in time to ensure the reduction and ultimately elimination of the pollution source. However, the complexity of computation in resolving high-resolution wind fields left the simulations impractical to test different time lengths and model configurations. This study presents a preliminar…
▽ More
To mitigate global warming, greenhouse gas sources need to be resolved at a high spatial resolution and monitored in time to ensure the reduction and ultimately elimination of the pollution source. However, the complexity of computation in resolving high-resolution wind fields left the simulations impractical to test different time lengths and model configurations. This study presents a preliminary development of a physics-informed super-resolution (SR) generative adversarial network (GAN) that super-resolves the three-dimensional (3D) low-resolution wind fields by upscaling x9 times. We develop a pixel-wise self-attention (PWA) module that learns 3D weather dynamics via a self-attention computation followed by a 2D convolution. We also employ a loss term that regularizes the self-attention map during pretraining, capturing the vertical convection process from input wind data. The new PWA SR-GAN shows the high-fidelity super-resolved 3D wind data, learns a wind structure at the high-frequency domain, and reduces the computational cost of a high-resolution wind simulation by x89.7 times.
△ Less
Submitted 20 December, 2023;
originally announced December 2023.
-
AI Foundation Models for Weather and Climate: Applications, Design, and Implementation
Authors:
S. Karthik Mukkavilli,
Daniel Salles Civitarese,
Johannes Schmude,
Johannes Jakubik,
Anne Jones,
Nam Nguyen,
Christopher Phillips,
Sujit Roy,
Shraddha Singh,
Campbell Watson,
Raghu Ganti,
Hendrik Hamann,
Udaysankar Nair,
Rahul Ramachandran,
Kommy Weldemariam
Abstract:
Machine learning and deep learning methods have been widely explored in understanding the chaotic behavior of the atmosphere and furthering weather forecasting. There has been increasing interest from technology companies, government institutions, and meteorological agencies in building digital twins of the Earth. Recent approaches using transformers, physics-informed machine learning, and graph n…
▽ More
Machine learning and deep learning methods have been widely explored in understanding the chaotic behavior of the atmosphere and furthering weather forecasting. There has been increasing interest from technology companies, government institutions, and meteorological agencies in building digital twins of the Earth. Recent approaches using transformers, physics-informed machine learning, and graph neural networks have demonstrated state-of-the-art performance on relatively narrow spatiotemporal scales and specific tasks. With the recent success of generative artificial intelligence (AI) using pre-trained transformers for language modeling and vision with prompt engineering and fine-tuning, we are now moving towards generalizable AI. In particular, we are witnessing the rise of AI foundation models that can perform competitively on multiple domain-specific downstream tasks. Despite this progress, we are still in the nascent stages of a generalizable AI model for global Earth system models, regional climate models, and mesoscale weather models. Here, we review current state-of-the-art AI approaches, primarily from transformer and operator learning literature in the context of meteorology. We provide our perspective on criteria for success towards a family of foundation models for nowcasting and forecasting weather and climate predictions. We also discuss how such models can perform competitively on downstream tasks such as downscaling (super-resolution), identifying conditions conducive to the occurrence of wildfires, and predicting consequential meteorological phenomena across various spatiotemporal scales such as hurricanes and atmospheric rivers. In particular, we examine current AI methodologies and contend they have matured enough to design and implement a weather foundation model.
△ Less
Submitted 19 September, 2023; v1 submitted 19 September, 2023;
originally announced September 2023.
-
TensorBank: Tensor Lakehouse for Foundation Model Training
Authors:
Romeo Kienzler,
Leonardo Pondian Tizzei,
Benedikt Blumenstiel,
Zoltan Arnold Nagy,
S. Karthik Mukkavilli,
Johannes Schmude,
Marcus Freitag,
Michael Behrendt,
Daniel Salles Civitarese,
Naomi Simumba,
Daiki Kimura,
Hendrik Hamann
Abstract:
Storing and streaming high dimensional data for foundation model training became a critical requirement with the rise of foundation models beyond natural language. In this paper we introduce TensorBank, a petabyte scale tensor lakehouse capable of streaming tensors from Cloud Object Store (COS) to GPU memory at wire speed based on complex relational queries. We use Hierarchical Statistical Indices…
▽ More
Storing and streaming high dimensional data for foundation model training became a critical requirement with the rise of foundation models beyond natural language. In this paper we introduce TensorBank, a petabyte scale tensor lakehouse capable of streaming tensors from Cloud Object Store (COS) to GPU memory at wire speed based on complex relational queries. We use Hierarchical Statistical Indices (HSI) for query acceleration. Our architecture allows to directly address tensors on block level using HTTP range reads. Once in GPU memory, data can be transformed using PyTorch transforms. We provide a generic PyTorch dataset type with a corresponding dataset factory translating relational queries and requested transformations as an instance. By making use of the HSI, irrelevant blocks can be skipped without reading them as those indices contain statistics on their content at different hierarchical resolution levels. This is an opinionated architecture powered by open standards and making heavy use of open-source technology. Although, hardened for production use using geospatial-temporal data, this architecture generalizes to other use case like computer vision, computational neuroscience, biological sequence analysis and more.
△ Less
Submitted 21 March, 2024; v1 submitted 5 September, 2023;
originally announced September 2023.
-
On polynomial grammars extended with substitution
Authors:
Janusz Schmude
Abstract:
We investigate decidability of equivalence of register transducers, also called copyful Streaming String Transducers in case of string input, extended with an operation of substituting a register for all occurrences of a given letter in another register. We reduce to zeroness of polynomial grammars (over ring of polynomials) extended with analogous substitution operation by encoding strings into p…
▽ More
We investigate decidability of equivalence of register transducers, also called copyful Streaming String Transducers in case of string input, extended with an operation of substituting a register for all occurrences of a given letter in another register. We reduce to zeroness of polynomial grammars (over ring of polynomials) extended with analogous substitution operation by encoding strings into polynomials; a similar method was used successfully by Seidl et al. in 2018. We give two restrictions under which register transducers with substitution have decidable equivalence. They seem to be very restrictive but on the other hand, they seem to be on the edge of the scope of this "polynomial" method, as in the third result we give a rather restricted model of polynomial grammars with substitution that has undecidable equivalence.
△ Less
Submitted 17 February, 2021;
originally announced February 2021.
-
Reducing Transducer Equivalence to Register Automata Problems Solved by "Hilbert Method"
Authors:
Adrien Boiret,
Radosław Piórkowski,
Janusz Schmude
Abstract:
In the past decades, classical results from algebra, including Hilbert's Basis Theorem, had various applications in formal languages, including a proof of the Ehrenfeucht Conjecture, decidability of HDT0L sequence equivalence, and decidability of the equivalence problem for functional tree-to-string transducers. In this paper, we study the scope of the algebraic methods mentioned above, particular…
▽ More
In the past decades, classical results from algebra, including Hilbert's Basis Theorem, had various applications in formal languages, including a proof of the Ehrenfeucht Conjecture, decidability of HDT0L sequence equivalence, and decidability of the equivalence problem for functional tree-to-string transducers. In this paper, we study the scope of the algebraic methods mentioned above, particularily as applied to the equivalence problem for register automata. We provide two results, one positive, one negative. The positive result is that equivalence is decidable for MSO transformations on unordered forests. The negative result comes from a try to extend this method to decide equivalence on macro tree transducers. We reduce macro tree transducers equivalence to an equivalence problem for some class of register automata naturally relevant to our method. We then prove this latter problem to be undecidable.
△ Less
Submitted 4 January, 2019; v1 submitted 12 June, 2018;
originally announced June 2018.
