-
Generative Data Assimilation of Sparse Weather Station Observations at Kilometer Scales
Authors:
Peter Manshausen,
Yair Cohen,
Jaideep Pathak,
Mike Pritchard,
Piyush Garg,
Morteza Mardani,
Karthik Kashinath,
Simon Byrne,
Noah Brenowitz
Abstract:
Data assimilation of observational data into full atmospheric states is essential for weather forecast model initialization. Recently, methods for deep generative data assimilation have been proposed which allow for using new input data without retraining the model. They could also dramatically accelerate the costly data assimilation process used in operational regional weather models. Here, in a…
▽ More
Data assimilation of observational data into full atmospheric states is essential for weather forecast model initialization. Recently, methods for deep generative data assimilation have been proposed which allow for using new input data without retraining the model. They could also dramatically accelerate the costly data assimilation process used in operational regional weather models. Here, in a central US testbed, we demonstrate the viability of score-based data assimilation in the context of realistically complex km-scale weather. We train an unconditional diffusion model to generate snapshots of a state-of-the-art km-scale analysis product, the High Resolution Rapid Refresh. Then, using score-based data assimilation to incorporate sparse weather station data, the model produces maps of precipitation and surface winds. The generated fields display physically plausible structures, such as gust fronts, and sensitivity tests confirm learnt physics through multivariate relationships. Preliminary skill analysis shows the approach already outperforms a naive baseline of the High-Resolution Rapid Refresh system itself. By incorporating observations from 40 weather stations, 10\% lower RMSEs on left-out stations are attained. Despite some lingering imperfections such as insufficiently disperse ensemble DA estimates, we find the results overall an encouraging proof of concept, and the first at km-scale. It is a ripe time to explore extensions that combine increasingly ambitious regional state generators with an increasing set of in situ, ground-based, and satellite remote sensing data streams.
△ Less
Submitted 19 June, 2024;
originally announced June 2024.
-
von Mises Quasi-Processes for Bayesian Circular Regression
Authors:
Yarden Cohen,
Alexandre Khae Wu Navarro,
Jes Frellsen,
Richard E. Turner,
Raziel Riemer,
Ari Pakman
Abstract:
The need for regression models to predict circular values arises in many scientific fields. In this work we explore a family of expressive and interpretable distributions over circle-valued random functions related to Gaussian processes targeting two Euclidean dimensions conditioned on the unit circle. The resulting probability model has connections with continuous spin models in statistical physi…
▽ More
The need for regression models to predict circular values arises in many scientific fields. In this work we explore a family of expressive and interpretable distributions over circle-valued random functions related to Gaussian processes targeting two Euclidean dimensions conditioned on the unit circle. The resulting probability model has connections with continuous spin models in statistical physics. Moreover, its density is very simple and has maximum-entropy, unlike previous Gaussian process-based approaches, which use wrap** or radial marginalization. For posterior inference, we introduce a new Stratonovich-like augmentation that lends itself to fast Markov Chain Monte Carlo sampling. We argue that transductive learning in these models favors a Bayesian approach to the parameters. We present experiments applying this model to the prediction of (i) wind directions and (ii) the percentage of the running gait cycle as a function of joint angles.
△ Less
Submitted 18 June, 2024;
originally announced June 2024.
-
Coupled Ocean-Atmosphere Dynamics in a Machine Learning Earth System Model
Authors:
Chenggong Wang,
Michael S. Pritchard,
Noah Brenowitz,
Yair Cohen,
Boris Bonev,
Thorsten Kurth,
Dale Durran,
Jaideep Pathak
Abstract:
Seasonal climate forecasts are socioeconomically important for managing the impacts of extreme weather events and for planning in sectors like agriculture and energy. Climate predictability on seasonal timescales is tied to boundary effects of the ocean on the atmosphere and coupled interactions in the ocean-atmosphere system. We present the Ocean-linked-atmosphere (Ola) model, a high-resolution (…
▽ More
Seasonal climate forecasts are socioeconomically important for managing the impacts of extreme weather events and for planning in sectors like agriculture and energy. Climate predictability on seasonal timescales is tied to boundary effects of the ocean on the atmosphere and coupled interactions in the ocean-atmosphere system. We present the Ocean-linked-atmosphere (Ola) model, a high-resolution (0.25°) Artificial Intelligence/ Machine Learning (AI/ML) coupled earth-system model which separately models the ocean and atmosphere dynamics using an autoregressive Spherical Fourier Neural Operator architecture, with a view towards enabling fast, accurate, large ensemble forecasts on the seasonal timescale. We find that Ola exhibits learned characteristics of ocean-atmosphere coupled dynamics including tropical oceanic waves with appropriate phase speeds, and an internally generated El Niño/Southern Oscillation (ENSO) having realistic amplitude, geographic structure, and vertical structure within the ocean mixed layer. We present initial evidence of skill in forecasting the ENSO which compares favorably to the SPEAR model of the Geophysical Fluid Dynamics Laboratory.
△ Less
Submitted 12 June, 2024;
originally announced June 2024.
-
DiffObs: Generative Diffusion for Global Forecasting of Satellite Observations
Authors:
Jason Stock,
Jaideep Pathak,
Yair Cohen,
Mike Pritchard,
Piyush Garg,
Dale Durran,
Morteza Mardani,
Noah Brenowitz
Abstract:
This work presents an autoregressive generative diffusion model (DiffObs) to predict the global evolution of daily precipitation, trained on a satellite observational product, and assessed with domain-specific diagnostics. The model is trained to probabilistically forecast day-ahead precipitation. Nonetheless, it is stable for multi-month rollouts, which reveal a qualitatively realistic superposit…
▽ More
This work presents an autoregressive generative diffusion model (DiffObs) to predict the global evolution of daily precipitation, trained on a satellite observational product, and assessed with domain-specific diagnostics. The model is trained to probabilistically forecast day-ahead precipitation. Nonetheless, it is stable for multi-month rollouts, which reveal a qualitatively realistic superposition of convectively coupled wave modes in the tropics. Cross-spectral analysis confirms successful generation of low frequency variations associated with the Madden--Julian oscillation, which regulates most subseasonal to seasonal predictability in the observed atmosphere, and convectively coupled moist Kelvin waves with approximately correct dispersion relationships. Despite secondary issues and biases, the results affirm the potential for a next generation of global diffusion models trained on increasingly sparse, and increasingly direct and differentiated observations of the world, for practical applications in subseasonal and climate prediction.
△ Less
Submitted 4 April, 2024;
originally announced April 2024.
-
SINR-Aware Deep Reinforcement Learning for Distributed Dynamic Channel Allocation in Cognitive Interference Networks
Authors:
Yaniv Cohen,
Tomer Gafni,
Ronen Greenberg,
Kobi Cohen
Abstract:
We consider the problem of dynamic channel allocation (DCA) in cognitive communication networks with the goal of maximizing a global signal-to-interference-plus-noise ratio (SINR) measure under a specified target quality of service (QoS)-SINR for each network. The shared bandwidth is partitioned into K channels with frequency separation. In contrast to the majority of existing studies that assume…
▽ More
We consider the problem of dynamic channel allocation (DCA) in cognitive communication networks with the goal of maximizing a global signal-to-interference-plus-noise ratio (SINR) measure under a specified target quality of service (QoS)-SINR for each network. The shared bandwidth is partitioned into K channels with frequency separation. In contrast to the majority of existing studies that assume perfect orthogonality or a one- to-one user-channel allocation map**, this paper focuses on real-world systems experiencing inter-carrier interference (ICI) and channel reuse by multiple large-scale networks. This realistic scenario significantly increases the problem dimension, rendering existing algorithms inefficient. We propose a novel multi-agent reinforcement learning (RL) framework for distributed DCA, named Channel Allocation RL To Overlapped Networks (CARLTON). The CARLTON framework is based on the Centralized Training with Decentralized Execution (CTDE) paradigm, utilizing the DeepMellow value-based RL algorithm. To ensure robust performance in the interference-laden environment we address, CARLTON employs a low-dimensional representation of observations, generating a QoS-type measure while maximizing a global SINR measure and ensuring the target QoS-SINR for each network. Our results demonstrate exceptional performance and robust generalization, showcasing superior efficiency compared to alternative state-of-the-art methods, while achieving a marginally diminished performance relative to a fully centralized approach.
△ Less
Submitted 17 February, 2024;
originally announced February 2024.
-
A Practical Probabilistic Benchmark for AI Weather Models
Authors:
Noah D. Brenowitz,
Yair Cohen,
Jaideep Pathak,
Ankur Mahesh,
Boris Bonev,
Thorsten Kurth,
Dale R. Durran,
Peter Harrington,
Michael S. Pritchard
Abstract:
Since the weather is chaotic, forecasts aim to predict the distribution of future states rather than make a single prediction. Recently, multiple data driven weather models have emerged claiming breakthroughs in skill. However, these have mostly been benchmarked using deterministic skill scores, and little is known about their probabilistic skill. Unfortunately, it is hard to fairly compare AI wea…
▽ More
Since the weather is chaotic, forecasts aim to predict the distribution of future states rather than make a single prediction. Recently, multiple data driven weather models have emerged claiming breakthroughs in skill. However, these have mostly been benchmarked using deterministic skill scores, and little is known about their probabilistic skill. Unfortunately, it is hard to fairly compare AI weather models in a probabilistic sense, since variations in choice of ensemble initialization, definition of state, and noise injection methodology become confounding. Moreover, even obtaining ensemble forecast baselines is a substantial engineering challenge given the data volumes involved. We sidestep both problems by applying a decades-old idea -- lagged ensembles -- whereby an ensemble can be constructed from a moderately-sized library of deterministic forecasts. This allows the first parameter-free intercomparison of leading AI weather models' probabilistic skill against an operational baseline. The results reveal that two leading AI weather models, i.e. GraphCast and Pangu, are tied on the probabilistic CRPS metric even though the former outperforms the latter in deterministic scoring. We also reveal how multiple time-step loss functions, which many data-driven weather models have employed, are counter-productive: they improve deterministic metrics at the cost of increased dissipation, deteriorating probabilistic skill. This is confirmed through ablations applied to a spherical Fourier Neural Operator (SFNO) approach to AI weather forecasting. Separate SFNO ablations modulating effective resolution reveal it has a useful effect on ensemble dispersion relevant to achieving good ensemble calibration. We hope these and forthcoming insights from lagged ensembles can help guide the development of AI weather forecasts and have thus shared the diagnostic code.
△ Less
Submitted 27 January, 2024;
originally announced January 2024.
-
Residual Diffusion Modeling for Km-scale Atmospheric Downscaling
Authors:
Morteza Mardani,
Noah Brenowitz,
Yair Cohen,
Jaideep Pathak,
Chieh-Yu Chen,
Cheng-Chin Liu,
Arash Vahdat,
Karthik Kashinath,
Jan Kautz,
Mike Pritchard
Abstract:
Predictions of weather hazard require expensive km-scale simulations driven by coarser global inputs. Here, a cost-effective stochastic downscaling model is trained from a high-resolution 2-km weather model over Taiwan conditioned on 25-km ERA5 reanalysis. To address the multi-scale machine learning challenges of weather data, we employ a two-step approach Corrector Diffusion (\textit{CorrDiff}),…
▽ More
Predictions of weather hazard require expensive km-scale simulations driven by coarser global inputs. Here, a cost-effective stochastic downscaling model is trained from a high-resolution 2-km weather model over Taiwan conditioned on 25-km ERA5 reanalysis. To address the multi-scale machine learning challenges of weather data, we employ a two-step approach Corrector Diffusion (\textit{CorrDiff}), where a UNet prediction of the mean is corrected by a diffusion step. Akin to Reynolds decomposition in fluid dynamics, this isolates generative learning to the stochastic scales. \textit{CorrDiff} exhibits skillful RMSE and CRPS and faithfully recovers spectra and distributions even for extremes. Case studies of coherent weather phenomena reveal appropriate multivariate relationships reminiscent of learnt physics: the collocation of intense rainfall and sharp gradients in fronts and extreme winds and rainfall bands near the eyewall of typhoons. Downscaling global forecasts successfully retains many of these benefits, foreshadowing the potential of end-to-end, global-to-km-scales machine learning weather predictions.
△ Less
Submitted 9 December, 2023; v1 submitted 24 September, 2023;
originally announced September 2023.
-
A Permutation-Based Heuristic for Buy Low, Sell High
Authors:
Yair Neuman,
Yochai Cohen
Abstract:
Buy low, sell high is one of the basic rules of thumb used in investment, although it is not considered to be a beneficial strategy. In this paper, we show how the appropriate permutation-based representation (i.e., the epistemic form) of a minute-by-minute trading time-series, alongside the use of a simple decision heuristic (i.e., the epistemic game), may surprisingly result in significant benef…
▽ More
Buy low, sell high is one of the basic rules of thumb used in investment, although it is not considered to be a beneficial strategy. In this paper, we show how the appropriate permutation-based representation (i.e., the epistemic form) of a minute-by-minute trading time-series, alongside the use of a simple decision heuristic (i.e., the epistemic game), may surprisingly result in significant benefits. Using our heuristic for selecting seven stocks, we ran two experiments on the data. The results provide empirical support for the possible benefit of using simple decision models in investment, even in the context of minute-by-minute trading.
△ Less
Submitted 4 July, 2022;
originally announced July 2022.
-
A Methodology for a Scalable, Collaborative, and Resource-Efficient Platform to Facilitate Healthcare AI Research
Authors:
Raphael Y. Cohen,
Vesela P. Kovacheva
Abstract:
Healthcare AI holds the potential to increase patient safety, augment efficiency and improve patient outcomes, yet research is often limited by data access, cohort curation, and tooling for analysis. Collection and translation of electronic health record data, live data, and real-time high resolution device data can be challenging and time-consuming. The development of real-world AI tools requires…
▽ More
Healthcare AI holds the potential to increase patient safety, augment efficiency and improve patient outcomes, yet research is often limited by data access, cohort curation, and tooling for analysis. Collection and translation of electronic health record data, live data, and real-time high resolution device data can be challenging and time-consuming. The development of real-world AI tools requires overcoming challenges in data acquisition, scarce hospital resources and high needs for data governance. These bottlenecks may result in resource-heavy needs and long delays in research and development of AI systems. We present a system and methodology to accelerate data acquisition, dataset development and analysis, and AI model development. We created an interactive platform that relies on a scalable microservice backend. This system can ingest 15,000 patient records per hour, where each record represents thousands of multimodal measurements, text notes, and high resolution data. Collectively, these records can approach a terabyte of data. The system can further perform cohort generation and preliminary dataset analysis in 2-5 minutes. As a result, multiple users can collaborate simultaneously to iterate on datasets and models in real time. We anticipate that this approach will drive real-world AI model development, and, in the long run, meaningfully improve healthcare delivery.
△ Less
Submitted 13 December, 2021;
originally announced December 2021.
-
Dimensionality Reduction of Longitudinal 'Omics Data using Modern Tensor Factorization
Authors:
Uria Mor,
Yotam Cohen,
Rafael Valdes-Mas,
Denise Kviatcovsky,
Eran Elinav,
Haim Avron
Abstract:
Precision medicine is a clinical approach for disease prevention, detection and treatment, which considers each individual's genetic background, environment and lifestyle. The development of this tailored avenue has been driven by the increased availability of omics methods, large cohorts of temporal samples, and their integration with clinical data. Despite the immense progression, existing compu…
▽ More
Precision medicine is a clinical approach for disease prevention, detection and treatment, which considers each individual's genetic background, environment and lifestyle. The development of this tailored avenue has been driven by the increased availability of omics methods, large cohorts of temporal samples, and their integration with clinical data. Despite the immense progression, existing computational methods for data analysis fail to provide appropriate solutions for this complex, high-dimensional and longitudinal data. In this work we have developed a new method termed TCAM, a dimensionality reduction technique for multi-way data, that overcomes major limitations when doing trajectory analysis of longitudinal omics data. Using real-world data, we show that TCAM outperforms traditional methods, as well as state-of-the-art tensor-based approaches for longitudinal microbiome data analysis. Moreover, we demonstrate the versatility of TCAM by applying it to several different omics datasets, and the applicability of it as a drop-in replacement within straightforward ML tasks.
△ Less
Submitted 28 November, 2021;
originally announced November 2021.
-
An Orchestration Platform that Puts Radiologists in the Driver's Seat of AI Innovation: A Methodological Approach
Authors:
Raphael Y. Cohen,
Aaron D. Sodickson
Abstract:
Current AI-driven research in radiology requires resources and expertise that are often inaccessible to small and resource-limited labs. The clinicians who are able to participate in AI research are frequently well-funded, well-staffed, and either have significant experience with AI and computing, or have access to colleagues or facilities that do. Current imaging data is clinician-oriented and is…
▽ More
Current AI-driven research in radiology requires resources and expertise that are often inaccessible to small and resource-limited labs. The clinicians who are able to participate in AI research are frequently well-funded, well-staffed, and either have significant experience with AI and computing, or have access to colleagues or facilities that do. Current imaging data is clinician-oriented and is not easily amenable to machine learning initiatives, resulting in inefficient, time consuming, and costly efforts that rely upon a crew of data engineers and machine learning scientists, and all too often preclude radiologists from driving AI research and innovation. We present the system and methodology we have developed to address infrastructure and platform needs, while reducing the staffing and resource barriers to entry. We emphasize a data-first and modular approach that streamlines the AI development and deployment process while providing efficient and familiar interfaces for radiologists, such that they can be the drivers of new AI innovations.
△ Less
Submitted 6 July, 2021;
originally announced July 2021.
-
Interactive Extractive Search over Biomedical Corpora
Authors:
Hillel Taub-Tabib,
Micah Shlain,
Shoval Sadde,
Dan Lahav,
Matan Eyal,
Yaara Cohen,
Yoav Goldberg
Abstract:
We present a system that allows life-science researchers to search a linguistically annotated corpus of scientific texts using patterns over dependency graphs, as well as using patterns over token sequences and a powerful variant of boolean keyword queries. In contrast to previous attempts to dependency-based search, we introduce a light-weight query language that does not require the user to know…
▽ More
We present a system that allows life-science researchers to search a linguistically annotated corpus of scientific texts using patterns over dependency graphs, as well as using patterns over token sequences and a powerful variant of boolean keyword queries. In contrast to previous attempts to dependency-based search, we introduce a light-weight query language that does not require the user to know the details of the underlying linguistic representations, and instead to query the corpus by providing an example sentence coupled with simple markup. Search is performed at an interactive speed due to efficient linguistic graph-indexing and retrieval engine. This allows for rapid exploration, development and refinement of user queries. We demonstrate the system using example workflows over two corpora: the PubMed corpus including 14,446,243 PubMed abstracts and the CORD-19 dataset, a collection of over 45,000 research papers focused on COVID-19 research. The system is publicly available at https://allenai.github.io/spike
△ Less
Submitted 7 June, 2020;
originally announced June 2020.
-
Forgotten @ Scale: A Practical Solution for Implementing the Right To Be Forgotten in Large-Scale Systems
Authors:
Abigail Goldsteen,
Tomer Douek,
Yaniv Cohen,
Igor Gokhman,
Ofir Keren-Ackerman,
Gadi Katsovich,
Grisha Weintraub,
Doron Ben-Ari
Abstract:
The European General Data Protection Regulation asserts data subjects' right to be forgotten, i.e., their right to request that all their personal data be deleted from an organizations' data stores. However, fulfilling such requests in large-scale systems is technically challenging. It requires that organizations keep track of all locations in which an individual's data is stored, be able to acces…
▽ More
The European General Data Protection Regulation asserts data subjects' right to be forgotten, i.e., their right to request that all their personal data be deleted from an organizations' data stores. However, fulfilling such requests in large-scale systems is technically challenging. It requires that organizations keep track of all locations in which an individual's data is stored, be able to access and delete it in a reasonable time frame, and be able to prove that all such data was in fact deleted. In addition, organizations must cope with complexities such as multiple, distributed, and continuously evolving systems of record, complex data retention policies and deletion approval workflows. We present a first design pattern and practical implementation of the right to be forgotten on a large scale in Big Data and cloud environments.
△ Less
Submitted 30 October, 2019;
originally announced October 2019.
-
Node-Centric Detection of Overlap** Communities in Social Networks
Authors:
Yehonatan Cohen,
Danny Hendler,
Amir Rubin
Abstract:
We present NECTAR, a community detection algorithm that generalizes Louvain method's local search heuristic for overlap** community structures. NECTAR chooses dynamically which objective function to optimize based on the network on which it is invoked. Our experimental evaluation on both synthetic benchmark graphs and real-world networks, based on ground-truth communities, shows that NECTAR prov…
▽ More
We present NECTAR, a community detection algorithm that generalizes Louvain method's local search heuristic for overlap** community structures. NECTAR chooses dynamically which objective function to optimize based on the network on which it is invoked. Our experimental evaluation on both synthetic benchmark graphs and real-world networks, based on ground-truth communities, shows that NECTAR provides excellent results as compared with state of the art community detection algorithms.
△ Less
Submitted 6 July, 2016;
originally announced July 2016.
-
Mining The Successful Binary Combinations: Methodology and A Simple Case Study
Authors:
Yuval Cohen
Abstract:
The importance of finding the characteristics leading to either a success or a failure is one of the driving forces of data mining. The various application areas of finding success/failure factors cover vast variety of areas such as credit risk evaluation and granting loans, micro array analysis, health factors and health risk factors, and parameter combination leading to a product success. This…
▽ More
The importance of finding the characteristics leading to either a success or a failure is one of the driving forces of data mining. The various application areas of finding success/failure factors cover vast variety of areas such as credit risk evaluation and granting loans, micro array analysis, health factors and health risk factors, and parameter combination leading to a product success. This paper presents a new approach for making inferences about dichotomous data. The objective is to determine rules that lead to a certain result. The method consists of four phases: in the first phase, the data is processed into a binary format of a truth table, in the second phase; rules are found by utilizing an algorithm that minimizes Boolean functions. In the third phase the rules are checked and filtered. In the fourth phase, simple rules that involve one to two features are revealed.
△ Less
Submitted 5 February, 2010;
originally announced February 2010.