-
A Benchmark Dataset for Tornado Detection and Prediction using Full-Resolution Polarimetric Weather Radar Data
Authors:
Mark S. Veillette,
James M. Kurdzo,
Phillip M. Stepanian,
John Y. N. Cho,
Siddharth Samsi,
Joseph McDonald
Abstract:
Weather radar is the primary tool used by forecasters to detect and warn for tornadoes in near-real time. In order to assist forecasters in warning the public, several algorithms have been developed to automatically detect tornadic signatures in weather radar observations. Recently, Machine Learning (ML) algorithms, which learn directly from large amounts of labeled data, have been shown to be hig…
▽ More
Weather radar is the primary tool used by forecasters to detect and warn for tornadoes in near-real time. In order to assist forecasters in warning the public, several algorithms have been developed to automatically detect tornadic signatures in weather radar observations. Recently, Machine Learning (ML) algorithms, which learn directly from large amounts of labeled data, have been shown to be highly effective for this purpose. Since tornadoes are extremely rare events within the corpus of all available radar observations, the selection and design of training datasets for ML applications is critical for the performance, robustness, and ultimate acceptance of ML algorithms. This study introduces a new benchmark dataset, TorNet to support development of ML algorithms in tornado detection and prediction. TorNet contains full-resolution, polarimetric, Level-II WSR-88D data sampled from 10 years of reported storm events. A number of ML baselines for tornado detection are developed and compared, including a novel deep learning (DL) architecture capable of processing raw radar imagery without the need for manual feature extraction required for existing ML algorithms. Despite not benefiting from manual feature engineering or other preprocessing, the DL model shows increased detection performance compared to non-DL and operational baselines. The TorNet dataset, as well as source code and model weights of the DL baseline trained in this work, are made freely available.
△ Less
Submitted 26 January, 2024;
originally announced January 2024.
-
A Deep Learning-based Velocity Dealiasing Algorithm Derived from the WSR-88D Open Radar Product Generator
Authors:
Mark S. Veillette,
James M. Kurdzo,
Phillip M. Stepanian,
Joseph McDonald,
Siddharth Samsi,
John Y. N. Cho
Abstract:
Radial velocity estimates provided by Doppler weather radar are critical measurements used by operational forecasters for the detection and monitoring of life-impacting storms. The sampling methods used to produce these measurements are inherently susceptible to aliasing, which produces ambiguous velocity values in regions with high winds, and needs to be corrected using a velocity dealiasing algo…
▽ More
Radial velocity estimates provided by Doppler weather radar are critical measurements used by operational forecasters for the detection and monitoring of life-impacting storms. The sampling methods used to produce these measurements are inherently susceptible to aliasing, which produces ambiguous velocity values in regions with high winds, and needs to be corrected using a velocity dealiasing algorithm (VDA). In the US, the Weather Surveillance Radar-1988 Doppler (WSR-88D) Open Radar Product Generator (ORPG) is a processing environment that provides a world-class VDA; however, this algorithm is complex and can be difficult to port to other radar systems outside of the WSR-88D network. In this work, a Deep Neural Network (DNN) is used to emulate the 2-dimensional WSR-88D ORPG dealiasing algorithm. It is shown that a DNN, specifically a customized U-Net, is highly effective for building VDAs that are accurate, fast, and portable to multiple radar types. To train the DNN model, a large dataset is generated containing aligned samples of folded and dealiased velocity pairs. This dataset contains samples collected from WSR-88D Level-II and Level-III archives, and uses the ORPG dealiasing algorithm output as a source of truth. Using this dataset, a U-Net is trained to produce the number of folds at each point of a velocity image. Several performance metrics are presented using WSR-88D data. The algorithm is also applied to other non-WSR-88D radar systems to demonstrate portability to other hardware/software interfaces. A discussion of the broad applicability of this method is presented, including how other Level-III algorithms may benefit from this approach.
△ Less
Submitted 30 March, 2023; v1 submitted 23 November, 2022;
originally announced November 2022.
-
Distributed Deep Learning for Precipitation Nowcasting
Authors:
Siddharth Samsi,
Christopher J. Mattioli,
Mark S. Veillette
Abstract:
Effective training of Deep Neural Networks requires massive amounts of data and compute. As a result, longer times are needed to train complex models requiring large datasets, which can severely limit research on model development and the exploitation of all available data. In this paper, this problem is investigated in the context of precipitation nowcasting, a term used to describe highly detail…
▽ More
Effective training of Deep Neural Networks requires massive amounts of data and compute. As a result, longer times are needed to train complex models requiring large datasets, which can severely limit research on model development and the exploitation of all available data. In this paper, this problem is investigated in the context of precipitation nowcasting, a term used to describe highly detailed short-term forecasts of precipitation and other hazardous weather. Convolutional Neural Networks (CNNs) are a powerful class of models that are well-suited for this task; however, the high resolution input weather imagery combined with model complexity required to process this data makes training CNNs to solve this task time consuming. To address this issue, a data-parallel model is implemented where a CNN is replicated across multiple compute nodes and the training batches are distributed across multiple nodes. By leveraging multiple GPUs, we show that the training time for a given nowcasting model architecture can be reduced from 59 hours to just over 1 hour. This will allow for faster iterations for improving CNN architectures and will facilitate future advancement in the area of nowcasting.
△ Less
Submitted 28 August, 2019;
originally announced August 2019.
-
Properties and numerical evaluation of the Rosenblatt distribution
Authors:
Mark S. Veillette,
Murad S. Taqqu
Abstract:
This paper studies various distributional properties of the Rosenblatt distribution. We begin by describing a technique for computing the cumulants. We then study the expansion of the Rosenblatt distribution in terms of shifted chi-squared distributions. We derive the coefficients of this expansion and use these to obtain the Lévy-Khintchine formula and derive asymptotic properties of the Lévy mea…
▽ More
This paper studies various distributional properties of the Rosenblatt distribution. We begin by describing a technique for computing the cumulants. We then study the expansion of the Rosenblatt distribution in terms of shifted chi-squared distributions. We derive the coefficients of this expansion and use these to obtain the Lévy-Khintchine formula and derive asymptotic properties of the Lévy measure. This allows us to compute the cumulants, moments, coefficients in the chi-square expansion and the density and cumulative distribution functions of the Rosenblatt distribution with a high degree of precision. Tables are provided and software written to implement the methods described here is freely available by request from the authors.
△ Less
Submitted 23 July, 2013;
originally announced July 2013.
-
Berry-Esseen and Edgeworth approximations for the tail of an infinite sum of weighted gamma random variables
Authors:
Mark S. Veillette,
Murad S. Taqqu
Abstract:
Consider the sum $Z = \sum_{n=1}^\infty λ_n (η_n - \mathbb{E}η_n)$, where $η_n$ are i.i.d.~gamma random variables with shape parameter $r > 0$, and the $λ_n$'s are predetermined weights. We study the asymptotic behavior of the tail $\sum_{n=M}^\infty λ_n (η_n - \mathbb{E}η_n)$ which is asymptotically normal under certain conditions. We derive a Berry-Essen bound and Edgeworth expansions for its di…
▽ More
Consider the sum $Z = \sum_{n=1}^\infty λ_n (η_n - \mathbb{E}η_n)$, where $η_n$ are i.i.d.~gamma random variables with shape parameter $r > 0$, and the $λ_n$'s are predetermined weights. We study the asymptotic behavior of the tail $\sum_{n=M}^\infty λ_n (η_n - \mathbb{E}η_n)$ which is asymptotically normal under certain conditions. We derive a Berry-Essen bound and Edgeworth expansions for its distribution function. We illustrate the effectiveness of these expansions on an infinite sum of weighted chi-squared distributions.
△ Less
Submitted 19 October, 2010;
originally announced October 2010.
-
Technique for computing the PDFs and CDFs of non-negative infinitely divisible random variables
Authors:
Mark S. Veillette,
Murad S. Taqqu
Abstract:
We present a method for computing the PDF and CDF of a non-negative infinitely divisible random variable $X$. Our method uses the Lévy-Khintchine representation of the Laplace transform $\mathbb{E} e^{-λX} = e^{-φ(λ)}$, where $φ$ is the Laplace exponent. We apply the Post-Widder method for Laplace transform inversion combined with a sequence convergence accelerator to obtain accurate results.…
▽ More
We present a method for computing the PDF and CDF of a non-negative infinitely divisible random variable $X$. Our method uses the Lévy-Khintchine representation of the Laplace transform $\mathbb{E} e^{-λX} = e^{-φ(λ)}$, where $φ$ is the Laplace exponent. We apply the Post-Widder method for Laplace transform inversion combined with a sequence convergence accelerator to obtain accurate results. We demonstrate this technique on several examples including the stable distribution, mixtures thereof, and integrals with respect to non-negative Lévy processes. Software to implement this method is available from the authors and we illustrate its use at the end of the paper.
△ Less
Submitted 14 May, 2010;
originally announced May 2010.
-
Distribution functions of Poisson random integrals: Analysis and computation
Authors:
Mark S. Veillette,
Murad S. Taqqu
Abstract:
We want to compute the cumulative distribution function of a one-dimensional Poisson stochastic integral $I(\krnl) = \displaystyle \int_0^T \krnl(s) N(ds)$, where $N$ is a Poisson random measure with control measure $n$ and $\krnl$ is a suitable kernel function. We do so by combining a Kolmogorov-Feller equation with a finite-difference scheme. We provide the rate of convergence of our numerical…
▽ More
We want to compute the cumulative distribution function of a one-dimensional Poisson stochastic integral $I(\krnl) = \displaystyle \int_0^T \krnl(s) N(ds)$, where $N$ is a Poisson random measure with control measure $n$ and $\krnl$ is a suitable kernel function. We do so by combining a Kolmogorov-Feller equation with a finite-difference scheme. We provide the rate of convergence of our numerical scheme and illustrate our method on a number of examples. The software used to implement the procedure is available on demand and we demonstrate its use in the paper.
△ Less
Submitted 29 April, 2010;
originally announced April 2010.
-
Using Differential Equations to Obtain Joint Moments of First-Passage Times of Increasing Levy Processes
Authors:
Mark S. Veillette,
Murad S. Taqqu
Abstract:
Let $\{D(s), s \geq 0 \}$ be a Lévy subordinator, that is, a non-decreasing process with stationary and independent increments and suppose that $D(0) = 0$. We study the first-hitting time of the process $D$, namely, the process $E(t) = \inf \{s: D(s) > t \}$, $t \geq 0$.
The process $E$ is, in general, non-Markovian with non-stationary and non-independent increments. We derive a partial differ…
▽ More
Let $\{D(s), s \geq 0 \}$ be a Lévy subordinator, that is, a non-decreasing process with stationary and independent increments and suppose that $D(0) = 0$. We study the first-hitting time of the process $D$, namely, the process $E(t) = \inf \{s: D(s) > t \}$, $t \geq 0$.
The process $E$ is, in general, non-Markovian with non-stationary and non-independent increments. We derive a partial differential equation for the Laplace transform of the $n$-time tail distribution function $P[E(t_1) > s_1,...,E(t_n) > s_n]$, and show that this PDE has a unique solution given natural boundary conditions. This PDE can be used to derive all $n$-time moments of the process $E$.
△ Less
Submitted 27 June, 2009;
originally announced June 2009.
-
Numerical Computation of First-Passage Times of Increasing Levy Processes
Authors:
Mark S. Veillette,
Murad S. Taqqu
Abstract:
Let $\{D(s), s \geq 0\}$ be a non-decreasing Lévy process. The first-hitting time process $\{E(t) t \geq 0\}$ (which is sometimes referred to as an inverse subordinator) defined by $E(t) = \inf \{s: D(s) > t \}$ is a process which has arisen in many applications. Of particular interest is the mean first-hitting time $U(t)=\mathbb{E}E(t)$. This function characterizes all finite-dimensional distri…
▽ More
Let $\{D(s), s \geq 0\}$ be a non-decreasing Lévy process. The first-hitting time process $\{E(t) t \geq 0\}$ (which is sometimes referred to as an inverse subordinator) defined by $E(t) = \inf \{s: D(s) > t \}$ is a process which has arisen in many applications. Of particular interest is the mean first-hitting time $U(t)=\mathbb{E}E(t)$. This function characterizes all finite-dimensional distributions of the process $E$. The function $U$ can be calculated by inverting the Laplace transform of the function $\widetilde{U}(λ) = (λφ(λ))^{-1}$, where $φ$ is the Lévy exponent of the subordinator $D$. In this paper, we give two methods for computing numerically the inverse of this Laplace transform. The first is based on the Bromwich integral and the second is based on the Post-Widder inversion formula. The software written to support this work is available from the authors and we illustrate its use at the end of the paper.
△ Less
Submitted 27 April, 2009;
originally announced April 2009.
