-
The future of cosmological likelihood-based inference: accelerated high-dimensional parameter estimation and model comparison
Authors:
Davide Piras,
Alicja Polanska,
Alessio Spurio Mancini,
Matthew A. Price,
Jason D. McEwen
Abstract:
We advocate for a new paradigm of cosmological likelihood-based inference, leveraging recent developments in machine learning and its underlying technology, to accelerate Bayesian inference in high-dimensional settings. Specifically, we combine (i) emulation, where a machine learning model is trained to mimic cosmological observables, e.g. CosmoPower-JAX; (ii) differentiable and probabilistic prog…
▽ More
We advocate for a new paradigm of cosmological likelihood-based inference, leveraging recent developments in machine learning and its underlying technology, to accelerate Bayesian inference in high-dimensional settings. Specifically, we combine (i) emulation, where a machine learning model is trained to mimic cosmological observables, e.g. CosmoPower-JAX; (ii) differentiable and probabilistic programming, e.g. JAX and NumPyro, respectively; (iii) scalable Markov chain Monte Carlo (MCMC) sampling techniques that exploit gradients, e.g. Hamiltonian Monte Carlo; and (iv) decoupled and scalable Bayesian model selection techniques that compute the Bayesian evidence purely from posterior samples, e.g. the learned harmonic mean implemented in harmonic. This paradigm allows us to carry out a complete Bayesian analysis, including both parameter estimation and model selection, in a fraction of the time of traditional approaches. First, we demonstrate the application of this paradigm on a simulated cosmic shear analysis for a Stage IV survey in 37- and 39-dimensional parameter spaces, comparing $Λ$CDM and a dynamical dark energy model ($w_0w_a$CDM). We recover posterior contours and evidence estimates that are in excellent agreement with those computed by the traditional nested sampling approach while reducing the computational cost from 8 months on 48 CPU cores to 2 days on 12 GPUs. Second, we consider a joint analysis between three simulated next-generation surveys, each performing a 3x2pt analysis, resulting in 157- and 159-dimensional parameter spaces. Standard nested sampling techniques are simply not feasible in this high-dimensional setting, requiring a projected 12 years of compute time on 48 CPU cores; on the other hand, the proposed approach only requires 8 days of compute time on 24 GPUs. All packages used in our analyses are publicly available.
△ Less
Submitted 21 May, 2024;
originally announced May 2024.
-
Learned harmonic mean estimation of the Bayesian evidence with normalizing flows
Authors:
Alicja Polanska,
Matthew A. Price,
Davide Piras,
Alessio Spurio Mancini,
Jason D. McEwen
Abstract:
We present the learned harmonic mean estimator with normalizing flows - a robust, scalable and flexible estimator of the Bayesian evidence for model comparison. Since the estimator is agnostic to sampling strategy and simply requires posterior samples, it can be applied to compute the evidence using any Markov chain Monte Carlo (MCMC) sampling technique, including saved down MCMC chains, or any va…
▽ More
We present the learned harmonic mean estimator with normalizing flows - a robust, scalable and flexible estimator of the Bayesian evidence for model comparison. Since the estimator is agnostic to sampling strategy and simply requires posterior samples, it can be applied to compute the evidence using any Markov chain Monte Carlo (MCMC) sampling technique, including saved down MCMC chains, or any variational inference approach. The learned harmonic mean estimator was recently introduced, where machine learning techniques were developed to learn a suitable internal importance sampling target distribution to solve the issue of exploding variance of the original harmonic mean estimator. In this article we present the use of normalizing flows as the internal machine learning technique within the learned harmonic mean estimator. Normalizing flows can be elegantly coupled with the learned harmonic mean to provide an approach that is more robust, flexible and scalable than the machine learning models considered previously. We perform a series of numerical experiments, applying our method to benchmark problems and to a cosmological example in up to 21 dimensions. We find the learned harmonic mean estimator is in agreement with ground truth values and nested sampling estimates. The open-source harmonic Python package implementing the learned harmonic mean, now with normalizing flows included, is publicly available.
△ Less
Submitted 9 May, 2024;
originally announced May 2024.
-
Differentiable and accelerated wavelet transforms on the sphere and ball
Authors:
Matthew A. Price,
Alicja Polanska,
Jessica Whitney,
Jason D. McEwen
Abstract:
Directional wavelet dictionaries are hierarchical representations which efficiently capture and segment information across scale, location and orientation. Such representations demonstrate a particular affinity to physical signals, which often exhibit highly anisotropic, localised multiscale structure. Many physically important signals are observed over spherical domains, such as the celestial sky…
▽ More
Directional wavelet dictionaries are hierarchical representations which efficiently capture and segment information across scale, location and orientation. Such representations demonstrate a particular affinity to physical signals, which often exhibit highly anisotropic, localised multiscale structure. Many physically important signals are observed over spherical domains, such as the celestial sky in cosmology. Leveraging recent advances in computational harmonic analysis, we design new highly distributable and automatically differentiable directional wavelet transforms on the $2$-dimensional sphere $\mathbb{S}^2$ and $3$-dimensional ball $\mathbb{B}^3 = \mathbb{R}^+ \times \mathbb{S}^2$ (the space formed by augmenting the sphere with the radial half-line). We observe up to a $300$-fold and $21800$-fold acceleration for signals on the sphere and ball, respectively, compared to existing software, whilst maintaining 64-bit machine precision. Not only do these algorithms dramatically accelerate existing spherical wavelet transforms, the gradient information afforded by automatic differentiation unlocks many data-driven analysis techniques previously not possible for these spaces. We publicly release both S2WAV and S2BALL, open-sourced JAX libraries for our transforms that are automatically differentiable and readily deployable both on and over clusters of hardware accelerators (e.g. GPUs & TPUs).
△ Less
Submitted 14 March, 2024; v1 submitted 2 February, 2024;
originally announced February 2024.
-
Scalable Bayesian uncertainty quantification with data-driven priors for radio interferometric imaging
Authors:
Tobías I. Liaudat,
Matthijs Mars,
Matthew A. Price,
Marcelo Pereyra,
Marta M. Betcke,
Jason D. McEwen
Abstract:
Next-generation radio interferometers like the Square Kilometer Array have the potential to unlock scientific discoveries thanks to their unprecedented angular resolution and sensitivity. One key to unlocking their potential resides in handling the deluge and complexity of incoming data. This challenge requires building radio interferometric imaging methods that can cope with the massive data size…
▽ More
Next-generation radio interferometers like the Square Kilometer Array have the potential to unlock scientific discoveries thanks to their unprecedented angular resolution and sensitivity. One key to unlocking their potential resides in handling the deluge and complexity of incoming data. This challenge requires building radio interferometric imaging methods that can cope with the massive data sizes and provide high-quality image reconstructions with uncertainty quantification (UQ). This work proposes a method coined QuantifAI to address UQ in radio-interferometric imaging with data-driven (learned) priors for high-dimensional settings. Our model, rooted in the Bayesian framework, uses a physically motivated model for the likelihood. The model exploits a data-driven convex prior, which can encode complex information learned implicitly from simulations and guarantee the log-concavity of the posterior. We leverage probability concentration phenomena of high-dimensional log-concave posteriors that let us obtain information about the posterior, avoiding MCMC sampling techniques. We rely on convex optimisation methods to compute the MAP estimation, which is known to be faster and better scale with dimension than MCMC sampling strategies. Our method allows us to compute local credible intervals, i.e., Bayesian error bars, and perform hypothesis testing of structure on the reconstructed image. In addition, we propose a novel blazing-fast method to compute pixel-wise uncertainties at different scales. We demonstrate our method by reconstructing radio-interferometric images in a simulated setting and carrying out fast and scalable UQ, which we validate with MCMC sampling. Our method shows an improved image quality and more meaningful uncertainties than the benchmark method based on a sparsity-promoting prior. QuantifAI's source code: https://github.com/astro-informatics/QuantifAI.
△ Less
Submitted 28 June, 2024; v1 submitted 30 November, 2023;
originally announced December 2023.
-
Differentiable and accelerated spherical harmonic and Wigner transforms
Authors:
Matthew A. Price,
Jason D. McEwen
Abstract:
Many areas of science and engineering encounter data defined on spherical manifolds. Modelling and analysis of spherical data often necessitates spherical harmonic transforms, at high degrees, and increasingly requires efficient computation of gradients for machine learning or other differentiable programming tasks. We develop novel algorithmic structures for accelerated and differentiable computa…
▽ More
Many areas of science and engineering encounter data defined on spherical manifolds. Modelling and analysis of spherical data often necessitates spherical harmonic transforms, at high degrees, and increasingly requires efficient computation of gradients for machine learning or other differentiable programming tasks. We develop novel algorithmic structures for accelerated and differentiable computation of generalised Fourier transforms on the sphere $\mathbb{S}^2$ and rotation group $\text{SO}(3)$, i.e. spherical harmonic and Wigner transforms, respectively. We present a recursive algorithm for the calculation of Wigner $d$-functions that is both stable to high harmonic degrees and extremely parallelisable. By tightly coupling this with separable spherical transforms, we obtain algorithms that exhibit an extremely parallelisable structure that is well-suited for the high throughput computing of modern hardware accelerators (e.g. GPUs). We also develop a hybrid automatic and manual differentiation approach so that gradients can be computed efficiently. Our algorithms are implemented within the JAX differentiable programming framework in the S2FFT software code. Numerous samplings of the sphere are supported, including equiangular and HEALPix sampling. Computational errors are at the order of machine precision for spherical samplings that admit a sampling theorem. When benchmarked against alternative C codes we observe up to a 400-fold acceleration. Furthermore, when distributing over multiple GPUs we achieve very close to optimal linear scaling with increasing number of GPUs due to the highly parallelised and balanced nature of our algorithms. Provided access to sufficiently many GPUs our transforms thus exhibit an unprecedented effective linear time complexity.
△ Less
Submitted 20 May, 2024; v1 submitted 24 November, 2023;
originally announced November 2023.
-
Fast emulation of anisotropies induced in the cosmic microwave background by cosmic strings
Authors:
Matthew A. Price,
Matthijs Mars,
Matthew M. Docherty,
Alessio Spurio Mancini,
Augustin Marignier,
Jason. D. McEwen
Abstract:
Cosmic strings are linear topological defects that may have been produced during symmetry-breaking phase transitions in the very early Universe. In an expanding Universe the existence of causally separate regions prevents such symmetries from being broken uniformly, with a network of cosmic string inevitably forming as a result. To faithfully generate observables of such processes requires computa…
▽ More
Cosmic strings are linear topological defects that may have been produced during symmetry-breaking phase transitions in the very early Universe. In an expanding Universe the existence of causally separate regions prevents such symmetries from being broken uniformly, with a network of cosmic string inevitably forming as a result. To faithfully generate observables of such processes requires computationally expensive numerical simulations, which prohibits many types of analyses. We propose a technique to instead rapidly emulate observables, thus circumventing simulation. Emulation is a form of generative modelling, often built upon a machine learning backbone. End-to-end emulation often fails due to high dimensionality and insufficient training data. Consequently, it is common to instead emulate a latent representation from which observables may readily be synthesised. Wavelet phase harmonics are an excellent latent representations for cosmological fields, both as a summary statistic and for emulation, since they do not require training and are highly sensitive to non-Gaussian information. Leveraging wavelet phase harmonics as a latent representation, we develop techniques to emulate string induced CMB anisotropies over a 7.2 degree field of view, with sub-arcminute resolution, in under a minute on a single GPU. Beyond generating high fidelity emulations, we provide a technique to ensure these observables are distributed correctly, providing a more representative ensemble of samples. The statistics of our emulations are commensurate with those calculated on comprehensive Nambu-Goto simulations. Our findings indicate these fast emulation approaches may be suitable for wide use in, e.g., simulation based inference pipelines. We make our code available to the community so that researchers may rapidly emulate cosmic string induced CMB anisotropies for their own analysis.
△ Less
Submitted 14 March, 2024; v1 submitted 10 July, 2023;
originally announced July 2023.
-
Proximal nested sampling with data-driven priors for physical scientists
Authors:
Jason D. McEwen,
Tobías I. Liaudat,
Matthew A. Price,
Xiaohao Cai,
Marcelo Pereyra
Abstract:
Proximal nested sampling was introduced recently to open up Bayesian model selection for high-dimensional problems such as computational imaging. The framework is suitable for models with a log-convex likelihood, which are ubiquitous in the imaging sciences. The purpose of this article is two-fold. First, we review proximal nested sampling in a pedagogical manner in an attempt to elucidate the fra…
▽ More
Proximal nested sampling was introduced recently to open up Bayesian model selection for high-dimensional problems such as computational imaging. The framework is suitable for models with a log-convex likelihood, which are ubiquitous in the imaging sciences. The purpose of this article is two-fold. First, we review proximal nested sampling in a pedagogical manner in an attempt to elucidate the framework for physical scientists. Second, we show how proximal nested sampling can be extended in an empirical Bayes setting to support data-driven priors, such as deep neural networks learned from training data.
△ Less
Submitted 28 July, 2023; v1 submitted 30 June, 2023;
originally announced July 2023.
-
Learned harmonic mean estimation of the marginal likelihood with normalizing flows
Authors:
Alicja Polanska,
Matthew A. Price,
Alessio Spurio Mancini,
Jason D. McEwen
Abstract:
Computing the marginal likelihood (also called the Bayesian model evidence) is an important task in Bayesian model selection, providing a principled quantitative way to compare models. The learned harmonic mean estimator solves the exploding variance problem of the original harmonic mean estimation of the marginal likelihood. The learned harmonic mean estimator learns an importance sampling target…
▽ More
Computing the marginal likelihood (also called the Bayesian model evidence) is an important task in Bayesian model selection, providing a principled quantitative way to compare models. The learned harmonic mean estimator solves the exploding variance problem of the original harmonic mean estimation of the marginal likelihood. The learned harmonic mean estimator learns an importance sampling target distribution that approximates the optimal distribution. While the approximation need not be highly accurate, it is critical that the probability mass of the learned distribution is contained within the posterior in order to avoid the exploding variance problem. In previous work a bespoke optimization problem is introduced when training models in order to ensure this property is satisfied. In the current article we introduce the use of normalizing flows to represent the importance sampling target distribution. A flow-based model is trained on samples from the posterior by maximum likelihood estimation. Then, the probability density of the flow is concentrated by lowering the variance of the base distribution, i.e. by lowering its "temperature", ensuring its probability mass is contained within the posterior. This approach avoids the need for a bespoke optimisation problem and careful fine tuning of parameters, resulting in a more robust method. Moreover, the use of normalizing flows has the potential to scale to high dimensional settings. We present preliminary experiments demonstrating the effectiveness of the use of flows for the learned harmonic mean estimator. The harmonic code implementing the learned harmonic mean, which is publicly available, has been updated to now support normalizing flows.
△ Less
Submitted 19 January, 2024; v1 submitted 30 June, 2023;
originally announced July 2023.
-
Scalable and Equivariant Spherical CNNs by Discrete-Continuous (DISCO) Convolutions
Authors:
Jeremy Ocampo,
Matthew A. Price,
Jason D. McEwen
Abstract:
No existing spherical convolutional neural network (CNN) framework is both computationally scalable and rotationally equivariant. Continuous approaches capture rotational equivariance but are often prohibitively computationally demanding. Discrete approaches offer more favorable computational performance but at the cost of equivariance. We develop a hybrid discrete-continuous (DISCO) group convolu…
▽ More
No existing spherical convolutional neural network (CNN) framework is both computationally scalable and rotationally equivariant. Continuous approaches capture rotational equivariance but are often prohibitively computationally demanding. Discrete approaches offer more favorable computational performance but at the cost of equivariance. We develop a hybrid discrete-continuous (DISCO) group convolution that is simultaneously equivariant and computationally scalable to high-resolution. While our framework can be applied to any compact group, we specialize to the sphere. Our DISCO spherical convolutions exhibit $\text{SO}(3)$ rotational equivariance, where $\text{SO}(n)$ is the special orthogonal group representing rotations in $n$-dimensions. When restricting rotations of the convolution to the quotient space $\text{SO}(3)/\text{SO}(2)$ for further computational enhancements, we recover a form of asymptotic $\text{SO}(3)$ rotational equivariance. Through a sparse tensor implementation we achieve linear scaling in number of pixels on the sphere for both computational cost and memory usage. For 4k spherical images we realize a saving of $10^9$ in computational cost and $10^4$ in memory usage when compared to the most efficient alternative equivariant spherical convolution. We apply the DISCO spherical CNN framework to a number of benchmark dense-prediction problems on the sphere, such as semantic segmentation and depth estimation, on all of which we achieve the state-of-the-art performance.
△ Less
Submitted 28 January, 2023; v1 submitted 27 September, 2022;
originally announced September 2022.
-
Bayesian model comparison for simulation-based inference
Authors:
A. Spurio Mancini,
M. M. Docherty,
M. A. Price,
J. D. McEwen
Abstract:
Comparison of appropriate models to describe observational data is a fundamental task of science. The Bayesian model evidence, or marginal likelihood, is a computationally challenging, yet crucial, quantity to estimate to perform Bayesian model comparison. We introduce a methodology to compute the Bayesian model evidence in simulation-based inference (SBI) scenarios (also often called likelihood-f…
▽ More
Comparison of appropriate models to describe observational data is a fundamental task of science. The Bayesian model evidence, or marginal likelihood, is a computationally challenging, yet crucial, quantity to estimate to perform Bayesian model comparison. We introduce a methodology to compute the Bayesian model evidence in simulation-based inference (SBI) scenarios (also often called likelihood-free inference). In particular, we leverage the recently proposed learnt harmonic mean estimator and exploit the fact that it is decoupled from the method used to generate posterior samples, i.e. it requires posterior samples only, which may be generated by any approach. This flexibility, which is lacking in many alternative methods for computing the model evidence, allows us to develop SBI model comparison techniques for the three main neural density estimation approaches, including neural posterior estimation (NPE), neural likelihood estimation (NLE), and neural ratio estimation (NRE). We demonstrate and validate our SBI evidence calculation techniques on a range of inference problems, including a gravitational wave example. Moreover, we further validate the accuracy of the learnt harmonic mean estimator, implemented in the HARMONIC software, in likelihood-based settings. These results highlight the potential of HARMONIC as a sampler-agnostic method to estimate the model evidence in both likelihood-based and simulation-based scenarios.
△ Less
Submitted 8 November, 2023; v1 submitted 8 July, 2022;
originally announced July 2022.
-
Machine learning assisted Bayesian model comparison: learnt harmonic mean estimator
Authors:
Jason D. McEwen,
Christopher G. R. Wallis,
Matthew A. Price,
Alessio Spurio Mancini
Abstract:
We resurrect the infamous harmonic mean estimator for computing the marginal likelihood (Bayesian evidence) and solve its problematic large variance. The marginal likelihood is a key component of Bayesian model selection to evaluate model posterior probabilities; however, its computation is challenging. The original harmonic mean estimator, first proposed by Newton and Raftery in 1994, involves co…
▽ More
We resurrect the infamous harmonic mean estimator for computing the marginal likelihood (Bayesian evidence) and solve its problematic large variance. The marginal likelihood is a key component of Bayesian model selection to evaluate model posterior probabilities; however, its computation is challenging. The original harmonic mean estimator, first proposed by Newton and Raftery in 1994, involves computing the harmonic mean of the likelihood given samples from the posterior. It was immediately realised that the original estimator can fail catastrophically since its variance can become very large (possibly not finite). A number of variants of the harmonic mean estimator have been proposed to address this issue although none have proven fully satisfactory. We present the \emph{learnt harmonic mean estimator}, a variant of the original estimator that solves its large variance problem. This is achieved by interpreting the harmonic mean estimator as importance sampling and introducing a new target distribution. The new target distribution is learned to approximate the optimal but inaccessible target, while minimising the variance of the resulting estimator. Since the estimator requires samples of the posterior only, it is agnostic to the sampling strategy used. We validate the estimator on a variety of numerical experiments, including a number of pathological examples where the original harmonic mean estimator fails catastrophically. We also consider a cosmological application, where our approach leads to $\sim$ 3 to 6 times more samples than current state-of-the-art techniques in 1/3 of the time. In all cases our learnt harmonic mean estimator is shown to be highly accurate. The estimator is computationally scalable and can be applied to problems of dimension $O(10^3)$ and beyond. Code implementing the learnt harmonic mean estimator is made publicly available
△ Less
Submitted 24 November, 2023; v1 submitted 24 November, 2021;
originally announced November 2021.
-
Bayesian variational regularization on the ball
Authors:
Matthew A. Price,
Jason D. McEwen
Abstract:
We develop variational regularization methods which leverage sparsity-promoting priors to solve severely ill posed inverse problems defined on the 3D ball (i.e. the solid sphere). Our method solves the problem natively on the ball and thus does not suffer from discontinuities that plague alternate approaches where each spherical shell is considered independently. Additionally, we leverage advances…
▽ More
We develop variational regularization methods which leverage sparsity-promoting priors to solve severely ill posed inverse problems defined on the 3D ball (i.e. the solid sphere). Our method solves the problem natively on the ball and thus does not suffer from discontinuities that plague alternate approaches where each spherical shell is considered independently. Additionally, we leverage advances in probability density theory to produce Bayesian variational methods which benefit from the computational efficiency of advanced convex optimization algorithms, whilst supporting principled uncertainty quantification. We showcase these variational regularization and uncertainty quantification techniques on an illustrative example. The C++ code discussed throughout is provided under a GNU general public license.
△ Less
Submitted 12 May, 2021;
originally announced May 2021.
-
Sparse image reconstruction on the sphere: a general approach with uncertainty quantification
Authors:
Matthew A. Price,
Luke Pratley,
Jason D. McEwen
Abstract:
Inverse problems defined naturally on the sphere are becoming increasingly of interest. In this article we provide a general framework for evaluation of inverse problems on the sphere, with a strong emphasis on flexibility and scalability. We consider flexibility with respect to the prior selection (regularization), the problem definition - specifically the problem formulation (constrained/unconst…
▽ More
Inverse problems defined naturally on the sphere are becoming increasingly of interest. In this article we provide a general framework for evaluation of inverse problems on the sphere, with a strong emphasis on flexibility and scalability. We consider flexibility with respect to the prior selection (regularization), the problem definition - specifically the problem formulation (constrained/unconstrained) and problem setting (analysis/synthesis) - and optimization adopted to solve the problem. We discuss and quantify the trade-offs between problem formulation and setting. Crucially, we consider the Bayesian interpretation of the unconstrained problem which, combined with recent developments in probability density theory, permits rapid, statistically principled uncertainty quantification (UQ) in the spherical setting. Linearity is exploited to significantly increase the computational efficiency of such UQ techniques, which in some cases are shown to permit analytic solutions. We showcase this reconstruction framework and UQ techniques on a variety of spherical inverse problems. The code discussed throughout is provided under a GNU general public license, in both C++ and Python.
△ Less
Submitted 11 May, 2021;
originally announced May 2021.
-
Efficient Generalized Spherical CNNs
Authors:
Oliver J. Cobb,
Christopher G. R. Wallis,
Augustine N. Mavor-Parker,
Augustin Marignier,
Matthew A. Price,
Mayeul d'Avezac,
Jason D. McEwen
Abstract:
Many problems across computer vision and the natural sciences require the analysis of spherical data, for which representations may be learned efficiently by encoding equivariance to rotational symmetries. We present a generalized spherical CNN framework that encompasses various existing approaches and allows them to be leveraged alongside each other. The only existing non-linear spherical CNN lay…
▽ More
Many problems across computer vision and the natural sciences require the analysis of spherical data, for which representations may be learned efficiently by encoding equivariance to rotational symmetries. We present a generalized spherical CNN framework that encompasses various existing approaches and allows them to be leveraged alongside each other. The only existing non-linear spherical CNN layer that is strictly equivariant has complexity $\mathcal{O}(C^2L^5)$, where $C$ is a measure of representational capacity and $L$ the spherical harmonic bandlimit. Such a high computational cost often prohibits the use of strictly equivariant spherical CNNs. We develop two new strictly equivariant layers with reduced complexity $\mathcal{O}(CL^4)$ and $\mathcal{O}(CL^3 \log L)$, making larger, more expressive models computationally feasible. Moreover, we adopt efficient sampling theory to achieve further computational savings. We show that these developments allow the construction of more expressive hybrid models that achieve state-of-the-art accuracy and parameter efficiency on spherical benchmark problems.
△ Less
Submitted 8 March, 2021; v1 submitted 9 October, 2020;
originally announced October 2020.
-
Spherical Bayesian mass-map** with uncertainties: full sky observations on the celestial sphere
Authors:
Matthew A. Price,
Jason D. McEwen,
L. Pratley,
Thomas D. Kitching
Abstract:
To date weak gravitational lensing surveys have typically been restricted to small fields of view, such that the $\textit{flat-sky approximation}$ has been sufficiently satisfied. However, with Stage IV surveys ($\textit{e.g. LSST}$ and $\textit{Euclid}$) imminent, extending mass-map** techniques to the sphere is a fundamental necessity. As such, we extend the sparse hierarchical Bayesian mass-m…
▽ More
To date weak gravitational lensing surveys have typically been restricted to small fields of view, such that the $\textit{flat-sky approximation}$ has been sufficiently satisfied. However, with Stage IV surveys ($\textit{e.g. LSST}$ and $\textit{Euclid}$) imminent, extending mass-map** techniques to the sphere is a fundamental necessity. As such, we extend the sparse hierarchical Bayesian mass-map** formalism presented in previous work to the spherical sky. For the first time, this allows us to construct $\textit{maximum a posteriori}$ spherical weak lensing dark-matter mass-maps, with principled Bayesian uncertainties, without imposing or assuming Gaussianty. We solve the spherical mass-map** inverse problem in the analysis setting adopting a sparsity promoting Laplace-type wavelet prior, though this theoretical framework supports all log-concave posteriors. Our spherical mass-map** formalism facilitates principled statistical interpretation of reconstructions. We apply our framework to convergence reconstruction on high resolution N-body simulations with pseudo-Euclid masking, polluted with a variety of realistic noise levels, and show a significant increase in reconstruction fidelity compared to standard approaches. Furthermore we perform the largest joint reconstruction to date of the majority of publicly available shear observational datasets (combining DESY1, KiDS450 and CFHTLens) and find that our formalism recovers a convergence map with significantly enhanced small-scale detail. Within our Bayesian framework we validate, in a statistically rigorous manner, the community's intuition regarding the need to smooth spherical Kaiser-Squires estimates to provide physically meaningful convergence maps. Such approaches cannot reveal the small-scale physical structures that we recover within our framework.
△ Less
Submitted 5 February, 2021; v1 submitted 16 April, 2020;
originally announced April 2020.
-
Sparse Bayesian mass-map** with uncertainties: peak statistics and feature locations
Authors:
Matthew A. Price,
Xiaohao Cai,
Jason D. McEwen,
Thomas D. Kitching
Abstract:
Weak lensing convergence maps - upon which higher order statistics can be calculated - can be recovered from observations of the shear field by solving the lensing inverse problem. For typical surveys this inverse problem is ill-posed (often seriously) leading to substantial uncertainty on the recovered convergence maps. In this paper we propose novel methods for quantifying the Bayesian uncertain…
▽ More
Weak lensing convergence maps - upon which higher order statistics can be calculated - can be recovered from observations of the shear field by solving the lensing inverse problem. For typical surveys this inverse problem is ill-posed (often seriously) leading to substantial uncertainty on the recovered convergence maps. In this paper we propose novel methods for quantifying the Bayesian uncertainty in the location of recovered features and the uncertainty in the cumulative peak statistic - the peak count as a function of signal to noise ratio (SNR). We adopt the sparse hierarchical Bayesian mass-map** framework developed in previous work, which provides robust reconstructions and principled statistical interpretation of reconstructed convergence maps without the need to assume or impose Gaussianity. We demonstrate our uncertainty quantification techniques on both Bolshoi N-body (cluster scale) and Buzzard V-1.6 (large scale structure) N-body simulations. For the first time, this methodology allows one to recover approximate Bayesian upper and lower limits on the cumulative peak statistic at well defined confidence levels.
△ Less
Submitted 5 February, 2021; v1 submitted 10 December, 2018;
originally announced December 2018.
-
Sparse Bayesian mass-map** with uncertainties: local credible intervals
Authors:
Matthew A. Price,
Xiaohao Cai,
Jason D. McEwen,
Marcelo Pereyra,
Thomas D. Kitching
Abstract:
Until recently mass-map** techniques for weak gravitational lensing convergence reconstruction have lacked a principled statistical framework upon which to quantify reconstruction uncertainties, without making strong assumptions of Gaussianity. In previous work we presented a sparse hierarchical Bayesian formalism for convergence reconstruction that addresses this shortcoming. Here, we draw on t…
▽ More
Until recently mass-map** techniques for weak gravitational lensing convergence reconstruction have lacked a principled statistical framework upon which to quantify reconstruction uncertainties, without making strong assumptions of Gaussianity. In previous work we presented a sparse hierarchical Bayesian formalism for convergence reconstruction that addresses this shortcoming. Here, we draw on the concept of local credible intervals (cf. Bayesian error bars) as an extension of the uncertainty quantification techniques previously detailed. These uncertainty quantification techniques are benchmarked against those recovered via Px-MALA - a state of the art proximal Markov Chain Monte Carlo (MCMC) algorithm. We find that typically our recovered uncertainties are everywhere conservative, of similar magnitude and highly correlated (Pearson correlation coefficient $\geq 0.85$) with those recovered via Px-MALA. Moreover, we demonstrate an increase in computational efficiency of $\mathcal{O}(10^6)$ when using our sparse Bayesian approach over MCMC techniques. This computational saving is critical for the application of Bayesian uncertainty quantification to large-scale stage IV surveys such as LSST and Euclid.
△ Less
Submitted 5 February, 2021; v1 submitted 10 December, 2018;
originally announced December 2018.
-
Sparse Bayesian mass-map** with uncertainties: hypothesis testing of structure
Authors:
Matthew A. Price,
Jason D. McEwen,
Xiaohao Cai,
Thomas D. Kitching,
Christopher G. R. Wallis
Abstract:
A crucial aspect of mass-map**, via weak lensing, is quantification of the uncertainty introduced during the reconstruction process. Properly accounting for these errors has been largely ignored to date. We present a new method to reconstruct maximum a posteriori (MAP) convergence maps by formulating an unconstrained Bayesian inference problem with Laplace-type l1-norm sparsity-promoting priors,…
▽ More
A crucial aspect of mass-map**, via weak lensing, is quantification of the uncertainty introduced during the reconstruction process. Properly accounting for these errors has been largely ignored to date. We present a new method to reconstruct maximum a posteriori (MAP) convergence maps by formulating an unconstrained Bayesian inference problem with Laplace-type l1-norm sparsity-promoting priors, which we solve via convex optimization. Approaching mass-map** in this manner allows us to exploit recent developments in probability concentration theory to infer theoretically conservative uncertainties for our MAP reconstructions, without relying on assumptions of Gaussianity. For the first time these methods allow us to perform hypothesis testing of structure, from which it is possible to distinguish between physical objects and artifacts of the reconstruction. Here we present this new formalism, demonstrate the method on simulations, before applying the developed formalism to two observational datasets of the Abel-520 cluster. Initial reconstructions of the Abel-520 catalogs reported the detection of an anomalous 'dark core' -- an over dense region with no optical counterpart -- which was taken to be evidence for self-interacting dark-matter. In our Bayesian framework it is found that neither Abel-520 dataset can conclusively determine the physicality of such dark cores at 99% confidence. However, in both cases the recovered MAP estimators are consistent with both sets of data.
△ Less
Submitted 15 May, 2021; v1 submitted 10 December, 2018;
originally announced December 2018.
-
Map** dark matter on the celestial sphere with weak gravitational lensing
Authors:
Christopher G. R. Wallis,
Matthew A. Price,
Jason D. McEwen,
Thomas D. Kitching,
Boris Leistedt,
Antoine Plouviez
Abstract:
Convergence maps of the integrated matter distribution are a key science result from weak gravitational lensing surveys. To date, recovering convergence maps has been performed using a planar approximation of the celestial sphere. However, with the increasing area of sky covered by dark energy experiments, such as Euclid, the Large Synoptic Survey Telescope (LSST), and the Wide Field Infrared Surv…
▽ More
Convergence maps of the integrated matter distribution are a key science result from weak gravitational lensing surveys. To date, recovering convergence maps has been performed using a planar approximation of the celestial sphere. However, with the increasing area of sky covered by dark energy experiments, such as Euclid, the Large Synoptic Survey Telescope (LSST), and the Wide Field Infrared Survey Telescope (WFIRST), this assumption will no longer be valid. We recover convergence fields on the celestial sphere using an extension of the Kaiser-Squires estimator to the spherical setting. Through simulations we study the error introduced by planar approximations. Moreover, we examine how best to recover convergence maps in the planar setting, considering a variety of different projections and defining the local rotations that are required when projecting spin fields such as cosmic shear. For the sky coverages typical of future surveys, errors introduced by projection effects can be of order tens of percent, exceeding 50% in some cases. The stereographic projection, which is conformal and so preserves local angles, is the most effective planar projection. In any case, these errors can be avoided entirely by recovering convergence fields directly on the celestial sphere. We apply the spherical Kaiser-Squires mass-map** method presented to the public Dark Energy Survey (DES) science verification data to recover convergence maps directly on the celestial sphere.
△ Less
Submitted 21 July, 2021; v1 submitted 27 March, 2017;
originally announced March 2017.
