-
SWIFT: A modern highly-parallel gravity and smoothed particle hydrodynamics solver for astrophysical and cosmological applications
Authors:
Matthieu Schaller,
Josh Borrow,
Peter W. Draper,
Mladen Ivkovic,
Stuart McAlpine,
Bert Vandenbroucke,
Yannick Bahé,
Evgenii Chaikin,
Aidan B. G. Chalk,
Tsang Keung Chan,
Camila Correa,
Marcel van Daalen,
Willem Elbers,
Pedro Gonnet,
Loïc Hausammann,
John Helly,
Filip Huško,
Jacob A. Kegerreis,
Folkert S. J. Nobels,
Sylvia Ploeckinger,
Yves Revaz,
William J. Roper,
Sergio Ruiz-Bonilla,
Thomas D. Sandnes,
Yolan Uyttenhove
, et al. (2 additional authors not shown)
Abstract:
Numerical simulations have become one of the key tools used by theorists in all the fields of astrophysics and cosmology. The development of modern tools that target the largest existing computing systems and exploit state-of-the-art numerical methods and algorithms is thus crucial. In this paper, we introduce the fully open-source highly-parallel, versatile, and modular coupled hydrodynamics, gra…
▽ More
Numerical simulations have become one of the key tools used by theorists in all the fields of astrophysics and cosmology. The development of modern tools that target the largest existing computing systems and exploit state-of-the-art numerical methods and algorithms is thus crucial. In this paper, we introduce the fully open-source highly-parallel, versatile, and modular coupled hydrodynamics, gravity, cosmology, and galaxy-formation code SWIFT. The software package exploits hybrid shared- and distributed-memory task-based parallelism, asynchronous communications, and domain-decomposition algorithms based on balancing the workload, rather than the data, to efficiently exploit modern high-performance computing cluster architectures. Gravity is solved for using a fast-multipole-method, optionally coupled to a particle mesh solver in Fourier space to handle periodic volumes. For gas evolution, multiple modern flavours of Smoothed Particle Hydrodynamics are implemented. SWIFT also evolves neutrinos using a state-of-the-art particle-based method. Two complementary networks of sub-grid models for galaxy formation as well as extensions to simulate planetary physics are also released as part of the code. An extensive set of output options, including snapshots, light-cones, power spectra, and a coupling to structure finders are also included. We describe the overall code architecture, summarise the consistency and accuracy tests that were performed, and demonstrate the excellent weak-scaling performance of the code using a representative cosmological hydrodynamical problem with $\approx$$300$ billion particles. The code is released to the community alongside extensive documentation for both users and developers, a large selection of example test problems, and a suite of tools to aid in the analysis of large simulations run with SWIFT.
△ Less
Submitted 29 March, 2024; v1 submitted 22 May, 2023;
originally announced May 2023.
-
Using the Sun to Measure the Primary Beam Response of the Canadian Hydrogen Intensity Map** Experiment
Authors:
CHIME Collaboration,
Mandana Amiri,
Kevin Bandura,
Anja Boskovic,
Jean-François Cliche,
Meiling Deng,
Matt Dobbs,
Mateus Fandino,
Simon Foreman,
Mark Halpern,
Alex S. Hill,
Gary Hinshaw,
Carolin Höfer,
Joseph Kania,
T. L. Landecker,
Joshua MacEachern,
Kiyoshi Masui,
Juan Mena-Parra,
Laura Newburgh,
Anna Ordog,
Tristan Pinsonneault-Marotte,
Ava Polzin,
Alex Reda,
J. Richard Shaw,
Seth R. Siegel
, et al. (5 additional authors not shown)
Abstract:
We present a beam pattern measurement of the Canadian Hydrogen Intensity Map** Experiment (CHIME) made using the Sun as a calibration source. As CHIME is a pure drift scan instrument, we rely on the seasonal North-South motion of the Sun to probe the beam at different elevations. This semiannual range in elevation, combined with the radio brightness of the Sun, enables a beam measurement which s…
▽ More
We present a beam pattern measurement of the Canadian Hydrogen Intensity Map** Experiment (CHIME) made using the Sun as a calibration source. As CHIME is a pure drift scan instrument, we rely on the seasonal North-South motion of the Sun to probe the beam at different elevations. This semiannual range in elevation, combined with the radio brightness of the Sun, enables a beam measurement which spans ~7,200 square degrees on the sky without the need to move the telescope. We take advantage of observations made near solar minimum to minimize the impact of solar variability, which is observed to be <10% in intensity over the observation period. The resulting data set is highly complementary to other CHIME beam measurements -- both in terms of angular coverage and systematics -- and plays an important role in the ongoing program to characterize the CHIME primary beam.
△ Less
Submitted 3 May, 2022; v1 submitted 27 January, 2022;
originally announced January 2022.
-
An Overview of CHIME, the Canadian Hydrogen Intensity Map** Experiment
Authors:
The CHIME Collaboration,
Mandana Amiri,
Kevin Bandura,
Anja Boskovic,
Tianyue Chen,
Jean-François Cliche,
Meiling Deng,
Nolan Denman,
Matt Dobbs,
Mateus Fandino,
Simon Foreman,
Mark Halpern,
David Hanna,
Alex S. Hill,
Gary Hinshaw,
Carolin Höfer,
Joseph Kania,
Peter Klages,
T. L. Landecker,
Joshua MacEachern,
Kiyoshi Masui,
Juan Mena-Parra,
Nikola Milutinovic,
Arash Mirhosseini,
Laura Newburgh
, et al. (18 additional authors not shown)
Abstract:
The Canadian Hydrogen Intensity Map** Experiment (CHIME) is a drift scan radio telescope operating across the 400-800 MHz band. CHIME is located at the Dominion Radio Astrophysical Observatory near Penticton, BC Canada. The instrument is designed to map neutral hydrogen over the redshift range 0.8 to 2.5 to constrain the expansion history of the Universe. This goal drives the design features of…
▽ More
The Canadian Hydrogen Intensity Map** Experiment (CHIME) is a drift scan radio telescope operating across the 400-800 MHz band. CHIME is located at the Dominion Radio Astrophysical Observatory near Penticton, BC Canada. The instrument is designed to map neutral hydrogen over the redshift range 0.8 to 2.5 to constrain the expansion history of the Universe. This goal drives the design features of the instrument. CHIME consists of four parallel cylindrical reflectors, oriented north-south, each 100 m $\times$ 20 m and outfitted with a 256 element dual-polarization linear feed array. CHIME observes a two degree wide stripe covering the entire meridian at any given moment, observing 3/4 of the sky every day due to Earth rotation. An FX correlator utilizes FPGAs and GPUs to digitize and correlate the signals, with different correlation products generated for cosmological, fast radio burst, pulsar, VLBI, and 21 cm absorber backends. For the cosmology backend, the $N_\mathrm{feed}^2$ correlation matrix is formed for 1024 frequency channels across the band every 31 ms. A data receiver system applies calibration and flagging and, for our primary cosmological data product, stacks redundant baselines and integrates for 10 s. We present an overview of the instrument, its performance metrics based on the first three years of science data, and we describe the current progress in characterizing CHIME's primary beam response. We also present maps of the sky derived from CHIME data; we are using versions of these maps for a cosmological stacking analysis as well as for investigation of Galactic foregrounds.
△ Less
Submitted 23 May, 2022; v1 submitted 19 January, 2022;
originally announced January 2022.
-
A Hybrid MPI+Threads Approach to Particle Group Finding Using Union-Find
Authors:
James S. Willis,
Matthieu Schaller,
Pedro Gonnet,
John C. Helly
Abstract:
The Friends-of-Friends (FoF) algorithm is a standard technique used in cosmological $N$-body simulations to identify structures. Its goal is to find clusters of particles (called groups) that are separated by at most a cut-off radius. $N$-body simulations typically use most of the memory present on a node, leaving very little free for a FoF algorithm to run on-the-fly. We propose a new method that…
▽ More
The Friends-of-Friends (FoF) algorithm is a standard technique used in cosmological $N$-body simulations to identify structures. Its goal is to find clusters of particles (called groups) that are separated by at most a cut-off radius. $N$-body simulations typically use most of the memory present on a node, leaving very little free for a FoF algorithm to run on-the-fly. We propose a new method that utilises the common Union-Find data structure and a hybrid MPI+threads approach. The algorithm can also be expressed elegantly in a task-based formalism if such a framework is used in the rest of the application. We have implemented our algorithm in the open-source cosmological code, SWIFT. Our implementation displays excellent strong- and weak-scaling behaviour on realistic problems and compares favourably (speed-up of 18x) over other methods commonly used in the $N$-body community.
△ Less
Submitted 25 March, 2020;
originally announced March 2020.
-
Hunting for Galaxies and Halos in simulations with VELOCIraptor
Authors:
Pascal J. Elahi,
Rodrigo Cañas,
Rhys J. J. Poulton,
Rodrigo J. Tobar,
James S. Willis,
Claudia del P. Lagos,
Chris Power,
Aaron S. G. Robotham
Abstract:
We present VELOCIraptor, a massively parallel galaxy/(sub)halo finder that is also capable of robustly identifying tidally disrupted objects and separate stellar halos from galaxies. The code is written in c++11, use the MPI and OpenMP API's for parallelisation, and includes python tools to read/manipulate the data products produced. We demonstrate the power of the VELOCIraptor (sub)halo finder, s…
▽ More
We present VELOCIraptor, a massively parallel galaxy/(sub)halo finder that is also capable of robustly identifying tidally disrupted objects and separate stellar halos from galaxies. The code is written in c++11, use the MPI and OpenMP API's for parallelisation, and includes python tools to read/manipulate the data products produced. We demonstrate the power of the VELOCIraptor (sub)halo finder, showing how it can identify subhalos deep within the host that have negligible density contrasts to their parent halo. We find a subhalo mass-radial distance dependence: large subhalos with mass ratios of $\gtrsim10^{-2}$ are more common in the central regions that smaller subhalos, a result of dynamical friction and low tidal mass loss rates. This dependence is completely absent in (sub)halo finders in common use, which generally search for substructure in configuration space, yet is present in codes that track particles belonging to halos as they fall into other halos, such as HBT+. VELOCIraptor largely reproduces the dependence seen without tracking, finding a similar radial dependence to HBT+ in well resolved halos from our limited resolution fiducial simulation.
△ Less
Submitted 28 March, 2019; v1 submitted 3 February, 2019;
originally announced February 2019.
-
An Efficient SIMD Implementation of Pseudo-Verlet Lists for Neighbour Interactions in Particle-Based Codes
Authors:
James S. Willis,
Matthieu Schaller,
Pedro Gonnet,
Richard G. Bower,
Peter W. Draper
Abstract:
In particle-based simulations, neighbour finding (i.e finding pairs of particles to interact within a given range) is the most time consuming part of the computation. One of the best such algorithms, which can be used for both Molecular Dynamics (MD) and Smoothed Particle Hydrodynamics (SPH) simulations, is the pseudo-Verlet list algorithm. This algorithm, however, does not vectorise trivially, an…
▽ More
In particle-based simulations, neighbour finding (i.e finding pairs of particles to interact within a given range) is the most time consuming part of the computation. One of the best such algorithms, which can be used for both Molecular Dynamics (MD) and Smoothed Particle Hydrodynamics (SPH) simulations, is the pseudo-Verlet list algorithm. This algorithm, however, does not vectorise trivially, and hence makes it difficult to exploit SIMD-parallel architectures. In this paper, we present several novel modifications as well as a vectorisation strategy for the algorithm which lead to overall speed-ups over the scalar version of the algorithm of 2.24x for the AVX instruction set (SIMD width of 8), 2.43x for AVX2, and 4.07x for AVX-512 (SIMD width of 16).
△ Less
Submitted 17 April, 2018;
originally announced April 2018.
