-
Second Analysis Ecosystem Workshop Report
Authors:
Mohamed Aly,
Jackson Burzynski,
Bryan Cardwell,
Daniel C. Craik,
Tal van Daalen,
Tomas Dado,
Ayanabha Das,
Antonio Delgado Peris,
Caterina Doglioni,
Peter Elmer,
Engin Eren,
Martin B. Eriksen,
Jonas Eschle,
Giulio Eulisse,
Conor Fitzpatrick,
José Flix Molina,
Alessandra Forti,
Ben Galewsky,
Sean Gasiorowski,
Aman Goel,
Loukas Gouskos,
Enrico Guiraud,
Kanhaiya Gupta,
Stephan Hageboeck,
Allison Reinsvold Hall
, et al. (44 additional authors not shown)
Abstract:
The second workshop on the HEP Analysis Ecosystem took place 23-25 May 2022 at IJCLab in Orsay, to look at progress and continuing challenges in scaling up HEP analysis to meet the needs of HL-LHC and DUNE, as well as the very pressing needs of LHC Run 3 analysis.
The workshop was themed around six particular topics, which were felt to capture key questions, opportunities and challenges. Each to…
▽ More
The second workshop on the HEP Analysis Ecosystem took place 23-25 May 2022 at IJCLab in Orsay, to look at progress and continuing challenges in scaling up HEP analysis to meet the needs of HL-LHC and DUNE, as well as the very pressing needs of LHC Run 3 analysis.
The workshop was themed around six particular topics, which were felt to capture key questions, opportunities and challenges. Each topic arranged a plenary session introduction, often with speakers summarising the state-of-the art and the next steps for analysis. This was then followed by parallel sessions, which were much more discussion focused, and where attendees could grapple with the challenges and propose solutions that could be tried. Where there was significant overlap between topics, a joint discussion between them was arranged.
In the weeks following the workshop the session conveners wrote this document, which is a summary of the main discussions, the key points raised and the conclusions and outcomes. The document was circulated amongst the participants for comments before being finalised here.
△ Less
Submitted 9 December, 2022;
originally announced December 2022.
-
funcX: Federated Function as a Service for Science
Authors:
Zhuozhao Li,
Ryan Chard,
Yadu Babuji,
Ben Galewsky,
Tyler Skluzacek,
Kirill Nagaitsev,
Anna Woodard,
Ben Blaiszik,
Josh Bryan,
Daniel S. Katz,
Ian Foster,
Kyle Chard
Abstract:
funcX is a distributed function as a service (FaaS) platform that enables flexible, scalable, and high performance remote function execution. Unlike centralized FaaS systems, funcX decouples the cloud-hosted management functionality from the edge-hosted execution functionality. funcX's endpoint software can be deployed, by users or administrators, on arbitrary laptops, clouds, clusters, and superc…
▽ More
funcX is a distributed function as a service (FaaS) platform that enables flexible, scalable, and high performance remote function execution. Unlike centralized FaaS systems, funcX decouples the cloud-hosted management functionality from the edge-hosted execution functionality. funcX's endpoint software can be deployed, by users or administrators, on arbitrary laptops, clouds, clusters, and supercomputers, in effect turning them into function serving systems. funcX's cloud-hosted service provides a single location for registering, sharing, and managing both functions and endpoints. It allows for transparent, secure, and reliable function execution across the federated ecosystem of endpoints--enabling users to route functions to endpoints based on specific needs. funcX uses containers (e.g., Docker, Singularity, and Shifter) to provide common execution environments across endpoints. funcX implements various container management strategies to execute functions with high performance and efficiency on diverse funcX endpoints. funcX also integrates with an in-memory data store and Globus for managing data that may span endpoints. We motivate the need for funcX, present our prototype design and implementation, and demonstrate, via experiments on two supercomputers, that funcX can scale to more than 130 000 concurrent workers. We show that funcX's container warming-aware routing algorithm can reduce the completion time for 3000 functions by up to 61% compared to a randomized algorithm and the in-memory data store can speed up data transfers by up to 3x compared to a shared file system.
△ Less
Submitted 23 September, 2022;
originally announced September 2022.
-
Towards Real-World Applications of ServiceX, an Analysis Data Transformation System
Authors:
KyungEon Choi,
Andrew Eckart,
Ben Galewsky,
Robert Gardner,
Mark S. Neubauer,
Peter Onyisi,
Mason Proffitt,
Ilija Vukotic,
Gordon T. Watts
Abstract:
One of the biggest challenges in the High-Luminosity LHC (HL- LHC) era will be the significantly increased data size to be recorded and analyzed from the collisions at the ATLAS and CMS experiments. ServiceX is a software R&D project in the area of Data Organization, Management and Access of the IRIS- HEP to investigate new computational models for the HL- LHC era. ServiceX is an experiment-agnost…
▽ More
One of the biggest challenges in the High-Luminosity LHC (HL- LHC) era will be the significantly increased data size to be recorded and analyzed from the collisions at the ATLAS and CMS experiments. ServiceX is a software R&D project in the area of Data Organization, Management and Access of the IRIS- HEP to investigate new computational models for the HL- LHC era. ServiceX is an experiment-agnostic service to enable on-demand data delivery specifically tailored for nearly-interactive vectorized analyses. It is capable of retrieving data from grid sites, on-the-fly data transformation, and delivering user-selected data in a variety of different formats. New features will be presented that make the service ready for public use. An ongoing effort to integrate ServiceX with a popular statistical analysis framework in ATLAS will be described with an emphasis of a practical implementation of ServiceX into the physics analysis pipeline.
△ Less
Submitted 5 July, 2021;
originally announced July 2021.
-
Learning from the Pandemic: the Future of Meetings in HEP and Beyond
Authors:
Mark S. Neubauer,
Todd Adams,
Jennifer Adelman-McCarthy,
Gabriele Benelli,
Tulika Bose,
David Britton,
Pat Burchat,
Joel Butler,
Timothy A. Cartwright,
Tomáš Davídek,
Jacques Dumarchez,
Peter Elmer,
Matthew Feickert,
Ben Galewsky,
Mandeep Gill,
Maciej Gladki,
Aman Goel,
Jonathan E. Guyer,
Bo Jayatilaka,
Brendan Kiburg,
Benjamin Krikler,
David Lange,
Claire Lee,
Nick Manganelli,
Giovanni Marchiori
, et al. (14 additional authors not shown)
Abstract:
The COVID-19 pandemic has by-and-large prevented in-person meetings since March 2020. While the increasing deployment of effective vaccines around the world is a very positive development, the timeline and pathway to "normality" is uncertain and the "new normal" we will settle into is anyone's guess. Particle physics, like many other scientific fields, has more than a year of experience in holding…
▽ More
The COVID-19 pandemic has by-and-large prevented in-person meetings since March 2020. While the increasing deployment of effective vaccines around the world is a very positive development, the timeline and pathway to "normality" is uncertain and the "new normal" we will settle into is anyone's guess. Particle physics, like many other scientific fields, has more than a year of experience in holding virtual meetings, workshops, and conferences. A great deal of experimentation and innovation to explore how to execute these meetings effectively has occurred. Therefore, it is an appropriate time to take stock of what we as a community learned from running virtual meetings and discuss possible strategies for the future. Continuing to develop effective strategies for meetings with a virtual component is likely to be important for reducing the carbon footprint of our research activities, while also enabling greater diversity and inclusion for participation. This report summarizes a virtual two-day workshop on Virtual Meetings held May 5-6, 2021 which brought together experts from both inside and outside of high-energy physics to share their experiences and practices with organizing and executing virtual workshops, and to develop possible strategies for future meetings as we begin to emerge from the COVID-19 pandemic. This report outlines some of the practices and tools that have worked well which we hope will serve as a valuable resource for future virtual meeting organizers in all scientific fields.
△ Less
Submitted 29 June, 2021;
originally announced June 2021.
-
Distributed statistical inference with pyhf enabled through funcX
Authors:
Matthew Feickert,
Lukas Heinrich,
Giordon Stark,
Ben Galewsky
Abstract:
In High Energy Physics facilities that provide High Performance Computing environments provide an opportunity to efficiently perform the statistical inference required for analysis of data from the Large Hadron Collider, but can pose problems with orchestration and efficient scheduling. The compute architectures at these facilities do not easily support the Python compute model, and the configurat…
▽ More
In High Energy Physics facilities that provide High Performance Computing environments provide an opportunity to efficiently perform the statistical inference required for analysis of data from the Large Hadron Collider, but can pose problems with orchestration and efficient scheduling. The compute architectures at these facilities do not easily support the Python compute model, and the configuration scheduling of batch jobs for physics often requires expertise in multiple job scheduling services. The combination of the pure-Python libraries pyhf and funcX reduces the common problem in HEP analyses of performing statistical inference with binned models, that would traditionally take multiple hours and bespoke scheduling, to an on-demand (fitting) "function as a service" that can scalably execute across workers in just a few minutes, offering reduced time to insight and inference. We demonstrate execution of a scalable workflow using funcX to simultaneously fit 125 signal hypotheses from a published ATLAS search for new physics using pyhf with a wall time of under 3 minutes. We additionally show performance comparisons for other physics analyses with openly published probability models and argue for a blueprint of fitting as a service systems at HPC centers.
△ Less
Submitted 31 August, 2021; v1 submitted 3 March, 2021;
originally announced March 2021.
