-
Coffea-Casa: Building composable analysis facilities for the HL-LHC
Authors:
Sam Albin,
Garhan Attebury,
Kenneth Bloom,
Brian Bockelman,
Carl Lundstedt,
Oksana Shadura,
John Thiltges
Abstract:
The large data volumes expected from the High Luminosity LHC (HL-LHC) present challenges to existing paradigms and facilities for end-user data analysis. Modern cyberinfrastructure tools provide a diverse set of services that can be composed into a system that provides physicists with powerful tools that give them straightforward access to large computing resources, with low barriers to entry. The…
▽ More
The large data volumes expected from the High Luminosity LHC (HL-LHC) present challenges to existing paradigms and facilities for end-user data analysis. Modern cyberinfrastructure tools provide a diverse set of services that can be composed into a system that provides physicists with powerful tools that give them straightforward access to large computing resources, with low barriers to entry. The Coffea-Casa analysis facility (AF) provides an environment for end users enabling the execution of increasingly complex analyses such as those demonstrated by the Analysis Grand Challenge (AGC) and capturing the features that physicists will need for the HL-LHC.
We describe the development progress of the Coffea-Casa facility featuring its modularity while demonstrating the ability to port and customize the facility software stack to other locations. The facility also facilitates the support of batch systems while staying Kubernetes-native. We present the evolved architecture of the facility, such as the integration of advanced data delivery services (e.g. ServiceX) and making data caching services (e.g. XCache) available to end users of the facility. We also highlight the composability of modern cyberinfrastructure tools. To enable machine learning pipelines at coffee-casa analysis facilities, a set of industry ML solutions adopted for HEP columnar analysis were integrated on top of existing facility services. These services also feature transparent access for user workflows to GPUs available at a facility via inference servers while using Kubernetes as enabling technology.
△ Less
Submitted 30 November, 2023;
originally announced December 2023.
-
Collaborative Computing Support for Analysis Facilities Exploiting Software as Infrastructure Techniques
Authors:
Maria Acosta Flechas,
Garhan Attebury,
Kenneth Bloom,
Brian Bockelman,
Lindsey Gray,
Burt Holzman,
Carl Lundstedt,
Oksana Shadura,
Nicholas Smith,
John Thiltges
Abstract:
Prior to the public release of Kubernetes it was difficult to conduct joint development of elaborate analysis facilities due to the highly non-homogeneous nature of hardware and network topology across compute facilities. However, since the advent of systems like Kubernetes and OpenShift, which provide declarative interfaces for building fault-tolerant and self-healing deployments of networked sof…
▽ More
Prior to the public release of Kubernetes it was difficult to conduct joint development of elaborate analysis facilities due to the highly non-homogeneous nature of hardware and network topology across compute facilities. However, since the advent of systems like Kubernetes and OpenShift, which provide declarative interfaces for building fault-tolerant and self-healing deployments of networked software, it is possible for multiple institutes to collaborate more effectively since resource details are abstracted away through various forms of hardware and software virtualization. In this whitepaper we will outline the development of two analysis facilities: "Coffea-casa" at University of Nebraska Lincoln and the "Elastic Analysis Facility" at Fermilab, and how utilizing platform abstraction has improved the development of common software for each of these facilities, and future development plans made possible by this methodology.
△ Less
Submitted 22 March, 2022; v1 submitted 18 March, 2022;
originally announced March 2022.
-
Coffea-casa: an analysis facility prototype
Authors:
Matous Adamec,
Garhan Attebury,
Kenneth Bloom,
Brian Bockelman,
Carl Lundstedt,
Oksana Shadura,
John Thiltges
Abstract:
Data analysis in HEP has often relied on batch systems and event loops; users are given a non-interactive interface to computing resources and consider data event-by-event. The "Coffea-casa" prototype analysis facility is an effort to provide users with alternate mechanisms to access computing resources and enable new programming paradigms. Instead of the command-line interface and asynchronous ba…
▽ More
Data analysis in HEP has often relied on batch systems and event loops; users are given a non-interactive interface to computing resources and consider data event-by-event. The "Coffea-casa" prototype analysis facility is an effort to provide users with alternate mechanisms to access computing resources and enable new programming paradigms. Instead of the command-line interface and asynchronous batch access, a notebook-based web interface and interactive computing is provided. Instead of writing event loops, the column-based Coffea library is used. In this paper, we describe the architectural components of the facility, the services offered to end-users, and how it integrates into a larger ecosystem for data access and authentication.
△ Less
Submitted 29 June, 2021; v1 submitted 2 March, 2021;
originally announced March 2021.
