-
Effectiveness and predictability of in-network storage cache for scientific workflows
Authors:
Caitlin Sim,
Kesheng Wu,
Alex Sim,
Inder Monga,
Chin Guok,
Frank Wurthwein,
Diego Davila,
Harvey Newman,
Justas Balcas
Abstract:
Large scientific collaborations often have multiple scientists accessing the same set of files while doing different analyses, which create repeated accesses to the large amounts of shared data located far away. These data accesses have long latency due to distance and occupy the limited bandwidth available over the wide-area network. To reduce the wide-area network traffic and the data access lat…
▽ More
Large scientific collaborations often have multiple scientists accessing the same set of files while doing different analyses, which create repeated accesses to the large amounts of shared data located far away. These data accesses have long latency due to distance and occupy the limited bandwidth available over the wide-area network. To reduce the wide-area network traffic and the data access latency, regional data storage caches have been installed as a new networking service. To study the effectiveness of such a cache system in scientific applications, we examine the Southern California Petabyte Scale Cache for a high-energy physics experiment. By examining about 3TB of operational logs, we show that this cache removed 67.6% of file requests from the wide-area network and reduced the traffic volume on wide-area network by 12.3TB (or 35.4%) an average day. The reduction in the traffic volume (35.4%) is less than the reduction in file counts (67.6%) because the larger files are less likely to be reused. Due to this difference in data access patterns, the cache system has implemented a policy to avoid evicting smaller files when processing larger files. We also build a machine learning model to study the predictability of the cache behavior. Tests show that this model is able to accurately predict the cache accesses, cache misses, and network throughput, making the model useful for future studies on resource provisioning and planning.
△ Less
Submitted 20 July, 2023;
originally announced July 2023.
-
Analyzing Transatlantic Network Traffic over Scientific Data Caches
Authors:
Z. Deng,
A. Sim,
K. Wu,
C. Guok,
D. Hazen,
I. Monga,
F. Andrijauskas,
F. Wuerthwein,
D. Weitzel
Abstract:
Large scientific collaborations often share huge volumes of data around the world. Consequently a significant amount of network bandwidth is needed for data replication and data access. Users in the same region may possibly share resources as well as data, especially when they are working on related topics with similar datasets. In this work, we study the network traffic patterns and resource util…
▽ More
Large scientific collaborations often share huge volumes of data around the world. Consequently a significant amount of network bandwidth is needed for data replication and data access. Users in the same region may possibly share resources as well as data, especially when they are working on related topics with similar datasets. In this work, we study the network traffic patterns and resource utilization for scientific data caches connecting European networks to the US. We explore the efficiency of resource utilization, especially for network traffic which consists mostly of transatlantic data transfers, and the potential for having more caching node deployments. Our study shows that these data caches reduced network traffic volume by 97% during the study period. This demonstrates that such caching nodes are effective in reducing wide-area network traffic.
△ Less
Submitted 17 July, 2023; v1 submitted 1 May, 2023;
originally announced May 2023.
-
Managed Network Services for Exascale Data Movement Across Large Global Scientific Collaborations
Authors:
Frank Würthwein,
Jonathan Guiang,
Aashay Arora,
Diego Davila,
John Graham,
Dima Mishin,
Thomas Hutton,
Igor Sfiligoi,
Harvey Newman,
Justas Balcas,
Tom Lehman,
Xi Yang,
Chin Guok
Abstract:
Unique scientific instruments designed and operated by large global collaborations are expected to produce Exabyte-scale data volumes per year by 2030. These collaborations depend on globally distributed storage and compute to turn raw data into science. While all of these infrastructures have batch scheduling capabilities to share compute, Research and Education networks lack those capabilities.…
▽ More
Unique scientific instruments designed and operated by large global collaborations are expected to produce Exabyte-scale data volumes per year by 2030. These collaborations depend on globally distributed storage and compute to turn raw data into science. While all of these infrastructures have batch scheduling capabilities to share compute, Research and Education networks lack those capabilities. There is thus uncontrolled competition for bandwidth between and within collaborations. As a result, data "hogs" disk space at processing facilities for much longer than it takes to process, leading to vastly over-provisioned storage infrastructures. Integrated co-scheduling of networks as part of high-level managed workflows might reduce these storage needs by more than an order of magnitude. This paper describes such a solution, demonstrates its functionality in the context of the Large Hadron Collider (LHC) at CERN, and presents the next-steps towards its use in production.
△ Less
Submitted 27 September, 2022;
originally announced September 2022.
-
Snowmass 2021 Computational Frontier CompF4 Topical Group Report: Storage and Processing Resource Access
Authors:
W. Bhimji,
D. Carder,
E. Dart,
J. Duarte,
I. Fisk,
R. Gardner,
C. Guok,
B. Jayatilaka,
T. Lehman,
M. Lin,
C. Maltzahn,
S. McKee,
M. S. Neubauer,
O. Rind,
O. Shadura,
N. V. Tran,
P. van Gemmeren,
G. Watts,
B. A. Weaver,
F. Würthwein
Abstract:
Computing plays a significant role in all areas of high energy physics. The Snowmass 2021 CompF4 topical group's scope is facilities R&D, where we consider "facilities" as the computing hardware and software infrastructure inside the data centers plus the networking between data centers, irrespective of who owns them, and what policies are applied for using them. In other words, it includes commer…
▽ More
Computing plays a significant role in all areas of high energy physics. The Snowmass 2021 CompF4 topical group's scope is facilities R&D, where we consider "facilities" as the computing hardware and software infrastructure inside the data centers plus the networking between data centers, irrespective of who owns them, and what policies are applied for using them. In other words, it includes commercial clouds, federally funded High Performance Computing (HPC) systems for all of science, and systems funded explicitly for a given experimental or theoretical program. This topical group report summarizes the findings and recommendations for the storage, processing, networking and associated software service infrastructures for future high energy physics research, based on the discussions organized through the Snowmass 2021 community study.
△ Less
Submitted 29 September, 2022; v1 submitted 19 September, 2022;
originally announced September 2022.
-
Studying Scientific Data Lifecycle in On-demand Distributed Storage Caches
Authors:
Julian Bellavita,
Alex Sim,
Kesheng Wu,
Inder Monga,
Chin Guok,
Frank Würthwein,
Diego Davila
Abstract:
The XRootD system is used to transfer, store, and cache large datasets from high-energy physics (HEP). In this study we focus on its capability as distributed on-demand storage cache. Through exploring a large set of daily log files between 2020 and 2021, we seek to understand the data access patterns that might inform future cache design. Our study begins with a set of summary statistics regardin…
▽ More
The XRootD system is used to transfer, store, and cache large datasets from high-energy physics (HEP). In this study we focus on its capability as distributed on-demand storage cache. Through exploring a large set of daily log files between 2020 and 2021, we seek to understand the data access patterns that might inform future cache design. Our study begins with a set of summary statistics regarding file read operations, file lifetimes, and file transfers. We observe that the number of read operations on each file remains nearly constant, while the average size of a read operation grows over time. Furthermore, files tend to have a consistent length of time during which they remain open and are in use. Based on this comprehensive study of the cache access statistics, we developed a cache simulator to explore the behavior of caches of different sizes. Within a certain size range, we find that increasing the XRootD cache size improves the cache hit rate, yielding faster overall file access. In particular, we find that increase the cache size from 40TB to 56TB could increase the hit rate from 0.62 to 0.89, which is a significant increase in cache effectiveness for modest cost.
△ Less
Submitted 11 May, 2022;
originally announced May 2022.
-
Access Trends of In-network Cache for Scientific Data
Authors:
Ruize Han,
Alex Sim,
Kesheng Wu,
Inder Monga,
Chin Guok,
Frank Würthwein,
Diego Davila,
Justas Balcas,
Harvey Newman
Abstract:
Scientific collaborations are increasingly relying on large volumes of data for their work and many of them employ tiered systems to replicate the data to their worldwide user communities. Each user in the community often selects a different subset of data for their analysis tasks; however, members of a research group often are working on related research topics that require similar data objects.…
▽ More
Scientific collaborations are increasingly relying on large volumes of data for their work and many of them employ tiered systems to replicate the data to their worldwide user communities. Each user in the community often selects a different subset of data for their analysis tasks; however, members of a research group often are working on related research topics that require similar data objects. Thus, there is a significant amount of data sharing possible. In this work, we study the access traces of a federated storage cache known as the Southern California Petabyte Scale Cache. By studying the access patterns and potential for network traffic reduction by this caching system, we aim to explore the predictability of the cache uses and the potential for a more general in-network data caching. Our study shows that this distributed storage cache is able to reduce the network traffic volume by a factor of 2.35 during a part of the study period. We further show that machine learning models could predict cache utilization with an accuracy of 0.88. This demonstrates that such cache usage is predictable, which could be useful for managing complex networking resources such as in-network caching.
△ Less
Submitted 11 May, 2022;
originally announced May 2022.
-
Data Transfer and Network Services management for Domain Science Workflows
Authors:
Tom Lehman,
Xi Yang,
Chin Guok,
Frank Wuerthwein,
Igor Sfiligoi,
John Graham,
Aashay Arora,
Dima Mishin,
Diego Davila,
Jonathan Guiang,
Tom Hutton,
Harvey Newman,
Justas Balcas
Abstract:
This paper describes a vision and work in progress to elevate network resources and data transfer management to the same level as compute and storage in the context of services access, scheduling, life cycle management, and orchestration. While domain science workflows often include active compute resource allocation and management, the data transfers and associated network resource coordination i…
▽ More
This paper describes a vision and work in progress to elevate network resources and data transfer management to the same level as compute and storage in the context of services access, scheduling, life cycle management, and orchestration. While domain science workflows often include active compute resource allocation and management, the data transfers and associated network resource coordination is not handled in a similar manner. As a result data transfers can introduce a degree of uncertainty in workflow operations, and the associated lack of network information does not allow for either the workflow operations or the network use to be optimized. The net result is that domain science workflow processes are forced to view the network as an opaque infrastructure into which they inject data and hope that it emerges at the destination with an acceptable Quality of Experience. There is little ability for applications to interact with the network to exchange information, negotiate performance parameters, discover expected performance metrics, or receive status/troubleshooting information in real time. Develo** mechanisms to allow an application workflow to obtain information regarding the network services, capabilities, and options, to a degree similar to what is possible for compute resources is the primary motivation for this work. The initial focus is on the Open Science Grid (OSG)/Compact Muon Solenoid (CMS) Large Hadron Collider (LHC) workflows with Rucio/FTS/XRootD based data transfers and the interoperation with the ESnet SENSE (Software-Defined Network for End-to-end Networked Science at the Exascale) system.
△ Less
Submitted 20 March, 2022; v1 submitted 15 March, 2022;
originally announced March 2022.
-
Deploying in-network caches in support of distributed scientific data sharing
Authors:
Alex Sim,
Ezra Kissel,
Chin Guok
Abstract:
The importance of intelligent data placement, management, and analysis has become apparent as scientific data volumes across the network continue to increase. To that end, we describe the use of in-network caching service deployments as a means to improve application performance and preserve available network bandwidth in a high energy physics data distribution environment. Details of the software…
▽ More
The importance of intelligent data placement, management, and analysis has become apparent as scientific data volumes across the network continue to increase. To that end, we describe the use of in-network caching service deployments as a means to improve application performance and preserve available network bandwidth in a high energy physics data distribution environment. Details of the software and hardware deployments, performance considerations, and cache usage analysis will be described. We include thoughts on possible future deployment models involving caching node installations at the edge along with methods to scale our approach.
△ Less
Submitted 13 March, 2022;
originally announced March 2022.
-
Analyzing scientific data sharing patterns for in-network data caching
Authors:
Elizabeth Copps,
Huiyi Zhang,
Alex Sim,
Kesheng Wu,
Inder Monga,
Chin Guok,
Frank Würthwein,
Diego Davila,
Edgar Fajardo
Abstract:
The volume of data moving through a network increases with new scientific experiments and simulations. Network bandwidth requirements also increase proportionally to deliver data within a certain time frame. We observe that a significant portion of the popular dataset is transferred multiple times to different users as well as to the same user for various reasons. In-network data caching for the s…
▽ More
The volume of data moving through a network increases with new scientific experiments and simulations. Network bandwidth requirements also increase proportionally to deliver data within a certain time frame. We observe that a significant portion of the popular dataset is transferred multiple times to different users as well as to the same user for various reasons. In-network data caching for the shared data has shown to reduce the redundant data transfers and consequently save network traffic volume. In addition, overall application performance is expected to improve with in-network caching because access to the locally cached data results in lower latency. This paper shows how much data was shared over the study period, how much network traffic volume was consequently saved, and how much the temporary in-network caching increased the scientific application performance. It also analyzes data access patterns in applications and the impacts of caching nodes on the regional data repository. From the results, we observed that the network bandwidth demand was reduced by nearly a factor of 3 over the study period.
△ Less
Submitted 3 May, 2021;
originally announced May 2021.
-
Software-Defined Network for End-to-end Networked Science at the Exascale
Authors:
Inder Monga,
Chin Guok,
John MacAuley,
Alex Sim,
Harvey Newman,
Justas Balcas,
Phil DeMar,
Linda Winkler,
Tom Lehman,
Xi Yang
Abstract:
Domain science applications and workflow processes are currently forced to view the network as an opaque infrastructure into which they inject data and hope that it emerges at the destination with an acceptable Quality of Experience. There is little ability for applications to interact with the network to exchange information, negotiate performance parameters, discover expected performance metrics…
▽ More
Domain science applications and workflow processes are currently forced to view the network as an opaque infrastructure into which they inject data and hope that it emerges at the destination with an acceptable Quality of Experience. There is little ability for applications to interact with the network to exchange information, negotiate performance parameters, discover expected performance metrics, or receive status/troubleshooting information in real time. The work presented here is motivated by a vision for a new smart network and smart application ecosystem that will provide a more deterministic and interactive environment for domain science workflows. The Software-Defined Network for End-to-end Networked Science at Exascale (SENSE) system includes a model-based architecture, implementation, and deployment which enables automated end-to-end network service instantiation across administrative domains. An intent based interface allows applications to express their high-level service requirements, an intelligent orchestrator and resource control systems allow for custom tailoring of scalability and real-time responsiveness based on individual application and infrastructure operator requirements. This allows the science applications to manage the network as a first-class schedulable resource as is the current practice for instruments, compute, and storage systems. Deployment and experiments on production networks and testbeds have validated SENSE functions and performance. Emulation based testing verified the scalability needed to support research and education infrastructures. Key contributions of this work include an architecture definition, reference implementation, and deployment. This provides the basis for further innovation of smart network services to accelerate scientific discovery in the era of big data, cloud computing, machine learning and artificial intelligence.
△ Less
Submitted 13 April, 2020;
originally announced April 2020.
