-
Creating a content delivery network for general science on the internet backbone using XCaches
Authors:
Edgar Fajardo,
Marian Zvada,
Derek Weitzel,
Mats Rynge,
John Hicks,
Mat Selmeci,
Brian Lin,
Pascal Paschos,
Brian Bockelman,
Igor Sfiligoi,
Andrew Hanushevsky,
Frank Würthwein
Abstract:
A general problem faced by computing on the grid for opportunistic users is that delivering cycles is simpler than delivering data to those cycles. In this project we show how we integrated XRootD caches placed on the internet backbone to implement a content delivery network for general science workflows. We will show that for some workflows on different science domains like high energy physics, g…
▽ More
A general problem faced by computing on the grid for opportunistic users is that delivering cycles is simpler than delivering data to those cycles. In this project we show how we integrated XRootD caches placed on the internet backbone to implement a content delivery network for general science workflows. We will show that for some workflows on different science domains like high energy physics, gravitational waves, and others the combination of data reuse from the workflows together with the use of caches increases CPU efficiency while decreasing network bandwidth use.
△ Less
Submitted 28 September, 2020; v1 submitted 2 July, 2020;
originally announced July 2020.
-
Designing a Multi-petabyte Database for LSST
Authors:
Jacek Becla,
Andrew Hanushevsky,
Sergei Nikolaev,
Ghaleb Abdulla,
Alex Szalay,
Maria Nieto-Santisteban,
Ani Thakar,
Jim Gray
Abstract:
The 3.2 giga-pixel LSST camera will produce approximately half a petabyte of archive images every month. These data need to be reduced in under a minute to produce real-time transient alerts, and then added to the cumulative catalog for further analysis. The catalog is expected to grow about three hundred terabytes per year. The data volume, the real-time transient alerting requirements of the L…
▽ More
The 3.2 giga-pixel LSST camera will produce approximately half a petabyte of archive images every month. These data need to be reduced in under a minute to produce real-time transient alerts, and then added to the cumulative catalog for further analysis. The catalog is expected to grow about three hundred terabytes per year. The data volume, the real-time transient alerting requirements of the LSST, and its spatio-temporal aspects require innovative techniques to build an efficient data access system at reasonable cost. As currently envisioned, the system will rely on a database for catalogs and metadata. Several database systems are being evaluated to understand how they perform at these data rates, data volumes, and access patterns. This paper describes the LSST requirements, the challenges they impose, the data access philosophy, results to date from evaluating available database technologies against LSST requirements, and the proposed database architecture to meet the data challenges.
△ Less
Submitted 27 April, 2006;
originally announced April 2006.
-
On the Verge of One Petabyte - the Story Behind the BaBar Database System
Authors:
Adeyemi Adesanya,
Tofigh Azemoon,
Jacek Becla,
Andrew Hanushevsky,
Adil Hasan,
Wilko Kroeger,
Artem Trunov,
Daniel Wang,
Igor Gaponenko,
Simon Patton,
David Quarrie
Abstract:
The BaBar database has pioneered the use of a commercial ODBMS within the HEP community. The unique object-oriented architecture of Objectivity/DB has made it possible to manage over 700 terabytes of production data generated since May'99, making the BaBar database the world's largest known database. The ongoing development includes new features, addressing the ever-increasing luminosity of the…
▽ More
The BaBar database has pioneered the use of a commercial ODBMS within the HEP community. The unique object-oriented architecture of Objectivity/DB has made it possible to manage over 700 terabytes of production data generated since May'99, making the BaBar database the world's largest known database. The ongoing development includes new features, addressing the ever-increasing luminosity of the detector as well as other changing physics requirements. Significant efforts are focused on reducing space requirements and operational costs. The paper discusses our experience with develo** a large scale database system, emphasizing universal aspects which may be applied to any large scale system, independently of underlying technology used.
△ Less
Submitted 4 June, 2003; v1 submitted 4 June, 2003;
originally announced June 2003.
