-
INDIGO-DataCloud:A data and computing platform to facilitate seamless access to e-infrastructures
Authors:
INDIGO-DataCloud Collaboration,
:,
Davide Salomoni,
Isabel Campos,
Luciano Gaido,
Jesus Marco de Lucas,
Peter Solagna,
Jorge Gomes,
Ludek Matyska,
Patrick Fuhrman,
Marcus Hardt,
Giacinto Donvito,
Lukasz Dutka,
Marcin Plociennik,
Roberto Barbera,
Ignacio Blanquer,
Andrea Ceccanti,
Mario David,
Cristina Duma,
Alvaro López-García,
Germán Moltó,
Pablo Orviz,
Zdenek Sustr,
Matthew Viljoen,
Fernando Aguilar
, et al. (40 additional authors not shown)
Abstract:
This paper describes the achievements of the H2020 project INDIGO-DataCloud. The project has provided e-infrastructures with tools, applications and cloud framework enhancements to manage the demanding requirements of scientific communities, either locally or through enhanced interfaces. The middleware developed allows to federate hybrid resources, to easily write, port and run scientific applicat…
▽ More
This paper describes the achievements of the H2020 project INDIGO-DataCloud. The project has provided e-infrastructures with tools, applications and cloud framework enhancements to manage the demanding requirements of scientific communities, either locally or through enhanced interfaces. The middleware developed allows to federate hybrid resources, to easily write, port and run scientific applications to the cloud. In particular, we have extended existing PaaS (Platform as a Service) solutions, allowing public and private e-infrastructures, including those provided by EGI, EUDAT, and Helix Nebula, to integrate their existing services and make them available through AAI services compliant with GEANT interfederation policies, thus guaranteeing transparency and trust in the provisioning of such services. Our middleware facilitates the execution of applications using containers on Cloud and Grid based infrastructures, as well as on HPC clusters. Our developments are freely downloadable as open source components, and are already being integrated into many scientific applications.
△ Less
Submitted 5 February, 2019; v1 submitted 6 November, 2017;
originally announced November 2017.
-
Improved Cloud resource allocation: how INDIGO-DataCloud is overcoming the current limitations in Cloud schedulers
Authors:
Alvaro Lopez Garcia,
Lisa Zangrando,
Massimo Sgaravatto,
Vincent Llorens,
Sara Vallero,
Valentina Zaccolo,
Stefano Bagnasco,
Sonia Taneja,
Stefano Dal Pra,
Davide Salomoni,
Giacinto Donvito
Abstract:
Performing efficient resource provisioning is a fundamental aspect for any resource provider. Local Resource Management Systems (LRMS) have been used in data centers for decades in order to obtain the best usage of the resources, providing their fair usage and partitioning for the users. In contrast, current cloud schedulers are normally based on the immediate allocation of resources on a first-co…
▽ More
Performing efficient resource provisioning is a fundamental aspect for any resource provider. Local Resource Management Systems (LRMS) have been used in data centers for decades in order to obtain the best usage of the resources, providing their fair usage and partitioning for the users. In contrast, current cloud schedulers are normally based on the immediate allocation of resources on a first-come, first-served basis, meaning that a request will fail if there are no resources (e.g. OpenStack) or it will be trivially queued ordered by entry time (e.g. OpenNebula). Moreover, these scheduling strategies are based on a static partitioning of the resources, meaning that existing quotas cannot be exceeded, even if there are idle resources allocated to other projects. This is a consequence of the fact that cloud instances are not associated with a maximum execution time and leads to a situation where the resources are under-utilized. These facts have been identified by the INDIGO-DataCloud project as being too simplistic for accommodating scientific workloads in an efficient way, leading to an underutilization of the resources, a non desirable situation in scientific data centers. In this work, we will present the work done in the scheduling area during the first year of the INDIGO project and the foreseen evolutions.
△ Less
Submitted 20 July, 2017;
originally announced July 2017.
-
Running CMS software on GRID Testbeds
Authors:
D. Bonacorsi,
P. Capiluppi,
A. Fanfani,
C. Grandi,
M. Corvo,
F. Fanzago,
M. Sgaravatto,
M. Verlato,
C. Charlot,
I. Semeniuok,
D. Colling,
B. MacEvoy,
H. Tallini,
M. Biasotto,
S. Fantinel,
E. Leonardi,
A. Sciaba',
O. Maroney,
I. Augustin,
E. Laure,
M. Schulz,
H. Stockinger,
V. Lefebure,
S. Burke,
J. J. Blaising
, et al. (5 additional authors not shown)
Abstract:
Starting in the middle of November 2002, the CMS experiment undertook an evaluation of the European DataGrid Project (EDG) middleware using its event simulation programs. A joint CMS-EDG task force performed a "stress test" by submitting a large number of jobs to many distributed sites. The EDG testbed was complemented with additional CMS-dedicated resources. A total of ~ 10000 jobs consisting o…
▽ More
Starting in the middle of November 2002, the CMS experiment undertook an evaluation of the European DataGrid Project (EDG) middleware using its event simulation programs. A joint CMS-EDG task force performed a "stress test" by submitting a large number of jobs to many distributed sites. The EDG testbed was complemented with additional CMS-dedicated resources. A total of ~ 10000 jobs consisting of two different computational types were submitted from four different locations in Europe over a period of about one month. Nine sites were active, providing integrated resources of more than 500 CPUs and about 5 TB of disk space (with the additional use of two Mass Storage Systems). Descriptions of the adopted procedures, the problems encountered and the corresponding solutions are reported. Results and evaluations of the test, both from the CMS and the EDG perspectives, are described.
△ Less
Submitted 4 June, 2003;
originally announced June 2003.
-
Sharing a conceptual model of grid resources and services
Authors:
Sergio Andreozzi,
Massimo Sgaravatto,
Cristina Vistoli
Abstract:
Grid technologies aim at enabling a coordinated resource-sharing and problem-solving capabilities over local and wide area networks and span locations, organizations, machine architectures and software boundaries. The heterogeneity of involved resources and the need for interoperability among different grid middlewares require the sharing of a common information model. Abstractions of different…
▽ More
Grid technologies aim at enabling a coordinated resource-sharing and problem-solving capabilities over local and wide area networks and span locations, organizations, machine architectures and software boundaries. The heterogeneity of involved resources and the need for interoperability among different grid middlewares require the sharing of a common information model. Abstractions of different flavors of resources and services and conceptual schemas of domain specific entities require a collaboration effort in order to enable a coherent information services cooperation.
With this paper, we present the result of our experience in grid resources and services modelling carried out within the Grid Laboratory Uniform Environment (GLUE) effort, a joint US and EU High Energy Physics projects collaboration towards grid interoperability. The first implementation-neutral agreement on services such as batch computing and storage manager, resources such as the hierarchy cluster, sub-cluster, host and the storage library are presented. Design guidelines and operational results are depicted together with open issues and future evolutions.
△ Less
Submitted 18 June, 2003;
originally announced June 2003.
-
The EU DataGrid Workload Management System: towards the second major release
Authors:
G. Avellino,
S. Barale,
S. Beco,
B. Cantalupo,
D. Colling,
F. Giacomini,
A. Gianelle,
A. Guarise,
A. Krenek,
D. Kouril,
A. Maraschini,
L. Matyska,
M. Mezzadri,
S. Monforte,
M. Mulac,
F. Pacini,
M. Pappalardo,
R. Peluso,
J. Pospisil,
F. Prelz,
E. Ronchieri,
M. Ruda,
L. Salconi,
Z. Salvet,
M. Sgaravatto
, et al. (4 additional authors not shown)
Abstract:
In the first phase of the European DataGrid project, the 'workload management' package (WP1) implemented a working prototype, providing users with an environment allowing to define and submit jobs to the Grid, and able to find and use the ``best'' resources for these jobs. Application users have now been experiencing for about a year now with this first release of the workload management system.…
▽ More
In the first phase of the European DataGrid project, the 'workload management' package (WP1) implemented a working prototype, providing users with an environment allowing to define and submit jobs to the Grid, and able to find and use the ``best'' resources for these jobs. Application users have now been experiencing for about a year now with this first release of the workload management system. The experiences acquired, the feedback received by the user and the need to plug new components implementing new functionalities, triggered an update of the existing architecture. A description of this revised and complemented workload management system is given.
△ Less
Submitted 13 June, 2003;
originally announced June 2003.
-
The first deployment of workload management services on the EU DataGrid Testbed: feedback on design and implementation
Authors:
G. Avellino,
S. Beco,
B. Cantalupo,
F. Pacini,
A. Terracina,
A. Maraschini,
D. Colling,
S. Monforte,
M. Pappalardo,
L. Salconi,
F. Giacomini,
E. Ronchieri,
D. Kouril,
A. Krenek,
L. Matyska,
M. Mulac,
J. Pospisil,
M. Ruda,
Z. Salvet,
J. Sitera,
M. Vocu,
M. Mezzadri,
F. Prelz,
A. Gianelle,
R. Peluso
, et al. (4 additional authors not shown)
Abstract:
Application users have now been experiencing for about a year with the standardized resource brokering services provided by the 'workload management' package of the EU DataGrid project (WP1). Understanding, sha** and pushing the limits of the system has provided valuable feedback on both its design and implementation. A digest of the lessons, and "better practices", that were learned, and that…
▽ More
Application users have now been experiencing for about a year with the standardized resource brokering services provided by the 'workload management' package of the EU DataGrid project (WP1). Understanding, sha** and pushing the limits of the system has provided valuable feedback on both its design and implementation. A digest of the lessons, and "better practices", that were learned, and that were applied towards the second major release of the software, is given.
△ Less
Submitted 31 May, 2003;
originally announced June 2003.
