-
CMS Workflow Execution using Intelligent Job Scheduling and Data Access Strategies
Authors:
Khawar Hasham,
Antonio Delgado Peris,
Ashiq Anjum,
Dave Evans,
Dirk Hufnagel,
Eduardo Huedo,
José M. Hernández,
Richard McClatchey,
Stephen Gowdy,
Simon Metson
Abstract:
Complex scientific workflows can process large amounts of data using thousands of tasks. The turnaround times of these workflows are often affected by various latencies such as the resource discovery, scheduling and data access latencies for the individual workflow processes or actors. Minimizing these latencies will improve the overall execution time of a workflow and thus lead to a more efficien…
▽ More
Complex scientific workflows can process large amounts of data using thousands of tasks. The turnaround times of these workflows are often affected by various latencies such as the resource discovery, scheduling and data access latencies for the individual workflow processes or actors. Minimizing these latencies will improve the overall execution time of a workflow and thus lead to a more efficient and robust processing environment. In this paper, we propose a pilot job based infrastructure that has intelligent data reuse and job execution strategies to minimize the scheduling, queuing, execution and data access latencies. The results have shown that significant improvements in the overall turnaround time of a workflow can be achieved with this approach. The proposed approach has been evaluated, first using the CMS Tier0 data processing workflow, and then simulating the workflows to evaluate its effectiveness in a controlled environment.
△ Less
Submitted 24 February, 2012;
originally announced February 2012.
-
The Grid[Way] Job Template Manager, a tool for parameter swee**
Authors:
Alejandro Lorca,
Eduardo Huedo,
Ignacio M. Llorente
Abstract:
Parameter swee** is a widely used algorithmic technique in computational science. It is specially suited for high-throughput computing since the jobs evaluating the parameter space are loosely coupled or independent.
A tool that integrates the modeling of a parameter study with the control of jobs in a distributed architecture is presented. The main task is to facilitate the creation and del…
▽ More
Parameter swee** is a widely used algorithmic technique in computational science. It is specially suited for high-throughput computing since the jobs evaluating the parameter space are loosely coupled or independent.
A tool that integrates the modeling of a parameter study with the control of jobs in a distributed architecture is presented. The main task is to facilitate the creation and deletion of job templates, which are the elements describing the jobs to be run. Extra functionality relies upon the GridWay Metascheduler, acting as the middleware layer for job submission and control. It supports interesting features like multi-dimensional swee** space, wildcarding of parameters, functional evaluation of ranges, value-skip** and job template automatic indexation.
The use of this tool increases the reliability of the parameter sweep study thanks to the systematic bookke** of job templates and respective job statuses. Furthermore, it simplifies the porting of the target application to the grid reducing the required amount of time and effort.
△ Less
Submitted 5 March, 2010;
originally announced March 2010.
-
Grid porting of Bhabha scattering code through a master-worker scheme
Authors:
Alejandro Lorca,
Jose Luis Vazquez-Poletti,
Eduardo Huedo,
Ignacio M. Llorente
Abstract:
A program calculating Bhabha scattering at high energy colliders is considered for porting to the EGEE Grid infrastructure. The program code, which is a result of the aITALC project, is ported by using a master-worker operating scheme. The job submission, execution and monitoring are implemented using the GridWay metascheduler. The unattended execution of jobs turned out to be complete and rathe…
▽ More
A program calculating Bhabha scattering at high energy colliders is considered for porting to the EGEE Grid infrastructure. The program code, which is a result of the aITALC project, is ported by using a master-worker operating scheme. The job submission, execution and monitoring are implemented using the GridWay metascheduler. The unattended execution of jobs turned out to be complete and rather efficient, even when pre-knowledge of the grid is absent. While the batch of jobs remains organized at the user's side, the actual computation was carried out within the phenogrid virtual organization. The scientific results support the use of the small angle Bhabha scattering for the luminosity measurements of the International Linear Collider project.
△ Less
Submitted 1 April, 2009;
originally announced April 2009.
