Workflows Community Summit 2022: A Roadmap Revolution
Authors:
Rafael Ferreira da Silva,
Rosa M. Badia,
Venkat Bala,
Debbie Bard,
Peer-Timo Bremer,
Ian Buckley,
Silvina Caino-Lores,
Kyle Chard,
Carole Goble,
Shantenu Jha,
Daniel S. Katz,
Daniel Laney,
Manish Parashar,
Frederic Suter,
Nick Tyler,
Thomas Uram,
Ilkay Altintas,
Stefan Andersson,
William Arndt,
Juan Aznar,
Jonathan Bader,
Bartosz Balis,
Chris Blanton,
Kelly Rosa Braghetto,
Aharon Brodutch
, et al. (80 additional authors not shown)
Abstract:
Scientific workflows have become integral tools in broad scientific computing use cases. Science discovery is increasingly dependent on workflows to orchestrate large and complex scientific experiments that range from execution of a cloud-based data preprocessing pipeline to multi-facility instrument-to-edge-to-HPC computational workflows. Given the changing landscape of scientific computing and t…
▽ More
Scientific workflows have become integral tools in broad scientific computing use cases. Science discovery is increasingly dependent on workflows to orchestrate large and complex scientific experiments that range from execution of a cloud-based data preprocessing pipeline to multi-facility instrument-to-edge-to-HPC computational workflows. Given the changing landscape of scientific computing and the evolving needs of emerging scientific applications, it is paramount that the development of novel scientific workflows and system functionalities seek to increase the efficiency, resilience, and pervasiveness of existing systems and applications. Specifically, the proliferation of machine learning/artificial intelligence (ML/AI) workflows, need for processing large scale datasets produced by instruments at the edge, intensification of near real-time data processing, support for long-term experiment campaigns, and emergence of quantum computing as an adjunct to HPC, have significantly changed the functional and operational requirements of workflow systems. Workflow systems now need to, for example, support data streams from the edge-to-cloud-to-HPC enable the management of many small-sized files, allow data reduction while ensuring high accuracy, orchestrate distributed services (workflows, instruments, data movement, provenance, publication, etc.) across computing and user facilities, among others. Further, to accelerate science, it is also necessary that these systems implement specifications/standards and APIs for seamless (horizontal and vertical) integration between systems and applications, as well as enabling the publication of workflows and their associated products according to the FAIR principles. This document reports on discussions and findings from the 2022 international edition of the Workflows Community Summit that took place on November 29 and 30, 2022.
△ Less
Submitted 31 March, 2023;
originally announced April 2023.
A Microservices Architecture for Distributed Complex Event Processing in Smart Cities
Authors:
Fernando Freire Scattone,
Kelly Rosa Braghetto
Abstract:
A considerable volume of data is collected from sensors today and needs to be processed in real time. Complex Event Processing (CEP) is one of the most important techniques developed for this purpose. In CEP, each new sensor measurement is considered an event and new event types can be defined based on other events occurrence. There exists several open-source CEP implementations currently availabl…
▽ More
A considerable volume of data is collected from sensors today and needs to be processed in real time. Complex Event Processing (CEP) is one of the most important techniques developed for this purpose. In CEP, each new sensor measurement is considered an event and new event types can be defined based on other events occurrence. There exists several open-source CEP implementations currently available, but all of them use orchestration to distribute event processing. This kind of architectural organization may harm system resilience, since it relies on a central core (i.e. the orchestrator). Any failures in the core might impact the whole system. Moreover, the core can become a bottleneck on system performance. In this work, a choreography-based microservices architecture is proposed for distributed CEP, in order to benefit from the low coupling and greater horizontal scalability this kind of architecture provides.
△ Less
Submitted 17 August, 2020;
originally announced August 2020.
