-
Microservices a Definition Analyzed by ßMACH
Authors:
Marcus Hilbrich,
Ninon De Mecquenem
Abstract:
Managing software artifacts is one of the most essential aspects of computer science. It enables to develop, operate, and maintain software in an engineer-like manner. Therefore, numerous concrete strategies, methods, best practices, and concepts are available. A combination of such methods must be adequate, efficient, applicable, and effective for a concrete project. Eelsewise, the developers, ma…
▽ More
Managing software artifacts is one of the most essential aspects of computer science. It enables to develop, operate, and maintain software in an engineer-like manner. Therefore, numerous concrete strategies, methods, best practices, and concepts are available. A combination of such methods must be adequate, efficient, applicable, and effective for a concrete project. Eelsewise, the developers, managers, and testers should understand it to avoid chaos. Therefore, we exemplify the ßMACH method that provides software guidance. The method can point out missing management aspects (e.g., the V-model is not usable for software operation), identify problems of knowledge transfer (e.g., how is responsible for requirements), provide an understandable management description (e.g., the developers describe what they do), and some more. The method provides a unified, knowledge-based description strategy applicable to all software management strategies. It provides a method to create a minimal but complete description. In this paper, we apply ßMACH to the microservice concept to explain both and to test the applicability and the advantages of ßMACH.
△ Less
Submitted 22 April, 2024;
originally announced April 2024.
-
Large Language Models to the Rescue: Reducing the Complexity in Scientific Workflow Development Using ChatGPT
Authors:
Mario Sänger,
Ninon De Mecquenem,
Katarzyna Ewa Lewińska,
Vasilis Bountris,
Fabian Lehmann,
Ulf Leser,
Thomas Kosch
Abstract:
Scientific workflow systems are increasingly popular for expressing and executing complex data analysis pipelines over large datasets, as they offer reproducibility, dependability, and scalability of analyses by automatic parallelization on large compute clusters. However, implementing workflows is difficult due to the involvement of many black-box tools and the deep infrastructure stack necessary…
▽ More
Scientific workflow systems are increasingly popular for expressing and executing complex data analysis pipelines over large datasets, as they offer reproducibility, dependability, and scalability of analyses by automatic parallelization on large compute clusters. However, implementing workflows is difficult due to the involvement of many black-box tools and the deep infrastructure stack necessary for their execution. Simultaneously, user-supporting tools are rare, and the number of available examples is much lower than in classical programming languages. To address these challenges, we investigate the efficiency of Large Language Models (LLMs), specifically ChatGPT, to support users when dealing with scientific workflows. We performed three user studies in two scientific domains to evaluate ChatGPT for comprehending, adapting, and extending workflows. Our results indicate that LLMs efficiently interpret workflows but achieve lower performance for exchanging components or purposeful workflow extensions. We characterize their limitations in these challenging scenarios and suggest future research directions.
△ Less
Submitted 6 November, 2023; v1 submitted 3 November, 2023;
originally announced November 2023.
-
Validity Constraints for Data Analysis Workflows
Authors:
Florian Schintke,
Ninon De Mecquenem,
David Frantz,
Vanessa Emanuela Guarino,
Marcus Hilbrich,
Fabian Lehmann,
Rebecca Sattler,
Jan Arne Sparka,
Daniel Speckhard,
Hermann Stolte,
Anh Duc Vu,
Ulf Leser
Abstract:
Porting a scientific data analysis workflow (DAW) to a cluster infrastructure, a new software stack, or even only a new dataset with some notably different properties is often challenging. Despite the structured definition of the steps (tasks) and their interdependencies during a complex data analysis in the DAW specification, relevant assumptions may remain unspecified and implicit. Such hidden a…
▽ More
Porting a scientific data analysis workflow (DAW) to a cluster infrastructure, a new software stack, or even only a new dataset with some notably different properties is often challenging. Despite the structured definition of the steps (tasks) and their interdependencies during a complex data analysis in the DAW specification, relevant assumptions may remain unspecified and implicit. Such hidden assumptions often lead to crashing tasks without a reasonable error message, poor performance in general, non-terminating executions, or silent wrong results of the DAW, to name only a few possible consequences. Searching for the causes of such errors and drawbacks in a distributed compute cluster managed by a complex infrastructure stack, where DAWs for large datasets typically are executed, can be tedious and time-consuming.
We propose validity constraints (VCs) as a new concept for DAW languages to alleviate this situation. A VC is a constraint specifying some logical conditions that must be fulfilled at certain times for DAW executions to be valid. When defined together with a DAW, VCs help to improve the portability, adaptability, and reusability of DAWs by making implicit assumptions explicit. Once specified, VC can be controlled automatically by the DAW infrastructure, and violations can lead to meaningful error messages and graceful behaviour (e.g., termination or invocation of repair mechanisms). We provide a broad list of possible VCs, classify them along multiple dimensions, and compare them to similar concepts one can find in related fields. We also provide a first sketch for VCs' implementation into existing DAW infrastructures.
△ Less
Submitted 15 May, 2023;
originally announced May 2023.
