-
TensorBank: Tensor Lakehouse for Foundation Model Training
Authors:
Romeo Kienzler,
Leonardo Pondian Tizzei,
Benedikt Blumenstiel,
Zoltan Arnold Nagy,
S. Karthik Mukkavilli,
Johannes Schmude,
Marcus Freitag,
Michael Behrendt,
Daniel Salles Civitarese,
Naomi Simumba,
Daiki Kimura,
Hendrik Hamann
Abstract:
Storing and streaming high dimensional data for foundation model training became a critical requirement with the rise of foundation models beyond natural language. In this paper we introduce TensorBank, a petabyte scale tensor lakehouse capable of streaming tensors from Cloud Object Store (COS) to GPU memory at wire speed based on complex relational queries. We use Hierarchical Statistical Indices…
▽ More
Storing and streaming high dimensional data for foundation model training became a critical requirement with the rise of foundation models beyond natural language. In this paper we introduce TensorBank, a petabyte scale tensor lakehouse capable of streaming tensors from Cloud Object Store (COS) to GPU memory at wire speed based on complex relational queries. We use Hierarchical Statistical Indices (HSI) for query acceleration. Our architecture allows to directly address tensors on block level using HTTP range reads. Once in GPU memory, data can be transformed using PyTorch transforms. We provide a generic PyTorch dataset type with a corresponding dataset factory translating relational queries and requested transformations as an instance. By making use of the HSI, irrelevant blocks can be skipped without reading them as those indices contain statistics on their content at different hierarchical resolution levels. This is an opinionated architecture powered by open standards and making heavy use of open-source technology. Although, hardened for production use using geospatial-temporal data, this architecture generalizes to other use case like computer vision, computational neuroscience, biological sequence analysis and more.
△ Less
Submitted 21 March, 2024; v1 submitted 5 September, 2023;
originally announced September 2023.
-
DevOps and Microservices in Scientific System development
Authors:
Maximillien de Bayser,
Vinicius Segura,
Leonardo Guerreiro Azevedo,
Leonardo P. Tizzei,
Raphael Melo Thiago,
Elton Soares,
Renato Cerqueira
Abstract:
There is a gap in scientific information systems development concerning modern software engineering and scientific computing. Historically, software engineering methodologies have been perceived as an unwanted accidental complexity to computational scientists in their scientific systems development. More recent trends, like the end of Moore's law and the subsequent diversification of hardware plat…
▽ More
There is a gap in scientific information systems development concerning modern software engineering and scientific computing. Historically, software engineering methodologies have been perceived as an unwanted accidental complexity to computational scientists in their scientific systems development. More recent trends, like the end of Moore's law and the subsequent diversification of hardware platforms, combined with the increasing multidisciplinarity of science itself have exacerbated the problem because self-taught "end user developers" are not familiar with the disciplines needed to tackle this increased complexity. On a more positive note, agile programming methods have approached software development practices to the way scientific software is produced. In this work, we present the experience of a multi-year industry research project where agile methods, microservices and DevOps were applied. Our goal is to validate the hypothesis that the use of microservices would allow computational scientists to work in the more minimalistic prototype-oriented way that they prefer while the software engineering team would handle the integration. Hence, scientific multidisciplinary systems would gain in a twofold way: (i) Subject Matter Experts(SME) use their preferable tools to develop the specific scientific part of the system; (ii) software engineers provide the high quality software code for the system delivery.
△ Less
Submitted 22 December, 2021;
originally announced December 2021.
-
Applying Software Craftsmanship Practices to a Scrum Project: an Experience Report
Authors:
Percival Lucena,
Leonardo P. Tizzei
Abstract:
The Software Craftsmanship manifesto has defined values and principles that software development teams should follow to deliver quality software that fulfills functional and non-functional requirements without dealing with high amounts of technical debt. Software craftsmanship approach to software development prioritizes technical practices in order to provide a clean code base. This work analyzes…
▽ More
The Software Craftsmanship manifesto has defined values and principles that software development teams should follow to deliver quality software that fulfills functional and non-functional requirements without dealing with high amounts of technical debt. Software craftsmanship approach to software development prioritizes technical practices in order to provide a clean code base. This work analyzes a set of practices that can be applied to a Scrum project that aims to incorporate Software Craftsmanship values. The process implementation described may be a useful contribution for software development teams who also intend to implement Software Craftsmanship on their projects.
△ Less
Submitted 17 November, 2016;
originally announced November 2016.
-
Job Placement Advisor Based on Turnaround Predictions for HPC Hybrid Clouds
Authors:
Renato L. F. Cunha,
Eduardo R. Rodrigues,
Leonardo P. Tizzei,
Marco A. S. Netto
Abstract:
Several companies and research institutes are moving their CPU-intensive applications to hybrid High Performance Computing (HPC) cloud environments. Such a shift depends on the creation of software systems that help users decide where a job should be placed considering execution time and queue wait time to access on-premise clusters. Relying blindly on turnaround prediction techniques will affect…
▽ More
Several companies and research institutes are moving their CPU-intensive applications to hybrid High Performance Computing (HPC) cloud environments. Such a shift depends on the creation of software systems that help users decide where a job should be placed considering execution time and queue wait time to access on-premise clusters. Relying blindly on turnaround prediction techniques will affect negatively response times inside HPC cloud environments. This paper introduces a tool to make job placement decisions in HPC hybrid cloud environments taking into account the inaccuracy of execution and waiting time predictions. We used job traces from real supercomputing centers to run our experiments, and compared the performance between environments using real speedup curves. We also extended a state-of-the-art machine learning based predictor to work with data from the cluster scheduler. Our main findings are: (i) depending on workload characteristics, there is a turning point where predictions should be disregarded in favor of a more conservative decision to minimize job turnaround times and (ii) scheduler data plays a key role in improving predictions generated with machine learning using job trace data---our experiments showed around 20% prediction accuracy improvements.
△ Less
Submitted 26 August, 2016; v1 submitted 22 August, 2016;
originally announced August 2016.
-
An SLA-based Advisor for Placement of HPC Jobs on Hybrid Clouds
Authors:
Kiran Mantripragada,
Leonardo P. Tizzei,
Alecio P. D. Binotto,
Marco A. S. Netto
Abstract:
Several scientific and industry applications require High Performance Computing (HPC) resources to process and/or simulate complex models. Not long ago, companies, research institutes, and universities used to acquire and maintain on-premise computer clusters; but, recently, cloud computing has emerged as an alternative for a subset of HPC applications. This poses a challenge to end-users, who hav…
▽ More
Several scientific and industry applications require High Performance Computing (HPC) resources to process and/or simulate complex models. Not long ago, companies, research institutes, and universities used to acquire and maintain on-premise computer clusters; but, recently, cloud computing has emerged as an alternative for a subset of HPC applications. This poses a challenge to end-users, who have to decide where to run their jobs: on local clusters or burst to a remote cloud service provider. While current research on HPC cloud has focused on comparing performance of on-premise clusters against cloud resources, we build on top of existing efforts and introduce an advisory service to help users make this decision considering the trade-offs of resource costs, performance, and availability on hybrid clouds. We evaluated our service using a real test-bed with a seismic processing application based on Full Waveform Inversion; a technique used by geophysicists in the oil & gas industry and earthquake prediction. We also discuss how the advisor can be used for other applications and highlight the main lessons learned constructing this service to reduce costs and turnaround times.
△ Less
Submitted 20 July, 2015;
originally announced July 2015.
-
A Self-adaptive Auto-scaling Method for Scientific Applications on HPC Environments and Clouds
Authors:
Kiran Mantripragada,
Alecio Binotto,
Leonardo P. Tizzei
Abstract:
High intensive computation applications can usually take days to months to finish an execution. During this time, it is common to have variations of the available resources when considering that such hardware is usually shared among a plurality of researchers/departments within an organization. On the other hand, High Performance Clusters can take advantage of Cloud Computing bursting techniques f…
▽ More
High intensive computation applications can usually take days to months to finish an execution. During this time, it is common to have variations of the available resources when considering that such hardware is usually shared among a plurality of researchers/departments within an organization. On the other hand, High Performance Clusters can take advantage of Cloud Computing bursting techniques for the execution of applications together with the on-premise resources. In order to meet deadlines, high intensive computational applications can use the Cloud to boost their performance when they are data and task parallel. This article presents an ongoing work towards the use of extended resources of an HPC execution platform together with Cloud. We propose an unified view of such heterogeneous environments and a method that monitors, predicts the application execution time, and dynamically shifts part of the domain -- previously running in local HPC hardware -- to be computed on the Cloud, meeting then a specific deadline. The method is exemplified along with a seismic application that, at runtime, adapts itself to move part of the processing to the Cloud (in a movement called bursting) and also auto-scales (the moved part) over cloud nodes. Our preliminary results show that there is an expected overhead for performing this movement and for synchronizing results, but our outcomes demonstrate it is an important feature for meeting deadlines in the case an on-premise cluster is overloaded or cannot provide the capacity needed for a particular project.
△ Less
Submitted 26 January, 2015; v1 submitted 19 December, 2014;
originally announced December 2014.
