-
Comparing Llama-2 and GPT-3 LLMs for HPC kernels generation
Authors:
Pedro Valero-Lara,
Alexis Huante,
Mustafa Al Lail,
William F. Godoy,
Keita Teranishi,
Prasanna Balaprakash,
Jeffrey S. Vetter
Abstract:
We evaluate the use of the open-source Llama-2 model for generating well-known, high-performance computing kernels (e.g., AXPY, GEMV, GEMM) on different parallel programming models and languages (e.g., C++: OpenMP, OpenMP Offload, OpenACC, CUDA, HIP; Fortran: OpenMP, OpenMP Offload, OpenACC; Python: numpy, Numba, pyCUDA, cuPy; and Julia: Threads, CUDA.jl, AMDGPU.jl). We built upon our previous wor…
▽ More
We evaluate the use of the open-source Llama-2 model for generating well-known, high-performance computing kernels (e.g., AXPY, GEMV, GEMM) on different parallel programming models and languages (e.g., C++: OpenMP, OpenMP Offload, OpenACC, CUDA, HIP; Fortran: OpenMP, OpenMP Offload, OpenACC; Python: numpy, Numba, pyCUDA, cuPy; and Julia: Threads, CUDA.jl, AMDGPU.jl). We built upon our previous work that is based on the OpenAI Codex, which is a descendant of GPT-3, to generate similar kernels with simple prompts via GitHub Copilot. Our goal is to compare the accuracy of Llama-2 and our original GPT-3 baseline by using a similar metric. Llama-2 has a simplified model that shows competitive or even superior accuracy. We also report on the differences between these foundational large language models as generative AI continues to redefine human-computer interactions. Overall, Copilot generates codes that are more reliable but less optimized, whereas codes generated by Llama-2 are less reliable but more optimized when correct.
△ Less
Submitted 11 September, 2023;
originally announced September 2023.
-
Analyzing the Performance Portability of Tensor Decomposition
Authors:
S. Isaac Geronimo Anderson,
Keita Teranishi,
Daniel M. Dunlavy,
Jee Choi
Abstract:
We employ pressure point analysis and roofline modeling to identify performance bottlenecks and determine an upper bound on the performance of the Canonical Polyadic Alternating Poisson Regression Multiplicative Update (CP-APR MU) algorithm in the SparTen software library. Our analyses reveal that a particular matrix computation, $Φ^{(n)}$, is the critical performance bottleneck in the SparTen CP-…
▽ More
We employ pressure point analysis and roofline modeling to identify performance bottlenecks and determine an upper bound on the performance of the Canonical Polyadic Alternating Poisson Regression Multiplicative Update (CP-APR MU) algorithm in the SparTen software library. Our analyses reveal that a particular matrix computation, $Φ^{(n)}$, is the critical performance bottleneck in the SparTen CP-APR MU implementation. Moreover, we find that atomic operations are not a critical bottleneck while higher cache reuse can provide a non-trivial performance improvement. We also utilize grid search on the Kokkos library parallel policy parameters to achieve 2.25x average speedup over the SparTen default for $Φ^{(n)}$ computation on CPU and 1.70x on GPU. We conclude our investigations by comparing Kokkos implementations of the STREAM benchmark and the matricized tensor times Khatri-Rao product (MTTKRP) benchmark from the Parallel Sparse Tensor Algorithm (PASTA) benchmark suite to implementations using vendor libraries. We show that with a single implementation Kokkos achieves performance comparable to hand-tuned code for fundamental operations that make up tensor decomposition kernels on a wide range of CPU and GPU systems. Overall, we conclude that Kokkos demonstrates good performance portability for simple data-intensive operations but requires tuning for algorithms with more complex dependencies and data access patterns.
△ Less
Submitted 6 July, 2023;
originally announced July 2023.
-
Evaluation of OpenAI Codex for HPC Parallel Programming Models Kernel Generation
Authors:
William F. Godoy,
Pedro Valero-Lara,
Keita Teranishi,
Prasanna Balaprakash,
Jeffrey S. Vetter
Abstract:
We evaluate AI-assisted generative capabilities on fundamental numerical kernels in high-performance computing (HPC), including AXPY, GEMV, GEMM, SpMV, Jacobi Stencil, and CG. We test the generated kernel codes for a variety of language-supported programming models, including (1) C++ (e.g., OpenMP [including offload], OpenACC, Kokkos, SyCL, CUDA, and HIP), (2) Fortran (e.g., OpenMP [including offl…
▽ More
We evaluate AI-assisted generative capabilities on fundamental numerical kernels in high-performance computing (HPC), including AXPY, GEMV, GEMM, SpMV, Jacobi Stencil, and CG. We test the generated kernel codes for a variety of language-supported programming models, including (1) C++ (e.g., OpenMP [including offload], OpenACC, Kokkos, SyCL, CUDA, and HIP), (2) Fortran (e.g., OpenMP [including offload] and OpenACC), (3) Python (e.g., numba, Numba, cuPy, and pyCUDA), and (4) Julia (e.g., Threads, CUDA.jl, AMDGPU.jl, and KernelAbstractions.jl). We use the GitHub Copilot capabilities powered by OpenAI Codex available in Visual Studio Code as of April 2023 to generate a vast amount of implementations given simple <kernel> + <programming model> + <optional hints> prompt variants. To quantify and compare the results, we propose a proficiency metric around the initial 10 suggestions given for each prompt. Results suggest that the OpenAI Codex outputs for C++ correlate with the adoption and maturity of programming models. For example, OpenMP and CUDA score really high, whereas HIP is still lacking. We found that prompts from either a targeted language such as Fortran or the more general-purpose Python can benefit from adding code keywords, while Julia prompts perform acceptably well for its mature programming models (e.g., Threads and CUDA.jl). We expect for these benchmarks to provide a point of reference for each programming model's community. Overall, understanding the convergence of large language models, AI, and HPC is crucial due to its rapidly evolving nature and how it is redefining human-computer interactions.
△ Less
Submitted 26 June, 2023;
originally announced June 2023.
-
Optimal Security Parameter for Encrypted Control Systems Against Eavesdropper and Malicious Server
Authors:
Kaoru Teranishi,
Kiminao Kogiso
Abstract:
A sample identifying complexity and a sample deciphering time have been introduced in a previous study to capture an estimation error and a computation time of system identification by adversaries. The quantities play a crucial role in defining the security of encrypted control systems and designing a security parameter. This study proposes an optimal security parameter for an encrypted control sy…
▽ More
A sample identifying complexity and a sample deciphering time have been introduced in a previous study to capture an estimation error and a computation time of system identification by adversaries. The quantities play a crucial role in defining the security of encrypted control systems and designing a security parameter. This study proposes an optimal security parameter for an encrypted control system under a network eavesdropper and a malicious controller server who attempt to identify system parameters using a least squares method. The security parameter design is achieved based on a modification of conventional homomorphic encryption for improving a sample deciphering time and a novel sample identifying complexity, characterized by controllability Gramians and the variance ratio of identification input to system noise. The effectiveness of the proposed design method for a security parameter is demonstrated through numerical simulations.
△ Less
Submitted 23 March, 2023;
originally announced March 2023.
-
Optimal Controller and Security Parameter for Encrypted Control Systems Under Least Squares Identification
Authors:
Kaoru Teranishi,
Kiminao Kogiso
Abstract:
Encrypted control is a framework for the secure outsourcing of controller computation using homomorphic encryption that allows to perform arithmetic operations on encrypted data without decryption. In a previous study, the security level of encrypted control systems was quantified based on the difficulty and computation time of system identification. This study investigates an optimal design of en…
▽ More
Encrypted control is a framework for the secure outsourcing of controller computation using homomorphic encryption that allows to perform arithmetic operations on encrypted data without decryption. In a previous study, the security level of encrypted control systems was quantified based on the difficulty and computation time of system identification. This study investigates an optimal design of encrypted control systems when facing an attack attempting to estimate a system parameter by the least squares method from the perspective of the security level. This study proposes an optimal $H_2$ controller that maximizes the difficulty of estimation and an equation to determine the minimum security parameter that guarantee the security of an encrypted control system as a solution to the design problem. The proposed controller and security parameter are beneficial for reducing the computation costs of an encrypted control system, while achieving the desired security level. Furthermore, the proposed design method enables the systematic design of encrypted control systems.
△ Less
Submitted 18 April, 2023; v1 submitted 23 February, 2023;
originally announced February 2023.
-
Towards Provably Secure Encrypted Control Using Homomorphic Encryption
Authors:
Kaoru Teranishi,
Kiminao Kogiso
Abstract:
Encrypted control is a promising method for the secure outsourcing of controller computation to a public cloud. However, a feasible method for security proofs of control has not yet been developed in the field of encrypted control systems. Additionally, cryptography does not consider certain types of attacks on encrypted control systems; therefore, the security of such a system cannot be guarantee…
▽ More
Encrypted control is a promising method for the secure outsourcing of controller computation to a public cloud. However, a feasible method for security proofs of control has not yet been developed in the field of encrypted control systems. Additionally, cryptography does not consider certain types of attacks on encrypted control systems; therefore, the security of such a system cannot be guaranteed using a secure cryptosystem. This study proposes a novel security definition for encrypted control under attack for control systems using cryptography. It applies the concept of provable security, which is the security of cryptosystems based on mathematical proofs, to encrypted control systems. Furthermore, this study analyzes the relation between the proposed security and the conventional security of cryptosystems. The results of the analysis demonstrated that the security of an encrypted control system can be enhanced by employing secure homomorphic encryption.
△ Less
Submitted 17 October, 2022;
originally announced October 2022.
-
Sample Identifying Complexity of Encrypted Control Systems Under Least Squares Identification
Authors:
Kaoru Teranishi,
Kiminao Kogiso
Abstract:
A sample identifying complexity has been introduced in the previous study to capture an adversary's estimation error of system identification. The complexity plays a crucial role in defining the security of encrypted control systems and designing a controller and security parameter for the systems. This study proposes a novel sample identifying complexity of encrypted control systems under an adve…
▽ More
A sample identifying complexity has been introduced in the previous study to capture an adversary's estimation error of system identification. The complexity plays a crucial role in defining the security of encrypted control systems and designing a controller and security parameter for the systems. This study proposes a novel sample identifying complexity of encrypted control systems under an adversary who identifies system parameters using a least squares method. The proposed complexity is characterized by a controllability Gramian and ratio of identification input variance to the noise variance. We examine the tightness of the proposed complexity and its changes associated with the Gramian and variance ratio through numerical simulations. The simulation results demonstrate that the proposed complexity captures a behavior of estimation error with a sufficient level. Moreover, it confirmed that the effect of controllability Gramian in the proposed complexity becomes larger as the variance ratio increases.
△ Less
Submitted 17 October, 2022;
originally announced October 2022.
-
Input-Output History Feedback Controller for Encrypted Control with Leveled Fully Homomorphic Encryption
Authors:
Kaoru Teranishi,
Tomonori Sadamoto,
Kiminao Kogiso
Abstract:
Protecting the parameters, states, and input/output signals of a dynamic controller is essential for securely outsourcing its computation to an untrusted third party. Although a fully homomorphic encryption scheme allows the evaluation of controller operations with encrypted data, an encrypted dynamic controller with the encryption scheme destabilizes a closed-loop system or degrades the control p…
▽ More
Protecting the parameters, states, and input/output signals of a dynamic controller is essential for securely outsourcing its computation to an untrusted third party. Although a fully homomorphic encryption scheme allows the evaluation of controller operations with encrypted data, an encrypted dynamic controller with the encryption scheme destabilizes a closed-loop system or degrades the control performance due to overflow. This paper presents a novel controller representation based on input-output history data to implement an encrypted dynamic controller that operates without destabilization and performance degradation. Implementation of this encrypted dynamic controller representation can be optimized via batching techniques to reduce the time and space complexities. Furthermore, this study analyzes the stability and performance degradation of a closed-loop system caused by the effects of controller encryption. A numerical simulation demonstrates the feasibility of the proposed encrypted control scheme, which inherits the control performance of the original controller at a sufficient level.
△ Less
Submitted 19 May, 2023; v1 submitted 22 September, 2021;
originally announced September 2021.
-
Parameter Sensitivity Analysis of the SparTen High Performance Sparse Tensor Decomposition Software: Extended Analysis
Authors:
Jeremy M. Myers,
Daniel M. Dunlavy,
Keita Teranishi,
D. S. Hollman
Abstract:
Tensor decomposition models play an increasingly important role in modern data science applications. One problem of particular interest is fitting a low-rank Canonical Polyadic (CP) tensor decomposition model when the tensor has sparse structure and the tensor elements are nonnegative count data. SparTen is a high-performance C++ library which computes a low-rank decomposition using different solv…
▽ More
Tensor decomposition models play an increasingly important role in modern data science applications. One problem of particular interest is fitting a low-rank Canonical Polyadic (CP) tensor decomposition model when the tensor has sparse structure and the tensor elements are nonnegative count data. SparTen is a high-performance C++ library which computes a low-rank decomposition using different solvers: a first-order quasi-Newton or a second-order damped Newton method, along with the appropriate choice of runtime parameters. Since default parameters in SparTen are tuned to experimental results in prior published work on a single real-world dataset conducted using MATLAB implementations of these methods, it remains unclear if the parameter defaults in SparTen are appropriate for general tensor data. Furthermore, it is unknown how sensitive algorithm convergence is to changes in the input parameter values. This report addresses these unresolved issues with large-scale experimentation on three benchmark tensor data sets. Experiments were conducted on several different CPU architectures and replicated with many initial states to establish generalized profiles of algorithm convergence behavior.
△ Less
Submitted 2 December, 2020;
originally announced December 2020.
-
Resiliency in Numerical Algorithm Design for Extreme Scale Simulations
Authors:
Emmanuel Agullo,
Mirco Altenbernd,
Hartwig Anzt,
Leonardo Bautista-Gomez,
Tommaso Benacchio,
Luca Bonaventura,
Hans-Joachim Bungartz,
Sanjay Chatterjee,
Florina M. Ciorba,
Nathan DeBardeleben,
Daniel Drzisga,
Sebastian Eibl,
Christian Engelmann,
Wilfried N. Gansterer,
Luc Giraud,
Dominik Goeddeke,
Marco Heisig,
Fabienne Jezequel,
Nils Kohl,
Xiaoye Sherry Li,
Romain Lion,
Miriam Mehl,
Paul Mycek,
Michael Obersteiner,
Enrique S. Quintana-Orti
, et al. (11 additional authors not shown)
Abstract:
This work is based on the seminar titled ``Resiliency in Numerical Algorithm Design for Extreme Scale Simulations'' held March 1-6, 2020 at Schloss Dagstuhl, that was attended by all the authors.
Naive versions of conventional resilience techniques will not scale to the exascale regime: with a main memory footprint of tens of Petabytes, synchronously writing checkpoint data all the way to backgr…
▽ More
This work is based on the seminar titled ``Resiliency in Numerical Algorithm Design for Extreme Scale Simulations'' held March 1-6, 2020 at Schloss Dagstuhl, that was attended by all the authors.
Naive versions of conventional resilience techniques will not scale to the exascale regime: with a main memory footprint of tens of Petabytes, synchronously writing checkpoint data all the way to background storage at frequent intervals will create intolerable overheads in runtime and energy consumption. Forecasts show that the mean time between failures could be lower than the time to recover from such a checkpoint, so that large calculations at scale might not make any progress if robust alternatives are not investigated.
More advanced resilience techniques must be devised. The key may lie in exploiting both advanced system features as well as specific application knowledge. Research will face two essential questions: (1) what are the reliability requirements for a particular computation and (2) how do we best design the algorithms and software to meet these requirements? One avenue would be to refine and improve on system- or application-level checkpointing and rollback strategies in the case an error is detected. Developers might use fault notification interfaces and flexible runtime systems to respond to node failures in an application-dependent fashion. Novel numerical algorithms or more stochastic computational approaches may be required to meet accuracy requirements in the face of undetectable soft errors.
The goal of this Dagstuhl Seminar was to bring together a diverse group of scientists with expertise in exascale computing to discuss novel ways to make applications resilient against detected and undetected faults. In particular, participants explored the role that algorithms and applications play in the holistic approach needed to tackle this challenge.
△ Less
Submitted 26 October, 2020;
originally announced October 2020.
-
RedThreads: An Interface for Application-level Fault Detection/Correction through Adaptive Redundant Multithreading
Authors:
Saurabh Hukerikar,
Keita Teranishi,
Pedro C. Diniz,
Robert F. Lucas
Abstract:
In the presence of accelerated fault rates, which are projected to be the norm on future exascale systems, it will become increasingly difficult for high-performance computing (HPC) applications to accomplish useful computation. Due to the fault-oblivious nature of current HPC programming paradigms and execution environments, HPC applications are insufficiently equipped to deal with errors. We bel…
▽ More
In the presence of accelerated fault rates, which are projected to be the norm on future exascale systems, it will become increasingly difficult for high-performance computing (HPC) applications to accomplish useful computation. Due to the fault-oblivious nature of current HPC programming paradigms and execution environments, HPC applications are insufficiently equipped to deal with errors. We believe that HPC applications should be enabled with capabilities to actively search for and correct errors in their computations. The redundant multithreading (RMT) approach offers lightweight replicated execution streams of program instructions within the context of a single application process. However, the use of complete redundancy incurs significant overhead to the application performance.
In this paper we present RedThreads, an interface that provides application-level fault detection and correction based on RMT, but applies the thread-level redundancy adaptively. We describe the RedThreads syntax and semantics, and the supporting compiler infrastructure and runtime system. Our approach enables application programmers to scope the extent of redundant computation. Additionally, the runtime system permits the use of RMT to be dynamically enabled, or disabled, based on the resiliency needs of the application and the state of the system. Our experimental results demonstrate how adaptive RMT exploits programmer insight and runtime inference to dynamically navigate the trade-off space between an application's resilience coverage and the associated performance overhead of redundant computation.
△ Less
Submitted 17 January, 2017; v1 submitted 6 October, 2016;
originally announced October 2016.
