-
The Emergence of Large Language Models in Static Analysis: A First Look through Micro-Benchmarks
Authors:
Ashwin Prasad Shivarpatna Venkatesh,
Samkutty Sabu,
Amir M. Mir,
Sofia Reis,
Eric Bodden
Abstract:
The application of Large Language Models (LLMs) in software engineering, particularly in static analysis tasks, represents a paradigm shift in the field. In this paper, we investigate the role that current LLMs can play in improving callgraph analysis and type inference for Python programs. Using the PyCG, HeaderGen, and TypeEvalPy micro-benchmarks, we evaluate 26 LLMs, including OpenAI's GPT seri…
▽ More
The application of Large Language Models (LLMs) in software engineering, particularly in static analysis tasks, represents a paradigm shift in the field. In this paper, we investigate the role that current LLMs can play in improving callgraph analysis and type inference for Python programs. Using the PyCG, HeaderGen, and TypeEvalPy micro-benchmarks, we evaluate 26 LLMs, including OpenAI's GPT series and open-source models such as LLaMA. Our study reveals that LLMs show promising results in type inference, demonstrating higher accuracy than traditional methods, yet they exhibit limitations in callgraph analysis. This contrast emphasizes the need for specialized fine-tuning of LLMs to better suit specific static analysis tasks. Our findings provide a foundation for further research towards integrating LLMs for static analysis tasks.
△ Less
Submitted 27 February, 2024;
originally announced February 2024.
-
Hidden Gems in the Rough: Computational Notebooks as an Uncharted Oasis for IDEs
Authors:
Sergey Titov,
Konstantin Grotov,
Ashwin Prasad S. Venkatesh
Abstract:
In this paper, we outline potential ways for the further development of computational notebooks in Integrated Development Environments (IDEs). We discuss notebooks integration with IDEs, focusing on three main areas: facilitating experimentation, adding collaborative features, and improving code comprehension. We propose that better support of notebooks will not only benefit the notebooks, but als…
▽ More
In this paper, we outline potential ways for the further development of computational notebooks in Integrated Development Environments (IDEs). We discuss notebooks integration with IDEs, focusing on three main areas: facilitating experimentation, adding collaborative features, and improving code comprehension. We propose that better support of notebooks will not only benefit the notebooks, but also enhance IDEs by supporting new development processes native to notebooks. In conclusion, we suggest that adapting IDEs for more experimentation-oriented notebook processes will prepare them for the future of AI-powered programming.
△ Less
Submitted 21 February, 2024;
originally announced February 2024.
-
TypeEvalPy: A Micro-benchmarking Framework for Python Type Inference Tools
Authors:
Ashwin Prasad Shivarpatna Venkatesh,
Samkutty Sabu,
Jiawei Wang,
Amir M. Mir,
Li Li,
Eric Bodden
Abstract:
In light of the growing interest in type inference research for Python, both researchers and practitioners require a standardized process to assess the performance of various type inference techniques. This paper introduces TypeEvalPy, a comprehensive micro-benchmarking framework for evaluating type inference tools. TypeEvalPy contains 154 code snippets with 845 type annotations across 18 categori…
▽ More
In light of the growing interest in type inference research for Python, both researchers and practitioners require a standardized process to assess the performance of various type inference techniques. This paper introduces TypeEvalPy, a comprehensive micro-benchmarking framework for evaluating type inference tools. TypeEvalPy contains 154 code snippets with 845 type annotations across 18 categories that target various Python features. The framework manages the execution of containerized tools, transforms inferred types into a standardized format, and produces meaningful metrics for assessment. Through our analysis, we compare the performance of six type inference tools, highlighting their strengths and limitations. Our findings provide a foundation for further research and optimization in the domain of Python type inference.
△ Less
Submitted 2 January, 2024; v1 submitted 28 December, 2023;
originally announced December 2023.
-
Static Analysis Driven Enhancements for Comprehension in Machine Learning Notebooks
Authors:
Ashwin Prasad Shivarpatna Venkatesh,
Samkutty Sabu,
Mouli Chekkapalli,
Jiawei Wang,
Li Li,
Eric Bodden
Abstract:
Jupyter notebooks enable developers to interleave code snippets with rich-text and in-line visualizations. Data scientists use Jupyter notebook as the de-facto standard for creating and sharing machine-learning based solutions, primarily written in Python. Recent studies have demonstrated, however, that a large portion of Jupyter notebooks available on public platforms are undocumented and lacks a…
▽ More
Jupyter notebooks enable developers to interleave code snippets with rich-text and in-line visualizations. Data scientists use Jupyter notebook as the de-facto standard for creating and sharing machine-learning based solutions, primarily written in Python. Recent studies have demonstrated, however, that a large portion of Jupyter notebooks available on public platforms are undocumented and lacks a narrative structure. This reduces the readability of these notebooks. To address this shortcoming, this paper presents HeaderGen, a novel tool-based approach that automatically annotates code cells with categorical markdown headers based on a taxonomy of ML operations, and classifies and displays function calls according to this taxonomy. For this functionality to be realized, HeaderGen enhances an existing call graph analysis in PyCG. To improve precision, HeaderGen extends PyCG's analysis with support for handling external library code and flow-sensitivity. The former is realized by facilitating the resolution of function return-types. The evaluation on 15 real-world Jupyter notebooks from Kaggle shows that HeaderGen's underlying call graph analysis yields high accuracy (95.6% precision and 95.3% recall). This is because HeaderGen can resolve return-types of external libraries where existing type inference tools such as pytype (by Google), pyright (by Microsoft), and Jedi fall short. The header generation has a precision of 85.7% and a recall rate of 92.8%. In a user study, HeaderGen helps participants finish comprehension and navigation tasks faster. To further evaluate the type inference capability of tools, we introduce TypeEvalPy, a framework for evaluating type inference tools with a micro-benchmark containing 154 code snippets and 845 type annotations. Our comparative analysis on four tools revealed that HeaderGen outperforms other tools in exact matches with the ground truth.
△ Less
Submitted 11 June, 2024; v1 submitted 11 January, 2023;
originally announced January 2023.
-
Security Implications Of Compiler Optimizations On Cryptography -- A Review
Authors:
A. P. Shivarpatna Venkatesh,
A. Bhat Handadi,
M. Mory
Abstract:
When implementing secure software, developers must ensure certain requirements, such as the erasure of secret data after its use and execution in real time. Such requirements are not explicitly captured by the C language and could potentially be violated by compiler optimizations. As a result, developers typically use indirect methods to hide their code's semantics from the compiler and avoid unwa…
▽ More
When implementing secure software, developers must ensure certain requirements, such as the erasure of secret data after its use and execution in real time. Such requirements are not explicitly captured by the C language and could potentially be violated by compiler optimizations. As a result, developers typically use indirect methods to hide their code's semantics from the compiler and avoid unwanted optimizations. However, such workarounds are not permanent solutions, as increasingly efficient compiler optimization causes code that was considered secure in the past now vulnerable. This paper is a literature review of (1) the security complications caused by compiler optimizations, (2) approaches used by developers to mitigate optimization problems, and (3) recent academic efforts towards enabling security engineers to communicate implicit security requirements to the compiler. In addition, we present a short study of six cryptographic libraries and how they approach the issue of ensuring security requirements. With this paper, we highlight the need for software developers and compiler designers to work together in order to design efficient systems for writing secure software.
△ Less
Submitted 4 July, 2019;
originally announced July 2019.
