-
An Empirical Study on the Effectiveness of Large Language Models for SATD Identification and Classification
Authors:
Mohammad Sadegh Sheikhaei,
Yuan Tian,
Shaowei Wang,
Bowen Xu
Abstract:
Self-Admitted Technical Debt (SATD), a concept highlighting sub-optimal choices in software development documented in code comments or other project resources, poses challenges in the maintainability and evolution of software systems. Large language models (LLMs) have demonstrated significant effectiveness across a broad range of software tasks, especially in software text generation tasks. Noneth…
▽ More
Self-Admitted Technical Debt (SATD), a concept highlighting sub-optimal choices in software development documented in code comments or other project resources, poses challenges in the maintainability and evolution of software systems. Large language models (LLMs) have demonstrated significant effectiveness across a broad range of software tasks, especially in software text generation tasks. Nonetheless, their effectiveness in tasks related to SATD is still under-researched. In this paper, we investigate the efficacy of LLMs in both identification and classification of SATD. For both tasks, we investigate the performance gain from using more recent LLMs, specifically the Flan-T5 family, across different common usage settings. Our results demonstrate that for SATD identification, all fine-tuned LLMs outperform the best existing non-LLM baseline, i.e., the CNN model, with a 4.4% to 7.2% improvement in F1 score. In the SATD classification task, while our largest fine-tuned model, Flan-T5-XL, still led in performance, the CNN model exhibited competitive results, even surpassing four of six LLMs. We also found that the largest Flan-T5 model, i.e., Flan-T5-XXL, when used with a zero-shot in-context learning (ICL) approach for SATD identification, provides competitive results with traditional approaches but performs 6.4% to 9.2% worse than fine-tuned LLMs. For SATD classification, few-shot ICL approach, incorporating examples and category descriptions in prompts, outperforms the zero-shot approach and even surpasses the fine-tuned smaller Flan-T5 models. Moreover, our experiments demonstrate that incorporating contextual information, such as surrounding code, into the SATD classification task enables larger fine-tuned LLMs to improve their performance.
△ Less
Submitted 10 May, 2024;
originally announced May 2024.
-
A Study of Update Request Comments in Stack Overflow Answer Posts
Authors:
Mohammad Sadegh Sheikhaei,
Yuan Tian,
Shaowei Wang
Abstract:
Comments play an important role in updating Stack Overflow (SO) posts. They are used to point out a problem (e.g., obsolete answer and buggy code) in a SO answer or ask for more details about a proposed answer. We refer to this type of comment as update request comments (URCs), which may trigger an update to the answer post and thus improve its quality.
In this study, we manually analyze a set o…
▽ More
Comments play an important role in updating Stack Overflow (SO) posts. They are used to point out a problem (e.g., obsolete answer and buggy code) in a SO answer or ask for more details about a proposed answer. We refer to this type of comment as update request comments (URCs), which may trigger an update to the answer post and thus improve its quality.
In this study, we manually analyze a set of 384 sampled SO answer posts and their associated 1,221 comments to investigate the prevalence of URCs and how URCs are addressed. We find that around half of the analyzed comments are URCs. While 55.3% of URCs are addressed within 24 hours, 36.5% of URCs remain unaddressed after a year. Moreover, we find that the current community-vote mechanism could not differentiate URCs from non-URCs. Thus many URCs might not be aware by users who can address the issue or improve the answer quality. As a first step to enhance the awareness of URCs and support future research on URCs, we investigate the feasibility of URC detection by proposing a set of features extracted from different aspects of SO comments and using them to build supervised classifiers that can automatically identify URCs. Our experiments on 377 and 289 comments posted on answers to JavaScript and Python questions show that the proposed URC classifier can achieve an accuracy of 90% and an AUC of 0.96, on average.
△ Less
Submitted 16 April, 2023;
originally announced April 2023.
-
Automated Self-Admitted Technical Debt Tracking at Commit-Level: A Language-independent Approach
Authors:
Mohammad Sadegh Sheikhaei,
Yuan Tian
Abstract:
Software and systems traceability is essential for downstream tasks such as data-driven software analysis and intelligent tool development. However, despite the increasing attention to mining and understanding technical debt in software systems, specific tools for supporting the track of technical debts are rarely available. In this work, we propose the first programming language-independent track…
▽ More
Software and systems traceability is essential for downstream tasks such as data-driven software analysis and intelligent tool development. However, despite the increasing attention to mining and understanding technical debt in software systems, specific tools for supporting the track of technical debts are rarely available. In this work, we propose the first programming language-independent tracking tool for self-admitted technical debt (SATD) -- a sub-optimal solution that is explicitly annotated by developers in software systems. Our approach takes a git repository as input and returns a list of SATDs with their evolution actions (created, deleted, updated) at the commit-level. Our approach also returns a line number indicating the latest starting position of the corresponding SATD in the system. Our SATD tracking approach first identifies an initial set of raw SATDs (which only have created and deleted actions) by detecting and tracking SATDs in commits' hunks, leveraging a state-of-the-art language-independent SATD detection approach. Then it calculates a context-based matching score between pairs of deleted and created raw SATDs in the same commits to identify SATD update actions. The results of our preliminary study on Apache Tomcat and Apache Ant show that our tracking tool can achieve a F1 score of 92.8% and 96.7% respectively.
△ Less
Submitted 16 April, 2023;
originally announced April 2023.
-
A First Look at Duplicate and Near-duplicate Self-admitted Technical Debt Comments
Authors:
Jerin Yasmin,
Mohammad Sadegh Sheikhaei,
Yuan Tian
Abstract:
Self-admitted technical debt (SATD) refers to technical debt that is intentionally introduced by developers and explicitly documented in code comments or other software artifacts (e.g., issue reports) to annotate sub-optimal decisions made by developers in the software development process.
In this work, we take the first look at the existence and characteristics of duplicate and near-duplicate S…
▽ More
Self-admitted technical debt (SATD) refers to technical debt that is intentionally introduced by developers and explicitly documented in code comments or other software artifacts (e.g., issue reports) to annotate sub-optimal decisions made by developers in the software development process.
In this work, we take the first look at the existence and characteristics of duplicate and near-duplicate SATD comments in five popular Apache OSS projects, i.e., JSPWiki, Helix, Jackrabbit, Archiva, and SystemML. We design a method to automatically identify groups of duplicate and near-duplicate SATD comments and track their evolution in the software system by mining the commit history of a software project. Leveraging the proposed method, we identified 3,520 duplicate and near-duplicate SATD comments from the target projects, which belong to 1,141 groups. We manually analyze the content and context of a sample of 1,505 SATD comments (by sampling 100 groups for each project) and identify if they annotate the same root cause. We also investigate whether duplicate SATD comments exist in code clones, whether they co-exist in the same file, and whether they are introduced and removed simultaneously. Our preliminary study reveals several surprising findings that would shed light on future studies aiming to improve the management of duplicate SATD comments. For instance, only 48.5% duplicate SATD comment groups with the same root cause exist in regular code clones, and only 33.9% of the duplicate SATD comment pairs are introduced in the same commit.
△ Less
Submitted 29 March, 2022;
originally announced March 2022.
