An Empirical Study on the Effectiveness of Large Language Models for SATD Identification and Classification

The goal of SATD identification is to determine whether a given code comment admitted that the corresponding code is a technical debt or not (Sheikhaei and Tian,, 2023). A few studies aim to detect SATD in other software artifacts, such as issues (Li et al.,, 2022; Li et al., 2023a, ), leveraging comment-based SATD approaches. Existing SATD identification approaches can be classified into two categories: 1) rule-based approaches (Potdar and Shihab,, 2014; Guo et al.,, 2021; Sridharan et al.,, 2023) which search for certain keywords (e.g., TODO) or phrases (e.g., “probably a bug”) in the code comment, and 2) supervised learning based approaches (Huang et al.,, 2017; Maldonado et al.,, 2017; Ren et al.,, 2019; Prenner and Robbes,, 2022) which train a model from labeled data and evaluate it on unseen code comments.

One of the most decent rule-based approaches is Matches task Annotation Tags (MAT) (Guo et al.,, 2021). In this approach, the authors showed that just matching a set of popular task annotation tags, i.e., TODO, FIXME, HACK, and XXX, provides similar or even superior performance for SATD identification compared with traditional machine learning approaches such as the natural language processing (NLP) approach (Maldonado et al.,, 2017) which leverages maximum entropy classifier, and the text-mining based approach (Huang et al.,, 2017) that employs Naïve Bayes Multinomial classifier. A more recent rule-based approach is PENTACET (Sridharan et al.,, 2023), where the authors extend the 64 SATD identification patterns introduced by Potdar and Shihab (Potdar and Shihab,, 2014), to 1,041 patterns using a tool named Sense2Vec (Trask et al.,, 2015). Sense2Vec captures contextually similar words with its word embedding and the authors use these extrapolated features (words) to evaluate the code comments for SATD. While it has shown that simple rule-based approaches, e.g., MAT, have a high precision, though not having a good recall, the performance of a complex rule-based approach such as PENTACET is not well studied. In this paper, we consider both MAT and PENTACET approaches as rule-based baselines, and the NLP approach (Maldonado et al.,, 2017) as a traditional machine-learning-based baseline for the SATD identification task.

More recently, supervised deep learning-based methods have emerged and achieved more promising performance compared to rule-based and traditional machine-learning approaches. Ren et al. (Ren et al.,, 2019) proposed a Convolutional Neural Network (CNN) based approach that outperforms traditional text-mining models. Yu et al. (Yu et al.,, 2021) proposed a Bidirectional Long Short-Term Memory (BiLSTM) model with a balanced cross-entropy loss function to overcome the class unbalance challenge. A more recent study by Prenner and Robbes (Prenner and Robbes,, 2022) showed that the BERT base models provide state-of-the-art results in different software-related tasks, including SATD identification. As the CNN and the BERT base models have achieved the state-of-the-art results in this domain, we use CNN as a strong non-LLM baseline, and BERT as a LLM baseline in our experiments for both SATD identification and classification tasks.

Maldonado and Shihab (Maldonado and Shihab,, 2015) introduced the task of SATD classification. They extracted 33,093 code comments from five Java projects and manually classified them into six categories: non-SATD, Design, Requirement, Defect, Test, and Documentation. Later, Maldonado et al., (2017) extended the dataset by adding another five Java projects with the same categories and proposed an NLP approach using the Java implementation of a maximum entropy classifier, Stanford Classifier (Manning and Klein,, 2003), to identify and classify SATDs by the code comments. Later, other researchers worked on that dataset and proposed another classification from a different perspective. Fucci et al. (Fucci et al.,, 2021) and Cassee et al. (Cassee et al.,, 2022) proposed a bottom-up strategy (i.e., what do SATD comments mention?) rather than top-down (i.e., how do SATD comments map into a software development life-cycle?) which was utlized in (Maldonado and Shihab,, 2015). They manually classified 1,038 SATD instances from Maldonado dataset into 41 categories and subcategories, such as “poor implemented choices” and “functional issues”. Chen et al. (Chen et al.,, 2022) leveraged chi-square to select representative features from the textual features, and applied the XGBoost model for the SATD classification task on the Maldonado dataset.

In addition to SATD classification for general software systems, there are studies that focus on identifying and classifying SATDs in specific domains such as Blockchain projects (Pinna et al.,, 2023), machine learning (ML) software (OBrien et al.,, 2022; Bhatia et al.,, 2023), and deep learning frameworks (Liu et al.,, 2020). Pinna et al. (Pinna et al.,, 2023) employed the NLP approach (Maldonado et al.,, 2017) to detect Design SATDs and Requirement SATDs in Blockchain projects. OBrien et al. (OBrien et al.,, 2022) manually classified the extracted SATDs into six general categories, which were previously introduced by Bavota and Russo (Bavota and Russo,, 2016): Requirement, Code, Test, Defect, Design, and Documentation. Then, they defined 23 specific categories for machine learning-related SATD and manually applied them to their dataset. Bhatia et al. (Bhatia et al.,, 2023) employed the text-mining tool (Liu et al.,, 2018) to identify SATDs in 318 ML and 318 non-ML open-source software projects. They then manually classified 611 randomly sampled SATDs based on the categories introduced by Bavota and Russo (Bavota and Russo,, 2016).

LLMs are deep learning models (Goodfellow et al.,, 2016), typically based on the Transformer architecture (Vaswani et al.,, 2017), and pre-trained on extensive corpora. One of the early milestones in LLMs was BERT (Bidirectional Encoder Representations from Transformers) (Devlin et al.,, 2019), which pioneered the use of bidirectional training to enhance the understanding of word context within sentences, significantly advancing performance across a range of natural language processing tasks. Following BERT’s encoder-only architecture, which excelled mainly in text classification, a new wave of LLMs emerged. These newer models adopted encoder-decoder (Raffel et al.,, 2019) or decoder-only architectures (Radford et al.,, 2019), catalyzing a revolution in text generation tasks. Concurrently, variations of the BERT model were introduced, such as CodeBERT (Feng et al.,, 2020), designed for understanding and generating code by blending natural language and programming language training, and RoBERTa (Liu et al.,, 2019), an optimized version of BERT with improved training methodology and larger datasets.

T5 (Text-to-Text Transfer Transformer) (Raffel et al.,, 2019) and Flan-T5 (Chung et al.,, 2022) are notable examples of LLMs utilizing the encoder-decoder architecture. T5, developed by Google, treats every language task as a text-to-text problem, converting tasks like translation, summarization, and question-answering into a unified framework. Flan-T5, an extension of T5, further enhances its capabilities through fine-tuning with a mixture of instruction-based tasks, improving its performance on various benchmarks. Both models are available in five different sizes, ranging from the small size with 77M parameters to the XXL size with 11.1B parameters. This variety enables researchers to study the effect of model size on their applications. Flan-T5 has been employed in various applications such as text summarization (Tam et al.,, 2023) and log parsing (Jiang et al.,, 2023) tasks and has demonstrated impressive performance, even when compared to some newer models like LLaMA (Touvron et al.,, 2023).

There are two general approaches for applying LLMs on downstream tasks: 1) fine-tuning: continuously train the LLM on the downstream task for a few more epochs, and 2) in-context learning (ICL): instructing the model on what it is expected to do, and if necessary, adding a few examples in the instruction to make the task clearer. The process of finding a good instruction to achieve good results is called prompt engineering. The main advantage of ICL over fine-tuning is that it doesn’t require investing time and processing resources to update model parameters for the downstream task. In this study, we consider both approaches for the identification and classification of SATDs.

LLMs have been successfully employed in different Software Engineering (SE) tasks. Prenner and Robbes (Prenner and Robbes,, 2022) applied BERT-based models on a selection of 13 smaller datasets from the SE literature, and achieved superior performance for tasks involving natural language. Gao et al. (Gao et al.,, 2023) leveraged the OpenAI’s Codex and GPT-3.5 models using the ICL approach for code intelligence tasks including code summarization, bug fixing, and program synthesis. These are just two examples in LLM4SE (Large Language Models for Software Engineering) domain. Hou et al. (Hou et al.,, 2023) conducted a systematic literature review on LLM4SE by studying 229 research papers from 2017 to 2023. The success of LLMs in various SE-related tasks motivated us to investigate their effectiveness in SATD tasks.

In this study, we employ two datasets: Maldonado-62k and OBrien. The usage of these datasets is as follows: for RQ1, RQ2, and RQ3, we utilize both datasets. For RQ4, we only use the OBrien dataset, because, unlike the Maldonado-62k dataset, which only provides the comment text for each entry, the OBrien dataset includes additional features such as the file path that can be utilized by LLMs to achieve better performance. More details for these two datasets are provided below.

For RQ1 and RQ4, we consider the following models:

• •

  BERT-base-uncased (110 million parameters) (Devlin et al.,, 2019): This model has demonstrated superior performance in the SATD identification task (Prenner and Robbes,, 2022), establishing a strong baseline for other LLMs in SATD identification and classification tasks.

• •

  CodeBERT (125 million parameters) (Feng et al.,, 2020): Differing from BERT, which primarily focuses on natural language, CodeBERT is a bimodal extension that utilizes both natural language and source codes. This makes it particularly suited for processing code comment SATD.

• •

  Flan-T5 (Chung et al.,, 2022): This is an enhanced version of T5 (Raffel et al.,, 2019), fine-tuned on over 1000 tasks. Available in various sizes, Flan-T5 allows for an exploration of the impact of model size on SATD tasks. Our infrastructure supports full fine-tuning of all Flan-T5 variants except the XXL version (11.1 billion parameters). Therefore, we will include Flan-T5 Small (77 million parameters), Base (248 million parameters), Large (783 million parameters), and XL (2.85 billion parameters) in our experiments.

For RQ2, we will utilize Flan-T5-XXL, the largest variant with 11.1 billion parameters. This model does not require fine-tuning for our purposes, as we will employ the ICL approach for the SATD identification and classification tasks. For RQ3, we evaluate the performance of the original Flan-T5 architecture against their altered version utilized in RQ1 and RQ4.

For the task of SATD identification, we employ four baseline methods: the NLP approach by (Maldonado et al.,, 2017), the MAT method (Guo et al.,, 2021), the technique introduced in the PENTACET study (Sridharan et al.,, 2023), and the convolutional neural networks (CNN) model proposed by (Ren et al.,, 2019). For the SATD classification task, we utilize two baselines: the NLP approach (Maldonado et al.,, 2017) and the CNN model (Ren et al.,, 2019).

Following the previous studies (Prenner and Robbes,, 2022; Ren et al.,, 2019; Maldonado et al.,, 2017), we report the F1 score for the SATD class in the SATD identification task. For the SATD classification task, we report overall performance using accuracy, and per class performance using the F1 score.

In this study, we propose an adaptation of the Flan-T5 architecture to address the classification task (see Figure 1). The original Flan-T5 model is inherently designed for text generation tasks, where the goal is to predict the probability distribution of subsequent words in a given text sequence. This is achieved through the model’s final layer, which is a text generation layer that outputs probabilities for each word in the vocabulary.

To tailor this architecture to classification objectives, we replace the text generation layer with a classification head, which consists of a dropout layer followed by a linear neural network layer with the size of number of classes. This new layer interprets the semantic representation of the input sequence from the last hidden state and produces a probability distribution over the predefined classes. For the loss function, we use cross-entropy which is a common choice for classification tasks. This modification leverages the pre-existing language understanding capabilities of the Flan-T5 model while aligning the output structure with the requirements of classification tasks. The updated architecture now effectively assigns input sequences to categories. We use this architecture in RQ1 and RQ4. For RQ2, we use the original architecture because for ICL approaches, there is no need to update the model parameters. In RQ3, we evaluate the effectiveness of the modified architecture against the original one.

We begin by downloading the official checkpoints for all evaluated models from Hugging Face. Subsequently, the Torch and Transformers packages are employed to conduct fine-tuning and ICL on a single Nvidia RTX 6000 GPU with 48 GB VRAM. In all models, we use the default 32-bit precision mode, and all experiments are done in Python 3.10.

For all baseline models, we employed their default settings. For selected LLMs, we adjusted the batch size to the highest power-of-two value that our VRAM could support. Additionally, since the learning rate significantly influences LLM performance, we conducted a few experiments to determine a near-optimal learning rate for each LLM. In the CNN and all LLMs, we configured the maximum length of input data for the Maldonado-62k dataset at 128 tokens. For the OBrien dataset, we set it at 512 tokens. The rationale for the higher token limit in the OBrien dataset is related to RQ4, where we explore the impact of incorporating contextual data into the input. This necessitates additional space to include this information for the models. All large language models are trained for eight epochs using the AdamW (Loshchilov and Hutter,, 2019) optimizer with a linear learning rate decay schedule. As there is no validation set to choose the best model across training epochs, we take the resulting model of the last epoch and report its performance on the test data. Table 3 presents the key parameter settings for each model applied to each dataset.

To train the selected large language models, we set the number of training epochs to eight. After training, we used the model from the final epoch to test its performance, because there was no validation set available to select the best-performing model. In this subsection, we present the performance of the trained model on the test set across epochs for RQ1.

Figure 2 illustrates the trend of the average F1 score across Maldonado-62k projects from epochs 1 to 8. The figure reveals that all six large language models start with a high performance (approximately 0.81 F1 score) in the first epoch, with subsequent epochs showing only minor fluctuations around this score. This consistency is likely due to the binary classification nature of the SATD identification task and the substantial volume of training data in the Maldonado-62k dataset, allowing these pre-trained models to reach near-optimal performance from the first epoch.

Figure 3 presents the results for the SATD classification task using the OBrien dataset. In contrast to Figure 2, where smaller models performed competitively with larger models, a more pronounced difference is observable in this figure. Additionally, the performance of all LLMs sharply increases from epoch 1 to epoch 5. Beyond epoch 5, the performance either stabilizes with minor fluctuations or gently continues to increase, suggesting that epoch 8 is an optimal point to stop training.