Automatic Bi-modal Question Title Generation for Stack Overflow with Prompt Learning

In this phase, we aim to construct the corpus by mining shared data dumps from Stack Overflow⁵⁵5https://archive.org/download/stackexchange, downloaded in March 2023. Due to a large number of posts in Stack Overflow, we refer to popularity statistics of tags⁶⁶6https://stackoverflow.com/tags?tab=popular, accessed in March 2023 and then determine the top-six popular programming languages (i.e., Python, Java, JavaScript, C#, PHP, and HTML). Moreover, we select two other programming languages (i.e., Go and Ruby), which can be used to evaluate the effectiveness of SOTitle+ in the low-resource scenario (discussed in Section 5.2).

Since there exist many low-quality posts in our initial gathered corpus, we consider three heuristic selection rules by following previous studies (Islam et al, 2019; Yazdaninia et al, 2021; Yin et al, 2018). For the convenience of introducing these selection rules, we use an example post⁷⁷7https://stackoverflow.com/questions/51560850 shown in Figure 4 to illustrate the main components of a Stack Overflow post. Specifically, this question post is related to the Python programming language, which contains a short question title, the problem description with the corresponding code snippet, the number of votes, and multiple tags. After introducing these components, we show the details of these three selection rules and their design motivations as follows.

• •

  Rule 1: The votes of the question post should not be smaller than 10. This rule is motivated by the previous study on Stack Overflow post analysis on deep learning bug characteristic analysis (Islam et al, 2019). They reduced the number of posts by considering the number of votes to focus on the high-quality posts. Through our preliminary empirical analysis, we found setting this value to 10 can help to ensure a certain level of post quality for the top six different popular programming languages.

• •

  Rule 2: The question post should contain code snippets. The reason for setting this rule is to verify the effectiveness of our approach by considering bi-modal information.

• •

  Rule 3: The question post should have an accepted answer. This rule is motivated by previous studies on mining Stack Overflow (Yazdaninia et al, 2021; Yin et al, 2018). We used the existence of an accepted answer as a criterion to potentially indicate question post quality, assuming that such questions might have clearer question titles.

After applying these three selection rules, we select 1.62% of posts (i.e., 179,119 posts from 11,023,365 posts). Then we extract the related programming language, the question title, the problem description, and the code snippet as the quadruplet $\langle$Lang, Title, Description, Code$\rangle$ for each question post. Notice as question posts were stored as a unified HTML format in the data dump, we use the Python Library lxml⁸⁸8https://lxml.de/ to extract the text wrapped by the “$<$code$>$” tag as the code snippet and the remaining text as the problem description.

In this phase, we aim to train the model for question title generation. Specifically, we first formalize question title generation for different programming languages as separate but related tasks and utilize multi-task learning (Liu et al, 2019b; Yang et al, 2023b) to solve these tasks. Then we consider the bi-modal information (i.e., problem description and code snippet), which is distinguished by hard prompts (e.g., “The problem description” and “The code snippet”). Meanwhile, we create soft prompts by replacing unimportant tokens with [SOFT] tokens. Based on the hybrid prompts (i.e., a mixture of the hard prompts and the soft prompts), our model is prompt-tuned based on a pre-trained Transformer model CodeT5 (Wang et al, 2021).

In this phase, for the problem posts that require title generation, we extract the problem description and the code snippet from the post body. Subsequently, based on the tuned hybrid prompt, we can construct the input and feed it into the PLM CodeT5 to obtain the generated question title.

In our empirical study, we want to answer the following five research questions (RQs):

RQ1: Can our proposed approach SOTitle+ generate higher-quality question titles than four state-of-the-art baselines via automatic evaluation?

Motivation. To show the competitiveness of SOTitle+ when compared to the baselines, we consider four performance measures to compare the similarity between the generated question title and the ground-truth question title in this RQ.

RQ2: Whether using the bi-modal information in the question body can improve the performance of SOTitle+?

Motivation. In this RQ, we want to conduct the ablation study to investigate which input modal contributes the most to the performance of SOTitle+.

RQ3: Whether using the prompt-tuning paradigm can improve the performance of SOTitle+ than using the fine-tuning paradigm?

Motivation. In this RQ, we want to conduct the ablation study to investigate whether using the prompt-tuning paradigm can achieve better performance of SOTitle+ than using the fine-tuning paradigm.

RQ4: Whether using hybrid prompts can improve the performance of SOTitle+?

Motivation. Prompt template design is a challenging and open problem in both natural language processing and software engineering tasks. In this RQ, we want to conduct the ablation study to show the competitiveness of using hybrid prompts in our proposed approach SOTitle+.

RQ5: Can our proposed approach SOTitle+ outperform state-of-the-art baselines via human study?

Motivation. There exist some disadvantages to automatic evaluation measures. For example, the text overlap-based measures used in our study may not effectively reflect the similarity between the generated question title and the ground-truth question title from a semantic perspective (Hu et al, 2020). Therefore, we want to conduct a human study to evaluate the competitiveness of SOTitle+ in this RQ.

After the corpus construction phase (introduced in Section 2.1), we finally select 43,056 posts for Python, 34,297 posts for Java, 37,900 posts for JavaScript, 34,846 posts for C#, 14,288 posts for HTML, and 14,732 posts for PHP. In the process of our corpus split, we consider the temporal relationship of posts. Specifically, splitting posts in chronological order is more applicable to the real-world application scenario and this setting can guarantee that the test set contains the latest posts. Therefore, this corpus split setting can relieve the data leakage problem caused by the homogeneous questions between the training set and the test set. Finally, we split the gathered corpus as the training set, the validation set, and the testing set according to 80%: 10%: 10% respectively. In Table 1, we show the detailed statistical information of our corpus split results. Finally, we also show the length statistics of code snippets, problem descriptions, and question titles in Table 2. Notice, some posts do not contain any problem description¹⁰¹⁰10https://stackoverflow.com/questions/1478248, so the minimum value of the description length is 0.

To show the competitiveness of SOTitle+, we consider four performance measures: Rouge-L (LIN, 2004), BLEU (Papineni et al, 2002), METEOR (Banerjee and Lavie, 2005), and CIDEr (Vedantam et al, 2015). Using these measures, we can measure the lexical similarity between the generated question title and the ground-truth question title. Moreover, these measures have been widely used in previous text generation tasks (Lin et al, 2023; Liu et al, 2019a; Li et al, 2022; Raffel et al, 2020; Gros et al, 2020; Yang et al, 2023a, 2022b; Li et al, 2021).

• •

  ROUGE-L. ROUGE (LIN, 2004) is a recall-based measure, which is used to calculate the lexical overlap between the generated sentence and the ground-truth sentence. In our study, we use ROUGE-L, which is based on the LCS (Longest Common Subsequence).

• •

  BLEU. BLEU (Papineni et al, 2002) is employed to assess the similarity degree between the generated sentence and the ground-truth sentence. Specifically, BLEU-$n$ calculates the number of $n$-grams that appear in the generated sentence.

• •

  METEOR. METEOR (Banerjee and Lavie, 2005) has been widely used to measure the quality of machine translation. It incorporates many aspects (e.g., vocabulary, grammar, and semantics) to evaluate the quality of the generated sentences.

• •

  CIDEr. CIDEr (Vedantam et al, 2015) is a measure that evaluates the quality of the generated sentences by analyzing the frequency of $n$-grams in the ground-truth sentences. To accomplish this goal, CIDEr employs TF-IDF weighting for each $n$-gram. Moreover, CIDEr calculates the $CIDEr_{n}$ score for each $n$-gram by averaging the cosine similarity between the generated sentence and the ground-truth sentence.

Notice the higher the value of these performance measures, the better the performance of the corresponding approach. In our study, we use the implementations provided by nlgeval¹¹¹¹11https://github.com/Maluuba/nlg-eval to calculate the value of these four performance measures.

For the Stack Overflow question title generation, we select four state-of-the-art approaches as our baselines to show the competitiveness of SOTitle+. We briefly introduce the characteristics of these baselines as follows.

• •

  Code2Que. Code2Que is the first study for Stack Overflow question title generation, which was proposed by  (Gao et al, 2020). Code2Que automatically generates question titles for Stack Overflow posts through an LSTM-based deep learning approach. The model’s encoder is a multi-layer, bidirectional LSTM network that processes the input code tokens sequentially. While the decoder is a single-layer LSTM that recursively generates predicted tokens.

• •

  SOTitle. To show the effectiveness of SOTitle+, we also consider our previously proposed approach SOTitle (Liu et al, 2022) as the baseline. SOTitle utilizes the pre-trained model T5 (Raffel et al, 2020), which follows a Transformer encoder-decoder architecture and employs multi-task learning to unify text-based language problems into a text-to-text paradigm.

• •

  CCBERT. CCBERT (Zhang et al, 2022) is another approach, which also considers bi-modal information. CCBERT considers the CodeBERT model and introduces customized copy attention layers on the encoder and decoder.

• •

  M3NSCT5. M3NSCT5 (Zhang et al, 2023) is a recently proposed approach that can improve the quality and diversity of the question titles generated from the code snippets. Zhang et al. fine-tuned the codeT5 and utilized the nucleus sampling and maximal marginal ranking strategy to make the model return a set of high-quality and diverse title candidates.

To alleviate the internal threats, we utilize the scripts shared by these four baselines and follow the hyperparameter settings used in their original studies. Since the baselines Code2Que and M3NSCT5 only consider code snippets as their inputs, we still use bi-modal information as their inputs to guarantee a fair comparison.

In our empirical study, we use Pytorch library¹²¹²12https://pytorch.org/ and transformers library¹³¹³13https://github.com/huggingface/transformers to construct our model. For prompt tuning, we use OpenPrompt library¹⁴¹⁴14https://github.com/thunlp/OpenPrompt. Specifically, by following the tutorial for OpenPrompt (Ding et al, 2022), we use the manual_template, SoftTemplate, MixedTemplate APIs to construct the hard prompt, the soft prompt, and the hybrid prompt templates, respectively. Based on the constructed prompt templates, we use PromptForGeneration API to perform prompt tuning. More details about our implementation can be found in our shared GitHub repository.

For the model hyperparameter setting, the word embedding dimension and hidden size are set to 768, and the number of attention heads and layers is set to 12. All parameters were optimized with AdamW (Loshchilov and Hutter, 2018), and the initial learning rate is set to 0.00005. During training, the batch size is set to 16. The maximum length of the encoder and decoder is set to 512 and 64 respectively. To implement heuristic search, we adopt beam search and set the beam size to 10. Finally, we also use the early stop method (Prechelt, 1998) to further alleviate the overfitting problem, and the weights with the highest performance on the validation set are taken as the final weights of the neural network.

We run all the experiments on a computer with an Intel(R) Core(TM) i5-13600K and a GeForce RTX4090 GPU with 24 GB memory. The running operation system platform is Windows Operation System. Based on our platform, we need about 24 hours to train the model by using SOTitle+.

The comparison results between SOTitle+ and baselines can be found in Table 3. In this table, we emphasize the best value in bold for each column. Based on the comparison results, we first find that our proposed approach SOTitle+ can outperform four baselines in terms of the performance measure Rouge-L for different programming languages. In particular, for our considered baselines, we find fine-tuning-based baselines (i.e., CCBERT (Zhang et al, 2022), SOTitle (Liu et al, 2022) and M3NSCT5 (Zhang et al, 2023)) can achieve better performance than Code2Que (Gao et al, 2020), which trains an LSTM model from scratch. The potential reason is that these three baselines are based on PLMs, which have been trained on the large-scale corpus. Moreover, CCBERT and SOTitle achieve considerable performance, which is better than M3NSCT5. We attribute the good performance of CCBERT and SOTitle to the reason that these two baselines were specifically designed to process bi-modal information. Finally, our approach can outperform the fine-tuning-based baselines. Specifically, compared to the baseline CCBERT, SOTitle+ can improve the performance of ROUGE-L by 18.04%, 14.65%, 9.41%, 13.43%, 20.19% and 7.98% for Python, Java, C#, JavaScript, PHP, and HTML respectively. Moreover, we also perform the Wilcoxon signed-rank test (Wilcoxon, 1992) at the confidence level of 95% to check whether the performance differences between SOTitle+ and baselines are significant. All the $p$-values are smaller than 0.05, which means SOTitle+ can significantly outperform the baselines.

Except for Rouge-L performance measure, we also compare SOTitle+ with baselines in terms of METEOR (Banerjee and Lavie, 2005), BLEU (Papineni et al, 2002) and CIDEr (Vedantam et al, 2015) performance measures. Final comparison results and the Wilcoxon signed-rank test results also show the competitive performance of SOTitle+ (i.e., the performance improvement is statistically significant).

Figure 5 shows the question titles generated by SOTitle+ and baselines for a question post related to Python programming language¹⁵¹⁵15https://stackoverflow.com/questions/66287470. In this example, we can find that the question titles generated by Code2Que and M3NSCT5 fail to capture the developer’s intent, and they mislocate key information to the “HTTP request”. The question titles generated by CCBERT and SOTitle lose the keywords “default” and “Asyncio”, respectively. However, the question title generated by SOTitle+ can accurately and fluently express the key information in this post.

In this RQ, we aim to investigate the contribution of different modal information to the performance of SOTitle+. Here we use different subscripts to distinguish different control approaches and introduce the meaning of these subscripts as follows.

• •

  code. The corresponding control approach only uses the code snippet along with its hard prompt as the input modality. We use $w/o\ desc$ to denote this control approach.

• •

  desc. The corresponding control approach only uses the problem description along with its hard prompt as the input modality. We use $w/o\ code$ to denote this control approach.

We show the performance of SOTitle+ with different modal information in Table 4. When only focusing on code snippets ($w/o\ desc$), the performance of SOTitle+ causes a significant decrease. In particular, in terms of Rouge-L, the average performance of SOTitle+ drops by 51.85%. On the other hand, when only focusing on problem descriptions ($w/o\ code$), there is a slight decrease in the performance of SOTitle+. These findings show that problem descriptions can provide more assistance in improving the performance of SOTitle+ than code snippets. Moreover, using both problem descriptions and code snippets simultaneously can help to achieve the best performance due to the complementarity between these two different modal information.

In this RQ, we want to investigate how much can prompt tuning contribute to the performance of SOTitle+. We design a control approach SOTitle+${}_{ft}$, which is fine-tuned on CodeT5. For SOTitle+${}_{ft}$, we remove all the prompts and use a special identifier ($<code>$) to distinguish problem descriptions and code snippets. The final input $X$ is defined as follows:

where $X_{desc}$ denotes the problem description and $x_{code}$ denotes the code snippet.

We show the comparison results between SOTitle+${}_{ft}$ and SOTitle+ in Table 5. In this table, We find that using the prompt-tuning paradigm can outperform using the fine-tuning paradigm for all programming languages. For example, for the Python programming language, using prompt tuning can improve the performance by 14.13%, 17.95%, 32.00%, and 24.47% in terms of ROUGE-L, METEOR, BLEU-4, and CIDEr respectively. The performance improvement is particularly obvious in the low-resource scenario (i.e., programming languages with fewer question posts). For example, for the PHP programming language, prompt tuning shows the highest performance improvement, with increase rate at 13.89%, 19.04%, 47.94%, and 27.97% in terms of Rouge-L, METEOR, BLEU-4, and CIDEr, respectively. Finally, we also show examples on our GitHub repository¹⁶¹⁶16https://github.com/shaoyuyoung/SOTitlePlus/blob/main/Appendix.md that can demonstrate the prompt-tuning contributions to bridge the gap between the fine-tuning paradigm and the prompt-tuning paradigm.

The prompt template design is a challenging and open problem. As (Liu et al, 2023d) reported even a single word difference in the hard prompt can make a huge performance difference. In this RQ, we want to investigate how much can our designed hybrid prompts contribute to SOTitle+. To answer this RQ, we design two control approaches that use only hard prompts or soft prompts, respectively. We show the details of them as follows:

• •

  Only Hard Prompts. This control approach only uses hard prompts as prompt templates. Specifically, we use Equation (1) as our hard prompt templates and use $w/$ $hard$ to denote this control approach.

• •

  Only Soft Prompts. This control approach only uses soft prompts as prompt templates. Specifically, we use Equation (2) as our soft prompt templates. We use $w/$ $soft$ to denote this control approach.

We show the performance of SOTitle+ with different types of prompt templates in Table 6. When only using hard prompts ($w/$ $hard$), the performance of SOTitle+ has a slight performance decrease. The potential reason is that considering soft prompts in the hybrid prompt may extend the ability of prompt templates. It is similar to only using soft prompts ($w/$ $soft$) for SOTitle+, which causes a slight performance decrease. The potential reason is that without original hard prompts (i.e., “The problem description:” and “The code snippet:”), which are the task-related important tokens, our proposed approach may lose the ability to effectively separate the problem description and the code snippet from the bi-modal information.

Since CCBERT (Zhang et al, 2022) and SOTitle (Liu et al, 2022) can achieve the best performance in our considered baselines (discussed in Section 4.1), we conduct a human study to assess the quality of the question titles generated by SOTitle+, CCBERT, and SOTitle respectively, To evaluate the quality of the generated question titles, we followed a human study methodology used by the previous studies (Wei et al, 2020; Liu et al, 2019a) for similar tasks. In our human study, we consider three quality criteria (i.e., similarity, naturalness, and informativeness) and the score values range from 1 to 4, where a higher score indicates better quality of the generated question titles. We show the meaning of these three criteria as follows:

• •

  Similarity. This criterion assesses the similarity between the generated question title and the ground-truth title.

• •

  Naturalness. This criterion evaluates the grammaticality and fluency of the generated question title.

• •

  Informativeness. This criterion quantifies the amount of content conveyed by the generated question title. Notice this criterion is irrespective of its naturalness.

We recruit six developers, including one Ph.D. student, and five Master students who are familiar with the usage of Stack Overflow and have at least three years of software development experience. They are not co-authors of this paper, which can avoid the bias of the human study results.

We randomly selected 100 posts for each programming language in the testing set, resulting in a total of 600 posts. For each post, we collected three question titles generated by SOTitle+, SOTitle, and CCBERT, respectively. We split posts for six programming languages into three groups (i.e., Python and Java, C#, and PHP, HTML, and JavaScript). Each group of posts is assigned to two different developers. Later, each developer manually evaluated 200 posts by considering the aforementioned three criteria. Notice the participants do not know which question titles are generated by SOTitle+. Since the evaluation process is labor-intensive, we limit each developer to only evaluating 100 posts within a half day. Finally, the evaluation scores provided by the two developers for each post are averaged. In our human study, to improve the score quality, the recruited developers are allowed to utilize the Internet to explore unfamiliar concepts related to the posts.

The evaluation results are shown in Figure 6. Regarding the similarity criterion, SOTitle+ outperforms SOTitle and CCBERT in terms of generating question titles with higher similarity scores, with an average value of 2.95. This indicates that the question titles generated by SOTitle+ are of good quality and can be used with minimal modifications. As for the naturalness criterion, all of these approaches achieve similar performance as expected. In terms of the informativeness criterion, SOTitle+ demonstrates the ability to generate more comprehensive question titles compared to SOTitle and CCBERT. To measure the consistency of the scoring results among different developers, we use Cohen’s Kappa (Cohen, 1960). The final Kappa value was 0.83, which indicates a substantial agreement among the developers. In addition to these scores, we also provided examples, of which SOTitle+ outperforms or does not outperform these two baselines, in our GitHub repository.

In this subsection, to investigate the advantage of using multi-task learning in SOTitle+, we design $w/o\ multitask$ as the control approach. In this control approach, we train separate models for each programming language. The comparison results can be found in Table 7. In this table, we find that by employing multi-task learning, SOTitle+ can outperform the models trained individually for each programming language. For example, SOTitle+ achieves a performance improvement of 17.06% at most in terms of BLEU_4 when compared to $w/o\ multitask$ for six programming languages. Notice that using multi-task learning can achieve higher performance improvement for low-resource programming languages (such as HTML and PHP). This advantage can be attributed to the limited training data available for these low-resource languages, which can hinder the control approach $w/o\ multitask$ from effectively learning helpful knowledge. However, by utilizing multi-task learning, SOTitle+ can effectively acquire related knowledge from programming languages with sufficient training data.

In Stack Overflow, there are some programming languages whose training data is limited and we call this scenario the low-resource scenario. In this subsection, we want to investigate the performance of SOTitle+ in this low-resource scenario. Specifically, we select other two low-resource programming languages (i.e., Ruby and Go). We use the same rules (introduced in Section 2) for corresponding corpus construction and corpus split. The statistical information of these two corpora is shown in Table 8.

In this experiment, we only consider two baselines (i.e., CCBERT (Zhang et al, 2022) and SOTitle (Liu et al, 2022)) due to their higher performance (discussed in Section 4.1). For SOTitle+, we directly use the model trained for answering RQ1 since we want to investigate the generalization of SOTitle+ on new programming languages. For the two baselines, we train SOTitle and CCBERT on corpus related to Rbuy and Go. We show the comparison results in Table 9. For the Ruby programming language, SOTitle+ can achieve 29.98% and 16.69% in terms of Rouge-L and BLEU-4, respectively. Similarly, for the Go programming language, SOTitle+ can achieve 31.22% and 15.02% in terms of Rouge-L and BLEU-4, respectively. Taking the Go programming language as an example, SOTitle+ demonstrates the performance improvements of 13.98%, 16.97%, and 31.87% in terms of the ROUGE-L, METEOR, and BLEU-4 measures, respectively when compared to SOTitle, which can achieve the best performance among these two baselines. These results verify the effectiveness of SOTitle+ for posts related to low-resource programming languages.

Recently, LLMs (Brown et al, 2020) (e.g., ChatGPT) have attracted great attention and achieved promising performance for many software engineering tasks, such as program repair (Xia and Zhang, 2023) and code generation (Liu et al, 2023a). In this subsection, we want to investigate the effectiveness of using ChatGPT in the SOQTG task and whether SOTitle+ can outperform ChatGPT on the posts related to Go and Ruby. Specifically, for ChatGPT, we use OpenAI official API¹⁷¹⁷17https://platform.openai.com/docs/api-reference. The version of ChatGPT we use is GPT-3.5-turbo. and the hard prompt introduced in Section 2.2.1 since ChatGPT does not support hybrid prompt. For SOTitle+, we directly use the model trained for answering RQ1. Notice both SOTitle+ and ChatGPT are not trained on the corpus related to Go and Ruby. Therefore, they are compared in the zero-shot scenario.

Table 10 shows the performance comparison results of ChatGPT and SOTitle+. In this table, we can find SOTitle+ can achieve better performance than ChatGPT in terms of all the performance measures. After our manual analysis, we found that ChatGPT tends to generate semantically rich but lengthy question titles. We use Figure 7 to show the length of question titles generated by ChatGPT and ground-truth question titles. Therefore, a potential solution is to design customized prompts, which can crop the rich but lengthy generated titles, in the future.

By following the study of Code2Que (Gao et al, 2021), we conduct a user study to investigate whether SOTitle+ can generate better question titles for low-quality questions. Specifically, we sampled 20 low-quality questions for each programming language. Notice the votes for these questions are less than 10 and these questions are not contained in our corpus. For each question, we gather the title written by users (i.e., ground-truth title) and the title generated by SOTitle+. Then we recruited the same six developers, who were also recruited in our human study, to determine which title was better in terms of three criteria (i.e., Clearness, Fitness, and Willingness). The detailed information of these six developers can be found in Section 4.5. Specifically, Clearness evaluates the clarity of a question title. Fitness measures whether a question title is reasonable in logic with the provided code snippet and the problem description. Willingness measures whether a user is willing to respond to this question.

We show the comparison results (i.e., Win, Loss, and Tie) for two programming languages in Table 11 and other programming languages in our GitHub repository. In this table, we find that titles generated by SOTitle+ can improve the title quality in terms of all three criteria in most cases. Moreover, we find that the titles generated by SOTitle+ can effectively increase the willingness of users to answer questions. Finally, we find that not all the titles of the low-quality questions can be improved. One of the reasons is that the post itself did not provide enough information. In summary, our preliminary user study can show the potential of SOTitle+ in title quality improvement.

Although SOTitle+ achieves promising performance via both automatic evaluation and human study, it should be noted that for some cases, SOTitle+ may generate question titles that show low similarity to the ground-truth question titles. To gain deeper insights into this issue, we randomly selected 100 posts and performed a manual analysis. This analysis aims to identify the specific types of posts that pose challenges for SOTitle+ and help to point out potential future studies.

The first limitation arises from the possibility of missing certain question words (e.g., how to, what, why) in the generated question titles. For example, Given this question post¹⁸¹⁸18https://stackoverflow.com/questions/51523765, the ground-truth title is “How to use OpenCV ConnectedComponents to get the images” while the title generated by SOTitle+ is “OpenCV ConnectedComponents in Python”. It appears that the model fails to capture the intention of this question post. One potential solution is to automatically identify the type of posts and provide this information for title generation.

The second challenge type is that the generated title and the ground truth convey the same meaning but are expressed using different ways. For example, given this question post¹⁹¹⁹19https://stackoverflow.com/questions/51579215, the ground-truth title is “Remove seaborn lineplot legend title”, the title generated by SOTitle+ is “Remove title from seaborn lineplot legend”. The sentence-level ROUGE-L score between these two titles is 26.56%. However, these two titles are semantically equivalent. This type of challenge can lead to lower scores in automatic evaluation metrics. One possible solution is to develop semantic-based evaluation metrics that can more accurately assess the semantic similarity between the generated title and the ground-truth title.

The third challenge type is that the problem description may contain noise and then mislead the model to generate the wrong question title. Given this question post²⁰²⁰20 https://stackoverflow.com/questions/59391560 , the ground-truth title is “How to run UVICORN in Heroku?”, while the title generated by SOTitle+ is “How to run a fastapi script in Heroku?”. After our manual analysis, we find that the keyword “fastapi” appears four times in the problem description and the keyword “UVICORN” appears only one time. However, in this post, the developer wants to ask how to use a command (i.e., “UVICORN”) but not a script (i.e., “fastapi”). One possible solution is to design an effective approach to identify the developer’s intent and use this information to guide title generation.

In this section, we mainly analyze the potential threats to the validity of our empirical study.

Ensuring the quality of question posts is essential for maintaining the usefulness of Q&A websites (such as Stack Overflow) (Yang et al, 2014; Ponzanelli et al, 2014; Duijn et al, 2015; Arora et al, 2015). Recently, researchers have focused on question title generation for Stack Overflow, as high-quality titles play a significant role in attracting potential experts to provide immediate answers. (Gao et al, 2020) were the first to study this problem and proposed Code2Que, which utilizes the LSTM network with the copy and coverage mechanisms to generate question titles. Similar to our previous study (Liu et al, 2022), (Zhang et al, 2022) found that considering both problem description and code snippet could significantly improve the quality of generated titles. Notice this study was conducted simultaneously with our previous study. But compared to our previous study, their study adopts a different architecture, which considers the CodeBERT model and introduces customized copy attention layers on the encoder and decoder. In addition, they simply merged the problem description and code snippet without distinguishing them with special identifiers explicitly in bi-modal construction. Later, (Zhang et al, 2023) further proposed the approach M3NSCT5, which utilizes the nucleus sampling and maximal marginal ranking strategy. In M3NSCT5, they fine-tuned CodeT5 and showed the promising performance of their proposed approach. Recently, (Liu et al, 2023b) further proposed an automatic question title reformulation approach QETRA by mining modification logs from Stack Overflow. Specifically, QETRA can further polish the draft titles generated by previous question title generation approaches (Gao et al, 2020; Liu et al, 2022; Zhang et al, 2022).

Prompt-tuning is a rapidly evolving technology, which directly adapts the pre-trained language model to cloze-style prediction or sequence-to-sequence generation through some additional prompts. As a new paradigm, prompt learning has achieved promising performances on various software engineering tasks. For example, (Wang et al, 2022) conducted an empirical study on three code intelligence tasks. They compared prompt-tuning with fine-tuning and reported that prompt-tuning has better performance on both classification tasks and generation tasks. (Zhu et al, 2023) proposed a context-aware prompt-tuning approach and applied this approach to two method naming tasks (i.e., method name recommendation and method name consistency checking). Recently, (Li et al, 2023) considered various types of prompt templates and combined prompt ensembling and transfer learning to improve the performance of prediction of vulnerability characteristics.

As the example shown in Figure 1, two posts have the same code snippet, but if the problem description is different, it will result in different titles. Compared to the previous study (Gao et al, 2020), our study aims to generate question titles by considering both the code snippet and the problem description. Our ablation study confirms the effectiveness of considering problem description and finds the problem description contributes more to the performance of SOTitle+. Moreover, as discussed in Figure 2, the inconsistent inputs and optimization objectives between the pre-training task and our investigated task may make fine-tuning hard to fully explore the knowledge of the pre-trained model. Therefore, SOTitle+ further prompt-tune CodeT5 with hybrid prompts (i.e., a mixture of hard and soft prompts) when compared to our preliminary study SOTitle (Liu et al, 2022). The effectiveness of using prompt tuning and hybrid prompts is confirmed in our ablation studies. Finally, to alleviate the insufficient training data issue for low-resource programming languages and fully utilize the knowledge in the related tasks, we formalize question title generation for different programming languages as separate but related tasks. Then we utilize multi-task learning to solve these tasks. The effectiveness of using multi-task learning is also confirmed in our ablation study and the promising performance is achieved when applying SOTitle+ to other two low-resource programming languages (i.e., Ruby and Go).