HDLdebugger: Streamlining HDL debugging with Large Language Models

The first step targets to generate a set of high-quality modification functions. These functions are then employed to modify HDL codes provided by industry engineers, thereby generating a diverse collection of buggy code examples. To ensure that the modified codes exhibit a range of realistic and diverse error patterns, we construct the modification functions by leveraging the capabilities of LLMs and a comprehensive collection of industrial HDL documents. Specifically, as shown in Fig. 2, we first collect a set of HDL documents, including the HDL manual, expert notes, and user logs. Subsequently, we then carefully design a prompt that guides the LLMs to extract and summarize prevalent and critical error patterns in HDL, such as syntax misuse or logical errors. With these insights, we proceed to develop the modification functions. The distilled functions focus on simple operations such as adding, deleting, modifying, and adjusting segments with HDL scripts, while the unique set of rules governing HDL ensures that similar operations can result in vastly different errors recorded in HDL documents. These functions explicitly introduce errors into the original correct HDL codes that mirror those commonly encountered errors in industry, thereby creating an invaluable dataset to fine-tune LLMs.

In this step, we gather a broad range of accurate and high-quality HDL codes $C=\{c_{i}\}_{i=1}^{|C|}$ from experienced chip engineers. These codes, which have been utilized across various chip designs, are comprehensive to encompass a wide range of functional testing scenarios for chips. These various HDL codes serve as the seed code for error case construction. Specifically, by applying the previous modification functions to these correct HDL codes, we systematically introduce errors, thereby producing a set of buggy codes. Particularly, we can apply various modification functions to one HDL code, which allows us to generate multiple instances of buggy code. Next, we employ an HDL compiler to compile these intentionally buggy codes on their respective design, which inevitably results in compilation errors. Then, for each correct HDL code $c_{i}\in C$, we can collect one of its buggy code $b_{i}$ with associated error message $m_{i}$ as an instance $I_{i}=(b_{i},m_{i},c_{i})$ of training data. Normally, diagnosing and rectifying HDL errors require the expertise of chip engineers. Given that our instructions systematically brought the errors, we can employ reverse engineering to identify the faults and produce solutions directly. This bypasses the need for manual error diagnosis, streaming the process of creating a vast array of comprehensive datasets of error scripts, error messages, and corresponding solutions. This method ensures a rich diversity in the types of errors produced, which is critical for creating an extensive and effective training dataset $D_{c}=\{I_{j}\}_{j=1}^{|D_{c}|}$.

As shown in Fig. 3 (a), we first collect a comprehensive collection of instructional documents for this HDL, encompassing language specifications, error diagnostics, and troubleshooting techniques. Then, we meticulously curate the content of documents, distilling a dedicated error database $D_{e}$ tailored to this HDL. This error database contains detailed error descriptions $d_{j}$, underlying reasons $r_{j}$, and suggested remedial strategies $s_{j}$ for each error id $e_{j}$. We illustrate examples of error information in error database in Tab. 7 in the Appendix. Given an error message query $m_{i}$, we first parse $m_{i}$ to extract $n_{e,i}$ constituent error codes $\{e_{j}\}_{j=1}^{n_{e,i}}$. Subsequently, we retrieve the descriptions, reasons, and potential solutions for each identified error code from our error database. Then, the document RAG of the query $m_{i}$ can be assembled as $rag_{i}^{d}=\{(e_{j},d_{j},r_{j},s_{j})\}_{j=1}^{n_{e,i}}$, thereby hel** LLMs to understand the error message $m_{i}$.

As shown in Fig. 3 (c), in the code retrieval component, we maintain a code database $D_{c}=\{I_{i}\}_{i=1}^{|D_{c}|}$, where each code instance $I_{i}=(b_{i},m_{i},c_{i})$ consists of a buggy code $b_{i}$, its associated error messages $m_{i}$, and the correct code $c_{i}$. Given a query $q=(b,m)$ that includes a snippet of buggy code $b$ with its associated error message $e$, the aim of the code RAG is to identify a subset $D_{c}^{q}\subseteq D_{c}$ of the top-$k$ code instances that have the most similar to the buggy code $b$ in the query. In such a way, the buggy code $b$ has similar buggy patterns with the buggy codes in each instance $I_{i}=(b_{i},m_{i},c_{i})\in D^{q}_{c}$. Thus, the correct code $c_{i}$ for each buggy code $b_{i}$ can be provided to LLMs. As a result, LLMs can use the learned patterns from the top-$k$ similar instances to fix for the buggy code $b$ in the query. Specifically, we first introduce how to learn low-dimensional vectors for buggy codes and then then introduce a two-stage ranker that retrieves the top-$k$ buggy code instances for code query $q$.

One straightforward way is to feed buggy code and error messages to LLMs and let LLMs directly predict the correct codes. However, this approach is insufficient for LLMs to deeply comprehend the problem and provide accurate solutions. Recent research (Wei et al., 2022; Yao et al., 2023) suggests that when LLMs are prompted to produce a series of intermediate and explanatory thought before finally outputting the solution to the given task, the performance of LLMs can be significantly improved. It is because these reasoning thoughts improve the understanding of LLMs on the input tasks and thus generate more relevant and accurate outputs. Therefore, before fine-tuning the LLMs, we propose to generate high-quality thought for each training code instance.

Specifically, as illustrated in Fig. 4, we first design a precise and explicit prompt $p_{t}$ to clarify the thought generation task for LLMs. Following this, we input the thought generation prompt $p_{t}$, the buggy code $b_{i}$, its associated error message $m_{i}$, and the document RAG $rag_{i}^{d}$, and its correct version $c_{i}$ into the LLM with its inference mode $LLM_{1}$. This enables the LLM to generate a thought $t_{i}$ on how to repair the buggy code $b_{i}$ into the correct code $c_{i}$ as follows:

where $\circ$ means concatenate operator. We omit the general task requirements prompt for simplification. Empirically, we find that when LLMs generate the thought $t_{i}$ only once, the output might be irrelevant to the buggy code $b_{i}$ or incorrect due to the hallucination phenomenon and randomness of LLMs. Therefore, to guarantee the quality of the generated thought, we iteratively generate $L$ different thoughts $T_{i}=\{t_{i,j}\}_{j=1}^{L}$ for each buggy code $b_{i}$ by running $LLM_{1}$ $L$ times following Eq. (3). This is achieved by modifying temperature parameters and the details are described in appendix.

To assess the quality of each thought $t_{i,j}\in T_{i}$, we adopt a self-guidance strategy to select the highest quality thought from the thought set $T_{i}$ for the buggy code $b_{i}$. Specifically, we feed the buggy code $b_{i}$, its associated error message $m_{i}$, and the document RAG $rag_{i}^{d}$ to the LLM. A prompt $p_{c}$, ”Based on the analysis, the correct script is,” is appended to guide the LLM towards generating the predicted correct script, represented as:

After we obtain the output $\hat{c}_{i,j}$ for each thought $t_{i,j}$, we employ the edit distance metric (Navarro, 2001)to evaluate the similarity between the predicted correct code $\hat{c}_{i,j}$ and the ground-truth of corrected code $c_{i}$ as

Intuitively, if the distance $d_{i,j}$ between the predicted correct code $\hat{c}_{i,j}$ and the ground truth of corrected code $c_{i}$ is smaller, the thought $t_{i,j}$ is more helpful to LLMs to repair the buggy code $b_{i}$. Therefore, we select the thought with the smallest edit distances for each $b_{i}$, i.e., $t_{i}=\min_{t_{i,j}\in T_{i}}{d_{i,j}}$. Finally, we can obtain the training dataset $\mathcal{D}=\{(b_{i},m_{i},rag_{i}^{d},rag^{c}_{i},t_{i},c_{i})\}_{i=1}^{|D_{c}|}$. The details of the thought generation are illustrated in Alg. 2 in the Appendix.

After obtaining the final training dataset $\mathcal{D}=\{(b_{i},m_{i},rag_{i}^{d},rag^{c}_{i},t_{i},c_{i})\}_{i=1}^{|D_{c}|}$, we supervise fine-tune the LLMs based on buggy codes and retrieval-augmented generation in Sec. 2.2. Specifically, given each training instance $D_{i}=(b_{i},m_{i},rag_{i}^{d},rag^{c}_{i},t_{i},c_{i})\in\mathcal{D}$, we first feed thought generation prompt $p_{t}$, buggy code $b_{i}$, its error message $m_{i}$, document RAG $rag_{i}^{d}$, code RAG $rag^{c}_{i}$, to a target LLM, $LLM_{2}$ to generate the predicted thought $\hat{t}_{i}$ as follows:

Then, based on the predicted thought $\hat{p}_{t}$ and code correction prompt $p_{c}$, LLM $LLM_{2}$ predicts the correct code $\hat{c}_{i}$ as follows:

Following (Wang et al., 2022; Taori et al., 2023), we fine-tune the target LLM with its training mode $LLM_{2}$ by using the conventional next-token prediction objective and minimize the cross-entropy loss $\mathcal{L}_{\mathcal{D}}$:

where the context $C=p_{t}\circ p_{c}\circ b_{i}\circ m_{i}\circ rag_{i}^{d}\circ rag^{c}_{i}$ denotes the concatenate of the inputs for clarification $w_{i,j}\in t_{i}$ denotes the $j$-th word in $t_{i}$ and $w_{i,<j}$ denotes a set of words in $t_{i}$ before $w_{i,j}$, and $p_{LLM_{2}}(w_{i,j}|w_{i,<j})$ denote the probability of $w_{i,j}$. $\mathcal{L}_{{t}_{i}}$ and $\mathcal{L}_{{c}_{i}}$ denote the prediction cross-entropy loss on thought ground truth $t_{i}$ and correct code ground truth $c_{i}$ regarding the buggy code $b_{i}$, respectively.

We gather a diverse collection of HDL code files from Huawei. These files are meticulously curated, with each file being specifically utilized for distinct chip design scenarios, reflecting the varied and specialized demands of circuit design. Each HDL code file contains an extensive array of data, including variable assignments, detailed circuit designs, clocking information, function testing protocols, and various test items that are critical for the chip design process. Based on these HDL code files, we use the data generation in Sec. 2.1 to generate 92,143 distinct HDL training code instances. Each HDL training code instance consists of the buggy HDL code, error messages, and the correct HDL code. Specifically, in the experiments, we split the data into training and testing sets at a ratio of 8:2, respectively.

We compare our proposed HDLdebugger with 13 baselines in three types of code debugging approaches as follows:

• •

  Five large language models: We compare HDLdebugger against two profound and state-of-the-art LLMs available through API services, including ChatGPT (Brown et al., 2020) and GPT-4 (Achiam et al., 2023). Additionally, we compare HDLdebugger with three open-source LLMs: OpenChat (Wang et al., 2023a), Orca2 (Mitra et al., 2023), and Mistral (Jiang et al., 2023). These open-source models have shown performance on par with ChatGPT across various open LLM benchmarks.

• •

  Four Code Debugging and HDL Models: We compare HDLdebugger with two code debugging and hardware code generation models, i.e., Self-debug (Chen et al., 2023) and RTLfixer (Tsai et al., 2023). Self-debug (Chen et al., 2023) is one of the most classical methods of code debugging. RTLfixer (Tsai et al., 2023) is proposed to solve HDL debugging problems. VeriGen (Thakur et al., 2023) and RTLCoder (Liu et al., 2023b) are two LLMs targeting hardware language.

• •

  Four Code Language Models: We compare four SOTA pretrained code LLMs, i.e., Deepseek (Bi et al., 2024), Starcoder (Li et al., 2023), Stablecode (Rombach et al., 2022), and WizardCoder (Luo et al., 2023).

For baselines except for ChatGPT (Brown et al., 2020) and GPT-4 (Achiam et al., 2023)., we adopt three strategies, i.e., the raw model, the raw model with RAG, and the raw model with supervised fine-tuning (SFT).

• •

  Raw model: We only feed buggy code and error messages to the model and enable raw models to infer the correct code directly.

• •

  Raw Model with RAG: For buggy code, we feed the buggy code, its error message, and the document RAG and code RAG obtained in Sec. 2.2 to the raw model and enable the raw models to infer the correct code directly.

• •

  Raw Model with SFT: We take the buggy code and error message as inputs of LLMs and use the correct code as ground-truth to fine-tune the raw models.

Specifically, we selected CodeLlama-13b as our base model from the available code LLMs. The choice of 13b was driven by its optimal model size, which strikes a balance between training and deployment, taking into account both performance and cost factors.

For the overall debug system, we mainly evaluate its pass rate for correcting codes, relative code runtimes, and edit distance between correct code and buggy code. The calculation of these metrics is listed below.

• •

  Pass-Rate: Pass rate for executing code file corrected by each method is defined as $P=\frac{\sum_{i=1}^{n_{c}}\mathbb{S}(y_{i})}{n_{c}}$, where $y_{i}$ stands for the corrected code, $\mathbb{S}$ denotes for executing code successfully.

• •

  Run-Time: The relatively average compilation time for Huawei’s internal HDL compiler to execute all test code files. The Run-Time for results from our HDLdebugger is set as the base unit.

• •

  Edit-Distance: Edit-Distance calculates the minimum number of operations (insertion, deletion, substitution) required to transform one code snippet into the other.

Also, we evaluate our code search engine in Sec. 2.2 by the hit ratio, mean average precision, and mean reciprocal rank metrics, which are formulated as follows.

• •

  H@K: Hit ratio for top-$K$ recommendation results on $n_{t}$ code queries is formulated as $H@K=\frac{1}{n_{t}}\sum_{i=1}^{n_{t}}\frac{1}{K}\sum_{k=1}^{K}\mathbb{I}(y_{i,k},y_{i})$, where $y_{i,k}$ denotes the error type of the retrieved $k$-th buggy code for query code $b_{i}$, and the indicator function $\mathbb{I}(y_{i,k},y_{i})=1$ if $y_{i,k}=y_{i}$

• •

  MAP@K: Mean average precision (MAP) for top-$K$ results on $n_{t}$ queries is defined as $MAP@K=\frac{1}{n_{t}}\sum_{i=1}^{n_{t}}\frac{1}{K}\sum_{k=1}^{K}\frac{\mathbb{I}(y_{i,k},y_{i})\cdot n(b_{i,\leq k})}{k}$, where $n(b_{i,\leq k})$ denotes the number of recommendations in the first top-$k$ that has the same error label with query $b_{i}$.

• •

  MRR@K: Mean reciprocal rank (MRR) for top-$K$ results on $n_{t}$ queries is formulated as $MRR@K=\frac{1}{n_{t}}\sum_{i=1}^{n_{t}}\frac{1}{K}\sum_{k=1}^{K}\frac{\mathbb{I}(y_{i,k},y_{i})}{k}$.

In our study, we conducted a comparative evaluation of both general-purpose LLMs such as ChatGPT, GPT-4, OpenChat, Orca, Mistral, Deepseek, Starcoder, Stablecode, WizardCoder, and specialized code LMs including Self-debug, RTLfixer, VeriGen, RTLCoder, within the HDL debugging scenario. Due to privacy concerns, certain code specifications have been omitted for testing purposes when using ChatGPT or GPT4. To distinguish these versions, we will refer to them as ChatGPT${}^{*}$ and GPT4${}^{*}$. It is evident that our approach exhibits superior performance against all 13 state-of-the-art benchmarks, including GPT-4 and other domain-specific hardware-based language models.

In this study, we implement three distinct evaluation strategies to assess the efficacy of various methodologies: with retrieval-augmented generation (RAG), with supervised fine-tuning (SFT), and via a direct approach. Within the context of HDL debugging, our analysis reveals that SFT holds greater significance and applicability across all evaluated baselines, including VeriGen, RTLCoder, Deepseek, Starcoder, Stablecode, and WizardCoder. In the majority of scenarios, we note that methods enhanced through SFT consistently outperform those augmented with RAG by a substantial margin. Besides, our proposed method integrates both RAG and SFT strategies, achieving unparalleled performance, indicating that it is better to incorporate SFT and RAG for the HDL debugging task.

In addition to general and code-specific Language Models (LLMs), methodologies such as Self-debug and RTLfixer are specifically devised to address code debugging scenarios. While these approaches demonstrate improvements over other general and code LLMs, their effectiveness still falls short of being fully satisfactory. Our analysis extends to evaluating our dataset with LLMs exclusively trained on hardware languages, namely VeriGen and RTLCoder. Contrary to expectations, these specialized hardware language models do not outperform their general and code LLM counterparts in our HDL debugging context, suggesting the possibility of an inherent task domain generalization issue within HDL debugging scenarios. On the other hand, our approach consistently surpasses domain-specific solutions regardless of the varied prompt engineering techniques employed or the domain-specific data used for training, which underscores the effectiveness of our methodology.

Figure 6 depicts the impact of varying the number of retrieved code instances on model performance. We find that performance improves with each additional instance between 1 to 5, achieving the most significant gains. However, beyond five instances, gains plateau or even decrease, indicating diminishing returns. This suggests that while adding retrieved code instances enhances model performance up to a point, increasing instances beyond this threshold leads to inefficiency. Moreover, we observe a tendency for the language model to generate repetitive or redundant content derived from previous input as the number of retrieved code instances increases. This phenomenon underscores a critical area for future exploration.

We assess the impact of incorporating the retrieved code instances on the additional inference budget.  Figure 6 illustrates the normalized inference times to clearly highlight the incremental budget required. A value of $0$ indicates the absence of retrieved code instances. Our observations reveal that as the number of retrieved code instances increases, the corresponding inference time exhibits a slow, logarithmic increase rather than a linear one. This pattern underscores the efficiency of our approach, demonstrating that integrating retrieved code instances significantly enhances performance without proportionally increasing the inference overhead.

We explore the effects of varying the temperature settings and the pass-rate$@$k for different values of $k$, where $k$ represents the number of answers generated by the LLM, as discussed in (Chen et al., 2021). Typically, a lower temperature setting yields more deterministic outcomes, whereas higher temperatures result in more varied outputs.  Figure 7 demonstrates that as $k$ increases, so does overall performance. Specifically, at a lower temperature, such as $0.1$, outputs are more consistent, leading to a narrower performance range. Conversely, at higher temperatures, like $1.2$, outputs become more varied, enhancing the likelihood of generating correct answers as $k$ increases. Notably, for a pass-rate$@$5, the performance at a temperature of $1.2$ surpasses that at $0.7$, indicating that increased temperature settings can improve outcomes, particularly at higher values of $k$.