Cross-Modality Program Representation Learning for Electronic Design Automation with High-Level Synthesis

Abstract

In recent years, domain-specific accelerators (DSAs) have gained popularity for applications such as deep learning and autonomous driving. To facilitate DSA designs, programmers use high-level synthesis (HLS) to compile a high-level description written in C/C++ into a design with low-level hardware description languages that eventually synthesize DSAs on circuits. However, creating a high-quality HLS design still demands significant domain knowledge, particularly in microarchitecture decisions expressed as pragmas. Thus, it is desirable to automate such decisions with the help of machine learning for predicting the quality of HLS designs, requiring a deeper understanding of the program that consists of original code and pragmas. Naturally, these programs can be considered as sequence data. In addition, these programs can be compiled and converted into a control data flow graph (CDFG). But existing works either fail to leverage both modalities or combine the two in shallow or coarse ways. We propose ProgSG, a model that allows interaction between the source code sequence modality and the graph modality in a deep and fine-grained way. To alleviate the scarcity of labeled designs, a pre-training method is proposed based on a suite of compiler’s data flow analysis tasks. Experimental results show that ProgSG reduces the RMSE of design performance predictions by up to $22\%$, and identifies designs with an average of $1.10\times$ and $1.26\times$ (up to $8.17\times$ and $13.31\times$) performance improvement in design space exploration (DSE) task compared to HARP and AutoDSE, respectively.

I Introduction

Over the past decades, the need for specialized computing systems to accelerate specific applications has grown, leading to the emergence of domain-specific accelerators (DSAs) like application-specific integrated circuits (ASICs) and field-programmable gate arrays (FPGAs). Designing DSAs is challenging because it involves using hardware description languages (HDLs) at the register-transfer level (RTL) with Verilog and VHDL, which are mainly familiar to circuit designers. High-level synthesis (HLS) was introduced to address this by raising the level of abstraction to C/C++/OpenCL/SystemC, allowing designers to describe high-level behavioral representations of their designs. Despite this, HLS tools still require significant hardware design knowledge through synthesis directives in the form of pragmas, which specify computation parallelization, data caching, memory buffer partitioning, etc. These optimizations are typically done by hardware programmers and are beyond the reach of average software programmers. Our objective is to automate and accelerate the optimization of integrated circuit (IC) design, making it more accessible to software programmers.

There is a growing trend to apply machine learning to IC design automation (Huang et al., 2021a). For example, researchers have developed learning-based methods to predict the quality of HLS designs (Sohrabizadeh et al., 2022a, 2023), to explore the HLS design space intelligently for optimal resource allocation (Wu et al., 2022a), etc. These methods fundamentally rely on an informative representation of an input design for high-quality performance prediction. We, therefore, focus on the representation learning for IC designs defined with HLS C/C++ (in short, we call them HLS designs) which are annotated with compiler directives/pragmas. Specifically, we aim to design an encoder-decoder framework where the encoder provides powerful representations for the input HLS designs so that the designs’ quality can be predicted accurately.

One limitation of the existing representation learning methods for programs and HLS designs is that they usually restrict the model to only using either the source code or the compiler-derived representation, but not both. For example, previous works (Sohrabizadeh et al., 2022a, 2023; Wu et al., 2022b) compile the HLS code into LLVM intermediate representation, which is then further transformed into a graph representation before a graph neural network (GNN) is used to encode it. Meanwhile, (Wang et al., 2021; Kanade et al., 2020; Feng et al., 2020; Guo et al., 2020) directly apply a large language model (LLM) to the source code to obtain the representations that catch the semantics of general computer programs.

However, we argue that only utilizing either one of the modalities is not good enough to obtain a comprehensive program representation. One the one hand, the graph modality tends to ignore the semantic information in the source code which is helpful to understand a program’s behavior. For example, in CDFG, it is difficult for GNN to understand the functionality of a call site, particularly to ones such as standard libraries (e.g., glibc). What is worse, a statement such as “A[i][j] *= beta;” would be converted to a relatively large and complex subgraph in the CDFG making it difficult for the model to understand the semantic meaning. On the other hand, two source code programs with similar semantics and functionalities could have significantly different latency and communication requirements. This is where the lower-level control-flow structure of the programs can help. Therefore, a novel model that effectively utilize information from both modalities could be the key to generating powerful representations of HLS designs and general programs.

In this paper, we propose ProgSG (Program representation learning combining the source Sequence and the control data flow Graph) for a unified representation learning that leverages both the source code modality and an enriched CDFG graph modality, with pre-training performed on both modalities. To handle the interaction between source code and CDFG graph, we propose two innovative designs in the architecture: (1) An attention-summary architecture for coarse interaction between the two modalities; (2) A fine-grained node-to-token message passing mechanism to enable further collaboration between the two modalities. We also propose a novel pre-training method based on predicting node-node relationships for compiler analysis tasks which helps the GNN encoder to address the label scarcity issue. Experiment results show the proposed ProgSG achieves a state-of-art performance on design quality prediction and design space exploration.

II Preliminaries

II-A HLS Design and Optimization Pragmas

The goal of this paper is to train a model to effectively predicts the quality of the HLS design, which is a C/C++ program with inserted pragmas serving as design specification. The quality of a design is measured by its latency in cycle counts (perf), the utilization rate of block RAM (util-BRAM), digital signal processors (util-DSP), flip-flop (util-FF), and lookup-tables (util-LUT) (Sohrabizadeh et al., 2022b, a).

We specifically consider the optimization pragmas of the Merlin Compiler, an open-source tool widely used for HLS designs. The Merlin Compiler provides three types of optimization pragmas, namely PIPELINE, PARALLEL, and TILE to define the desired microarchitecture (Sohrabizadeh et al., 2022b). As shown in Code LABEL:code:mvt in the Appendix, these pragmas can be applied at the loop level and offer control over the type of pipelining, the parallelization factor, and the amount of data caching. Table I summarizes the parameter space of these pragmas. For a given program $P$, any change in the option of any of the pragmas results in a different design $D$ with a unique microarchitecture. For example, the “fg” option in pipelining refers to the case where all the inner loops are unrolled (parallelized with separate logic) and each parallel unit is pipelined. The “cg” option, on the other hand, results in coarse-grained processing elements (PEs) that are pipelined together. For example, it can create pipelined load-compute-store units. The PARALLEL and TILE pragmas take numeric values that determine the degree of parallelization and loop tiling, respectively.

TABLE I: Target pragmas with their options.

Pragma	Parameter Name	Parameter Space
PARALLEL	factor	integer
PIPELINE	mode	“cg”, “fg”, off
TILE	factor	integer

II-B Hierarchical Graph Representation of HLS Designs 

Figure 1: An Illustration of HARP control data flow graph. Compared with a normal CDFG, it has additional block nodes and three types of edges: intra-block edges, block-flow edges, and hierarchy-level edges.

We leverage HARP’s approach (Sohrabizadeh et al., 2023) to generate the hierarchical graph representation of an HLS design, which is an enriched CDFG with extra block nodes and their connections. Figure 1 depicts an illustration of a HARP graph. Specifically, given the source code $C=(c_{1},\ldots,c_{I})$ ($c_{i}$, $i=1,\ldots,I$ denotes the $i$-th token of the source code), it is first transformed into an LLVM (Lattner and Adve, 2004) intermediate representation (IR), and further converted into a CDFG. Then to insert hierarchical information into the graph, auxiliary nodes are added into the graph where each auxiliary node represents a distinct LLVM IR block. Each of these blocks is a sequence of instructions that has a single entry point and a single exit point. Each auxiliary node has three types of edges: the edges to all instruction and data nodes within that block (intra-block edges), the edges to the previous and next block (block flow edges), and the edges building connections based on the hierarchy level of the "for" loops in the C/C++ code (hierarchy level edges). HARP (Sohrabizadeh et al., 2023) shows that the hierarchical graph representation helps propagate the long-range dependency information in the graph, which helps it learn a better graph representation.

III Proposed Method: ProgSG

In this section, we first describe the overall encoder-decoder architecture of ProgSG. Then, we focus on our novel encoder with a graph summary augmented sequence representation, and a fine-grained node-to-token alignment for the unification of the two modalities. Finally, we introduce a novel pre-training framework for program graphs.

III-A Overall Architecture for Design Quality Prediction

Given a design $D$ with source code $C$ and HARP graph $G$, the overall model $f(D)=f(C,G)$ first encodes designs into a set of embeddings, and then generates predictions $\hat{\bm{y}}$ with a multilayer perceptron (MLP) based decoder. Figure 2 depicts the overall diagrams of our model. Let $\bm{y}$ indicate the ground-truth targets (i.e., perf, util-BRAM, util-DSP, util-FF, and util-LUT). Our objective is to minimize the loss function that measures the mean squared error (MSE) between $\bm{y}$ and $\hat{\bm{y}}$, i.e., $\mathcal{L}_{\mathrm{task}}=||\hat{\bm{y}}-\bm{y}||^{2}$.

Since one modality is the source code sequence, and the other is the HARP graph, it is natural to adopt a transformer model on $C$ and a GNN model on $G$, which produce token representations $\{\bm{h}_{j}\in\mathbb{R}^{d}|j\in\{1,\ldots,I\}\}$ via the transformer’s self-attention mechanism, and node representations $\{\bm{h}_{k}\in\mathbb{R}^{d}|k\in\{1,\ldots,|V|\}\}$ via the message passing mechanism, respectively. $d$ denotes the embedding dimension. The starting token $c_{1}$’s embedding is then taken as the source code summary, $\bm{h}_{\mathrm{src}}\in\mathbb{R}^{d}$, and a graph-level aggregation can be performed on the node embeddings serving as the graph summary, $\bm{h}_{\mathrm{graph}}\in\mathbb{R}^{d}$. The encoder outputs the concatenation of the two modalities summaries, $\mathrm{concat}(\bm{h}_{\mathrm{src}},\bm{h}_{\mathrm{graph}})$, and lets the MLP-based decoder generate predictions.

This model serves as the foundation of our architecture. However, it solely relies on the MLP-based decoder to manage the interaction between the two modalities. We denote this simplified version of our model as ProgSG-ca.

III-B ProgSG-si: Graph-Summary-Augmented Sequence Representation

One limitation of the ProgSG-ca encoder is the shallow and ineffective modeling of the interaction between $C$ and $G$. We propose a novel yet simple way to address the issue, by making the following observation: The transformer operates on the sequence of tokens $C=(c_{1},\ldots,c_{I})$ by enabling every token to pay attention to every other token. That is,

$$\bm{h}_{\mathrm{src}}=AGG(g_{\mathrm{att}}\big{(}\bm{h}^{(0)}_{c_{1}},\ldots,\bm{h}^{(0)}_{c_{I}}\big{)})$$ (1)

where $AGG$ can be any aggregation function, and $g_{\mathrm{att}}$ denotes the multi-layer self-attention encoder of a transformer model, capturing the interaction between pairwise source code tokens, $\bm{h}^{(0)}_{c_{j}}$ stands for the $j$-th token’s initial embedding, $\bm{h}_{\mathrm{src}}$ denotes the final program-level source code embedding.

Based on the above observation, we propose to insert the graph summary $\bm{h}_{\mathrm{graph}}$ to the beginning of the sequence, forming an augmented sequence representation $C^{\mathrm{(aug)}}=(\bm{h}_{\mathrm{graph}},c_{1},\ldots,c_{I})$ as input to the transformer. Overall,

$$\bm{h}_{\mathrm{src}}=AGG(g_{\mathrm{att}}\big{(}\bm{h}_{\mathrm{graph}},\bm{h}^{(0)}_{c_{1}},\ldots,\bm{h}^{(0)}_{c_{I}}\big{)}).$$ (2)

We name such an encoder as ProgSG-si (Summary Interaction), since it first performs GNN with $L_{1}$ layers on $G$ to obtain a summary, and let the expressive transformer of $L_{2}$ layers handle the pairwise attention between tokens and that summary embedding, which efficiently allows cross-modality interaction. In other words, the graph is treated as a derivative of the source code whose summary embedding is used to augment the source code sequence. During training, the gradients back-propagate through $\bm{h}_{\mathrm{graph}}$ to the GNN, updating both the GNN and the transformer.

Figure 2: The overall diagrams of ProgSG. “GNN”, “TF”, and “Dec” refer to Graph Neural Network Layer, Transformer Layer, and Decoder, respectively.

III-C Full Model ProgSG: Leveraging Fine-grained Node Token Interaction

While ProgSG-si enables interaction between the graph and tokens, the graph-level summary is too coarse for the model to fully exploit the information from both modalities. Intuitively, the information exchange between two modalities would be more effective if the interaction happens in node/token level. A straightforward way is to utilize a cross attention module to all node embeddings $\bm{h}_{v_{1}},\ldots,\bm{h}_{v_{|V|}}$ and token embeddings $\bm{h}_{c_{1}},\ldots,\bm{h}_{c_{I}}$. However, since there could be thousands of nodes and tokens for an HLS design, the computation overhead is too expensive. So a more efficient way to leverage fine-grained node token interactions is needed.

Recall that there are auxiliary nodes in the HARP graph that stand for the LLVM-IR blocks (see Sec II-B for more details). Meanwhile, the source code is segmented into multiple chunks so that the length of each chunk is within the input length limit of the transformer. Let $\bm{h}_{v_{a_{1}}},\ldots,\bm{h}_{v_{a_{N}}}$ denote the embeddings of auxiliary block nodes and let $\bm{h}_{c_{s_{1}}},\ldots,\bm{h}_{c_{s_{M}}}$ indicate the embeddings of the summary tokens in source code chunks. Since the auxiliary block nodes in the graph modality and the chunks of source codes provide an intermediate granularity between graph/program and node/token level, we propose to utilize them to conduct a hierarchical node/token interaction, which is illustrated in Figure 3. The information between two modalities are first exchanged between the block nodes and the summary tokens via the following cross-modality message passing mechanism inspired by message passing GNNs:

$$\begin{array}[]{rl}\bm{h}^{\prime}_{v_{a_{k}}}&=\bm{h}_{v_{a_{k}}}+\mathrm{MLP}_{2}\Big{(}\sum_{j}\alpha_{k,j}\mathrm{MLP}_{1}(\bm{h}_{c_{s_{j}}})\Big{)},\\ \bm{h}^{\prime}_{c_{s_{j}}}&=\bm{h}_{c_{s_{j}}}+\mathrm{MLP}_{4}\Big{(}\sum_{k}\alpha_{j,k}\mathrm{MLP}_{3}(\bm{h}_{v_{a_{k}}})\Big{)},\end{array}$$ (3)

where the attention coefficients are computed via a dot product attention with learnable weight matrices $\bm{W}_{1}\in\mathbb{R}^{d\times d}$ and $\bm{W}_{2}\in\mathbb{R}^{d\times d}$, $\alpha_{k,j}=\textrm{Softmax}\left(\frac{(\bm{W}_{1}\bm{h}_{v_{k}})^{\top}(\bm{W}_{2}\bm{h}_{c_{j}})}{\sqrt{d}}\right).$

Then, the exchanged information is propagated to each node and token via a GNN and a transformer layer, respectively. Specifically, for a node $v_{i}$ (token $c_{i}$) that is not a block node (summary token), let $\bm{h}^{\prime}_{v_{i}}=\bm{h}_{v_{i}}$ ($\bm{h}^{\prime}_{c_{i}}=\bm{h}_{c_{i}}$), the second step can be written as

$$\begin{array}[]{cc}&\bm{h}^{\prime\prime}_{v_{i}}=\bm{h}^{\prime}_{v_{i}}+\mathrm{MLP}_{6}\Big{(}\sum_{j}\alpha_{i,j}\mathrm{MLP}_{5}(\bm{h}^{\prime}_{v_{j}})\Big{)}\\ &\bm{h}^{\prime\prime}_{c_{1}},\ldots,\bm{h}^{\prime\prime}_{c_{I}}=Attention(\bm{h}^{\prime}_{c_{1}},\ldots,\bm{h}^{\prime}_{c_{I}})\end{array}$$ (4)

Such cross-modality interaction enables fine-grained interaction between the two modalities so that more informative embeddings for the final prediction task can be generated. As an additional benefit, the interaction step is significantly more efficient than the full cross-attention because the number of auxiliary nodes and summary tokens is usually small. To allow deep cross-modality interaction, we perform the above node-token message passing $L_{2}$ times where $L_{2}$ is the number of transformer layers, e.g., 6 for the pre-trained CodeT5 model used in our experiments. In each of the $L_{2}$ layers, ProgSG performs the self-attention encoder on $C^{(aug)}$, and executes GNN on $G$, followed by the node-token interaction.

Figure 3: Illustration of the node-token message passing mechanism. The cross-modality information is first exchanged via block nodes and block tokens. Then the information is propagated to normal nodes and tokens through the GNN and transformer layers, respectively.

III-D Pretraining GNNs for Graph Modality

Generating ground-truth targets with an HLS simulator is slow, resulting in a scarcity of labeled data. To mitigate this issue, we propose utilizing pre-training tasks. While there is extensive work on pre-training transformer models with code (Wang et al., 2021; Feng et al., 2020; Kanade et al., 2020), our focus is on pre-training GNNs for graph modality. Existing self-supervised tasks for GNNs are for general graphs instead of CDFG; thus, we propose employing data flow analyses as self-supervised learning tasks. Data flow analysis is fundamental to modern compiler technology (Cummins et al., 2021) and necessitates that GNNs extract crucial information from a program’s structure. Furthermore, these tasks can be effectively addressed by non-ML techniques, allowing us to easily obtain a substantial set of labeled data for pre-training.

In particular, we select four data analyses tasks: (1) reachability: if a node can be reached from another node, (2) dominators: if every control-flow path to an instruction node passes through another node, (3) data dependencies: if a variable is defined in an instruction and used in another instruction, and (4) liveness: if a variable is live-out of a statement $n$. More detailed definitions of these tasks can be found in (Cummins et al., 2021). These tasks cover a full range of forward and backward analyses, and control and data analyses. In addition, these tasks focus on predicting the relationship between two nodes in a CDFG. Such node-level tasks help the GNN to learn meaningful node embeddings, which is the foundation of generating good graph embeddings. Each task can be viewed as a binary classification problem. Given a pair of nodes $v_{i},v_{j}$ and a label $y_{ij}$ which is a binary label indicating if the nodes have a particular relationship, we employ the cross entropy loss for pre-training loss.

Normally after pre-training, we would directly fine-tune the pre-trained GNN for the downstream task. However, the pre-training dataset does not contain any pragma nodes, which is important for predicting the quality of the HLS design. Therefore, we propose to use the pre-trained node embeddings as guidance to train a new (target) GNN for the downstream task. Specifically, given a graph with pragma nodes, denoted as $G$, we would generate a corresponding graph without pragma nodes, designated as $G^{\prime}$. Then for a node $v$ that appears in both $G$ and $G^{\prime}$, we would compute its embedding in $G^{\prime}$ with the pre-trained GNN and compute its embedding in $G$ with the GNN to be trained. Then, we would maximize the cosine similarity between the two embeddings with the following loss $\mathcal{L}_{\mathrm{guide}}=1-\cos\langle g_{\mathrm{cont}}(\bm{h}_{v,G}),\bm{h}_{v,G^{\prime}}\rangle$ where $g_{\mathrm{cont}}$ is a continuous function (e.g., MLP, identity function). In this way, the target GNN would learn how to extract useful node-level information from the pre-trained GNN, which would in turn improve the quality of graph-level embeddings.

IV Experiments

Here we present the main experiment results. Additional experiments are provided in the Appendix.

IV-A Model Hyperparameters and Training Details

During training, we combine the proposed loss functions including $\mathcal{L}_{\mathrm{total}}=\mathcal{L}_{\mathrm{task}}+\gamma_{1}\mathcal{L}_{\mathrm{fineAlign}}+\gamma_{2}\mathcal{L}_{\mathrm{coarseAlign}}+\gamma_{3}\mathcal{L}_{\mathrm{guide}}$, where $\gamma$s are hyperparameters controlling the weight for the different loss terms. During inference, we apply the encoder-decoder architecture to obtain $\hat{\bm{Y}}$.

We set the maximum number of tokens to 64 for the tokenizer, and chunk each source code into multiple subsequences to handle the long input source code sequence. We leave the exploration using more advanced modeling for long sequences such as Xiao et al. (2023) as future work. Since the task is on the whole program level, for each subsequence, we use the final embedding of the initial token (“[cls]”) as the summary of each subsequence (for ProgSG-si and ProgSG, an additional MLP is applied to project $\bm{h}_{\mathrm{src}}=\bm{H}_{\mathrm{src}}[0:1]$ from dimension 1024 to 512), and aggregate all summaries into a final sequence-level embedding (denoted as $\bm{h}_{\mathrm{src}}$ in the main paper) which is fed into the decoder. For the two-modality models, the decoder receives the concatenation of $\bm{h}_{\mathrm{src}}$ and $\bm{h}_{\mathrm{CDFG}}$ as described in the main paper.

The decoder consists of 6 sequentially stacked layers that project the input to a scalar. If the model is of a single modality, the MLP decoder has hidden dimensions 512-256-128-64-32-16-1. If the model is of two modalities, the MLP decoder has hidden dimensions 1024-768-512-256-128-61-1. The above scheme is administered consistently to all the methods for a fair comparison. Since we have 5 target metrics to predict as mentioned in Section 4.1 of the main paper, we use 5 MLPs applied on the input embeddings to transform them into the final $\hat{\bm{y}}\in\mathbb{R}^{5}$. We use the Exponential Linear Unit (ELU) function Clevert et al. (2015).

Our framework is implemented with PyTorch, PyTorch Geometric, Transformers, etc. Training is performed on a server with NVIDIA Tesla V100 GPUs. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with the initial learning rate tuned for each model using a validation set. We perform training with $\gamma_{1}=\gamma_{2}=\gamma_{3}=1$ over 1000 epochs with the best model selected based on a validation set for final adaptation and testing.

For the pre-trained GNN, we use utilize GNN with 5 transformer convolutional layers (Shi et al., 2021) as encoders and a 2-layer MLP as the decoder for each data analysis task. We use a training set with 276,197 graphs. The $\beta$ in focal loss is set to 2. We employ a validation set with 500 graphs to select the best pre-trained GNN. The $\beta$ in focal loss is set to 2.

IV-B Dataset and Evaluation Protocol

For the purpose of this study, we assembled a database of medium-complexity kernels that function as fundamental building blocks for larger applications. We selected a total of 42 kernels from two well-known benchmark suites, namely, the MachSuite benchmark (Reagen et al., 2014) and the Polyhedral benchmark suite (Polybench) (Yuki and Pouchet, [n.d.]). The kernels in the database were chosen to have a broad range of computation intensities, including linear algebra operations on matrices and vectors (e.g., BLAS kernels), data mining kernels (e.g., correlation and covariance), stencil operations, encryption, and a dynamic programming application.

The database is a new version of datasets released in Bai et al. (2023), generated by the AMD/Xilinx HLS tool version 2021 to implement the design, with the AMD/Xilinx Alveo U200 as the target FPGA and a working frequency of 250MHz. For each kernel, we perform a random split with the training, validation, and testing ratio being 70:15:15. For each design point, we recorded the latency in terms of cycle counts, as well as the resource utilization for DSP, BRAM, LUT, and FF. These targets are normalized following the same procedure in Bai et al. (2023); Sohrabizadeh et al. (2022a). The statistics of the dataset are presented in Table II. The dataset will be available upon paper acceptance.

IV-C Model Setup and Hyperparameters

We follow (Sohrabizadeh et al., 2023) to generate the HARP graphs. We adopt $L_{1}=8$ layers of TransformerConv (Shi et al., 2021) with a jum** knowledge network (Xu et al., 2018) as the final node embedding aggregation method. The embedding dimension $d=512$. For the source code, we use CodeT5 (Wang et al., 2021) with $L_{2}=6$ layers to embed the source code. AutoDSE defines a variable for each pragma, as shown in Code LABEL:code:hls, that is a placeholder for the option of the pragma. Since the pragmas $\zeta$ must be reflected in the input source code, for each design, we add the pragma options to their respective variables, e.g., we change “__PARA__L0__” to “__PARA__L0=1”, “__PIPE__L2” to “__PIPE__L2=flatten”, etc. We set the maximum number of tokens to 64 for the tokenizer, and chunk each source code into multiple subsequences to handle the long input source code sequence. The summaries of all subsequences are aggregated into the final representation for the decoder. We report the full hyperparameters in the appendix.

TABLE II: Dataset statistics. “#D”, “#P”, “A#P”, “A#T”, “A#N”, “A#E”, and “A#MP” denote “# designs”, “# programs”, “avg # pragmas per design”, “avg # tokens per program”, “avg # nodes per program’s CDFG”, and “avg # edges per program’s CDFG”, respectively.

Dataset	#D	#P	A#P	A#T	A#N	A#E
Vitis 2021	10,868	40	8.1	1286.3	354.7	1246.4

TABLE III: Rooted mean sqaure error (RMSE) of different methods in predicting target values.

		perf	util-LUT	util-FF	util-DSP	util-BRAM	total
Single Modalities Model	Code2vec	1.0641	0.5462	0.3103	0.9989	0.1555	3.6150
	HARP	0.2671	0.1043	0.0565	0.1584	0.0611	0.6474
	CodeT5	0.2077	0.0985	0.0619	0.1881	0.0597	0.6159
Cross Modalities Model	GreaseLM	0.2033	0.0805	0.0499	0.1349	0.0459	0.5146
	ProgSG-ca	0.2181	0.1232	0.0532	0.1381	0.0334	0.5660
	ProgSG-si	0.1591	0.1630	0.0514	0.1558	0.0335	0.5628
	ProgSG	0.1481	0.0709	0.0406	0.1084	0.0242	0.3923

IV-D Performance Prediction Results

We compare the accuracy of performance prediction of ProgSG against three categories of baselines: (1) models of source code modality, Code2vec (Alon et al., 2019) and CodeT5 (Wang et al., 2021); (2) models of graph modality, HARP (Sohrabizadeh et al., 2023); and (3) models of both modalities, GreaseLM (Zhang et al., 2022). GreaseLM is designed for knowledge graph augmented question-task. It exchanges information from two modalities in the program/graph level, which is too coarse. We also include ProgSG-ca, which is a simple concatenation of the summary representations described (Section III-A), and ProgSG-si which combines the two modalities without fine-grained interaction (Section III-B).

Table III provides a detailed breakdown of the prediction accuracy across different target variables. Notably, our results consistently reveal that the cross-modality model outperforms the single-modality model in terms of rooted mean square error (RMSE). This finding strongly supports our argument for the benefits of integrating multiple modalities within our model architecture. Furthermore, the comparison between the error rates of ProgSG-si and ProgSG-ca highlights the effectiveness of our graph-summary-augmented sequence representation. Moreover, ProgSG surpasses ProgSG-ca, ProgSG-si, and GreaseLM. This outcome underscores the superiority of our fine-grained node token interaction module, enabling more accurate predictions across a diverse range of target variables. In summary, our experimental results validate the effectiveness of our novel cross-modality program encoder.

IV-E Design Space Exploration Results

In addition, we evaluate how our method performs in finding the best design of a given kernel, i.e., design space exploration. Following previous studies (Sohrabizadeh et al., 2022a, 2023), for each kernel we have each model to verify as many designs points as possible in an hour following a heuristic order. The design points with the top 10 predicted performance is recorded. Then we run an HLS simulation to get the ground-truth performance of the selected design points and compare them with the best design point found by running AutoDSE (Sohrabizadeh et al., 2022b) for 25 hours. We use the average speedup between design points found by the model and those found by AutoDSE as the metric to evaluate the performance of each model on DSE task. Figure 4(a) shows the average and geomean of the DSE performance of HARP, CodeT5, and ProgSG. ProgSG outperforms the two single-modality baselines, revealing that our cross-modality model is superior. In addition, CodeT5 is worse than HARP in the DSE task, though it has a smaller RMSE in the regression task. We think it is because running CodeT5 for inference is slower than running HARP, as the GNN is much smaller. As a result, the number of designs verified by CodeT5 is smaller. Meanwhile, although ProgSG also suffers from the slow inference, it still manages to find better design points due to its better prediction accuracy.

One way to handle the slow inference speed of CodeT5 and ProgSG is to do a two-level design space exploration. That is, HARP is first run for an hour to find 1,000 candidate designs with the best predicted performance, then the larger model (CodeT5 or ProgSG) is used to select the top 10 designs from them. This two-level approach can simultaneously utilize the efficiency of the GNN model and the effectiveness of larger cross-modality model. Figure 4(b) shows the DSE performance of this two-level approach. It is clear that the performance of CodeT5 and ProgSG are significantly improved, showing the advantage of the two-level design space exploration. However, CodeT5 still cannot outperform HARP, demonstrating that LLM itself might not be powerful enough for our task.

(a) Running each model for one hour.

(b) Running each model on 1K candidates returned by HARP.

Figure 4: Relative performance improvement of best design found by our model compared to running AutoDSE for twenty-five hours.

IV-F Effects of Pre-training

In addition to our main analysis, we conducted an ablation study to delve deeper into the impact of our Graph Neural Network (GNN) pre-training strategy on model prediction accuracy. The results, as presented in Table IV, provide insightful observations. Notably, while there is a slight decrease in the accuracy of performance prediction, the prediction accuracy for the other four targets shows a notable improvement ranging from 10% to 15%. The drop of accuracy in performance is because we train the model to predict multiple targets simultaneously, and the accuracy of one target might drop while the overall prediction effectiveness improves. Furthermore, when considering the overall prediction accuracy, we observe an improvement of 5.57%. This substantial boost reaffirms the efficacy of our pre-training approach in refining the model’s effectiveness across diverse prediction tasks.

TABLE IV: Effects of pre-training to the prediction RMSE of our model.

Targets	wo pretrain	with pretrain	relative impr.
perf	0.1387	0.1481	-6.8%
util-LUT	0.0830	0.0709	14.6%
util-FF	0.0461	0.0406	11.9%
util-DSP	0.1084	0.1022	9.97%
util-BRAM	0.0281	0.0242	13.9%
total	0.4163	0.3923	5.77%

IV-G Attention Visualization

To better understand if the transformer model learns to attend tokens that are relevant to HLS pragma configurations, we visualize the average attention scores of some of the pragma-related tokens in the Gemm-n kernel (shown in Code LABEL:code:hls) for the transformer before and after training for our regression task (illustrated in Figure 5). We can see that 11 out of 15 tokens have higher attention scores after fine-tuning. For the 4 tokens that have lower attention scores (i.e., “__PIPE__”, “ACCEL”, “TILE”, and “FACTOR”), we can see that they often appear simultaneously with other keywords such as “PIPELINE”, “__TILE__”, and “PARALLEL”, which makes them somewhat redundant. If we compute the summation of their attention scores with the attention scores of tokens that simultaneously appear with them (e.g., “__PIPE__” and “PIPELINE”), we find that the summed attention score increases after training. So the changes in attention score suggest that the transformer model does learn to attend to the pragma-related tokens, which are important to predicting the quality of an HLS pragma configuration, even though these tokens are not included in its pre-training stage.

⬇

void gemm_N(double m1[4096],double m2[4096],double prod[4096])

{

int i,j,k,k_col,i_col;

double mult;

#pragma ACCEL PIPELINE auto{__PIPE__L0}

#pragma ACCEL TILE FACTOR=auto{__TILE__L0}

#pragma ACCEL PARALLEL FACTOR=auto{__PARA__L0}

for (i = 0; i < 64; i++) {

#pragma ACCEL PIPELINE auto{__PIPE__L1}

#pragma ACCEL TILE FACTOR=auto{__TILE__L1}

#pragma ACCEL PARALLEL FACTOR=auto{__PARA__L1}

for (j = 0; j < 64; j++) {

i_col = i * 64;

double sum = (double )0;

#pragma ACCEL PARALLEL reduction=sum FACTOR=auto{__PARA__L2}

for (k = 0; k < 64; k++) {

k_col = k * 64;

mult = m1[i_col + k] * m2[k_col + j];

sum += mult;

}

prod[i_col + j] = sum;

}}}

Code 1: Code snippet of the Gemm-Ncubed kernel with its pragmas starting with “#pragma”.

Figure 5: Bar plots of the average attention scores of pragma-related tokens before (ProgSG-rand) and after (ProgSG) being fine-tuned.

IV-H Training with Multiple Versions of Data

In addition, HARP Sohrabizadeh et al. (2023) revealed that training the model with data obtained through multiple versions of HLS tools can improve the performance of the model. In their experiments, HARP is first trained with data of one version, then fine-tuned with data of another version. To investigate if the conclusion is true for ProgSG, we conduct a similar experiment with data of three different versions using HARP and ProgSG. Each model is first trained with data of the first version (HLS v18) for 1,000 epochs, then fine-tuned with the data of the second version (HLS v20) for 200 epochs, and finally fine-tuned with the data of third versions (HLS v21) for 400 epochs. Figure 6 illustrates the DSE performance of HARP and ProgSG trained with 1 version and 3 versions of data. It is clear that training ProgSG with multiple versions of data significantly increases its performance.

Figure 6: DSE results of HARP and ProgSG trained with 1 version (v21, denoted as 1V) and three versions (v18, v20, and v21, designated as 3V) of HLS tools.

(a) HARP embeddings

(b) CodeT5 embeddings

(c) ProgSG embeddings

Figure 7: Embedding visualizations with different methods for “Correlation” kernel using t-SNE. The color indicates the value of “perf” target.

IV-I Embedding Visualization

To gain further insight into why ProgSG outperforms CodeT5 and HARP, we visualize the embeddings of valid “correlation” kernel designs in Figure 7. The colors represent the ground-truth performance targets. All methods form distinctive clusters with similar performance within each cluster. However, HARP’s clusters are more crowded, likely due to the larger sizes of CodeT5 and ProgSG, which can better differentiate designs. Additionally, ProgSG’s clusters align more closely with performance targets, as evidenced by the closer proximity of the purple points in ProgSG’s embeddings compared to those in CodeT5’s embeddings. This suggests that ProgSG’s embeddings more accurately reflect design performance, thereby explaining its superior DSE and prediction results.

V Related Work

Machine Learning for Electronic Design Automation  Machine learning (ML) for electronic design automation (EDA) is a rising research area (Huang et al., 2021a) with applications at various stages of hardware design, such as design verification (Vasudevan et al., 2021; Xu et al., 2020; Liang et al., 2023), high-level synthesis (HLS) (Ustun et al., 2020; Sohrabizadeh et al., 2022a; Bai et al., 2022; Wu et al., 2022a; Sohrabizadeh et al., 2023; Fu et al., 2023), circuit design (Ren et al., 2020; Wang et al., 2022, 2020; Yang et al., 2022), etc. This work focuses on obtaining representations of HLS designs using information from both the source code and the CDFG graph for FPGA design quality regression. Many works depict the input design/circuit as graphs (Ustun et al., 2020; Ren et al., 2020; Sohrabizadeh et al., 2022a). Recently, large language models (LLMs) are used to directly generate EDA scripts (Liu et al., 2023a, b). However, their results show that LLMs can only generate a few lines of scripts without considering the quality of the design. This work is among the first to combine both the source code and the graph modalities.

Representation Learning for Programs  Based on the modality of data, current methods can be divided into source-code-based methods and data-structure-based methods. Source-code-based methods (Kanade et al., 2020; Feng et al., 2020; Wang et al., 2021; Svyatkovskiy et al., 2020) employ language models (Devlin et al., 2018; Radford et al., 2019; Raffel et al., 2020; Zheng et al., 2023; Li et al., 2023a; Gunasekar et al., 2023; Roziere et al., 2023; Fu et al., 2024) on source code to perform various types of tasks. However, it has not been demonstrated that these language models can predict the program’s runtime, let alone predicting the corresponding hardware design performance. The data-structure-based methods (Alon et al., 2019; Sohrabizadeh et al., 2022a, 2023) obtain the program embeddings from the data structure that represents a program. But the sizes of the models are usually small, restricting their prediction ability.

Multi-modal Learning with Transformers  Modality-wise, transformers have been employed in cross-modality tasks spanning across vision (Li et al., 2021; Wu et al., 2021; Huang et al., 2021b), language (Zhang et al., 2019; Lee et al., 2022), source code (Dai et al., 2022), knowledge graphs (Yasunaga et al., 2022; Rao et al., 2023), audio (Arandjelovic and Zisserman, 2017; Gan et al., 2020), point clouds (Afham et al., 2022), etc. In fact, multi-modal learning using transformers has recently been considered possible for achieving generalist artificial intelligence (Moor et al., 2023; Mai et al., 2023). More thorough surveys on graphs and transformers can be found at Liu et al. (2023c); Li et al. (2023b); ** et al. (2023).

GreaseLM (Zhang et al., 2022) combines GNN and transformer for knowledge-graph augmented QA task. It is similar to our task in that it also aims to combine the graph and the text modality. However, there are some differences in the targeted tasks that lead to distinct challenges and model design choices: (1) The differences in program structures are subtle. Two designs with the same functionality can have a huge performance gap due to slight differences in the program or pragmas. So the global-level interaction in GreaseLM is not effective enough. A more fine-grained interaction between the modalities is needed. (2) Efficiency is important to our task. As a larger inference overhead means slower design space exploration. Our program-derived graph is also much larger, with thousands of nodes and tokens. So the model needs to be efficient, forbidding the usage of full cross-attention between nodes and tokens. (3) The programs are inherently hierarchical, making it possible to do interactions at multiple levels. Given these unique challenges, we propose a novel model that interacts modalities at both global and block levels to maintain effectiveness while kee** it efficient. It is fair to say that we are the first to explore the cross-modality model for program representation learning.

Graph Neural Networks Pre-training   Existing self-supervised learning methods (Xie et al., 2023) can be divided into two categories: contrastive methods (Sun et al., 2019; You et al., 2020; Veličković et al., 2019; Qiu et al., 2020) and predictive methods (Xie et al., 2023; Kipf and Welling, 2016; Hu et al., 2020; Peng et al., 2020; Rong et al., 2020; Sun et al., 2020; Hu et al., 2021). To our knowledge, we are the first to explore pre-training GNNs with CDFGs.

VI Conclusion

We propose ProgSG, a novel two-modality program representation learning method for IC design (defined with HLS C/C++) optimization. The key assumption is that there is critical information in both the source code modality and the assembly code modality, which must be captured jointly. To achieve that, we propose a graph-summary-augmented sequence representation for the source code transformer, a fine-grained alignment utilization method, and a novel pre-training method for the GNN encoder for the CDFG. Experiments confirm the superiority of the proposed ProgSG over baselines. We believe the core idea of using both modalities together with their alignment is general and can be adapted for other tasks.

-A Additional Background in HLS Design

⬇

void kernel_mvt(double x1[120], double x2[120], double y_1[120], double y_2[120], double A[120][120]) {

int i, j;

#pragma ACCEL PIPELINE auto{__PIPE__L0}

#pragma ACCEL TILE FACTOR=auto{__TILE__L0}

#pragma ACCEL PARALLEL FACTOR=auto{__PARA__L0}

for (i = 0; i < 120; i++) {

#pragma ACCEL PARALLEL reduction = x1 FACTOR=auto{__PARA__L2}

for (j = 0; j < 120; j++) {

x1[i] += A[i][j] * y_1[j];

}}

#pragma ACCEL PIPELINE auto{__PIPE__L1}

#pragma ACCEL TILE FACTOR=auto{__TILE__L1}

#pragma ACCEL PARALLEL FACTOR=auto{__PARA__L1}

for (i = 0; i < 120; i++) {

#pragma ACCEL PARALLEL reduction = x2 FACTOR=auto{__PARA__L3}

for (j = 0; j < 120; j++) {

x2[i] += A[j][i] * y_2[j];

}}}

Code 2: Code snippet of the mvt kernel (Matrix Vector Product and Transpose) with its 8 pragmas starting with “#pragma”.

-B Extra Embedding Visualization

In Figure 8 we visualize the embeddings of different models for “symm-opt-medium” kernel, where ProgSG achieves more than eight times speed up compared to HARP. Similar to the embeddings of “Correlation” kernel, the embeddings of HARP are more crowded, suggesting weaker generalization ability. Moreover, comparing the embeddings of CodeT5 and ProgSG, we can see that the yellow point (which represents the design point with the best performance) in ProgSG’s embeddings are further away from other points than in CodeT5’s embeddings. It suggests ProgSG can better distinguish the good design points.

(a) HARP embeddings

(b) CodeT5 embeddings

(c) ProgSG embeddings

Figure 8: Embedding visualizations with different methods for “Symm-Opt-Medium kernel” using t-SNE. The color indicates the value of “perf” target.

-C Case Studies of Best Design Points in DSE experiments

In this section, we show the design points returned by AutoDSE (25 hours), HARP (1 hour), and ProgSG (1 hour) in the DSE experiments for some kernels (“Correlation”, “Symm-opt-medium”, “Gemver-medium”) where ProgSG outperforms HARP significantly. We show the source code of these kernels in Code LABEL:code:hls-cor, Code LABEL:code:hls-symm, and Code LABEL:code:hls-gemver. And we show the design points in Table V, Table VI, and Table VII.

For correlation kernel, the values of “__PARA__L5” in AutoDSE and ProgSG’s design points are much larger than the value in HARP’s design point. So the design point returned by HARP leads to sub-optimal data loading procedures, which takes up 96,000 cycles of the total latency. While the design point returned by ProgSG does not have this issue. For symm-opt-medium kernel, the design point returned by HARP has smaller parallelization factor than the design points returned by AutoDSE and ProgSG, leading to worse efficiency. For gemver-medium kernel, the parameter “__PARA__L4” is 64 in the design point returned by HARP, which can not divide 400, which is the total number of the for-loop. As the result, the loop takes up 120,000 cycles of the total latency, leading to worse performance. On the other hand, The “__PARA__L4” is 25 in the design point returned by ProgSG, which can divide 400, thus avoiding the problem.

⬇

void kernel_correlation(double float_n,double data[100][80],double corr[80][80],double mean[80],double stddev[80])

{

int i;

int j;

int k;

double eps = 0.1;

#pragma ACCEL PIPELINE auto{__PIPE__L0}

#pragma ACCEL TILE FACTOR=auto{__TILE__L0}

#pragma ACCEL PARALLEL FACTOR=auto{__PARA__L0}

for (j = 0; j < 80; j++) {

mean[j] = 0.0;

#pragma ACCEL PARALLEL FACTOR=auto{__PARA__L4}

for (i = 0; i < 100; i++) {

mean[j] += data[i][j];

}

mean[j] /= float_n;

}

#pragma ACCEL PIPELINE auto{__PIPE__L1}

#pragma ACCEL TILE FACTOR=auto{__TILE__L1}

#pragma ACCEL PARALLEL FACTOR=auto{__PARA__L1}

for (j = 0; j < 80; j++) {

stddev[j] = 0.0;

#pragma ACCEL PARALLEL FACTOR=auto{__PARA__L5}

for (i = 0; i < 100; i++) {

stddev[j] += pow(data[i][j] - mean[j],(double )2);

}

stddev[j] /= float_n;

stddev[j] = sqrt(stddev[j]);

stddev[j] = (stddev[j] <= eps?1.0 : stddev[j]);

}

#pragma ACCEL PIPELINE auto{__PIPE__L2}

#pragma ACCEL TILE FACTOR=auto{__TILE__L2}

#pragma ACCEL PARALLEL FACTOR=auto{__PARA__L2}

for (i = 0; i < 100; i++) {

#pragma ACCEL PARALLEL FACTOR=auto{__PARA__L6}

for (j = 0; j < 80; j++) {

data[i][j] -= mean[j];

data[i][j] /= sqrt(float_n) * stddev[j];

}

}

#pragma ACCEL PIPELINE auto{__PIPE__L3}

#pragma ACCEL TILE FACTOR=auto{__TILE__L3}

#pragma ACCEL PARALLEL FACTOR=auto{__PARA__L3}

for (i = 0; i < 80 - 1; i++) {

corr[i][i] = 1.0;

#pragma ACCEL PIPELINE auto{__PIPE__L7}

for (j = i + 1; j < 80; j++) {

corr[i][j] = 0.0;

#pragma ACCEL PARALLEL FACTOR=auto{__PARA__L7_0}

for (k = 0; k < 100; k++) {

corr[i][j] += data[k][i] * data[k][j];

}

corr[j][i] = corr[i][j];

}

}

corr[80 - 1][80 - 1] = 1.0;

}

Code 3: Code snippet of the Correlation kernel with its pragmas starting with “#pragma”.

	AutoDSE	HARP	ProgSG
__PARA__L0	1	1	1
__PARA__L1	1	1	1
__PARA__L2	1	1	1
__PARA__L3	1	1	1
__PARA__L4	1	5	4
__PARA__L5	32	5	25
__PARA__L6	1	10	4
__PARA__L7_0	1	1	1
__PIPE__L0	fg	off	off
__PIPE__L1	off	off	off
__PIPE__L2	off	off	off
__PIPE__L3	off	off	off
__PIPE__L7	fg	fg	fg
__TILE__L0	1	1	1
__TILE__L1	1	1	1
__TILE__L2	1	1	1
__TILE__L3	1	1	1
perf	60,237	165,135	61,287

TABLE V: Best design points returned by AutoDSE, HARP, and ProgSG on “Correlation” kernel.

⬇

void kernel_symm(double alpha,double beta,double C[200][240],double A[200][200],double B[200][240])

{

int i,j,k;

#pragma ACCEL PIPELINE auto{__PIPE__L0}

#pragma ACCEL TILE FACTOR=auto{__TILE__L0}

#pragma ACCEL PARALLEL FACTOR=auto{__PARA__L0}

for (i = 0; i < 200; i++) {

#pragma ACCEL PIPELINE auto{__PIPE__L1}

#pragma ACCEL TILE FACTOR=auto{__TILE__L1}

#pragma ACCEL PARALLEL FACTOR=auto{__PARA__L1}

for (j = 0; j < 240; j++) {

double tmp = B[i][j];

#pragma ACCEL PARALLEL reduction=C FACTOR=auto{__PARA__L2}

for (k = 0; k < 200; k++) {

if (k < i) {

C[k][j] += alpha * tmp * A[i][k];

}

}

double temp2 = (double )0;

#pragma ACCEL PARALLEL reduction=temp2 FACTOR=auto{__PARA__L3}

for (k = 0; k < 200; k++) {

if (k < i) {

temp2 += B[k][j] * A[i][k];

}

}

C[i][j] = beta * C[i][j] + alpha * B[i][j] * A[i][i] + alpha * temp2;

}

}

}

Code 4: Code snippet of the Symm-opt-medium kernel with its pragmas starting with “#pragma”.

	AutoDSE	HARP	ProgSG
__PARA__L0	1	1	1
__PARA__L1	1	1	1
__PARA__L2	25	25	25
__PARA__L3	200	32	200
__PIPE__L0	cg	off	cg
__PIPE__L1	off	cg	off
__TILE__L0	1	1	1
__TILE__L1	1	8	1
perf	4,345,927	35,536,546	4,345,927

TABLE VI: Best design points returned by AutoDSE, HARP, and ProgSG on “Symm-OPT-Medium” kernel.

⬇

void kernel_gemver(int n,double alpha,double beta,double A[400][400],double u1[400],double v1[400],double u2[400],double v2[400],double w[400],double x[400],double y[400],double z[400])

{

int i,j;

#pragma ACCEL PIPELINE auto{__PIPE__L0}

#pragma ACCEL TILE FACTOR=auto{__TILE__L0}

#pragma ACCEL PARALLEL FACTOR=auto{__PARA__L0}

for (i = 0; i < 400; i++) {

#pragma ACCEL PARALLEL reduction=A FACTOR=auto{__PARA__L4}

for (j = 0; j < 400; j++) {

A[i][j] += + u1[i] * v1[j] + u2[i] * v2[j];

}

}

#pragma ACCEL PIPELINE auto{__PIPE__L1}

#pragma ACCEL TILE FACTOR=auto{__TILE__L1}

#pragma ACCEL PARALLEL FACTOR=auto{__PARA__L1}

for (i = 0; i < 400; i++) {

#pragma ACCEL PARALLEL reduction=x FACTOR=auto{__PARA__L5}

for (j = 0; j < 400; j++) {

x[i] += beta * A[j][i] * y[j];

}

}

#pragma ACCEL PARALLEL FACTOR=auto{__PARA__L2}

for (i = 0; i < 400; i++) {

x[i] = x[i] + z[i];

}

#pragma ACCEL PIPELINE auto{__PIPE__L3}

#pragma ACCEL TILE FACTOR=auto{__TILE__L3}

#pragma ACCEL PARALLEL FACTOR=auto{__PARA__L3}

for (i = 0; i < 400; i++) {

#pragma ACCEL PARALLEL reduction=w FACTOR=auto{__PARA__L6}

for (j = 0; j < 400; j++) {

w[i] += alpha * A[i][j] * x[j];

}}}

Code 5: Code snippet of the Gemver-medium kernel with its pragmas starting with “#pragma”.

	AutoDSE	HARP	ProgSG
__PARA__L0	1	1	1
__PARA__L1	1	1	1
__PARA__L2	1	1	1
__PARA__L3	1	8	8
__PARA__L4	2	64	25
__PARA__L5	1	10	10
__PARA__L6	25	20	25
__PIPE__L0	off	off	off
__PIPE__L1	fg	off	cg
__PIPE__L3	off	cg	off
__TILE__L0	1	1	1
__TILE__L1	1	1	1
__TILE__L3	8	1	1
perf	210,335	265,686	167,270

TABLE VII: Best design points returned by AutoDSE, HARP, and ProgSG on “Gemver-medium” kernel.