Embedded Prompt Tuning: Towards Enhanced Calibration of Pretrained Models for Medical Images

Let $\sigma$ be the softmax operators. A bold lower case character, e.g. $\mathbf{x}$, denotes a vector. A bold upper case character, e.g. $\mathbf{W}$, denotes a matrix while $\mathbf{W}_{i,j}$, $\mathbf{W}_{i,{:}}$ and $\mathbf{W}_{:,j}$ is the $(i,j)$-th element, $i$-th row, $j$-th column, respectively. We use ${ReLU}(\mathbf{v})=\max(\mathbf{v},\mathbf{0})$ to denote the ReLU activation function where $\max(\cdot)$ function is applied entry wise to a vector.

Only the first layer of the Transformer is embedded with prompts, therefore all the Transformer layers of EPT-SHALLOW are represented as

All Transformer layers are embedded with prompts, and the Transformer layers of EPT-DEEP are represented as

For a matrix $\mathbf{K^{T}Q}$, we select the vector from the $i$-th column to illustrate. Let $\mathbf{u}=\mathbf{{K^{T}}Q}_{:,i}=(\mathbf{{K^{T}}Q})_{:,i}$, which corresponds to the multiplication of the query of the $i$-th patch and the keys of all patches. Then, the softmax value of $\mathbf{u}$ can be represented as

For the prompted matrix $\left[\begin{array}[]{c}\mathbf{P_{E}}\\ \mathbf{K^{\top}Q}\end{array}\right]$, let $\mathbf{u_{p}}=\left[\begin{array}[]{l}\mathbf{p}\\ \mathbf{u}\end{array}\right]\in\mathbb{R}^{{d_{p}}+{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}$ be the $i$-th column of the prompted matrix, where $\mathbf{p}\in\mathbb{R}^{d_{p}}$ and $\mathbf{u}\in\mathbb{R}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}$ represent the prompt and the original vector respectively. After the softmax operation, the value of the $i$-th column vector of the original matrix becomes

From equations (9) and (10), we can observe that the softmax operation applied to the prompt enables the adjustment of each patch-level feature, corresponding to patch-wise scaling. It is worth noting that the dimensions of the input and output remain unchanged during the softmax operation. The advantages of patch-wise scaling will be further analyzed and compared in Sec 3.3.2.

We evaluate on the MedFMC2023 grand challenge public classification dataset [23] which covers three different modalities: (1)Thoracic Disease Screening (ChestDR). 2,140 for training, 2,708 for validation, and 3,869 for testing. (2)Pathological Tumor Tissue Classification (Colon). 5,654 for training, 4,355 for validation and 7,651 for testing. (3)Endoscopy Images for Lesion Classification (Endo). 1,810 for training, 2,055 for validation and 2,936 for testing.

Each task is formulated as N-Way K-shot classification tasks. Due to the backbone network is pre-trained on ImageNet-21k, which owns a domain gap with medical images, the classification tasks are cross-domain few-shot classification tasks. The few-shot medical classification task, especially the multi-label classification task, particularly highlights the generalization performance of these PEFT methods. We mainly focus on few-shot scenerios rather than fully supervised training, and K is set to 1,5,10. 1-shot refers to every N images that sample from each class, rather than one image. Among the three datasets, ChestDR and Endo are multi-label classification tasks with 19 and 4 categories respectively, while Colon is a single-label classification task with 2 categories.

To evaluate the performance of experimental results, we compute the overall accuracy (Acc) for the multi-class classification tasks in the datasets of ColonPath, and the mean average precision (mAP) for the multi-label classification tasks in the datasets of ChestDR and Endo.

We follow MedFMC and compare EPT with other widely applied PEFT methods. We compare the results of \deletedtwoVision Transformer architecture\deleted, Vision Transformer [44] (ViT)\deleted and Swin Transformer(Swin), on image classification in Sec 4.2. Due to the fact that the EPT method is also a prompt-based fine-tuning approach, it shares many similarities with VPT and VP. Additionally, given the superior performances of VPT and VP, we primarily compare our results with VPT and VP. Unlike selecting the Full method as the primary reference in [24], our experiments chose the Partial-1[45] fine-tuning method as the primary reference. The reason is that for few-shot fine-tuning, fine-tuning all parameters incurs computational costs that do not match the data volume, and our experiments have found that the Full method often yields the worst results. Therefore, we chose Partial-1 as the primary reference method. \deletedFor Adapter[36] and Lora[27], we set the dimension $r$ for downsampling to 4. The default prompted layers for EPT and VPT are both set to 12, meaning the prompt is used through all the layers. The prompt length for prompt-based fine-tuning methods is set to achieve the best performance individually. If not otherwise specified, we adopt ViT-Base/16 pretrained on ImageNet-21k as our default backbone model, which is also the default model for ViT in the MedFMC Challenge.

For the two relative embedding orders of prompts in the EPT method: EPT and EPT-C, we select different relative embedding orders for each task and choose the best result as the final result of our approach. We do this because we found that EPT-C often exhibits better generalization performance, while EPT tends to have better fitting ability. This leads to each method being more suitable for relatively simple datasets (Colon) and relatively complex datasets (Chest) respectively, and this difference brings about significant performance superiority for our method. Furthermore, we conducted a more in-depth analysis of this phenomenon in Sec 4.3. In terms of fairness, our standalone EPT-C or EPT can surpass most other PEFT methods, especially the VPT that we specifically compared against. More results and details are provided in Sec 4.2 and Appendix.

We present average results of various PEFT methods for 1-shot, 5-shot and 10-shot classification on three datasets: Chest, Colon and Endo, as shown in Tab 1. Several main findings can be observed.

First, EPT outperforms other PEFT methods by a substantial margin on three datasets in most cases. Specifically, it surpasses the second-best method Adapter by 2.05% and the third-best method VPT by 2.74%, respectively. EPT still achieves second place in the Colon 5-shot task, with a performance gap of only 0.24 from Bias. This demonstrates that the superior performance of EPT when adapting pre-trained foundation models to cross-domain few-shot scenarios, e.g., medicial image analysis. Second, in the category of prompt tuning, EPT surpasses VPT and VP by 2.74% and 5.09%, respectively. In addition, EPT outperforms LoRA by 5.04%. This not only indicates that EPT has addressed previous shortcomings in the way of introducing prompts, embedding prompts in the channel direction is a more promising approach. Moreover, EPT has better approximation capabilities on Transformer architectures, thereby breaking through limitations of prompt tuning. Third, EPT, VPT and VP surpasses Linear by 6.24%, 3.50%, and 1.15%, respectively, and prompt tuning achieves very competitive ranks among all PEFT methods. This shows that prompt tuning helps to alleviate the negative impact of foundation models in learning pre-training data distributions, and calibrates their performance in new domains. Fourth, Full is 9.34% lower than Linear, potentially due to overfitting. When adapting foundation models to few-shot scenarios, fine-tuning more parameters is not always better, and PEFT methods are very meaningful.

As shown in Fig 2, we present visualization results of feature distribution after fine-tuning on the Colon dataset.

In Fig 2 (a), it displays the feature distribution of samples from the same class on the first principal component obtained by principal component analysis (PCA). We \deletedrandomlyselect 2,628 samples labeled as existing tumours, and four methods were compared: Full, Linear(Pretrained backbones), VPT, and EPT. The x-axis represents feature values, and the y-axis represents the number of samples. Results show that: (1) EPT exhibits a more narrower range of sample distribution, with more pronounced feature values, demonstrating a more concentrated intra-class feature distribution. (2) Foundation models only pre-trained on natural images displays a clear different behavior, and lack continuity, suggesting that samples may cluster into two distinct groups that are significantly far from each other. (3) Full displays that samples are overly dispersed and exhibit lower feature values, which could be due to overfitting and the failure of capturing common features.

In Fig 2 (b), it visualizes the two-dimensional feature distribution of samples from the same class by t-SNE. Our selcted samples and methods are same as Fig 2 (a). The x-axis and y-axis represent feature values of two different dimensions, respectively. Results show that: (1) Pretrain shows that the density contour lines on the left and right sides do not connect and close, indicating the discontinuity and a huge distance within the distribution. (2) Full shows that the density contour lines are irregular, indicating that samples are dispersed throughout the space, and not sufficiently concentrated. (3) VPT shows a clear outlier distribution on the top, far from the center. (4) EPT shows that the area enclosed by the density contour lines is the smallest, indicating that samples are more concentrated. Fig 2 (a) and Fig 2 (b) both illustrate that EPT has a more continuous and concentrated intra-class feature distribution. This may be due to the patch-wise scaling operation, which reduces the distribution scopes and extracts more universal and effective features.

In Fig 2 (c), it displays the two-dimensional feature distribution of all classes by t-SNE. We randomly select 5,256 samples from two categories, maintaining a 1:1 ratio. Results show that: (1) Pretrain not only exhibits a larger intra-class distribution but also shows inter-class linear inseparability. (2) Prompt tuning demonstrates a continuouts and concentrated intra-class distribution, and the separability between classes are also distinct. This may be due to the introduction of prompt tokens, which expand the dimensionality of the input space, thereby increasing the separability of features.

These visualization results provide strong evidence for the view that prompt tuning acts as a distribution calibrator.

For the purpose of observing the performances of various tuning methods at larger backbone scales, we conducted experiments on ViT-Large/16. The experimental results, as shown in Tab 2, indicate that our EPT method still demonstrates the most significant advantages on all three datasets and outperforms other methods. Additionally, we can observe that fine-tuning with ViT-Large/16 as the backbone generally yields better classification performance compared to ViT-Base/16. Even linear methods show improved classification results. However, both MLP-3 and Adapter methods exhibit a decrease in performance, which differs from the results in Tab 1. Furthermore, the average classification results of EPT demonstrate an increasing trend compared to Tab 1. This proves the effectiveness of our EPT method on larger-scale pre-trained models.

We vary the training data from 1, 5, 10-shot to 20% of the entire training set. The average accuracy of different fine-tuning methods on various data scales is presented in Fig 3. First, we observe that our method consistently outperforms others across all data scales, displaying superior data scalability. Moreover, as the data scale increases, all fine-tuning methods exhibit a significant increase in accuracy, but their performance growth rate tend to slow down. Further observation reveals that the performance on Colon converges faster than on Chest and Endo. This might be because that multi-label tasks require more data to learn sufficient discriminative information.

The average accuracy of different prompt tuning methods on various prompt length is presented in Fig 4. We vary prompt length from 1, 5, 10, 20, 50, 100 to 150. Since EPT has a different way of introducing prompts from VPT, the number of prompt parameters from VPT is four times that of EPT when introducing a prompt with length of 1 in a single Transformer layer. Therefore, to maintain relatively consistent number of parameters, we use relative prompt length on the x-axis. This means when the relative prompt length is 1, the actual prompt length of VPT is 1, and the actual prompt length of EPT is 4.

As the prompt length increases, the performance of VPT generally declines across three datasets. In contrast, EPT shows a clear upward trend on Colon and Endo, and remains relatively stable on Chest. This suggests that EPT can better address overfitting issues. Moreover, when the prompt length is only 1, the overall performance of EPT and VPT is nearly identical. As the prompt length and the number of prompt parameters increase, yet remain within the scope of PEFT, EPT gradully begins to outperform VPT by a large margin. This indicates that EPT has better parameter scalability, with a much higher upper limit on approximation capabilities compared to VPT. We can also observe that both EPT and VPT can achieve better performance at shorter prompt length compared to longer ones in some cases. This suggests that longer prompt length is not always necessarily better, we can explore locally optimal solutions to balance performance and efficiency.

The average accuracy of different EPT variants on various prompt embedding ways is presented in Fig 5. We present four prompt embedding ways: add, multiply, concat purely, multiply and concat. "add" means the sum of ${P_{E}}$ and $K^{\top}Q$. "multiply" means the product of ${P_{E}}$ and $K^{\top}Q$. "concat purely" means the concatenation of ${P_{E}}$ and $K^{\top}Q$. "multiply and concat" is a novel EPT variant based on the insight of patch-wise scaling. We first obtain a scaling vector $\alpha\in\mathbb{R}^{n}$ by calculating difference between the maximum and minimum values in each patch of $K^{\top}Q$. Then, we multiply ${P_{E}}$ with $\alpha$, and concat the product with $K^{\top}Q$.

The outcome shows that $"Multi\_Cat"$ achieves the best performance across three datasets. After applying the scaling vector $\alpha$, the values of prompts become larger, and original values become smaller according to eq 10. Large original values are reduced more significantly, while small original values are reduced less. Therefore, the absolute distance between values decrease, enhancing characteristics of patch-wise scaling and further reducing the distance of intra-class feature distribution. Moreover, $"Pure\_Cat"$ has better performance compared to $"Add"$ and $"Multi"$, validating our analysis of prompt introducing ways again. The way of concatenation not only can helps to preserve the information of original tokens, but also introduces additional useful context in a more fine-grained manner.

The average accuracy of EPT on various prompt depth and embedding orders is presented in Fig 6. Prompt depth is varying from 1 to 12, and embedding orders are varying from top to bottom (orange line) and bottom to top (blue line). Two valuable questions are mentioned below:

Shallow or deep? Overall, as the number of prompted layers increase, the performance of EPT improves. As the depth increases, prompts can acquire a more complex and abstract representation capability and adjust features from both shallow and deep layers.

Top or bottom? We can observe that the performance of EPT improves when embedding prompts from top to bottom, which is contrary to the conclusion of VPT. This may be related to prompt introducing ways. The foreground and background across different medical images are similar, which means there are fewer low-level and coarse-grained semantic features that are discriminative. Instead, high-level and fine-grained semantic features are more crucial. Since EPT directly embeds prompts in the channel direction, it can correlate with tokens more fine-grainedly and increase the dimension of representation. This results in greater enhancement of the high-level semantic features from deeper layers.

The impact of grads of prompt parameters is shown in Tab 3. The outcome demonstrates that the performance of foundation models still improves in downstream tasks compared to models that are only pre-trained on natural images, even though we freeze the prompt parameters of VPT. This suggests that prompt tuning enhances the feature separability and strengthen the distribution calibration characteristics by introducing additional prompt tokens and increasing the input dimensions.

The visualization results of convergence distribution are showned in Fig 7. Results indicate that EPT has the highest average values and the smallest variance in the convergence interval across various tasks, making it a superior and reliable PEFT method. Moreover, both VPT and EPT achive better convergence than Full, proving the superiority of PEFT in cross-domain few-shot scenarios.

For further exploration of the performance of fine-tuning methods in few-shot medical image analysis, we also conducted experiments on the ISIC 2018 dataset [46](see Appendix for details). From Tab 4, it can be observed that some fine-tuning methods failed, such as MLP-3 and VP, both exhibiting worse performance compared to the Linear method. However, methods targeting backbone fine-tuning, such as LoRA, Bias, and Adapter, still showed more significant improvements compared to Linear. VPT and EPT demonstrated even more notable enhancements, with EPT outperforming other fine-tuning methods by nearly 10% points relative to Linear, thus confirming the effectiveness of the EPT method.