Gentle-CLIP: Exploring Aligned Semantic In Low-Quality Multimodal Data With Soft Alignment

Abstract

Multimodal fusion breaks through the barriers between diverse modalities and has already yielded numerous impressive performances. However, in various specialized fields, it is struggling to obtain sufficient alignment data for the training process, which seriously limits the use of previously elegant models. Thus, semi-supervised learning attempts to achieve multimodal alignment with fewer matched pairs but traditional methods like pseudo-labeling are difficult to apply in domains with no label information. To address these problems, we transform semi-supervised multimodal alignment into a manifold matching problem and propose a new method based on CLIP, named Gentle-CLIP. Specifically, we design a novel semantic density distribution loss to explore implicit semantic alignment information from unpaired multimodal data by constraining the latent representation distribution with fine granularity, thus eliminating the need for numerous strictly matched pairs. Meanwhile, we introduce multi-kernel maximum mean discrepancy as well as self-supervised contrastive loss to pull separate modality distributions closer and enhance the stability of the representation distribution. In addition, the contrastive loss used in CLIP is employed on the supervised matched data to prevent negative optimization. Extensive experiments conducted on a range of tasks in various fields, including protein, remote sensing, and the general vision-language field, demonstrate the effectiveness of our proposed Gentle-CLIP.

1 Introduction

As a pivotal foundation for numerous tasks[11, 10], multimodal learning has become the focus of many researches[8, 9]. By integrating information from diverse modalities such as texts, images and so forth, multimodal models can acquire more comprehensive information to enhance the generalization of the learned representations[36]. Meanwhile, multimodal fusion enables networks to emulate human-like multiple perceptual capabilities and address the inherent challenges like data scarcity, noise and ambiguity in various domains, from computer vision to healthcare[14, 3].

In order to better harness the latent alignment information, most previous researches have primarily focused on the frameworks and the pretraining objectives, which can guide models to effectively understand the relationships between different modalities. Result from the sufficient development of image and text field, many studies have made great progress in the vision-language field[6, 51, 31, 33].Thereinto, CLIP employs a contrastive pretraining task [4] on large-scale datasets and gets robust multimodal representation. Due to simple framework and general pretraining task, it has good portability and scalability as well as competitive performance compared with supervised methods, hence it becomes the baseline for various vision-language works[53, 34]. Besides the aforesaid traditional field, in other intersecting domains, great breakthroughs have been also made by applying CLIP. EchoCLIP[13] improves the performance of cardiac imaging models by learning the association between cardiac ultrasound images and expert texts while ProtST[1] can capture more protein function information by aligning protein sequences and textual property descriptions. Moreover, in the field of zero-shot video recognition, Open-VCLIP[22] also shows excellent performance by leveraging the similar paradigm, which proves the powerful effects of CLIP.

(a) CLIP

(b) S-CLIP

(c) Gentle-CLIP

Figure 1: Comparison among CLIP, S-CLIP as well as Gentle-CLIP on how to adopt unpaired multimodal data. (a) CLIP only takes use of the matched data for multimodal fushion while ignoring the valuable information in unlabeled data. (b) S-CLIP attempts to improve the alignment performance by two novel pseudo-labeling methods but it limits to the language modality and heavily relies on the way how to measure similarity between samples. (c) Gentle-CLIP try to explore latent alignment from unmatched multimodal data based on the characteristics of data themselves, which needs less expert knowledge and can extend to various fields.

Nonetheless, there are still many specialized fields where it is usually difficult to obtain sufficient alignment data[15] while traditional multimodal models like CLIP can only learn from matched pairs, which greatly limites the performance of previously elaborate models. In order to break the above dilemma, several studies paid attention to these specialized fields and attempted to reduce the number of matched pairs for pretraining[16, 17, 18, 19]. The main idea is to modify the loss function of CLIP and apply semi-supervised learning method[37, 38] to explore the latent alignment information in the unlabeled data. Recently, a new research improves the original CLIP and proposes S-CLIP[28] which introduces two novel pseudo-labeling methods for unlabeled images and achieve state-of-the-art in various specialized semi-supervised vision-language fields.

However, pseudo-labeling learning methods may only be limited to the fields with class information and have difficulties scaling to other specialized multimodal domains where pseudo-labels are struggling to obtain. Meanwhile, the knowledge of generating pseudo-labels only relies on the insufficient labeled data, which leads to narrow ken and may loss much potential alignment information. In addition, the quality of pseudo-label has a great impact on the final performance so the learning process is unstable and even negative[39]. In order to solve these problems, it is necessary to design new semi-supervised methods for multimodalities, which can capture latent alignment information in unpaired data and be well extended to various multimodal domains.

Therefore, we propose a novel semi-supervised learning method for multimodal alignment based on CLIP, named as Gentle-CLIP. We believe that ultimate representation is composed of modality, structure as well as semantic and the key to multimodal alignment is to capture the same semantic representation while ignoring the other two representations. On the premise of two modal data with the same semantic distribution, we design a new pretraining task based on manifold matching and design a novel loss called semantic density distribution(SDD) to better concentrate on the implicit alignment information among vast unpaired multimodal data. Moreover, we introduce multi-kernel maximum mean discrepancy(MK-MMD) to eliminate the difference between modality representations while self-supervised contrastive loss is used to prevent mode collapse and enhance the robustness of semantic distribution. At the same time, we apply contrastive loss from CLIP on the matched multimodal pairs to keep the correct learning direction. Gentle-CLIP tries to explore alignment relationship in latent space and it can be extended to various multimodal domains due to task irrelevance. Through end-to-end learning, the mutual constraints between losses prevent negative optimization and implicitly expand the knowledge range. Our approach attempts to achieve multimodal alignment from essence and it can be transferred to different multimodal frameworks[49, 50, 52]. The comparison between Gentle-CLIP and other learning strategies is shown in Figure 1.

In short, our contributions are summarized as follows: (1) We contribute a groundbreaking perspective for the semi-supervised multimodal alignment problem by transforming it into a manifold matching problem, which brings a new pathway to exploit the implicit alignment information in the rich, yet largely unmatched multimodal data. (2) We design a novel semantic density distribution loss with less computing cost and it can be applied in various specialized fields as well as different multimodal frameworks. We introduce other objectives based on theoretical analysis about the components of representation and propose Gentle-CLIP to realize multimodal alignment with less supervised pairs. Moreover, our method can be applied to other domains with two-stream networks [35], such as knowledge distillation[47, 46], self-supervised learning[48] and domain adaptation. (3) We conduct extensive experiments in various fields and prove the effectiveness of our method. And Gentle-CLIP outperforms the existing semi-supervised methods in specialized fields because of larger knowledge range. We also explain the effects of key modules and provide a feasible usage paradigm for the specialized fields without enough supervised pairs.

2 Related Works

Multimodal alignment. Multimodality enhances understanding and decision-making by integrating information from multiple sensorymodalities[81, 82]. Thereinto, ALBEF [77] aligns visual and language representations, using momentum distillation to improve multimodal embeddings. FLAVA [79] enhances multitask and cross-modal learning by jointly pretraining text and images. ALIGN [80] jointly trains language and image encoders, significantly enhancing performance across various vision and text benchmarks. In recent years, researches around CLIP has further optimized computational efficiency and model representation capabilities. For instance, FLIP[53] brings lower computation and faster training times by randomly removing a large number of image patches during training process while SoftCLIP[7] applies fine-grained interior self-similarity as a softening target for cross-modal learning to alleviate the strict mutual exclusion problem. Moreover, latent diffusion models[11] generates reliable text embeddings as condition by using pretrained text encoder of CLIP and CLIPSelf [12] enhances region-level representation through self-distillation from CLIP’s image encoder, which proves the powerful effects of CLIP.

Semi-supervised learning. Semi-supervised learning [61, 69, 70] uses both labeled and unlabeled data to improve model training, encompassing strategies like pseudo-labeling [62, 73, 74], where models self-label their training data, and self-supervised learning [63, 64, 76, 75], which generates its own labels, thereby broadening its application in various fields. vONTSS [65] utilizes the von Mises-Fisher distribution and optimal transport for semi-supervised neural topic modeling to improve topic extraction in text datasets. SSGD [66] proposes a new semi-supervised domain generalization method that enhances model robustness under domain shifts through stochastic modeling and style augmentation. SS-ORL [68] employs a semi-supervised offline reinforcement learning approach, improving learning outcomes by utilizing unlabeled trajectories and limited complete action data. Semi-supervised learning ensures performance with fewer samples[71, 72], and the CLIP algorithm focuses on the complementarity and integration of different modalities. However, within the specialized domains, how to achieve these complementarity and integration across different modalities remains an issue that needs attention and resolution.

3 Method

3.1 Problem Description and Distribution Assumption

Different from the general vision-lanuage field, there could be only limited available matched pairs between specific associated modalities while it is relatively simple to get a large amount of data with similar semantic distribution in each modality. Therefore, we propose Gentle-CLIP, which uses massive unmatched data as well as limited matched pairs to realize more generalized alignment through semi-supervised learning. Formally, for any two modalities $\bf A$ and $\bf B$, we employ a small number of matched pairs $\left\{a_{i},b_{i}\right\}_{i=1}^{N}$ and a large number of unmatched data $\left\{a_{j}\in\bf A\right\}_{j=1}^{M_{1}}$ as well as $\left\{b_{j}\in\bf B\right\}_{j=1}^{M_{2}}$ to train our model. Through sampling respectively from unpaired two sets, we acquire $\left\{a_{j}\in\bf A\right\}_{j=1}^{M}$ and $\left\{b_{j}\in\bf B\right\}_{j=1}^{M}$ as unsupervised training data which are only considered to have similar semantic distribution rather than strict one-to-one matching. Based on a natural assumption, models can be fully trained on adequate unmatched multimodal data for robust representation.

Assumption 1 (Semantic Distribution Similarity Assumption, SDSA). We suppose that the latent embedding is a combination representation of modality, structure as well as semantic and more detailed analysis will be displayed in Appendix A. The goal of multimodal alignment is to find the same semantic representation and get rid of the interference from the other two representations. If the overall semantic distributions of $\left\{a_{j}\in\bf A\right\}_{j=1}^{M}$ and $\left\{b_{j}\in\bf B\right\}_{j=1}^{M}$ are similar, we can find a embedding space $\bf{S}\subseteq\bf{R}^{K}$ where $u_{j}$ and $v_{j}$ are the embedding representations respectively from $\bf{A}$ and $\bf{B}$. When the density distributions of $\left\{u_{j}\in\bf S\right\}_{j=1}^{M}$ as $\bf U$ and $\left\{v_{j}\in\bf S\right\}_{j=1}^{M}$ as $\bf V$ are similar, this space $\bf S$ is the semantic embedding space of $\bf A$ and $\bf B$. Consequently, when datasets from two modalities have the similar semantic distribution and their volumes are large enough, we can find the aligned semantic space by narrowing the gap between the density distribution from two modalities rather than strict matching relationship or pseudo-labeling method[45]. Through the above assumption, the semi-supervised multimodal alignment can be transformed into a manifold matching problem.

3.2 The Framework of Gentle-CLIP

Figure 2 introduces the conceptual overview of Gentle-CLIP. Due to the convenience and efficiency of CLIP, our proposed method follows to design the two-stream network[35]. Each stream includes an encoder network $F_{i}(\cdot),i\in\{\bf A,\bf B\}$ and a projection head network $H_{i}(\cdot),i\in\{\bf A,\bf B\}$, which are applied to map the data from original space into embedding space. The network from different streams adopt different backbone with unshared weight. We introduce multi-kernel maximum mean discrepancy loss(MK-MMD) as well as self-supervised contrastive loss(SSL) and design a novel semantic density distribution loss(SDD) to learn potential alignment relationship in large amounts of unpaired data. Through contrastive metrix, we apply contrastive loss(CL) on limited supervised pairs to guarantee proper optimization of the model. Moreover, this framework could be extended to various modalities and the objectives in Gentle-CLIP can also migrate to other multimodal alignment works[31, 32]. A detailed description of loss functions will be shown below.

Figure 2: The overall framework of Gentle-CLIP. Thereinto, $N$ is the number of matched pairs while $M$ denotes the unlabeled scale. Based on CLIP, we redesign new objectives which work on latent space to acquire robust representation by exploring potential alignment from unmatched data.

3.3 Objective loss

Supervised Alignment Guidance: Based on problem description, there are limited matched pairs $\left\{a_{i},b_{i}\right\}_{i=1}^{N}$ and a large number of unmatched data $\left\{a_{j},b_{j}\right\}_{j=1}^{M}$. Due to the lack of sufficient data, we are supposed to learn generalized representations through unsupervised data and take advantage of explicit alignment relationship as ground-truth to achieve precise alignment. We apply contrastive loss $\mathcal{L}_{\mathrm{CL}}$ in CLIP, which is to maximize the representation similarity between matched pairs while minimize the similarity between negative pairs. The format of this loss function is shown as follows:

$$\mathcal{L}_{\mathrm{CL}}=-\frac{1}{2n}\sum_{i=1}^{n}\left(\log\frac{\exp\left(\mathcal{S}(u_{i},v_{i})/\tau\right)}{\sum_{j=1}^{n}\exp\left(\mathcal{S}(u_{i},v_{j})/\tau\right)}+\log\frac{\exp\left(\mathcal{S}(v_{i},u_{i})/\tau\right)}{\sum_{j=1}^{n}\exp\left(\mathcal{S}(v_{i},u_{j})/\tau\right)}\right)$$ (1)

where $n=\lfloor\frac{N}{M+N}B\rfloor$ and $B$ is the batch size. $u_{i}$ and $v_{i}$ are representations in latent space respectively from two modalities. $\tau$ is a learnable temperature parameter and $\mathcal{S}(\cdot,\cdot)$ denotes cosine similarity. We expect to apply supervised as well as unsupervised data in every batch to jointly train the model due to the reason that a mass of unsupervised data can bring richer alignment information while matched pairs could lead to more accurate learning.

Coarse-Grained Modality Adaptation: MK-MMD[44] is used to measure the gap between two probability distributions $P$ as well as $Q$ and the core idea of this method is that samples $\{p_{1},p_{2},\ldots,p_{m}\}$ as well as $\{q_{1},q_{2},\ldots,q_{n}\}$ drawn from $P$ and $Q$ should keep similar statistical properties if the two distributions are the same. Specifically, MK-MMD maps the data from original space to Reproducing Kernel Hilbert Spac(RKHS) by kernel functions[43] and we can compare the difference between distributions in this space. Through linear combination of multiple kernel functions, we could get a more robust map** function to RKHS where we can easily distinguish two distributions even though they are similar in original space. The formula is shown as follows:

$$\mathcal{L}_{\mathrm{MK-MMD}}=\left\|\frac{1}{B}\sum_{i=1}^{B}\phi\left(u_{i}\right)-\frac{1}{B}\sum_{j=1}^{B}\phi\left(v_{j}\right)\right\|_{\mathcal{H}_{k}}^{2}$$ (2)

where $B$ is batch size while $u_{i}$ and $v_{j}$ are latent representations from two modalities. $\mathcal{H}_{k}$ is RKHS induced by kernel function $k$ and $\phi(\cdot)$ is implicit function used to map the original space data to $\mathcal{H}_{k}$. For multi-kernel cases, kernel function $k$ is a linear combination of $d$ basic kernel functions $\{k_{1},k_{2},\ldots,k_{d}\}$ and the format is $k=\sum_{i=1}^{d}\beta_{i}k_{i}$. learnable kernel weight $\beta_{i}$ is obtained through optimization to effectively represent differences between distributions. In our method, $d$ equals to $2$ while we choose Gaussian Kernel and Polynomial Kernel as basic kernel function.

Fine-grained Semantic Distribution Alignment: Since MK-MMD focuses on the whole distribution rather than sample level so it is imprecise and can only achieve macro alignment which is not enough for representation alignment. Consequently, we propose a new objective named as semantic density distribution loss(SDD) to explore more fine-grained information from unpaired data and realize more refined alignment. SDD is inspired from the perspective of probability density distribution estimation[29], hence it could keep an eye on specific sample representation while take the whole semantic distribution alignment into consideration at the same time. The formula is shown as follows:

$$\mathcal{L}_{\mathrm{SDD}}=\frac{1}{2}[\Gamma(U,V)+\Gamma(V,U)]$$ (3)

where $\mathcal{L}_{\mathrm{SDD}}$ works on the embedding space to measure the difference between two representation distributions more accurately in a symmetrical way and the models are trained to minimize the loss value to realize latent semantic alignment. $U$ and $V$ denotes embedding distributions and the format of $\Gamma(\cdot,\cdot)$ is shown as follows:

$$\Gamma(T,R)=\sum_{i=1}^{B}\{\frac{\kappa(t_{i},T)}{\sum_{j=1}^{B}\kappa(t_{j},T)}\log\frac{\kappa(t_{i},T)/\sum_{j=1}^{B}\kappa(t_{j},T)}{\kappa(t_{i},R)/\sum_{j=1}^{B}\kappa(t_{j},R)}\}$$ (4)

here for generality and convenience, we define three intermediate variables, $T=\{t_{i}\}_{i=1}^{B}$ while $R=\{r_{j}\}_{j=1}^{B}$ are sets composed of latent represenations and $x$ denotes the latent representation of a sample. $B$ is the size of batch which is a combination of matched pairs and unmatched data and Kullback-Leibler divergence is introduced to measure the dis-similarity between the density values of a specific sample from two distributions. The format of $\kappa(\cdot,\cdot)$ is displayed in the following formula.

$$\kappa(x,T)=\frac{\sum_{i=1}^{B}\exp\left(-\frac{\|x-t_{i}\|^{2}}{b^{2}\sigma(T)}\right)}{2Bb^{2}\pi}$$ (5)

here we apply exponential function as probability density function and $b$ denotes bandwidth used to control the smoothness. $\sigma(\cdot)$ denotes the variance of distribution and the format is shown as follows.

$$\sigma(T)=\frac{1}{B-1}\sum_{i=1}^{B}\left\|t_{i}-\frac{\sum_{j=1}^{B}t_{i}}{B}\right\|^{2}$$ (6)

where $t_{i}$ is the sample from set $T$ and we apply sample variance with Bessel’s Correction. $\sigma(\cdot)$ can lead model to pay more attention to narrowing the gap between semantic distributions from different modalities while avoid close cluster. Through training, $\mathcal{L}_{\mathrm{SDD}}$ can make the semantic aligned data from different modalities keep the similar probability density distribution in latent space. Meanwhile, the time complexity of $\mathcal{L}_{\mathrm{SDD}}$ is $\mathcal{O}(B^{2})$ which is the same as $\mathcal{L}_{\mathrm{CL}}$. In addition, Sinkhorn algorithm can also achieve fine-grained alignment[84] but it brings huge computing costs and may not fit for low-quality multimodal data due to slashing alignment. More details will show in Appendix B.

Self-supervised Distribution Stability: Rely on self-supervised contrastive loss(SSL)[41, 40], we can adequately find out implicit information from single modality and get robust feature representation. In the field of multimodal alignment with limit matched pairs, we find that it is essential to apply this objective because it can pull away the representations of different samples in the latent space with incomplete alignment guidance. In other words, if SSL is not employed, the data without alignment constraint may gather into a tight cluster. To be specific, we apply augmentation to generate positive pairs and the format of $\mathcal{L}_{\mathrm{SSL}}$ is displayed as follows:

$$\mathcal{L}_{\mathrm{SSL}}=-\frac{1}{B}\sum_{i=1}^{B}\log\frac{\exp(\mathcal{S}(z_{i},z_{i}^{+})/\tau)}{\sum_{j=1}^{B}\exp(\mathcal{S}(z_{i},z_{j})/\tau)}$$ (7)

where $z$ is latent embedding and $z_{i}^{+}$ denotes the representation of corresponding positive sample. In our method, each modality is supposed to apply this loss while $\mathcal{S}(\cdot,\cdot)$ is calculated by cosine similarity. We denote $u_{i}$ and $v_{i}$ as practical latent representation respectively from different two modalities and corresponding objectives are named as $\mathcal{L}_{\mathrm{SSL-U}}$ as well as $\mathcal{L}_{\mathrm{SSL-V}}$. According to the above constraint, we propose new loss named as $\mathcal{L}_{GC}$ and the formula is shown as follows:

$$\mathcal{L}_{GC}=\mathcal{L}_{CL}+\mu\mathcal{L}_{SSL-U}+\\ \mu\mathcal{L}_{SSL-V}$$ (8)

where $\mu$ is a hyperparameter and $\mathcal{L}_{GC}$ is used to guide the training process with accurate supervised alignment information rather than semantic distribution similarity, which is necessary for avoiding negative optimization. Meanwhile, if the data from different modalities can achieve augmentation according to the common semantics rather than the pattern in the single modality, the performance of related method may realize further growth[83].

The Overall Pretraining Objective: Our method aims to adopt matched pairs as well as unsupervised data in a batch at the same time. In this way, during the pretraining process, we can utilize comprehensive unsupervised data as well as the alignment constraint from matched pairs to realize robust and stable optimization process. Moreover, through samantic distribution alignment, the knowledge learned from unsupervised data and matched pairs can potentially interact with each other which could guide to enlarge the range of knowledge. For overall pretraining objective, we seek to minimize the loss functions of all pretraining tasks simultaneously:

$$\mathop{\min}_{\theta}~{}\alpha\mathcal{L}_{\mathrm{GC}}+\delta\mathcal{L}_{\mathrm{MK-MMD}}+\eta\mathcal{L}_{\mathrm{SDD}}$$ (9)

where $\theta$ denotes all learnable parameters in encoder networks and projection head networks. $\alpha$, $\delta$ and $\eta$ are hyperparameters which is used to control the influence of different pretraining tasks.

4 Experiments

In order to evaluate the effectiveness of our proposed method, we conduct extensive experiments in various specialized fields, including protein representation, remote sensing as well as general vision-language field. In addition, we analyze the roles of key modules by ablation experiments.

4.1 Quantitative Analysis About Sampling Size

As mentioned above, there exists implicit alignment information between different modalities with similar semantic distribution even if there is no definite matched pairs. Therefore, if we can acquire unimodal batches which reflect the real distribution of original data by stochastic sampling in each modality, it is derivable that each batch from different modalities also keeps similar semantic distribution and can be used for subsequent training process. Obviously, sampling size significantly influence the ability whether batches are on behalf of original distributions. Hence We attempt to quantitatively analyze the ability of different scales of sampling size for representing the original distribution, which can guide to choose the proper size. By applying soft Parzen-window method which is used for non-parametric density estimation, we can calculate the representing confidence of specific sample batch with given size. Through experimental verification, it could be concluded that sample batches will be able to represent original complicated distribution effectively when sampling size is over $64$. Furthermore, if different modal data is from the same semantic distribution, the batches with sampling size over $64$ will also keep the similar semantic distribution. More Detailed process of method and relevant analysis will be displayed in Appendix E.

Table 1: Benchmark results on protein representation field. Bold denotes the best results while underline represents the second best value. Two-stream networks outperform in most tasks and Gentle-CLIP can effectively explore latent alignment from unmatched data.

Input	Method	Gene Ontology			Enzyme Commission	Average
		BP	MF	CC
1D	ProtBert	0.279	0.456	0.408	0.838	0.495
	ESM-1b	0.452	0.659	0.477	0.869	0.614
	ESM-2	0.472	0.662	0.472	0.874	0.620
(3+1)D	GVP	0.326	0.426	0.420	0.489	0.415
	GearNet	0.356	0.503	0.414	0.730	0.501
	CDConv	0.453	0.654	0.479	0.820	0.602
	CLIP(1/2 CATH)	0.456	0.661	0.485	0.881	0.621
	Gentle-CLIP(Ours)	0.459	0.667	0.491	0.884	0.625
	CLIP(CATH)	0.463	0.665	0.493	0.885	0.627

4.2 Evaluation On Single Protein Function Prediction

Overview of tasks and training setup: To examine the efficacy of Gentle-CLIP in non-vision-language multimodal domains with insufficient alignment data, we conduct experiments in the protein representation field. Proteins can be defined using a multi-level structure and most previous works take aligned sequence and structure as input for single-stream network to capture the invariance features[26]. Due to limited aligned data, these intricately designed models struggle into trouble. Following [30], we consider sequence and structure as two modalities and apply Gentle-CLIP to realize multimodal fushion by pulling semantic distributions closer from extensive unsupervised data. Structure encoder is designed based on CDConv[23] while ESM-2[91] is selected as sequence encoder. We adopt CATH $4.2$ dataset for pretraining and this process lasts $100$ epochs. According to the same settings in [23], we evaluate the proposed method on the following four tasks: protein fold classification[26], enzyme reaction classification[26], gene ontology (GO) term prediction[27] and enzyme commission (EC) number prediction[27]. More details is shown in Appendix F and D.

Results: The performance of downstream tasks are shown in Table 1 while results of previous baselines are from [23] and [1]. Thereinto, CLIP(1/2 CATH) is a two-stream network with 50% CATH data for pretraining while CLIP(CATH) is pretrained on the whole CATH datasets. Gentle-CLIP(Ours) adopts 50% CATH data as supervised pairs while the rest are considered as unlabeled data. Moreover, we add an Average item to evaluate the overall performance. We first verify the effect of two-stream network compared to single-stream model and the results are displayed at the last line in Table 1. We can find that CLIP achieve better results at most downstream tasks and show superiority especially in EC number prediction. Further, it is obvious that Gentle-CLIP dramatically narrow the overall gap between CLIP pretrained on the whole CATH and the performance is even better at a few downstream tasks. This may be due to the fact that Gentle-CLIP can explore implicit alignment through the fine-grained semantic distribution constraint of SDD while CLIP only focuses on local representation match which may loss some global distribution information.

4.3 Evaluation On Remote Sensing Datasets

Overview of tasks and training setup: The models in remote sensing field can acquire comprehensive knowledge by jointly learning satellite images and corresponding captions. However, the training datasets are usually composed of web-crawled data without integrality and annotating captions may also need various expert knowledge, which can be expensive and time-consuming. So it is essential to evaluate the performance of Gentle-CLIP on limited matched pairs which is hard to tackle by traditional methods. Following [28], Gentle-CLIP is pretrained on the union of RSICD[55], UCM[56] and Sydney[57] with zero-shot classification and image-text retrieval as downstream tasks. ResNet[2] and transformer[5] are chosen as single modal encoder and Gentle-CLIP is pretrained for $25$ epochs. We subsample $10\%$ of image-text pairs for supervised learning while the remaining data is served unlabeled but conform to the same semantic distribution. Similarly, Top-1 classification accuracy is used to evaluate the performance on zero-shot classification while recall is applied for image-text retrieval tasks. Detailed descriptions will be presented in the Appendix C and D.

Table 2: Benchmark results on remote sensing field. Bold is the best average results and Gentle-CLIP improves the performance at most datasets through learning from unsupervised data.

Method	RSICD-CLS	UCM-CLS	WHU-RS19	RSSCN7	AID
CLIP(original)	45.3	50.5	65.5	58.9	47.8
CLIP(fine-tune)	58.3$\pm$0.3	63.5$\pm$3.4	76.5$\pm$3.2	61.9$\pm$1.2	63.1$\pm$1.3
Hard-PL	56.6$\pm$3.5	61.6$\pm$2.2	78.1$\pm$2.5	63.9$\pm$2.1	63.2$\pm$2.6
Soft-PL	62.5$\pm$0.8	65.7$\pm$2.7	83.7$\pm$2.7	65.7$\pm$0.6	68.0$\pm$0.7
S-CLIP	66.9$\pm$1.7	66.7$\pm$1.6	86.9$\pm$2.0	66.2$\pm$1.1	73.0$\pm$0.3
Gentle-CLIP(ours)	69.2$\pm$0.8	67.5$\pm$1.1	89.0$\pm$1.6	66.2$\pm$0.9	76.2$\pm$0.9

Results: Table 2 displays the results of zero-shot classification and the first five items are from [28]. In this experiment, Gentle-CLIP is designed based on S-CLIP and trained to narrow the embedding distribution gap between paired texts and unlabeled images under the guidance of SDD. The whole distributions of batches from different modalities may keep similar even though the explicit matching relationship is unknown between specific samples. For zero-shot classification, our method shows outstanding performance except RSSCN7. We can also find that existing methods are all difficult to bring much gain compared with other datasets, so it is believed that RSSCN7 may have significant gap with training set resulting in greater difficulty for inference. As shown in Appendix G, Gentle-CLIP consistently improves the results in image-text retrieval, which proves that less distribution gap between texts and unlabeled images brings robust pseudo-labels and stable distribution structure.

4.4 Evaluation On General Vision-Language Retrieval

Overview of tasks and training setup: Besides above specialized domains, we also evaluate the performance of Gentle-CLIP in general vision-language field. Experiments are carried on Flickr-8k[20] and Mini COCO while image-text retrieval is adopted as downstream task. Moreover, the vision encoder employs ResNet-50 or ViT-32[24] and BERT[25] acts as the text encoder. We choose the first description of each image as corresponding caption while dropout ratio is set to $0.3$ for augmentation following [40]. Pretraining process lasts $50$ epochs and batch size is $64$ with learning rate equalling to $0.001$. Furthermore, $1/3$ multimodal data is considered to be matched while the rest data is unlabeled. Datailed statements of datasets will be displayed in Appendix C.

Table 3: Benchmark results on general vision-language field. CLIP(1/3) is the baseline while values highlighted in green indicate the improvements. Gentle-CLIP enlarges the knowledge range and especially brings advantages in Mini COCO dataset with VIT as image encoder.

Model	Method	Flickr-8k				Mini COCO
		Image $\rightarrow$ Text		Text $\rightarrow$ Image		Image $\rightarrow$ Text		Text $\rightarrow$ Image
		top1	top5	top1	top5	top1	top5	top1	top5
ResNet	CLIP(1/3)	11.6	35.5	10.9	34.2	7.9	30.6	8.2	30.3
	CLIP(1)	+9.5	+17.4	+8.9	+18.1	+6.5	+16.3	+6.0	+16.2
	Gentle-CLIP	+1.7	+5.4	+1.2	+5.2	+3.8	+8.5	+3.1	+8.5
VIT	CLIP(1/3)	14.7	42.7	14.5	41.7	10.5	37.1	9.7	36.5
	CLIP(1)	+10.9	+14.7	+10.0	+14.5	+6.9	+14.4	+6.6	+12.1
	Gentle-CLIP	+2.1	+2.9	+1.9	+3.6	+4.3	+12.3	+5.1	+12.6

Results: We evaluate the performance with different models as well as datasets and the results of image-text retrieval is shown in Table 3. CLIP(1/3) is trained by 1/3 dataset while CLIP(1) learns alignment on the whole dataset. Gentle-CLIP employs 1/3 matched data for supervised learning and enlarge knowledge range from rest unpaired data. For better observing effects of different strategies, we adopt the improvement value as yardstick and baseline is CLIP(1/3). We can find that Gentle-CLIP continuously brings gains regardless of settings and the overall performance of VIT is better than ResNet for given datasets. From the experimental results, we may conclude that multimodal fushion could be simple when different modality models have similar distance measurement.

4.5 Ablation

Gentle-CLIP applies different kinds of objectives to explore implicit semantic alignment from low-quality multimodal data and shows excellent performances in various experiments. Then, we will further analyze the internal mechanism by replying to the following noteworthy questions.

Q1: How will SDD and SSL influence the final effectiveness? To answer this question, we conduct experiments in protein representation field and evaluate the performance at protein fold classification as well as enzyme commission number prediction. We focus on the effects of these two objectives and results are shown in Table 4.5. We can find that Gentle-CLIP trained with both objectives shows better performance except superfamily classification while the model only trained by SDD outperforms at this item but fall into negative optimization in EC number prediction. Single SSL brings weak improvement compared with the baseline so it also acquires additional alignment information during pretraining process. Through above results, we believe that the combination of these two objectives brings more advantages rather than simple stack of respective effect, in other words, these two losses interact and depend on each other. To be specific, SDD can exploit alignment information in unsupervised data but may cause mode collapse and negative learning as shown at the third line. While adding SSL can further constrain the optimization direction and increase the stability of the overall distribution. However, SSL will also affect the latent distribution of similar samples, hence it is necessary to balance the relationship to achieve better performance[83].

Q2: How is the performance if we change some modules of SDD? With fine-grained distribution similarity measurement, SDD plays a critical role in semi-supervised fushion. So it is essential to deconstruct SDD and analyse which settings may lead to better effects. We make retrieval experiment on Flickr-8k with Top-3 recall and Table 4.5 display the results. We focus on how to calculate the distance between samples as well as the difference between distributions. Relative distance(RD) $\|x-t_{i}\|^{2}/\sigma(T)$ in eq. 5 can eliminate the indistinguishability in tight latent cluster compared to absolute distance while Kullback-Leibler Divergence(KL) in eq. 4 may be more suitable for distribution contrast with MSE. It is clear that models achieve the greatest improvement when adding RD and KL simultaneously. Moreover, if improved Softmax function in [47] can be employed to modify the distance $\|x-t_{i}\|^{2}$ with different attention, we will control distribution alignment granularity through different temperatures. Other ablation results will be shown in Appendix I.

tableAblation results evaluated on protein representation field to analyze the roles of SSL and SDD. Thereinto, Super denotes Superfamily task and EC is EC number prediction. The values in bold is the best result. SSL SDD Fold Classification EC Fold Super Family 2613 2613 57.7 78.6 99.6 0.881 ✓ 2613 57.9 78.7 99.6 0.881 2613 ✓ 58.5 80.1 99.6 0.878 ✓ ✓ 59.1 79.7 99.6 0.884

tableAblation results on Flickr-8k with ResNet as image encoder to analyze the effects of key modules in SDD. Bold indicates the best result. RD denotes relative distance while $50$ and $100$ represent the retrieval sizes. RD KL I$\rightarrow$T R@3 T$\rightarrow$I R@3 50 100 50 100 2613 2613 28.4 18.4 29.6 18.5 2613 ✓ 29.1 18.7 30.6 18.6 ✓ 2613 28.7 18.8 29.9 18.9 ✓ ✓ 29.8 20.2 31.1 20.3

5 Conclusion

We propose a semi-supervised multimodal alignment method named as Gentle-CLIP and design a novel objective SDD which can measure the fine-grained difference between latent distributions. Besides, we introduce other loss functions to realize rubost generalization through the analysis of latent embedding. We demonstrate the superiority of our method across various fields and provide a possible way for multimodal fushion in specialized domains with insufficient aligned data by exploring latent alignment information from unimodality.

Appendix A The analysis of latent embedding

We hold the opinion that the ultimate embedding is a combination representation of modality, structure as well as semantic. If data is from different modalities, the modality representation may have difference and lead to the gap between ultimate embeddings. Meanwhile, if the data is from single modality, the representation may be separated due to the gap of semantic rather than modality. In addition, even though the data keeps similar semantic and are from the same modality, the embedding may also have difference because of the internal structural gap. The visualization of relationship between these three representations are shown in Figure 3.

Figure 3: The visualization of relationship between different representations.

Appendix B Difference between SDD and Sinkhorn Algorithm

For Sinkhorn Algorithm, we will analyse its role on the basis of SuperGlue[84]. In the attention-based graph neural network processing, each pair of keypoints (one from image A and one from image B) gets a combined feature representation. SuperGlue constructs a score matrix $S$ by calculating the inner product of these feature representations, where each element $S_{i,j}$ indicates the matching score between the $i$-th keypoint in image A and the $j$-th keypoint in image B. In order to handle the unmatched keypoints elegantly, it introduces an additional "dustbin" to the score matrix, allowing keypoints which find no match from the other image to be assigned to this dustbin.

The score matrix $S$ is transformed using the exponential function to obtain $K=e^{S}$. Sinkhorn algorithm iteratively adjusts $K$ through alternating row and column normalizations:

• Row normalization: $K\leftarrow K\oslash(K\mathbf{1}_{N})$, where $\mathbf{1}_{N}$ is a length-$N$ vector of ones, and $\oslash$ denotes element-wise division.

• Column normalization: $K\leftarrow K\oslash(\mathbf{1}_{M}K)$, where $\mathbf{1}_{M}$ is a length-$M$ vector of ones.

This procedure ensures that the matrix $K$ approaches a doubly stochastic matrix, where each row and each column sum approximately to $1$. As for the time complexity, since each row or column normalization operation involves processing $M\times N$ elements (where $M$ and $N$ are the numbers of keypoints in images A and B respectively), each normalization requires traversing the entire matrix, hence the complexity of a single normalization is $\mathcal{O}(MN)$. Sinkhorn algorithm requires multiple iterations to converge. If the number of iterations is T, then the total time complexity is $\mathcal{O}(T\times MN)$. In addition, it requires strict alignment of data points from different modalities, which may not be applicable for low-quality aligned multi-modal data and may bring wrong information to the optimization process, hence it requires strong monitoring with supervised data. Contrastively, we implicitly align data points by measuring the gap between these two semantic distributions with a gentle and comprehensive way. SDD takes the features of overall distribution into account, which reflects the essence of the problem, and the time complexity is $\mathcal{O}(MN)$.

Appendix C Datasets

C.1 The datasets in protein representation fields

CATH 4.2. These dataset[85], part of the CATH (Class, Architecture, Topology, Homologous superfamily) protein structure classification system, is divided into training (18,204 proteins), validation (608 proteins), and testing (1,120 proteins) sets. In downstream protein design tasks, the training set fine-tunes pretrained models, and the test set evaluates them. It includes updates and classifications detailing protein classes, architectures, topologies, and homologous superfamilies. Although smaller in number, the CASP14 dataset better reflects real-world blind testing and competition environments.

Protein Fold Classification. We utilized the SCOPe 1.75 dataset[86] for protein folding classification across training, validation, and testing phases. Comprising 16,712 proteins, the dataset is classified into 1,195 folding categories, with 3D protein coordinates derived from the SCOPe 1.75 database. It employs three distinct evaluation schemes: the fold scheme, which excludes proteins from the same superfamily during training; the superfamily scheme, which withholds proteins from the same family in training; and the family scheme, where proteins from the same family are accessible during training. Average accuracy serves as the evaluation metric.

Enzyme Reaction Classification. The dataset described by Hermosilla et al.[87]includes 37,428 proteins categorized into 384 enzyme-catalyzed reaction classes based on Enzyme Commission (EC) numbers. It is structured into training, validation, and testing sets with proteins clustered by 50% sequence similarity to ensure that similar sequences are contained within the same data partition. This dataset is used primarily for develo** and evaluating machine learning models that predict enzyme functions from protein sequences.

Gene Ontology Term Prediction. This dataset, based on the research by Gligorijević et al. [88], is used for predicting Gene Ontology (GO) terms with protein functions. It categorizes proteins into three GO categories: Molecular Function (MF), Biological Process (BP), and Cellular Component (CC). It includes 1,943 biological process categories, 489 molecular function categories, and 320 cellular component categories. Additionally, the dataset comprises 29,898 proteins for training, 3,322 for validation, and 3,415 for testing, with the maximum F score($F_{\text{max}}$) as evaluation metric.

Enzyme Commission Number Prediction. Unlike enzyme reaction classification, it aims to predict the three-level and four-level 538 EC numbers. The dataset is divided into training, validation, and test sets as described by Gligorijević et al. [88], comprising 15,550 proteins for training, 1,729 proteins for validation, and 1,919 proteins for testing. EC number prediction is also a multi-label classification task, with the maximum F score ($F_{\text{max}}$) used as the evaluation metric. For GO term and EC number prediction, we follow the multi-cutoff splits in Gligorijević et al. to ensure that the test set only contains PDB chains with sequence identity no more than 95% to the training set.

C.2 The datasets in image-text retrieval task

Flickr-8K. This dataset is a well-known resource in the field of computer vision, particularly used for tasks like image captioning and cross-modal retrieval. It consists of 8,000 images collected from Flickr, each accompanied by five human-annotated captions that describe the images in detail.

MiniCOCO. This dataset is a smaller subset derived from the large COCO2014 dataset, designed for more efficient training and test. It comprises 25,000 images, which includes approximately 184,000 annotations across 80 object categories. In addition, this dataset is carefully sampled to maintain a balanced representation of object instances, size distributions, and class-specific object ratios.

RSICD. This dataset is a specialized resource in remote sensing, primarily used for tasks like image captioning and cross-modal retrieval. It contains approximately 10,000 images sourced from various global locations, each paired with five expert-generated captions detailing the geographical and environmental features visible in the images. This dataset is designed to advance the automatic description of complex geographical scenes from aerial or satellite perspectives.

UCM. This dataset is a collection from the University of California, Merced, consisting of 2,100 high-resolution satellite images across 21 land use categories including airports, golf courses, and residential areas. Each category contains 100 images, all meticulously labeled for precise land use classification tasks. This dataset provides a detailed visualization of diverse land use scenarios, facilitating advanced studies in geographic information systems and machine learning.

Sydney. This dataset comprises 643 high-resolution aerial images categorized into seven distinct urban elements such as parks, residential areas, and so forth. This dataset is used for urban planning and management, offering a rich visual repository to support research in urban feature analysis and geographic information systems applications, ensuring a comprehensive study of urban landscapes.

C.3 The datasets in zero-shot classification

RSCID-CLS. Derived from the broader RSICD, this dataset is tailored for image classification with a focus on geographical features. While the exact number of images is not specified, it is understood to include several thousand images, typically a significant subset of RSICD. This classification-focused dataset encompasses varied landscapes such as urban environments, water bodies, and agricultural regions, all categorized for targeted machine learning applications. The dataset aims to improve algorithms’ accuracy in discerning complex geographical patterns from satellite imagery.

UCM-CLS. This dataset adapts the original 2,100 high-resolution satellite images from the UCM dataset into a format specifically intended for classification tasks. Each image, measuring $256\times 256$ pixels, represents one of 21 distinct land use types including airports, orchards, and industrial zones. This granular categorization supports precise model training in land use recognition, providing a critical tool for geographic analysis and urban planning research.

WHU-RS19. Consisting of approximately 1,000 high-resolution images, this dataset captures 19 urban classifications such as transportation networks, and green spaces. This diversity offers a robust platform for testing urban classification algorithms against a backdrop of global cityscapes, thereby aiding in the development of more adaptive and effective remote sensing technologies.

RSSCN7. With 2,800 images distributed across 7 categories, each containing around 400 images, this dataset challenges scene classification systems with its variety of land covers. These categories include industrial sites, commercial regions, and parks, aiming to mirror the complexity of real-world environments. The dataset tests the ability of models to identify and classify scene characteristics accurately, enhancing their usability in practical remote sensing applications.

AID. This dataset extends over 10,000 images across 30 diverse scene types such as beaches, railways, and residential areas. The extensive collection of images is particularly designed to challenge and refine the scene recognition capabilities of aerial image analysis algorithms. Each category in AID is chosen to represent a wide array of aerial perspectives, promoting comprehensive training and evaluation of models on a scale that mimics real-world aerial surveillance tasks.

Appendix D Model Architecture

ESM2. ESM2 is an advanced protein language model designed to extract meaningful representations from protein sequences. It is based on a 24-layer transformer encoder architecture, employing rotary position embeddings to better handle dependencies in long sequences. ESM2 was pre-trained on 65 million unique UniRef50 sequences using a masked language model, enabling it to learn sequence patterns and their structural implications from a large amount of data. As the model scales from 8 million to 15 billion parameters, ESM2 shows significant improvements in structural knowledge, especially in low-resolution and atomic-level structure predictions. Its variant, ESMFold, can directly predict high-precision three-dimensional structures from single sequences without the need for multiple sequence alignment steps, greatly increasing prediction speed. These features make ESM2 not only excel in structural prediction but also applicable to functional prediction and mutation effect analysis in various biological research areas.

CDConv. CDConv is specifically designed to handle the complexity of protein sequences and geometries. The architecture includes two main components: discrete processing for sequences and continuous encoding for geometric coordinates. Through this approach, CDConv independently adjusts processing strategies for different spaces (sequence and geometry) and optimizes their respective weights. Specifically, the architecture employs discrete weights to capture the regularity in sequences while directly encoding continuous irregular displacements in geometries, thereby effectively reducing the impact of geometric irregularity on sequence modeling. Additionally, this architecture implements a hierarchical deep convolutional neural network, which progressively refines and abstracts protein features through multiple layers to support various biological function prediction tasks. This unique modeling approach has demonstrated excellent performance in multiple protein modeling tasks, proving its effectiveness in capturing the complex relationships between protein structure and function.

ResNet. ResNet (Residual Network), proposed by Microsoft Research in 2015, addresses the vanishing gradient problem in deep neural networks by introducing residual blocks with skip connections. These connections allow gradients to flow through multiple layers, hel** the network learn identity functions and ensuring that deeper networks perform at least as well as shallower ones. ResNet architectures, such as ResNet-34, ResNet-50, ResNet-101, and ResNet-152, range from 34 to 152 layers, effectively balancing performance and computational efficiency for various image recognition tasks. ResNet-34 handles medium-scale tasks, while ResNet-50, known for its balance of performance and efficiency, is widely used in image processing. ResNet-101 captures finer details for advanced visual recognition, and ResNet-152 excels in complex image processing tasks. Variants like Pre-activated ResNet and Wide ResNet further optimize performance and efficiency, making ResNet widely applicable in computer vision.

Vit. ViT (Vision Transformer), proposed by Google Research in 2020, adapts the Transformer model from NLP to computer vision by dividing images into patches and processing them with self-attention mechanisms. Unlike traditional CNNs that focus on local features, ViT captures global relationships between patches, enhancing its understanding of complex spatial hierarchies in images. This approach allows ViT to perform comparably to or even surpass state-of-the-art methods in various image processing tasks. ViT’s scalability allows it to adapt to different computational and performance needs by adjusting Transformer layers, model size, or input patch resolution. Variants like ViT-32 and ViT-64 demonstrate this flexibility, with ViT-32 using 32x32 patches for larger images and ViT-64 using 64x64 patches for coarser segmentation. This versatility and efficiency make ViT a powerful tool in computer vision.

Bert. BERT (Bidirectional Encoder Representations from Transformers) is a deep learning model developed by Google AI in 2018 for natural language processing (NLP) tasks. Its bidirectional Transformer architecture allows it to consider both forward and backward context, enhancing performance in tasks like question answering, language inference, and sentiment analysis. BERT’s pre-training involves "Masked Language Model" (MLM) and "Next Sentence Prediction" (NSP), followed by fine-tuning on specific tasks. Improved models like RoBERTa[92], ALBERT[93], SpanBERT[94], DeBERTa[95], and ELECTRA[96] have built upon BERT’s foundation. RoBERTa optimizes training by removing NSP and using larger datasets. ALBERT reduces memory consumption with cross-layer parameter sharing. SpanBERT enhances span predictions for tasks like question answering. DeBERTa introduces disentangled attention for better context-aware representations, and ELECTRA uses a discriminator model for more efficient pre-training. These advancements have significantly advanced NLP capabilities.

Appendix E Detailed quantitative analysis for sampling size.

The sampling size may have a great influence on the experiment. On the one hand, if the sampling size is not large enough, the sampling batch may not represents the original distribution which leads to different semantic distribution between batches from different modalities. On the other hand, larger sampling size may bring greater cost even though it can better measure the true distribution. Hence, it is significant to select appropriate sampling size and we attempt to design a new approach to realize quantitative analysis. To be specific, for the original distribution, we apply the same scales of sampling size and acquire various sampling batches while we compute the difference between every two batch distributions according to the following formula:

$$D=\frac{1}{B}\sum_{i=1}^{B}(\kappa(t_{i},T)-\kappa(t_{i},R))^{2}+\frac{1}{B}\sum_{i=1}^{B}(\kappa(r_{i},R)-\kappa(r_{i},T))^{2}$$ (10)

where $T=\{t_{i}\}_{i=1}^{B}$ and $R=\{r_{i}\}_{i=1}^{B}$ are two sampling batches from the same distribution and $B$ is sampling size. $\kappa(x,T)$ denotes the density estimate of sample $x$ in $T$ and the formula is as follows:

$$\kappa(x,T)=\frac{\sum_{i=1}^{B}\exp\left(-\frac{\|x-t_{i}\|^{2}}{\sigma(T)}\right)}{B\pi}$$ (11)

where the format of $\sigma(\cdot)$ is shown in Eq. 6 and we adopt relative distance to avoid indistinguishable density estimates of different samples due to the tiny absolute distance. The smaller of the average gap between every two sampling batches is, the more similar the sampling distributions are and we can further deduce that the batch can better represent the original distribution. We choose random distribution as real distribution because it can be seen as the most cluttered distribution. If the batch well represents the real distribution with the specific sampling size, in theory, we can adopt the same scale of batches to realize representation for any other distributions. We select different scales of size and analyse the ability of corresponding sampling batches for representing the original distribution. The results are shown in Figure 4.

(a) The average gap of different sample sizes.

(b) The visual display for various sample sizes.

Figure 4: The average gaps between batches and their visual display with different sample sizes.

In Fig. 4a, abscissa denotes different sampling sizes and ordinate represents the average gap between sampling batches from the same distribution. Different colors denotes the different scales of sample feature dimensions. As we can see, with sampling sizes increasing, the average gap decreases sharply. If we suppose that the dissimilarity between batches is $100\%$ when sampling size is $2$, the dissimilarity drops to $0.9\%$ when the size reaches $64$ and we can apply the sampling batches to represent the original distribution with $99\%$ confidence coefficient. In Fig. 4b, we acquire two sampling batches from random distribution and visualize each of them with different colors after dimensionality reduction. With constantly increasing sample sizes, visual distribution areas from different batches are getting coincident.

Appendix F More details in protein representation field

In the field of protein representation, there is no widely accepted two-stream network and most of the previous works input the sequence and structure as two properties of the protein into a single-stream network. Here, following [30], we consider sequence and structure as two modalities for different encoders and refer to Gentle-CLIP to realize multimodal fusion. Due to the good performance of Continuous-Discrete Convolution (CDConv)[23], structure encoder is designed based on CDConv and we adopt ESM-2 as sequence encoder. The definition of neighborhood is not changed because it is critical for the node representation. Result from CDConv as structure encoder, we modify its input module to represent structure information as follows:

$$\mathrm{S}[i]=\begin{cases}Concat([0,0,0],\mathcal{P}[i+1]-\mathcal{P}[i])&i=0\\ Concat([\mathcal{P}[i-1]-\mathcal{P}[i],\mathcal{P}[i+1]-\mathcal{P}[i])&1\leq i<N\\ Concat([\mathcal{P}[i-1]-\mathcal{P}[i],[0,0,0])&i=N\end{cases}$$ (12)

Where $\mathcal{P}[i]$ denotes the coordinate matrix of node $i$ and $N$ represents the total number of nodes. Through the above formulas, we can obtain the structural information between the current node and its neighbors, encompassing the positional relationship and angles. Then we apply an embedding layer for adapting to model inputs and we set original sequence information to random noise together with structure information as input. Although CDConv does not directly acquire sequence information, it learns sequence knowledge transmitted by ESM-2 through the fusion process, making the model gradually learn the comprehensive representation. In the pretraining process, we respectively train Gentle-CLIP and CLIP on CATH $4.2$ for comparison and the batch size is set to $64$. The pretraining process last $100$ epochs and learning rate is $0.001$. According to the same settings in [23], we evaluate the effect of pretraining process in the downstream tasks and the results are shown in Table 4.

Table 4: Using supervised models, as well as our Gentle-CLIP model with half supervised and half unsupervised learning, we compared the performance on Fold Classification and Enzyme Reaction tasks. The best performer in each column is highlighted in bold.

Input	Method	Fold Classification			Enzyme Reaction	Average
		Fold	Superfamily	Family
1D	CNN	11.3	13.4	53.4	51.7	32.5
	ResNet	10.1	7.21	23.5	24.1	16.2
	LSTM	6.41	4.33	18.1	11.0	9.9
	Transformer	9.22	8.81	40.4	26.6	21.3
3D	GCN	16.8	21.3	82.8	67.3	47.1
	GAT	12.4	16.5	72.7	55.6	39.3
	3D CNN	31.6	45.4	92.5	72.2	60.4
3D+Topo	IEConv(atom level)	45.0	69.7	98.9	87.2	75.2
(3+1)D	GraphQA	23.7	32.5	84.4	60.8	50.4
	GVP	16.0	22.5	83.8	65.5	46.9
	IEConv(residue level)	47.6	70.2	99.2	87.2	76.1
	GearNet	28.4	42.6	95.3	79.4	61.4
	GearNet-IEConv	42.3	64.1	99.1	83.7	72.3
	GearNet-Edge	44.0	66.7	99.1	86.6	74.1
	GearNet-Edge-IEConv	48.3	70.3	99.5	85.3	75.9
	CDConv	56.7	77.7	99.6	88.5	80.6
	CLIP(1/2 CATH)	57.7	78.6	99.6	88.6	81.1
	Gentle-CLIP(Ours)	59.1	79.7	99.6	88.8	81.8
	CLIP(CATH)	58.5	81.3	99.7	88.8	82.1

Appendix G More detailed experiment and analysis in remote sensing

As shown in Table 5, the performance of several CLIP-based methods on cross-modal retrieval tasks for image-to-text and text-to-image scenarios is compared. The datasets used include RSICD, UCM, and Sydney. The methods listed include the original CLIP model, fine-tuned CLIP model, Hard-PL, Soft-PL, S-CLIP methods, and our Gentle-CLIP. The performance metric is R@5 (Recall at top 5), detailing each method’s performance across different datasets. Notably, the Gentle-CLIP algorithm excels in all test scenarios, demonstrating its effectiveness and superiority in cross-modal retrieval tasks. This table provides quantitative evidence of the potential advantages of our Gentle-CLIP method in practical applications.

Table 5: In the field of remote sensing retrieval, the results produced by the best performer in each column are boldfaced.

Method	Image $\rightarrow$ text R@5			Text $\rightarrow$ image R@5
	RSICD	UCM	Sydney	RSICD	UCM	Sydney
CLIP(original)	9.4	34.3	36.2	10.1	24.8	51.7
CLIP(fine-tune)	15.4$\pm$1.7	41.3$\pm$1.8	47.1$\pm$6.5	15.1$\pm$1.0	40.9$\pm$1.6	56.1$\pm$2.4
Hard-PL	16.1$\pm$0.2	40.8$\pm$2.9	43.1$\pm$3.0	15.7$\pm$0.7	40.5$\pm$3.0	47.7$\pm$5.3
Soft-PL	17.0$\pm$0.9	43.2$\pm$3.9	42.0$\pm$4.3	16.5$\pm$0.1	42.9$\pm$3.3	50.2$\pm$4.9
S-CLIP	18.4$\pm$0.6	45.7$\pm$1.4	50.0$\pm$3.0	16.8$\pm$1.2	43.5$\pm$1.5	55.1$\pm$2.0
Gentle-CLIP(Ours)	19.6$\pm$0.7	46.3$\pm$1.1	51.1$\pm$1.7	17.4$\pm$0.9	44.1$\pm$1.2	55.2$\pm$2.3

Appendix H Visualization experiment in general vision-language field

As shown in Figure 5, benchmark results on image-text matching task. In these figures, circle denotes the representation of image in latent space while triangle denotes the latent embedding of text. Visualization points with the same color means the similar semantic. So the goal of this task is to make the circles and triangles with the same color closer while pull away the other points. We can find that Gentle-CLIP can better get the distinguishing representations due to applying the self-supervised contrastive loss.

Figure 5: We conducted experiments on the traditional image-text dataset Flickr8K, where points of the same color represent text and images of the same category. The image architecture used is ResNet, and the text architecture used is BERT. We compared our Gentle-CLIP model, which uses half supervised and half unsupervised data, against CLIP using half supervised data and CLIP using fully supervised data. The results show that CLIP using half the amount of supervised data performs poorly in clustering images and text, as indicated by the blue and black points not clustering completely. In contrast, our model, which uses half supervised and half unsupervised data, achieves results close to those of CLIP with fully supervised data, and in some cases (e.g., the blue points), it even outperforms the fully supervised CLIP in clustering effectiveness.

Appendix I More ablation experiment

Q3: Can we simplify the format of SDD while guarantee competitive performance?
The question focus on whether it is necessary to design such a delicate objective for semantic distributions in a fine-grained manner. In other words, it is significant to analyse whether SDD can acquire more alignment information than other coarse-grained methods. Centroid method is chosen as comparison and it adopts the centroid distance to represent the difference between distributions. Experiments is carried out on the remote sensing field and the results are shown in Table 6.

S-CLIP uses a pseudo-labeling algorithm for alignment, while the Centroid method calculates the centroid of the distribution and aligns the centroids. Since our model performs matching at a finer granularity, it achieves the best results.

Table 6: In the field of remote sensing retrieval, we conducted ablation experiments on the alignment method, the results produced by the best performer in each column are boldfaced.

Method	Image $\rightarrow$ text R@5			Text $\rightarrow$ image R@5
	RSICD	UCM	Sydney	RSICD	UCM	Sydney
S-CLIP	18.4$\pm$0.6	45.7$\pm$1.4	50.0$\pm$3.0	16.8$\pm$1.2	43.5$\pm$1.5	55.1$\pm$2.0
Centroid method	18.6$\pm$0.8	45.9$\pm$1.1	50.7$\pm$2.1	17.0$\pm$1.1	43.9$\pm$1.3	55.1$\pm$1.7
Gentle-CLIP	19.6$\pm$0.7	46.3$\pm$1.1	51.1$\pm$1.7	17.4$\pm$0.9	44.1$\pm$1.2	55.2$\pm$2.3

Appendix J Experimental Resource Allocation

Experimental Resource Allocation. In the protein domain, the related experiments were conducted using 8 A100 GPUs, each with 80GB of memory, and the training lasted for 3 days. In the image-text domain, the experiments were performed using 4 A100 GPUs, each with 80GB of memory, and the training lasted for 1 day. For the remote sensing domain, the experiments utilized 4 A100 GPUs, each with 80GB of memory, and the training also lasted for 1 day.