Visual Language Model based Cross-modal Semantic Communication Systems

Deep learning techniques are commonly employed in the construction of encoders and decoders for ISC systems. In [9], a comprehensive SC system based on CNNs was initially introduced, showcasing superior performance in Peak Signal-to-Noise Ratio (PSNR) when compared to traditional compression algorithms. In [10], a novel Nonlinear Transform Source-Channel Coding (NTSCC) for SC systems was proposed, which leveraged a Variational AutoEncoder (VAE) to map the source signal to the latent space, and executed nonlinear transformation and channel coding in the space. Additionally, [11] presented an innovative SC system incorporating Semantic Slice-Models (SeSM) to facilitate adaptable model resemblance under diverse requirements. Furthermore, [12] introduced a Reinforcement Learning-based Adaptive Semantic Coding (RL-ASC) for image data. RL-ASC utilized a combination of VAE, RL, and generative adversarial networks (GANs) to encode, allocate, and decode semantic concepts.

Although convolutional and ViT-based autoencoders have shown promising results, their feature extraction capabilities are limited compared to state-of-the-art VLMs. This limitation arises from constraints posed by model parameters and the availability of training data.

VLMs are a class of large AI models capable of simultaneously processing both image and text information [13]. They find extensive application across various visual language tasks, encompassing image description, visual question answering, text-to-image generation, and other multimodal tasks. In [14], a contrastive loss function was utilized to train both image encoders and text encoders. This loss function aimed to minimize the feature space distance between matching image-text pairs, enabling the learning of semantically relevant visual language features while reducing the dependence on large amounts of annotated data. In [15], images were treated as prefixes in language models. They were decomposed into multiple blocks, concatenated with text sequences as input, and used to predict the subsequent parts of the text sequences. Furthermore, in [16], a cross-attention mechanism was employed to integrate visual and language features. This mechanism allowed the two modalities to reference and enhance each other, facilitating the learning of more comprehensive and refined visual language features. The approach demonstrated applicability to various downstream tasks.

VLMs aim to understand the correlation between images and text, enabling accurate visual description or image generation. Future research involves deep integration of self-supervised pre-training techniques and VLMs. This integration will help extract cross-modal relationships between visual and language features, providing a stronger foundation for downstream tasks.

Continual learning can effectively mitigate the problem of catastrophic forgetting in dynamic environments. In [17], the authors discuss continual learning in Mobile Edge Computing (MEC) networks, focusing on age-aware optimization for data selection and aggregator placement. They also present a prototype implementation involving diverse user equipment and cloudlets. In [18], the authors propose a continual learning digital predistortion algorithm for linearizing radio frequency power amplifiers in 6G wireless communications. The algorithm demonstrates effectiveness in adapting to both new and known operating states with low long-term complexity. In [19], the authors address the challenge of forgetting tasks in cross-edge federated learning by preserving past knowledge through continual learning. They achieve enhanced accuracy across various tasks with minimal storage cost. Furthermore, in [20], the authors employ continual learning to enable adaptive downlink beamforming optimization in dynamic environments. The proposed approach addresses task mismatch and exhibits good adaptability with low complexity.

Recent advancements in continual learning have been directed towards more challenging scenarios, specifically those where task boundaries are unknown. In these contexts, researchers have focused on develo** sample selection strategies to identify which samples should be stored in the buffer for model training. This approach aims to improve the efficiency and effectiveness of continual learning in handling unknown task boundaries.

The input to the transmitter is an image represented by the matrix $\bm{\mathrm{x}}\in\mathbb{R}^{H\times W\times C}$, whose size is $H(height)\times W(weight)\times C(channel)$. In the transmitter, the input image $\bm{\mathrm{x}}$ is mapped to symbols $\bm{\mathrm{y}}$ for transmission over the physical channel. The transmitter consists of three independent components: a CKB for cross-modal semantic extraction, a semantic encoder, and a channel encoder. The CKB is used to extract semantic information from the image and represent it as the corresponding textual information. The semantic encoder and channel encoder are responsible for semantic coding, and channel coding and modulation, ensuring that the encoded semantic information can be smoothly transmitted over the physical channel. The encoded symbol sequence $\bm{\mathrm{y}}$ can be represented as:

where $K_{\theta}(\cdot)$ is the CKB with the parameter set $\theta$, $S_{\alpha}(\cdot)$ is the semantic encoder with the parameter set $\alpha$, and $C_{\beta}(\cdot)$ is the channel encoder with the parameter set $\beta$, $\mu$ is the channel SNR that can be estimated and fed back to the semantic encoder and channel encoder.

The transmitter sends encoded symbols $\bm{\mathrm{y}}$, which is transmitted through the physical channel to the receiver. The channel output sequence $\bm{\mathrm{\hat{y}}}$ at the receiver can be expressed as:

where $\bm{\mathrm{h}}$ represents the channel gain, and $\bm{\mathrm{n}}$ is Additive White Gaussian Noise (AWGN).

Similar to the transmitter, the receiver consists of three components: a channel decoder, a semantic decoder, and a cross-modal knowledge base for semantic reconstruction. The semantic decoder and channel decoder are used to decode textual information from received symbols, while the cross-modal knowledge base is employed for image reconstruction based on the corresponding textual information. The decoded image can be represented as:

where $C_{\gamma}^{-1}(\cdot)$ is the channel decoder with the parameter set $\gamma$, $S_{\delta}^{-1}(\cdot)$ is the semantic decoder with the parameter set $\delta$ and $K_{\theta^{\prime}}^{-1}(\cdot)$ is the cross-modal knowledge base with the parameter set $\theta^{\prime}$.

For the purpose of reconstructing image information from the semantic level, maintaining the consistency of textual semantics between $\bm{\mathrm{s}}$ and $\bm{\mathrm{\hat{s}}}$ is crucial. Here, $\bm{\mathrm{s}}=K_{\theta}(\bm{\mathrm{x}})$ represents the extracted textual semantic information from the image, and $\bm{\mathrm{\hat{s}}}=S_{\delta}^{-1}(C_{\gamma}^{-1}(\bm{\mathrm{\hat{y}}},\mu),\mu)$ represents the recovered textual semantic information after decoding. We utilize Cross-Entropy (CE) as the loss function:

where $q(w_{l})$ denotes the real probability of the appearance of the $l$-th word $w_{l}$ in the sentence $\mathbf{s}$, and $p(w_{l})$ represents the predicted probability of the appearance of the $l$-th word $w_{i}$ in the sentence $\bm{\mathrm{\hat{s}}}$. CE is employed to measure the difference between two probability distributions. By minimizing the CE loss, the semantic encoder and decoder can learn the word distribution $q(w_{l})$ in the source sentence $\bm{\mathrm{s}}$, which represents the meaning of words in terms of grammar, phrases, and contextual information. Hence, the goal pf the CSC system is to determine the parameters of the semantic/channel encoder and decoder ${\alpha}^{\ast}$, ${\beta}^{\ast}$, ${\delta}^{\ast}$ and ${\gamma}^{\ast}$ that minimize the expected distortion as follows:

where ${\alpha}^{\ast}$ is the optimal semantic encoder parameters, ${\beta}^{\ast}$ is the optimal channel encoder parameters, ${\gamma}^{\ast}$ is the optimal channel decoder parameters, and ${\delta}^{\ast}$ is the optimal semantic decoder parameters. $p(\mathbf{s},\bm{\mathrm{\hat{s}}})$ represents the joint probability distribution of the $\bm{\mathrm{s}}$ and $\bm{\mathrm{\hat{s}}}$, and $p(\mu)$ represents the probability distribution of the SNR.

The BLIP model, introduced by Salesforce AI Research, is a sophisticated VLM designed for understanding and generating content that involves both visual and textual elements [22]. The BLIP model possesses rich visual-linguistic knowledge and utilizes multiple knowledge components such as text encoders, image encoders, and image-grounded text decoders and decoders to perform various visual-linguistic tasks, such as image captioning, visual question answering, and multimodal classification. At the transmitter, we employ the BLIP model to construct the CKB and utilize the image encoder and image-grounded text decoder (abbreviated as text decoder) in the CKB to transform original image data into detailed textual descriptions containing image semantic information. The workflow of the BLIP-based CKB is illustrated in Fig. 3.

For a given image $\bm{\mathrm{x}}$, the process of extracting semantic information from image data and generating textual representation $\bm{\mathrm{s}}$ is as follows:

The SD model is an elaborate VLM collaboratively developed by Stability AI, which possesses rich visual-linguistic knowledge and is applicable to diverse tasks such as text-to-image and image-to-image generation [26]. At the receiver, we use the SD to construct the CKB and utilize the text-to-image components in the CKB to reconstruct images. The semantic reconstructor is composed of a text encoder, a feature generator, and an image decoder.

For a given semantic text $\bm{\mathrm{\hat{s}}}$, the image reconstruction process through the SD model is illustrated in Fig. 4 and is described as follows:

In dynamic environments, both the distribution of the transmitted contents and channel states will change over time. This necessitates that the CSC system continuously adjusts based on new input data and channel states to adapt to the evolving data distribution. However, such adjustments may lead to parameter updates in the encoder and decoder of the CSC system, potentially causing the catastrophic forgetting issue where old parameter values are overwritten or ignored [5]. Hence, continual learning diminishes the robustness of the encoder and decoder in the CSC system.

The memory-based learning strategy addresses the catastrophic forgetting problem in continual learning by diversifying the memorized content [27]. We design a MED method with STM and LTM for both semantic encoder and decoder. Below, we present the workflow of the MED as follows:

We denote $\mathcal{M}_{stm}=\{\bm{\mathrm{s}}_{i}^{stm}\}_{i=1}^{n_{stm}}$ and $\mathcal{M}_{ltm}=\{\bm{\mathrm{s}}_{j}^{ltm}\}_{j=1}^{n_{ltm}}$ as the sets representing dynamic samples stored in STM and LTM. $\bm{\mathrm{s}}_{i}^{stm}$ denotes the $i$-th sample in STM, and $\bm{\mathrm{s}}_{j}^{ltm}$ represents the $j$-th sample in LTM. $n_{stm}$ and $n_{ltm}$ denote the current number of samples, respectively. When the STM pool becomes full, it is necessary to select representative samples from it and transfer them to the LTM. Hence, let $n_{stm}^{Max}$ represent the maximum number of samples that can be stored in $\mathcal{M}_{stm}$. The sample selection process can be illustrated in Fig. 5 and described as follows:

Inspired by the feature attention module in [7], we propose a NAM based on SNR values. The NAM leverages a new noise attention network to determine the importance of each feature vector during the process of encoding and decoding, assigning weights to semantic coding and channel coding. This allows for achieving integrated encoding of both semantic and channel information according to the current SNR.

Specifically, in unfavorable channel conditions, higher weights are allocated to the channel encoder and lower weights are allocated to the semantic encoder for the same source information. This allocation strategy enhances robustness in the channel encoder to mitigate the effects of severe channel noise. Conversely, in favorable channel conditions, lower weights are assigned to the channel encoder and higher weights are assigned to the semantic encoder for the same source information. This increased allocation of weights to the semantic encoder aims to enhance semantic quality.

The structure of the NAM is illustrated in Fig. 6, and a detailed description of the workflow is provided below:

The datasets employed in this study include publicly available Kaggle datasets such as CIFAR, BIRDS, CATSvsDOGS, and EPPs [28]. The configuration of the experiments is detailed as follows:

The pretrained BLIP has 129MB parameters, and the pretrained SD model has 1.99GB parameters. The semantic encoder comprises three transformer encoder layers alternated with NAMs. Each transformer encoder layer has 8 heads and the feature dimension is 128. The channel encoder is composed of two FF hidden layers alternating with NAMs, where the first hidden layer has 256 neurons and the second FNN layer has 128 neurons. To maintain information consistency, the semantic and channel decoder employs a structure opposite to that of the encoder. In NAM, the four FF layers have neuron quantities of 56, 128, 56, and 56, respectively. Additionally, the maximum sample size for STM is 500, and the threshold for sample selection is 0.05.

The experimental training and testing environment involves the Windows 2016 server with Python3.8, PyTorch 1.8.0 and CUDA 11.6. Computational resources are provided by an Intel(R) Xeon(R) Silver 4210R CPU @ 2.40GHz and NVIDIA Tesla T4.

The proposed VLM-CSC system transforms image data to textual semantic data through the BLIP-based knowledge base, encodes it using a semantic encoder, decodes it at the receiver, and finally reconstructs the image through the SD-based knowledge base. To assess the performance of the VLM-CSC system, two corresponding metrics are designed: (1) Image-level, examining the accuracy of semantic reconstruction for image data; (2) Text-level, examining the accuracy of semantic recovery for text data.

To evaluate the performance of extracting semantic information from images using KBs, we employ three VLMs (BLIP, LEMON[31], and RAM[32]) to construct the sender-side KBs in the CSC system. The receiver-side KB is uniformly implemented using the SD model. Subsequently, we assess the CSC system’s performance on the AWGN channel. SSQ is utilized as the evaluation metric on the CATSvsDOGS dataset [28]. The experimental outcomes are illustrated in Fig. 7.

From Fig. 7, it is evident that the CSC system based on BLIP exhibits the highest SSQ, followed by the one based on LEMON, while the CSC system based on RAM performs the poorest, significantly lower than the CSC systems based on BLIP and LEMON. Furthermore, the CSC system based on BLIP maintains robust performance even at low SNR values. The experimental results indicate that the CSC system constructed based on BLIP accurately extracts image semantics and sustains commendable performance across different SNR levels.

To demonstrate the performance of the proposed MED, we conduct experiments comparing VLM-CSC with the MED module against VLM-CSC without the MED module. The evaluation is performed across different image datasets. The image datasets include Cifar, Birds, and CatsVSDogs [28].BLEU scores for semantic similarity serve as the evaluation metric. Additionally, when assessing the performance of VLM-CSC on image datasets with different distributions, the channel is fixed to Rayleigh. The continual learning map, originally proposed by Google, is employed to visualize the performance changes of existing tasks when a new task is introduced. The experimental results are illustrated by the continual learning map in Fig. 8.

Figure Fig. 8 (a) and (b) illustrate a significant performance drop in the VLM-CSC system without the MED module on the previous Cifar dataset after learning subsequent datasets such as Birds and CatsVSDogs. In contrast, Fig. 8 (c) and (d) reveal that the VLM-CSC system with the MED module only exhibits a marginal decline in performance on the previous Cifar dataset after learning subsequent datasets like Birds and CatsVSDogs.

The experimental results from Fig. 8 underscore that the proposed MED module enables the CSC system to overcome catastrophic forgetting during the continual learning process. This facilitates knowledge learning from multiple image datasets, enhancing the generalization of the CSC system in dynamic environments.

To demonstrate the performance of the proposed NAM, we conduct an experimental comparison between VLM-CSC with and without NAM. Semantic similarity, measured by BLEU score, serves as the evaluation metric. Specifically, the proposed VLM-CSC system is trained under a uniform distribution of $SNR_{train}$ ranging from 0 dB to 10 dB, while the VLM-CSC system without NAM is trained at specific $SNR_{train}$ values of 1 dB, 4 dB, 7 dB, and 10 dB. Subsequently, the performance of the VLM-CSC system is evaluated at specific $SNR_{test}$ values ranging from 0 dB to 10 dB. The experimental results are depicted in Fig. 9.

The findings depicted in Figure 9 demonstrate that the performance of the proposed VLM-CSC system outperforms any VLM-CSC system without NAM, specifically trained at distinct $SNR_{train}$ values. This observation highlights the capability of the VLM-CSC system, equipped with NAM, to address the performance degradation challenges caused by the mismatch between the SNR during training and deployment stages in conventional ISC systems. This improvement contributes to the robustness of the VLM-CSC system across different SNR values.

To evaluate the performance of the VLM-CSC system in image classification tasks, we compare it with JSCC based on CNN [33] and WITT based on ViT [34]. The metric used for performance evaluation is classification accuracy. Additionally, we assess the bandwidth-saving capabilities of VLM-CSC by considering the compression ratio between transmitted data and original images as the evaluation metric. The experimental results are presented in Fig. 10.

Fig. 10 (a) clearly demonstrates that, at low SNR levels, the superior performance of VLM-CSC in the classification task with the CATSvsDOGS dataset, and WITT shows slightly lower results, particularly with decreased performance compared to VLM-CSC. At high SNR levels, WIIT and JSCC exhibit superior SSQ compared to VLM-CSC due to their direct transmission of images. Fig. 10 (b) depicts the compression ratio and trainable parameters, with VLM-CSC achieving the lowest of all, followed by JSCC, while WITT attains the highest compression ratio and trainable parameters. Fig. 10 (c) illustrates that the reconstructed image highly aligns with the original image and the image description, validating the VLM-CSC system’s ability to ensure semantic consistency across modalities.

The experimental results depicted in Fig. 10 demonstrate that the proposed VLM-CSC exhibits overall superior performance in image classification tasks compared to other ISC systems at low SNR levels. Then, the compression ratio of transmitted data is significantly lower for VLM-CSC compared to other ISC systems, indicating that VLM-CSC can effectively conserve transmission bandwidth while preserving high-quality semantic transmission. Moreover, due to the absence of training VLMs, the VLM-CSC system exhibits the minimum number of trainable parameters, resulting in the lowest training complexity.