CoAVT: A Cognition-Inspired Unified Audio-Visual-Text Pre-Training Model for Multimodal Processing

Audio and video are two naturally aligned and closely related modalities of human perception, offering different but complementary information. Inspired by previous work [16, 15, 17], our joint audio-visual encoder incorporates an audio-visual encoder pair, and a shallow joint encoder layer. This design is intended to encode audio-visual synchronization information together with the audio and visual content, promoting enhanced cross-modal understanding and representation learning.

Given a video and its corresponding audio, we follow the pre-processing and tokenization in AST [29] and ViT [30] for audio and image inputs, respectively. Specifically, for audio, all audio clips are first randomly cropped or padded to 10 seconds, and then the 128-dimensional log Mel-filterbank features are extracted with a 25ms Hanning window every 10ms, which results in a 1024 $\times$ 128 spectrogram. We then split the spectrogram into 16 $\times$ 16 square patches $\mathbf{a}=[a^{1},a^{2},\cdots,a^{512}]$ as the input of the audio encoder. For video, we uniformly sample 10 RGB frames from each video, and randomly select one frame as input during pre-training. For each frame, we resize and center crop it to 224 $\times$ 224 size, and then split it into 16 $\times$ 16 square patches $\mathbf{v}=[v^{1},v^{2},\cdots,v^{196}]$ as the input of the visual encoder. The joint encoder employ a multi-stream forward pass strategy, in which we input the output of the audio encoder $E_{A}^{{}^{\prime}}$, the output of the visual encoder $E_{v}^{{}^{\prime}}$, and the concatenated audio-visual representation $[E_{A}^{{}^{\prime}},E_{V}^{{}^{\prime}}]$ in three independent forward passes and obtain the final audio-only embedding $E_{A}$, visual-only embedding $E_{V}$, and joint audio-visual embedding $E_{AV}$.

We use BERT${}_{base}$ [31] as our text encoder. Given a text input of $N$ tokens, the text encoder outputs an embedding sequence $\{t_{cls},t_{1},\cdots,t_{N}\}$, with $t_{i}\in\mathbb{R}^{d}$ and $t_{cls}$ is the embedding of the text [CLS] token. Following BERT, we choose the $t_{cls}$ as the global text representation.

To mitigate the modality gap, we propose to use the query encoder as a bridge between the joint audio-visual encoder and text encoder, and align the three modalities. The query encoder inserts one additional cross-attention layer between the self-attention layer and the feed-forward network for each transformer block of the text encoder. We first create a set number of predetermined learnable query embeddings as input to the query encoder. The queries can interact with each other through self-attention layers. More importantly, these queries can further interact with the embeddings (e.g., $E_{A},E_{V},E_{AV}$) from the joint audio-visual encoder through cross-attention layers, enabling better alignment between modalities. Except for cross-attention layers, the query encoder shares parameters with the text encoder, thereby these queries can additionally interact with the text through the same self-attention layers as well. The output of query encoder is $Q\in\mathbb{R}^{N_{q}\times C}$, where $N_{q}$ is the number of queries and $C$ is the hidden size. For the inputs of $E_{A},E_{V},E_{AV}$, the outputs of query encoder are $Q_{A},Q_{V},Q_{AV}$, respectively.

We first build the coarse-grained alignment between modality $X$ and text via $X$-text contrastive learning, where $X$ represents different modalities including audio (A), visual (V) and joint audio-visual (AV). It aims to align the embedding space of the query encoder and the text encoder by encouraging positive $X$-text pairs to have similar representations in contrast to the negative pairs.

Formally, we align the output query representation $Q_{X}$ with the text representation $t_{cls}$. Since $Q_{X}$ contains multiple output embeddings (e.g., 16), we first compute the pairwise similarity between each query output $q_{i}$ and $t_{cls}$, and then select the highest one as the similarity score:

where $g_{q}(\cdot)$ and $g_{t}(\cdot)$ are linear projections that transforms the query embedding and [CLS] embedding to a common normalized low-dimensional space. The $X$-text contrastive loss consists of two symmetric terms, one for $X$-to-text classification:

and the other for text-to-$X$ classification:

where $\tau$ is a learnable temperature parameter, and $B$ is the batch size. The final X-text contrastive loss is then denoted as: $\mathcal{L}_{XTC}=\frac{1}{2}(\mathcal{L}_{X2T}+\mathcal{L}_{T2X})$.

$X$-text matching (XTM) is a binary classification task where the model is asked to predict whether $X$-text is matched or unmatched given the corresponding query features and text features. Here we employ a bi-directional self-attention mask so that all queries and texts can attend to each other, which enables the query encoder to effectively capture the multimodal information between the text and modality $X$ through the query embeddings. Finally, we feed each output query embedding into a two-class linear classifier to obtain a matching probability $p_{i}$, and then average the probabilities across all queries as the final matching score:

where $y$ is 1 when the input $X$-text pair is matched and 0 otherwise.

Language modeling aims to generate text given the representations of modality $X$ as the condition. Specifically, as there exists no direct interactions between the text encoder and the joint audio-visual encoder, the information required for generating the text must be first extracted through the query encoder, and then passed to the text tokens via self-attention layers. During this process, we employ a multimodal causal self-attention mask to control the interaction between the queries and text, in which the queries can only attend to each other but not the text tokens, and each text token can attend to both all queries and its preceding tokens. This masking strategy ensures a coherent and effective flow of information from the queries to the text, enabling the generation of contextually relevant and coherent text given the modality $X$ representations. This casual language modeling loss with modality $X$ as condition can be formulated as:

where $D$ denotes the training batch, $y_{i}$ denotes the current predicted token, and $y_{\textless i}$ represents the previous predicted tokens. The final pre-training objective for modality $X$ is the sum of above three objectives:

Besides the foundational audiovisual-text tri-modal alignment, we further utilize the correspondences between audio and vision with language (e.g., audio-text and visual-text) to enhance the audiovisual-text alignment, thereby enabling better multimodal representation learning. Therefore, the overall pre-training objective of our CoAVT consists of three pair-wise losses based on Eq. (6):

Video retrieval aims to retrieve the relevant video segment given a free form natural language query. Unlike most existing video retrieval methods that solely focus on aligning text with visual elements and disregard audio information, our proposed model, empowered with tri-modality encoding ability, enables a holistic exploration of both visual and audio information for text-to-video retrieval. Moreover, we also consider audio retrieval for downstream evaluation. During the fine-tuning process, we adhere to the same objectives as pre-training. Without encoding audio information, existing video retrieval works only focus the matching between text and vision modality. Benefiting from the tri-modality encoding ability of our model, we fully explore both vision and audio information in the video for text-to-video retrieval.

Besides the retrieval task, audio-visual event classification is another challenging task on video understanding, which requires good joint audio-visual representation. To conduct classification, we apply average pooling on top of the query encoder followed by a randomly initialized linear layer. Specifically, we fine-tune the model using audio-only data (A), video-only data (V), and audio-visual data (AV) to evaluate the single modal and multi-modal representation quality.

For pre-training, we use the publicly available large-scale multimodal dataset AudioSet [32]. It contains over 2 million 10-second YouTube video clips, and each clip is labeled with event labels from a set of 527 distinct labels in a non-exclusive way. After filtering out those unavailable data, we finally downloaded 1,450,529 audio-video-text pairs for training. To generate coherent captions from discrete labels, we opt for a simply concatenation without any prompt.

We evaluate the pre-trained CoAVT on retrieval and classification benchmarks, including AudioCaps [33], Clotho [34], AudioSet-20K [32] and VGGSound [35].

• •

  AudioCaps is an audio-centric video dataset, whose videos are mainly in event scenarios with duration shorter than 10 seconds from YouTube. Each training sample contains one caption, while five captions per sample are used in validation and test sets. We use this dataset for text-video retrieval task on both zero-shot and fine-tuning settings. After filtering out the videos that are no longer available, we finally obtain 32,747 training, 442 validation, 753 test samples.

• •

  Clotho is an audio-only dataset of described sounds, which are sourced from Freesound platform [36]. This dataset consists of a development set and evaluation set of 2893 and 1045 audio samples respectively, and every audio sample is accompanied by 5 captions. For fair comparison, we follow [37] and treat each of the 5 captions per test audio as a separate query. We use this dataset to validate the generalization ability of our CoAVT on text-to-audio retrieval.

• •

  AudioSet-20K and VGGSound  For the audio-visual event classification task, we conduct experiments on AudioSet-20K [32] and VGGSound [35]. AudioSet-20K is a subset of AudioSet-2M with a more balanced class distribution. We downloaded 18,063 training and 16,690 evaluation samples. VGGSound is a collection of 200K 10-second YouTube video clips annotated with 309 classes, and we downloaded 162,567 training and 13,483 test samples.

To demonstrate the effectiveness of our proposed CoAVT model, we first evaluate it for video retrieval on AudioCaps benchmark. During inference, we follow [8, 10], which first select $k=128$ candidates based on the similarity scores, then followed by a re-ranking based on pairwise matching scores. The results are provided in Table I. We mainly compare the CoAVT with three tri-modal pre-training methods, including 1) CLIP4VLA [1], which incorporates audio into VL pre-training framework CLIP and trains on both AudioSet and HowTo100M datasets; 2) VALOR [3], which uses three separate encoders and trains on four large-scale datasets, containing 27.5M samples in total; 3) AVR [17], which uses a dual-stream model but trains with simple contrastive loss between audiovisual and text features on VideoCC3M dataset. For fair comparison, we build another two baseline models: $Baseline$ model use the same objective as AVR but trains with our AudioSet dataset, while $Vanilla$ model employs a query encoder between the joint audiovisual encoder and text encoder, ignoring the bi-modal audio-text and visual-text alignments, and trains with three objectives on AudioSet.

Table I compares the video retrieval performances of different approaches on the AudioCaps corpus. The results show that the proposed CoAVT model achieves state-of-the-art performance with significant improvement over existing methods in both zero-shot and fine-tuned settings, either with only audio representations or joint audiovisual representations. When compared to the strong baseline AVR model, our $Baseline$ model obtains similar performance under the zero-shot setting (e.g., 10.6/45.3 $\rightarrow$ 10.5/48.7), despite that it is pre-trained on a much larger video-text dataset [17], which contains more than 9M samples. Furthermore, our $Vanilla$ model, which employs the query encoder to address the modality gap, outperforms the AVR model (e.g., 10.6/45.3 $\rightarrow$ 14.3/57.6), which demonstrates the effectiveness of the query encoder. Finally, when our final model is further equipped with audio-text and video-text correspondences during pre-training, we obtain the state-of-the-art retrieval results. This indicates that our model is capable of learning more effective semantic representations across audio, vision and text modalities.

We further evaluate our model by conducting audio-only retrieval on the Clotho dataset. As the results shown in Table III, our CoAVT model outperforms the AVR model, but falls short of the VALOR model. It should be noted that VALOR model uses a significantly larger dataset during pre-training, with up to 27.5M samples, compared to the 1.4M samples used in the pre-training of our CoAVT model. Despite this difference in pre-training sample size, the CoAVT model still demonstrates competitive performance, underscoring its efficiency and potential.

To further validate whether our CoAVT model could learn a good joint audio-visual representation, we conduct audio-visual event classification experiments. Table II summarizes the performance comparison on AudioSet and VGGSound datasets. We report accuracy for fine-tuning using the audio(A), video(V) and joint audio-visual representation (A+V). Our CoAVT sets new state-of-the-art performance on audio-only and audio-visual classification on both AudioSet-20K and VGGSound datasets. As shown in the table, our $Vanilla$ model already outperforms recent competitive models, such as CAV-MAE [16] and MAViL [15] by a large margin (e.g., 42.0/42.5 $\rightarrow$ 45.0 and 65.5/63.8 $\rightarrow$ 66.2) on audio-visual based event classification. Furthermore, when we include the audio-text and visual-text bi-modal information during pre-training, our CoAVT also surpasses these models on audio-only based event classification (e.g., 37.7/39.0 $\rightarrow$ 42.5 and 59.5/59.9 $\rightarrow$ 60.7). This indicates that our CoAVT model not only captures a good joint audio-visual representation, but also effectively learns audio-only representation.

These results on video retrieval and audio-visual event classification validate the effectiveness of our CoAVT model in handling multimodal understanding tasks and highlight its potential for improving performance in both audio-visual and audio-only scenarios. The inclusion of bi-modal audio-text and visual-text bi-modal correspondences during pre-training provides additional benefits, indicating that the integration of bi-modal information can lead to more robust and versatile multimodal representations.

In this section, we further investigate whether our CoAVT model also learns a well-coordinated representation that captures audio-visual correspondence via text for audio-visual retrieval task. For fair comparison, we use the same subset of AudioSet and VGGSound evaluation set as in  [16], which includes 1,725 and 1,545 audio-visual samples from the AudioSet and VGGSound evaluation sets, respectively. Specifically, we input audio and image to the model in two independent forward passes and take the mean-pooled query encoder outputs as audio and visual representation, respectively. We then calculate the retrieval recall at rank 1,5 and 10 (R@1, R@5, R@10) based on the cosine similarity of the audio and visual representation. The results are shown in Table IV. As demonstrated in the table, compared to CAV-MAE, which learns the joint audio-visual representation through self-supervised learning, our model could leverage additional textual information as a bridge and achieves significantly better audio-visual retrieval performance. This result indicates that our CoAVT model is not only capable of learning high-quality joint audio-visual representations, but also excels in coordinating these representations to capture audio-visual correspondences effectively. The use of text as a bridge between audio and visual modalities enhances the model’s performance in the audio-visual retrieval task.

The learnable query embeddings of the query encoder plays an important role in extracting the audio-visual features that are informative of the corresponding text. To investigate the effect of the number of queries, we vary the number from 8 to 32 during pre-training based on our $Vanilla$ model. The results are shown in Table V. From these results, it becomes clear that the number of queries has a significant impact on the model’s performance. The model achieves the best performance when the number of queries is set to 16. Interestingly, when the number of queries is increased from 16 to 32, the performance degrades dramatically. This might be due to that increasing the number of queries will introduce more noise and redundancy in the model. These additional queries may capture irrelevant or redundant information, which can interfere with the learning process and hinder the model’s ability to extract meaningful multimodal representations, ultimately leading to a decrease in performance.

Masking is a crucial technique used during pre-training, which involves randomly hiding a portion of the input data and prompting the model to predict the masked data based on the context provided by the unmasked data. This encourages the model to learn robust and generalizable representations of the data. In our CoAVT model, we perform both audio and video masking during pre-training. To investigate the impact of masking, we conduct an ablation study by varying the masking ratio. As shown in Table VI, the model yields best performance when the masking ratio of audio is 0.75 and the masking ratio of video is 0.5. Compared to the models that do not use any masking, the model with masking not only makes the pre-training process more efficient but also improves the generalization ability of the model.

To enhance the multimodal representation learning, we introduce the audio-text and visual-text bi-modal alignments in addition to the base audiovisual-text tri-modal alignment in the CoAVT model. These alignments aim to capture the rich semantic correspondences between different modalities, thus improving the model’s ability to understand and integrate multimodal information. In order to evaluate the contribution of audio-text and visual-text alignment separately, we conduct an ablation study where we remove these alignments one by one during pre-training. The results are shown in Table VII. When we remove the visual-text alignment, the overall performance drops, especially in the video-only based retrieval task. This indicates that the visual-text alignment plays a crucial role in capturing the semantic correspondences between visual and textual data, which is particularly important for tasks that rely heavily on visual information. When we further remove the audio-text alignment, the performance of retrieval tasks degrades accordingly. This suggests that the audio-text alignment also contributes significantly to the model’s performance, hel** to establish meaningful correspondences between audio and textual data.

Overall, these results reveal that the bi-modal correspondences help the model capture rich semantic representations between modalities, leading to better performance in downstream tasks. This underscores the importance of incorporating bi-modal alignments in multimodal representation learning models.

In Figure 3, we show some qualitative results of our CoAVT and the $baseline$ model on video-to-text retrieval task. Specifically, we present the query videos with matched texts. Our CoAVT model demonstrates a stronger cross-modal video-text understanding capability compared to the baseline model. For example, consider the second video in the figure. While the baseline model returns a more generic or less accurate description, the CoAVT model returns the text “an aircraft motor”, which is a more specific and accurate description of the audiovisual content in the video. These qualitative results suggest the superiority of our CoAVT model over the baseline model in the downstream task. This reinforces the effectiveness of our approach, which leverages bi-modal and tri-modal alignments for enhanced multimodal representation learning. The results also highlight the model’s ability to capture fine-grained semantic correspondences between video and text, which is crucial for various downstream tasks.