Conceptual Codebook Learning for Vision-Language Models

Abstract

In this paper, we propose Conceptual Codebook Learning (CoCoLe), a novel fine-tuning method for vision-language models (VLMs) to address the challenge of improving the generalization capability of VLMs while fine-tuning them on downstream tasks in a few-shot setting. We recognize that visual concepts, such as textures, shapes, and colors are naturally transferable across domains and play a crucial role in generalization tasks. Motivated by this interesting finding, we learn a conceptual codebook consisting of visual concepts as keys and conceptual prompts as values, which serves as a link between the image encoder’s outputs and the text encoder’s inputs. Specifically, for a given image, we leverage the codebook to identify the most relevant conceptual prompts associated with the class embeddings to perform the classification. Additionally, we incorporate a handcrafted concept cache as a regularization to alleviate the overfitting issues in low-shot scenarios. We observe that this conceptual codebook learning method is able to achieve enhanced alignment between visual and linguistic modalities. Extensive experimental results demonstrate that our CoCoLe method remarkably outperforms the existing state-of-the-art methods across various evaluation settings, including base-to-new generalization, cross-dataset evaluation, and domain generalization tasks. Detailed ablation studies further confirm the efficacy of each component in CoCoLe.

1 Introduction

Figure 1: (a) Visualization of the chosen prompts of the same image. (b) Visualization of the same prompts on different images. Grad-CAM [30] is used for the visualization.

Pre-trained Vision-Language Models (VLMs), e.g., CLIP [26] and ALIGN [15], have achieved exceptional zero-shot performance in various downstream tasks. Trained on large-scale image-text datasets with contrastive optimization objectives, these models effectively align and embed different modalities into a shared vector space. Despite their impressive performance, adapting these models to diverse downstream tasks remains challenging due to their substantial size. As a result, recent research has concentrated on improving the adaptation of pre-trained VLMs to downstream tasks by adjusting additional parameters while kee** the foundation model fixed. Prompt-tuning methods, e.g. CoOp [42] and ProGrad [43], replace manual prompts with learnable ones to obtain task-specific knowledge, while adapter-based methods utilize extra modules directly on the top of VLMs, such as Clip-adapter [9] and Tip-adapter [38]. These methods have made significant advancements with limited labeled data.

However, we observe that the current fine-tuning methods for CLIP, such as CoOp [42] and CPL [40] demonstrate relatively low performance on fine-grained datasets such as FGVCAircraft [21] (aircraft classification), and UCF101 [31] (action classification). To address the challenge of improving the generalization capability of VLMs in a few-shot settings, in this paper, we propose a novel fine-tuning method called Conceptual Codebook Learning (CoCoLe). Our idea stems from the observation that visual concepts are naturally transferable across domains. As illustrated in Fig. 1, within a single image, there exist multiple distinct visual concepts focusing on different regions. For example, the selected prompts highlight the claws, ears of the koala, and the branches where the koala stands. Moreover, there are similar concepts in images from different classes; for example, the "firetruck", "racer", and "bicycle" classes possess the compound concept of "wheel" in common.

Motivated by this interesting finding, we propose to learn a conceptual codebook consisting of visual concepts as keys and conceptual prompts as values, which serve as a link between the image encoder’s outputs and the text encoder’s inputs. Specifically, for a given image, we leverage the codebook to identify the most relevant conceptual prompts associated with the class embeddings to perform the classification. Additionally, we incorporate a handcrafted concept cache as a regularization to alleviate the overfitting issues in low-shot scenarios. As shown in Fig. 2, we observe that this conceptual codebook learning method can achieve enhanced alignment between visual and linguistic modalities. Our contributions could be summarized as:

• We proposed a novel fine-tuning method named CoCoLe for vision-language models to solve the problem of performance degradation on generalization tasks.

• CoCoLe introduces a conceptual codebook to adaptively learn visual concepts and their corresponding conceptual prompts with regularization to further guarantee the generalization capability.

• Extensive experimental results demonstrate the outstanding performance of the proposed CoCoLe compared to existing state-of-the-art methods in base-to-novel generalization, cross-dataset transfer and domain generalization tasks.

Figure 2: Examples and accuracy comparisons on base-to-novel generalization, cross-dataset transfer and domain generalization tasks. S and T represent source and target datasets respectively.

2 Related Work

2.1 Vision-Language Models

In recent times, pre-trained vision-language models (VLMs) has emerged as a notable trend [19, 29, 5, 26]. Capitalizing on tremendous image-text data, these large-scale models can effectively acquire visual representations using contrastive loss, enabling them to grasp both visual and textual semantics and achieve successful modality alignment. Current studies [42, 37] have showcased that by harnessing extensive sets of image-text pairs, VLMs exhibit outstanding performance across a range of downstream visual tasks [13, 6, 16]. For example, Derived through contrastive learning on 400 million online image-text pairs, CLIP [26] demonstrated remarkable zero-shot accuracy on classification tasks. Our method aims to utilize the comprehensive capability of CLIP to perform knowledge-guide fine-tuning for better adaptation to downstream tasks.

2.2 Prompt Tuning for VLMs

As text input for pre-trained vision-language models, prompt functions as the guidance for the downstream tasks, thereby distilling task-relevant information from inherent knowledge of VLMs [41, 42]. Setting a precedent in this field, CoOp [42] performs end-to-end optimization on the prompt context by a set of learnable vectors but fails to generalize to unseen classes. To address this issue, CoCoOp [41] improved CoOp’s generalization by generating conditional prompts. Further, KgCoOp [36] enhances the generalization by minimizing the discrepancy between learned and handcrafted prompts, and CoPrompt [28] constrains the trainable models by pre-trained ones to avoid overfitting on the downstream task. Meanwhile, there are methods exploring diverse forms of prompts. MaPLe [17] tunes both vision and language branches via a coupling function to induce cross-modal synergy, whereas CPL [40] leverages the powerful generalization of CLIP to build a visual concept cache with a projector to capture multi-level visual features.

In our work, we mainly focus on prompt-tuning ways and meticulously manipulate learnable vectors by a learnable codebook with the regularization of a handcrafted concept cache. Among existing methods, the most related to ours are CoOp and CPL. Compared with CoOp, the proposed CoCoLe introduces an adaptive codebook rather than fixed to specific classes or tasks. On the other hand, CoCoLe leverages the transferability of concepts across domains, with optimal handcrafted concept-based prompts as a regularization to prevent the codebook from overfitting.

2.3 Visual Concept Learning

Previous research has identified two major approaches to visual concept learning. The first one typically uses manual concept annotations (e.g., colors, textures, and fabric) for the training images [24, 25], while the other method utilizes unsupervised learning to design data-driven concepts [8, 20, 14]. However, these approaches may introduce biases into the learned concepts, thereby limiting their overall performance. Recent studies have sought complementary prompting methods to capture unbiased visual concepts [34, 33]. Notably, CPL [40] first utilizes the capabilities of CLIP [26] to design an unsupervised concept cache. Furthermore, in this work, we introduce a learnable codebook supervised by a handcrafted concept cache, which automatically selects conceptual prompts that are aligned with visual concepts.

3 Method

In this section, we present the details of our method. First, we provide the background and an overview of our proposed CoCoLe. Second, we delve into the specifics of CoCoLe. Finally, we introduce the training and inference processes of CoCoLe.

Figure 3: An overview of the proposed CoCoLe. (a) shows the establishing process of handcrafted concept cache. (b) displays the handcrafted concept-based prompting process. (c) presents the training pipeline for CoCoLe. Within CoCoLe, only the keys and values in the Conceptual Codebook are learnable.

3.1 Background and Overview

CLIP and CoOp.

CLIP [26] is composed of two encoders: a visual encoder denoted as $E_{v}$ responsible for handling image input $x$, and a text encoder referred to as $E_{t}$ designed to process the corresponding textual prompt $t_{c}$ built as “a photo of $[\text{CLS}]_{c}$", where $[\text{CLS}]_{c}$ represents the word embedding for the class $c$. During training, CLIP learns to optimize the resemblance between the image feature and the prompt embeddings associated with the true label. CoOp [42] further replaces manual prompt construction with learnable prompts by utilizing a collection of $n$ adaptable context vectors $\{[V_{1}],[V_{2}],\cdots,[V_{n}]\}$, each having the same dimension as word embeddings. These vectors are iteratively updated through gradient descent. For a specific class $c$, the respective prompt can be represented as $t_{c}=\{[V_{1}],[V_{2}],\cdots,[V_{n}],[\text{CLS}]_{c}\}$.

Overview of CoCoLe.

In Fig. 3, we present an overview of our proposed CoCoLe method. Figure 3 (a) shows the establishing process of handcrafted concept cache. We first construct a list of text concepts $\Omega_{t}$ that describes major visual concepts. Then we leverage CLIP’s robust text-image correlation capability to discover the image feature $v_{j}$ with Top-$K_{1}$ highest similarity score for each text concept feature $c_{t}^{i}\in C_{t}$. These Top-$K_{1}$ "matched" features are averaged and stored in the visual concepts cache as keys, with their corresponding text concepts $\omega_{i}\in\Omega_{t}$ as values. Figure 3 (b) shows the handcrafted concept-based prompting process: we first extract the image feature $v$ by $E_{v}$, then use the image feature as the query to find Top-$K_{2}$ similar keys using cosine distance, and finally we utilize the corresponding values with the class embeddings as input for LLM (e.g., GPT [2]) to generate the optimal handcrafted concept-based prompts denoted as $\mathcal{P}^{h}\triangleq\{P_{h_{i}}\}_{i=1}^{N^{C}}$, where $N^{C}$ is the number of class.

Figure 3 (c) presents the training pipeline for CoCoLe. We first extract the visual features $f_{v}$ for a given image $x$ using the visual encoder, Next, we follow (b) to generate the handcrafted concept-based prompts $\mathcal{P}^{h}$ and extract text features by $E_{t}$, denoted as $\mathcal{F}_{h}=E_{t}(\mathcal{P}^{h})$. The visual concepts (keys) and the conceptual prompts (values) in the conceptual codebook are trainable parameters optimized by four loss functions. $\mathcal{L}_{ce}$ is the classification loss adopted to maximize the similarity between image feature $f_{v}$ and the corresponding text features $f_{t}$. $\mathcal{L}_{ma}$ is designed to shorten the distance between the selected keys (Top-$K_{3}$ similar visual concepts) and the image feature $f_{v}$, enabling the keys to learn generalizable concepts. $\mathcal{L}_{cc}$ works as a regularization for diminishing the overfitting problem, ensuring the text features produced by the selected learned prompts do not deviate significantly from those generated by the handcrafted concept-based prompts. Finally, $\mathcal{L}_{or}$ makes the text features of the prompts orthogonal to increase the diversity of the prompts.

3.2 Conceptual Codebook Learning (CoCoLe)

Learnable Conceptual Codebook.

In the CoOp framework, each class embedding is associated with a single set of prompt vectors. Nevertheless, images belonging to the same class often encompass a variety of concepts. Conflating these varied concepts into a single set of prompts can result in significant knowledge loss. Moreover, the encoded information within CoOp’s prompts lacks inter-class interaction, since concepts from one class may assist in identifying another class with similar concepts. For instance, when presented with an image of a cat in the tree, the concept of "in the tree" might also apply to images of other animals (e.g., a koala in the tree). We hypothesize that fine-tuning prompts based on image concepts can facilitate the learning of textual descriptions associated with these concepts, thereby improving generalization across datasets.

As such, we propose CoCoLe, as depicted in Fig. 3. The key insight of CoCoLe is a trainable concept codebook, empowering the image to autonomously determine the prompts it should learn based on its inherent concepts. For each training input, only a subset of prompts that align with the current image concepts are chosen and trained individually. The learnable concept codebook stores visual concepts as keys and conceptual prompts as values, which consists of $N$ (key, value) pairs, denoted as $\Psi_{cc}\triangleq\{(\mathbf{V}_{i},\mathbf{P}_{i})\}_{i=1}^{N}$, where $\Psi_{cc}$ denotes the learnable concept codebook, each $\mathbf{V}_{i}\!\in\!\mathbb{R}^{D}$ has the same dimensionality as the image feature $f_{v}$, and each $\mathbf{P}_{i}\!=\![\mathbf{p}_{i}]_{1}\dots[\mathbf{p}_{i}]_{M}\!\in\!\mathbb{R}^{D\times M}$ is composed of $M$ learnable vectors. Denoting the set of learnable visual concepts as $\mathcal{V}\!=\!\{\mathbf{V}_{i}\}_{i=1}^{N}$ and the set of all learnable conceptual prompts as $\mathcal{P}\!=\!\{\mathbf{P}_{i}\}_{i=1}^{N}$. In an optimal scenario, we anticipate that the image itself should determine the prompts to be selected, guided by the concepts it encompasses, in order to steer the prediction process. To this end, given an input image $\mathbf{x}_{j}$, we first obtain its image feature $f_{v_{j}}\!=\!E_{v}(\mathbf{x}_{j})$, where $j$ is the index of the image. Then we calculate the cosine similarity score between $f_{v_{j}}$ and $\mathbf{V}_{i}\in\mathcal{V}$, denoted as,

$$S_{c}=\frac{f_{v_{j}}\cdot\mathbf{V}_{i}}{||f_{v_{j}}||~{}||\mathbf{V}_{i}||},$$ (1)

Next, we choose the keys with Top-$K_{3}$ cosine similarity score to form set $\mathcal{V}_{j}$, which denotes the subset of Top-$K_{3}$ visual concepts selected from $\mathcal{V}$ specifically for the $j$-th image. We then choose the corresponding conceptual prompts that are paired with these visual concepts, denoted as $\mathcal{P}_{j}\!=\!\{\mathbf{P}_{j_{i}}\}_{i=1}^{K_{3}}$, where $\mathbf{P}_{j_{i}}$ is the $i$-th prompt selected specifically for $\mathbf{x}_{j}$. These prompts are attached to the class name embedding of $\mathbf{x}_{j}$ as illustrated in Fig. 3, and the text input for text encoder can be denoted as, $\mathbf{T}(\mathcal{P}_{j})=\operatorname{concat}(\mathbf{P}_{j_{1}};\dots;\mathbf{P}_{j_{K_{3}}};[\text{CLS}]_{d}),$ where $\operatorname{concat}(\cdot)$ denotes concatenation. Therefore, given test image $\mathbf{x}_{j}$ and the prompts $\mathcal{P}_{j}$ selected according to the concepts of $\mathbf{x}_{j}$, the text feature $f_{t_{j}}$ can be obtained by $f_{t_{j}}\triangleq E_{t}(\mathbf{T}(\mathcal{P}_{j}))$. The probability of predicting the image as class $y_{i}$ is finally computed as:

$$\textbf{}p(y_{i}|\mathbf{x}_{j})=\frac{e^{\langle f_{v_{j}},f_{t_{j}}\rangle/\tau}}{\sum_{d=1}^{D}e^{\langle f_{v_{j}},f_{t_{d}}\rangle/\tau}}.\textbf{}$$ (2)

From a broader viewpoint, the suggested adaptable concept codebook serves as a link connecting the outcomes of the image encoder and the inputs of the text encoder. The keys are fine-tuned to closely align with the identified image features, which hold abundant high-level information, such as image concepts. Meanwhile, the prompts are refined to encompass textual details associated with the respective image concepts, facilitating improved guidance for the model predictions alongside the class name embeddings.

Handcrafted Concept Cache.

In Figure 3 (a), following [39], we start by constructing a comprehensive list $\Omega_{t}$ comprising $I=2000$ descriptive text concepts gathered from established visual concept datasets [40, 39]. The descriptions encompass words representing texture, colors, brightness, density etc., and we categorize these descriptions into 50 classes. Illustrations of these terms can be found in Fig. 4. The dictionary is represented as $\Omega_{t}\triangleq\{\omega_{i}\}_{i=1}^{I}$. Adhering to CLIP’s zero-shot setup, we begin by appending $\omega_{i}$ to a manually designed prompt $\phi=$ “The photo is …" to form a concept-specific textual input $\{\pi;\omega_{i}\}$. Consequently, utilizing the text encoder $E_{t}$, we generate text concept features $C_{t}\triangleq\{c_{t}^{i}\}_{i=1}^{I}$, denoted as $c_{t}^{i}=E_{t}({\pi;\omega_{i}})$.

We denote the handcrafted concept cache as $\Phi_{mc}\triangleq\{(key,value)_{i}\}_{i=1}^{I}$. The key and value are the visual concepts and textual concept words respectively. The visual concepts are discovered by leveraging the text concept features $C_{t}$ and the CLIP model, derived from the training images. In the scenario of $H$-shot $D$-class few-shot learning, where there exist $H$ labeled images within each of the $D$ classes. Utilizing the CLIP visual encoder $E_{v}$, we generate their respective image features $V\triangleq\{v_{j}\}_{j=1}^{HD}$, expressed as $v_{j}=E_{v}(x_{j})$. For every text concept feature $c_{t}\in C_{t}$, the similarity score $S_{t}$ is calculated against all visual features in $V$ using the formula $S_{t}=\mathrm{sim}\left(c_{t},v_{j}\right)=c_{t}v_{j}$, where both $c_{t}$ and $v_{j}$ are normalized. Subsequently, we identify Top-$K_{1}$ image features with the highest similarity score and regard their average as the key and its corresponding text concept word $\omega_{t}$ as the associated value, stored within the handcrafted concept cache.

Figure 4: Examples of text concepts from established visual concept datasets, including descriptive terms of texture, color, transparency, motion and brightness.

Conceptual Codebook Learning with Regularization.

Figure 3 (b) shows the handcrafted concept-based prompting process, we first extract the image feature $f_{v}$ by $E_{v}$, then use the image feature as the query to find Top-$K_{2}$ similar keys using cosine distance, and finally we get the corresponding values (conceptual words). Together with the class name, these concept words are input to an LLM (eg. GPT [2]) to generate optimal handcrafted concept-based prompts. Therefore, our approach addresses the challenge of diminished generalization on downstream tasks by introducing a regularization, ensuring the text features produced by the selected learned prompts are not significantly different from their counterpart generated by the handcrafted concept-based prompts. We enforce this consistency by utilizing the Euclidean distance as a constraint between the text features generated from the hand-crafted concept-based prompts ($f_{t_{d}}^{h}$) and those obtained from selected learned prompts ($f_{t_{d}}^{l}$). While alternative measures such as cosine distance could also serve as constraints, our empirical findings suggest that Euclidean distance yields superior performance. We can represent the consistency constraint as:

$$\mathcal{L}_{cc}=\frac{1}{D}\sum_{d=1}^{D}||f_{t_{d}}^{l}-f_{t_{d}}^{h}||^{2}_{2},$$ (3)

where $||\cdot||$ is the euclidean distance, $D$ is the number of seen classes.

3.3 Training and Inference

Training Objective.

Based on Eq. 2, the image classification loss is formulated as:

$$\mathcal{L}_{ce}=\mathbb{E}[-\text{log}\frac{e^{\langle f_{v_{j}},f_{t_{j}}\rangle/\tau}}{\sum_{d=1}^{D}e^{\langle f_{v_{j}},f_{t_{d}}\rangle/\tau}}].$$ (4)

In addition to $\mathcal{L}_{ce}$, a matching loss is needed to pull the matched top-$K_{3}$ keys $\mathcal{K}_{j}$ closer to the image embedding $\mathbf{z}_{j}$, so that the keys learn rich concepts from the samples. We use cosine distance as the matching loss. However, other similar criteria, like Euclidean distance, can also be used as a matching loss. We empiricaly observe that cosine distance as the matching loss yields the best performance. Therefore The matching loss adopted to optimize the keys is defined as:

$$\mathcal{L}_{ma}=\sum_{i=1}^{C}(1-\frac{f_{v_{j}}\cdot\mathbf{V}_{j_{i}}}{||f_{v_{j}}||~{}||\mathbf{V}_{j_{i}}||}).$$ (5)

Finally, in order to make the learned prompts more semantically diverse, we adopt a third loss to orthogonalize the embeddings of different prompts to increase the diversity of the prompts:

$$\mathcal{L}_{or}=\frac{1}{N(N-1)}\sum_{i=1}^{N}\sum_{j=i+1}^{N}|\langle E_{t}(\mathbf{P}_{i}),E_{t}(\mathbf{P}_{j})\rangle|,$$ (6)

where $\langle\cdot,\cdot\rangle$ denotes the cosine similarity. In this way, the overall optimization objective is defined as:

$$\mathcal{L}=\mathcal{L}_{ce}+\mathcal{L}_{ma}+\mathcal{L}_{or}+\mathcal{L}_{cc},$$ (7)

The keys are optimized by $\mathcal{L}_{ma}$, and the prompts by $\mathcal{L}_{ce}$, $\mathcal{L}_{or}$ and $\mathcal{L}_{cc}$.

Inference.

Once visual and textual concepts (keys and values) are learned through training, they can be shipped with CLIP for downstream tasks with a standard zero-shot CLIP inference setup. As shown in Figure 3, we first generate the visual features $f_{vt}$ of the test image using visual Encoder $E_{v}$, then we use $f_{vt}$ to choose top-$K_{3}$ similar visual concepts(keys) by cosine similarity, next, we concatenate the corresponding prompts(values) of the keys to obtain the conceptual prompt, which is fused with each given name to produce conceptual prompt text features $\{f_{tt}^{i}\}_{i=1}^{C}$. Finally, zero-shot inference is performed with the conceptual prompted text features and the input image feature $f_{vt}$ to produce classification scores on the test images.

4 Experiments

In this section, We present a thorough evaluation of our method, including experimental results, performance comparisons with state-of-the-art methods, and ablation studies.

4.1 Experimental Setup

Benchmark Settings.

We follow previous works to extensively evaluate our proposed method on three challenging tasks:

• Base to Novel Class generalization. We evaluate the generalization capability of our method in zero-shot scenarios within a dataset. The dataset is evenly divided into base and novel classes. We train our model with few-shot images on the base classes and then evaluate on both base classes and unseen novel classes.

• Cross-Dataset Evaluation. The cross-dataset transfer is a much more challenging generalization task compared to base-to-novel generalization, since the latter only transfers within a single dataset while the former transfers across different datasets, e.g., from object recognition to texture classification. In this experiment, we follow previous works to train our model in a few-shot setting on 1000 ImageNet classes and subsequently evaluate its performance on ten other unseen datasets.

• Domain Generalization. We evaluate our model’s performance on out-of-distribution generalization. Similarly, we assess our model trained on ImageNet directly on four ImageNet variants that contain the same classes but from different distributions

Datasets.

For conducting experiments on base-to-novel generalization and cross-dataset transfer tasks, we adhere to the setting of prior studies [26, 42, 41]. Specifically, we evaluate our approach across 11 diverse image classification datasets. These datasets encompass a wide range of tasks, including generic object classification (e.g., ImageNet [4] and Caltech101 [7]), fine-grained classification (e.g., OxfordPets [23], StanfordCars [18], Flowers102 [22], Food101 [1], and FGVCAircraft [21]), scene recognition (SUN397 [35]), action recognition (UCF101 [31]), texture classification (DTD [3]), and satellite image recognition (EuroSAT [10]). For the domain generalization task, we employ ImageNet as the source dataset and evaluate our method’s performance on four ImageNet variants: ImageNet-A [12], ImageNet-R [11], ImageNet-V2 [27], and ImageNet-Sketch [32].

Table 1: Comparison with state-of-the-art methods on base-to-novel generalization. The best accuracies are bolded and the second best results are underlined. HM indicates the harmonic mean.

(a) Average

	Base	Novel	HM
CLIP	69.34	74.22	71.70
CoOp	82.69	63.22	71.66
Co-CoOp	80.47	71.69	75.83
KgCoOp	80.73	73.60	77.00
MaPLe	82.28	75.14	78.55
CoPrompt	84.00	77.23	80.48
CPL	84.38	78.03	81.08
Ours	85.22	80.31	82.70
	+0.84	+2.28	+1.62

(b) ImageNet

	Base	Novel	HM
CLIP	72.43	68.14	70.22
CoOp	76.47	67.88	71.92
Co-CoOp	75.98	70.43	73.10
KgCoOp	75.83	69.96	72.78
MaPLe	76.66	70.54	73.47
CoPrompt	77.67	71.27	74.33
CPL	78.74	72.03	75.24
Ours	79.25	74.58	76.84
	+0.51	+2.55	+1.60

(c) Caltech101

	Base	Novel	HM
CLIP	96.84	94.00	95.40
CoOp	98.00	89.81	93.73
Co-CoOp	97.96	93.81	95.84
KgCoOp	97.72	94.39	96.03
MaPLe	97.74	94.36	96.02
CoPrompt	98.27	94.90	96.55
CPL	98.35	95.13	96.71
Ours	98.17	95.67	96.90
	-0.18	+0.54	+0.19

(d) OxfordPets

	Base	Novel	HM
CLIP	91.17	97.26	94.12
CoOp	93.67	95.29	94.47
Co-CoOp	95.20	97.69	96.43
KgCoOp	94.65	97.76	96.18
MaPLe	95.43	97.76	96.58
CoPrompt	95.67	98.10	96.87
CPL	95.86	98.21	97.02
Ours	96.21	98.55	97.37
	+0.35	+0.34	+0.35

(e) StanfordCars

	Base	Novel	HM
CLIP	63.37	74.89	68.65
CoOp	78.12	60.40	68.13
Co-CoOp	70.49	73.59	72.01
KgCoOp	71.76	75.04	73.36
MaPLe	72.94	74.00	73.47
CoPrompt	76.97	74.40	75.66
CPL	79.31	76.65	77.96
Ours	80.32	78.84	79.57
	+1.01	+2.19	+1.61

(f) Flowers102

	Base	Novel	HM
CLIP	72.08	77.80	74.83
CoOp	97.60	59.67	74.06
Co-CoOp	94.87	71.75	81.71
KgCoOp	95.00	74.73	83.65
MaPLe	95.92	72.46	82.56
CoPrompt	97.27	76.60	85.71
CPL	98.07	80.43	88.38
Ours	97.72	81.04	88.60
	-0.35	+0.61	+0.22

(g) Food101

	Base	Novel	HM
CLIP	90.10	91.22	90.66
CoOp	88.33	82.26	85.19
Co-CoOp	90.70	91.29	90.99
KgCoOp	90.50	91.70	91.09
MaPLe	90.71	92.05	91.38
CoPrompt	90.73	92.07	91.40
CPL	91.92	93.87	92.88
Ours	92.23	94.28	93.24
	+0.31	+0.41	+0.36

(h) FGVCAircraft

	Base	Novel	HM
CLIP	27.19	36.29	31.09
CoOp	40.44	22.30	28.75
Co-CoOp	33.41	23.71	27.74
KgCoOp	36.21	33.55	34.83
MaPLe	37.44	35.61	36.50
CoPrompt	40.20	39.33	39.76
CPL	42.27	38.85	40.49
Ours	43.86	42.65	43.25
	+1.59	+3.32	+2.76

(i) SUN397

	Base	Novel	HM
CLIP	69.36	75.35	72.23
CoOp	80.60	65.89	72.51
Co-CoOp	79.74	76.86	78.27
KgCoOp	80.29	76.53	78.36
MaPLe	80.82	78.70	79.75
CoPrompt	82.63	80.03	81.31
CPL	81.88	79.65	80.75
Ours	83.97	82.24	83.10
	+1.34	+2.21	+1.79

(j) DTD

	Base	Novel	HM
CLIP	53.24	59.90	56.37
CoOp	79.44	41.18	54.24
Co-CoOp	77.01	56.00	64.85
KgCoOp	77.55	54.99	64.35
MaPLe	80.36	59.18	68.16
CoPrompt	83.13	64.73	72.79
CPL	80.92	62.27	70.38
Ours	82.46	68.38	74.76
	-0.67	+3.65	+1.97

(k) EuroSAT

	Base	Novel	HM
CLIP	56.48	64.05	60.03
CoOp	92.19	54.74	68.69
Co-CoOp	87.49	60.04	71.21
KgCoOp	85.64	64.34	73.48
MaPLe	94.07	73.23	82.35
CoPrompt	94.60	78.57	85.84
CPL	94.18	81.05	87.12
Ours	95.03	84.17	89.27
	+0.43	+3.12	+2.15

(l) UCF101

	Base	Novel	HM
CLIP	70.53	77.50	73.85
CoOp	84.69	56.05	67.46
Co-CoOp	82.33	73.45	77.64
KgCoOp	82.89	76.67	79.65
MaPLe	83.00	78.66	80.77
CoPrompt	86.90	79.57	83.07
CPL	86.73	80.17	83.32
Ours	88.30	83.05	85.60
	+1.40	+2.88	+2.28

Implementation Details.

To ensure a fair comparison, we employ the ViT-B/16 CLIP model across all three benchmark tasks. For base-to-novel generalization, we train our model with 16-shot images on base classes and then evaluate both base classes and novel classes. For cross-dataset evaluation and domain generalization, both use the model trained with 16-shot ImageNet and test on each target dataset. Throughout the training, we keep both the visual and textual encoders fixed. Our data preprocessing follows CLIP’s protocol, including resizing and random crop** operations, among others. For the base-to-novel generation task, we train for 30 epochs on ImageNet and 20 epochs on other datasets. $K_{1}$ and $K_{2}$ are set to 3 and 10 respectively. We set prompt length $M$ to 8, the number of concepts in the bank $N$ to 100, and the number of selected concepts $K_{3}$ to 4. Training utilizes a batch size of 8 with an initial learning rate of $10^{-3}$. We utilize the AdamW optimizer alongside a cosine annealing scheduler. Results are averaged over 3 runs for all methods.

4.2 Base-to-Novel Generalization

In this section, We compare our proposed method with seven baselines: zero-shot CLIP [26], CoOp [42], CoCoOp [41], MaPLe [17], KgCoOp [36], CoPrompt [28] and CPL [40]. The experimental results regarding base-to-novel generalization across 11 datasets with 16-shot samples are presented in Table 1. We have highlighted the best results in bold and marked the improvement over the second-best performance in blue. As we can see from Table 1(a), the average of all 11 datasets shows that our method outperforms all the baselines by a large margin for both base and novel classes. In comparison to CoOp and CoCoOp, which are the pioneering prompt learning methods, the performance gain of our method even reached +11% and +6.9% respectively. Our method outperforms the previous state-of-the-art (CPL) by +2.28% on novel classes and +1.62% on the harmonic mean (HM). These results demonstrate the strong zero-shot generalization capability of our proposed method. Also, our method outperforms CPL on base classes by +0.84%, which shows a strong few-shot learning capability.

For the performance of individual datasets, our method outperforms CPL on all the datasets for novel class and HM. For the base classes, our method achieves superior performance gains compared to CPL on 8 out of 11 datasets. Even for Catech101, Flower102, and DTD, where there is a slight performance drop, it remains marginal. This highlights the enhanced generalization capability of our method towards novel classes without compromising performance on base classes. Notably, aside from CPL, our method outperforms all other methods by a significant margin across all datasets. Compared to the second-best performing baseline, our method surpasses it by up to +3.32%, +3.65%, and +2.55% on FGVCAircraft, DTD, and ImageNet, respectively. These observations indicate that our method can effectively learn diverse and discriminative visual and textual concepts, thereby enhancing CLIP’s adaptation for generalization tasks.

4.3 Cross-Dataset Evaluation

The comparison results with CoOp, CocoOp, MaPLe, CoPrompt, and CPL are presented in Table 2. Overall, our CoCoLe method marks the best performance on both source and target datasets with a target average of 68.91%, and outperforms CPL by 0.84%. Notably, we obtain the highest improvement of 2.26% over CPL on UCF101, an action image dataset whose fundamentals are distinctive from ImageNet. This suggests that conceptual codebook learning in our method facilitates better generalization.

Table 2: Comparison of our method with existing approaches on cross-dataset evaluation. Overall, our method demonstrates superior generalization capabilities with the highest average accuracy on 10 datasets.

	Source	Target
	ImageNet	Caltech101	OxfordPets	StanfordCars	Flowers102	Food101	Aircraft	SUN397	DTD	EuroSAT	UCF101	Average
CoOp [42]	71.51	93.70	89.14	64.51	68.71	85.30	18.47	64.15	41.92	46.39	66.55	63.88
CoCoOp [41]	71.02	94.43	90.14	65.32	71.88	86.06	22.94	67.36	45.73	45.37	68.21	65.74
MaPLe [17]	70.72	93.53	90.49	65.57	72.23	86.20	24.74	67.01	46.49	48.06	68.69	66.30
CoPrompt [28]	70.80	94.50	90.73	65.67	72.30	86.43	24.00	67.57	47.07	51.90	69.73	67.00
CPL [40]	73.53	95.52	91.64	66.17	73.35	87.68	27.36	68.24	48.96	51.25	70.52	68.07
Ours	73.88	95.88	91.93	67.79	74.17	87.97	28.83	68.75	49.26	51.75	72.78	68.91

4.4 Domain Generalization

In Table 3, we provide the classification accuracy across the source domain and target domains, as well as the average accuracy within target domains. Our approach consistently outperforms all baselines on both source and target datasets, achieving a new state-of-the-art average accuracy of 61.85% on this task. Notably, our method beats CPL [40] by +1.51% on ImageNet-R. This indicates the remarkable robustness of our model against distribution shifts.

Table 3: Comparison with other methods on robustness ($\%$) to natural distribution shifts. The best results are in bold and the second-best results are underlined.

Method	Source	Target
	ImageNet	-V2	-Sketch	-A	-R	Ave.
CLIP [26]	66.73	60.83	46.15	47.77	73.96	57.17
CoOp [42]	71.51	64.20	47.99	49.71	75.21	59.28
CoCoOp [41]	71.02	64.07	48.75	50.63	76.18	59.90
KgCoOp [36]	71.20	64.10	48.97	50.69	76.70	60.11
MaPLe [17]	70.72	64.07	49.15	50.90	76.98	60.27
CoPrompt [28]	70.80	64.25	49.43	50.50	77.51	60.42
CPL [40]	73.53	65.18	49.92	50.73	77.38	60.80
Ours	73.88	65.86	50.89	51.75	78.89	61.85

4.5 Ablation Studies

To systematically evaluate our proposed method, we provide an empirical analysis of our design choices and illustrate the effects of different components of our method in this section. If not otherwise specified, our experiments are conducted on the ImageNet for the Base-to-Novel Task, and we report the harmonic mean.

Contributions of major algorithm components.

In Tab. 4, $\mathcal{L}_{ma}$ is the matching loss adopted to optimize the keys, $\mathcal{L}_{cc}$ is for consistency constraint, $\mathcal{L}_{or}$ is used to orthogonalize the embeddings of different prompts. We present an ablation study by removing different components of the proposed method to understand the importance of each of them. For inference, in the first row of the table, we present the final performance of CoCoLe, which has a harmonic mean of 76.84%. In the first ablation experiment, we eliminate the $\mathcal{L}_{or}$ from CoCoLe, leading to a performance drop of 0.46%. This highlights the importance of $\mathcal{L}_{or}$ for CoCoLe. Next, we remove $\mathcal{L}_{cc}$, effectively enforcing consistency between the learnable conceptual prompts and handcrafted concept-based prompts. This results in a 1.98% drop in performance, suggesting the importance of consistency regularization. Finally, we remove $\mathcal{L}_{ma}$, which leads to a performance drop of 2.39%. This shows that the learnable conceptual codebook plays a crucial role in CoCoLe. In general, the overall results highlight the significant contribution of all components to enhanced performance.

Table 4: The ablation study on each component of CoCoLe. The experiments are conducted on ImageNet. We report the harmonic mean accuracy of the base-to-novel task.

$\mathcal{L}_{ma}$	$\mathcal{L}_{cc}$	$\mathcal{L}_{or}$	Accuracy(HM)
✓	✓	✓	76.84
✓	✓	✗	76.38
✓	✗	✓	74.86
✗	✓	✓	74.45
✓	✗	✗	74.57
✗	✓	✗	74.26
✗	✗	✗	72.12

Table 5: Ablation study of the length of prompts $M$, the size of conceptual codebook $N$, the number $K_{3}$ of visual concepts for training at a time.

Value of $M$	4	8	12	16	20
Accuracy	75.83	76.84	76.53	75.97	75.75
Value of $N$	50	100	150	200	250
Accuracy	75.07	76.84	76.73	76.51	76.25
Value of $K_{3}$	1	2	4	6	8
Accuracy	75.30	75.86	76.84	76.72	76.33

The value of $M$, $N$, and $K_{3}$.

As defined in Sec. 3.2, $M$ is the length of prompts, $N$ is the size of the conceptual codebook, and $K_{3}$ is the number of learned visual concepts (i.e., keys in the codebook) selected for training at the same time. As indicated in Table 5, the model performs optimally with $M=8$. If the prompt is overly long, it raises both training time and computational costs. Concerning $N$, the model excels with $N=100$. Additionally, experimenting with various $K_{3}$ values shows that saturating training with excessive keys and prompts simultaneously diminishes model performance. The model achieves its peak performance at $K_{3}=4$.

Comparison on the training time.

As shown in Tab. 6, our proposed CoCoLe exhibits a significant performance advantage over other methods. Though our CoCoLe requires more training time compared to CoOp, it still outperforms CoOp by 11%. Furthermore, when compared to CoCoOp and CPL, our method achieves substantial performance gains with less time. This demonstrates the efficiency and effectiveness of our method.

Table 6: Comparison on the training time. We report the average accuracy across 11 datasets on base-to-novel tasks.

Methods	Prompts	Accuracy			Training-time
		Base	New	H
CLIP	handcrafted	69.34	74.22	71.70	-
CoOp	textual	82.69	63.22	71.66	6ms/image
CoCoOp	textual+visual	80.47	71.69	75.83	160ms/image
CPL	textual+visual	84.38	78.03	81.08	25ms/image
CoCoLe	textual+visual	85.22	80.31	82.70	10ms/image

Visualization.

As shown in Fig. 1, in order to confirm that various prompts indeed capture distinct image concepts, we employ Grad-CAM [30] to visually represent the image contents associated with different prompts. Observing Fig. 1(a), it’s apparent that various prompts highlight distinct regions within the same image, showcasing the diversity of the acquired prompts. For instance, different prompts applied to the Koala image emphasize different areas such as the head, claws, and tree, illustrating the versatility of the learned prompts. To assess whether the learned prompts indeed encapsulate image concepts with higher-level semantics, we visualize the content of specific prompts ($\textbf{P}_{3}$, $\textbf{P}_{23}$, $\textbf{P}_{48}$) across different images in Fig. 1(b). It’s evident that $\textbf{P}_{3}$ mainly captures the concept "wheels," $\textbf{P}_{23}$ encapsulates the concept "grass," while $\textbf{P}_{48}$ focuses on the "screen" of the devices. This highlights the effectiveness of prompts in learning key concepts that can be generalized across images, thereby enhancing performance in generalization tasks. In the Supplemental Materials, we provide additional details of the proposed method and experimental results.

5 Conclusion

To address the challenge of improving the generalization capability of VLMs while fine-tuning them on downstream tasks, we propose Conceptual Codebook Learning (CoCoLe). The learned conceptual codebook consists of visual concepts as keys and conceptual prompts as values, which serves as a link between the image encoder’s outputs and the text encoder’s inputs. Additionally, we incorporate a handcrafted concept cache as a regularization to alleviate the overfitting issues in low-shot scenarios. Extensive experimental results demonstrate that our CoCoLe method remarkably outperforms the existing state-of-the-art methods.