CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept Matching

Abstract

Diffusion models have demonstrated great success in the field of text-to-image generation. However, alleviating the misalignment between the text prompts and images is still challenging. We break down the problem into two causes: concept ignorance and concept mismap**. To tackle the two challenges, we propose CoMat, an end-to-end diffusion model fine-tuning strategy with the image-to-text concept matching mechanism. Firstly, we introduce a novel image-to-text concept activation module to guide the diffusion model in revisiting ignored concepts. Additionally, an attribute concentration module is proposed to map the text conditions of each entity to its corresponding image area correctly. Extensive experimental evaluations, conducted across three distinct text-to-image alignment benchmarks, demonstrate the superior efficacy of our proposed method, CoMat-SDXL, over the baseline model, SDXL [38]. We also show that our method enhances general condition utilization capability and generalizes to the long and complex prompt despite not specifically training on it.

Figure 1: Current text-to-image diffusion model still struggles to produce images well-aligned with text prompts, as shown in the generated images of SDXL [38]. Our proposed method CoMat significantly enhances the baseline model on text condition following, demonstrating superior capability in text-image alignment. All the pairs are generated with the same random seed.

1 Introduction

Figure 2: Visualization of token activation and attention map. We compare the tokens’ attention activation value and attention map before and after applying our methods. Our method improves token activation and encourages the missing concept ‘gown’ to appear. Furthermore, the attention map of the attribute token ‘red’ better aligns with its region in the image.

The area of text-to-image generation has witnessed considerable progress with the introduction of diffusion models [17, 41, 40, 44, 46] recently. These models have demonstrated remarkable performance in creating high-fidelity and diverse images based on textual prompts. However, it still remains challenging for these models to faithfully align with the prompts, especially for the complex ones. For example, as shown in Fig. 1, current state-of-the-art open-sourced model SDXL [38] fails to generate entities or attributes mentioned in the prompts, e.g., the feathers made of lace and dwarfs in the top row. Additionally, it fails to understand the relationship in the prompt. In the middle row of Fig. 1, it mistakenly generates a Victorian gentleman and a quilt with a river on it.

We break down this misalignment problem into two causes: concept ignorance and concept mismap**. The concept ignorance problem is caused by the diffusion model’s omission of certain concepts in the text prompt. Even though the concept token is activated, the diffusion model often fails to map it to the correct area in the image, which is termed the concept mismap** problem. Actually, the misalignment originally stems from the training paradigm of the text-to-image diffusion models: Given the text condition $c$ and the paired image $x$, the training process aims to learn the conditional distribution $p(x|c)$. However, the text condition only serves as additional information for the denoising loss. Without explicit guidance in learning each concept in the text, the diffusion model could easily fail to understand the concepts in the prompt correctly.

Recently, to alleviate the misalignment, various works have proposed to incorporate linguistics prior [42, 6] to heuristically address the concept omission or concept mismap** problem. However, a specific design is required for each type of misalignment problem. Other works use the Large Language Model (LLM) [32, 61] to split the prompt into single entities and generate each of them. Although this method promotes the congruence between the image’s global structure and the text prompt, it still suffers from local misalignment of the single entity. Hence, we ask the question: Is there a universal solution to address various global and local misalignment problems?

In this work, we propose CoMat, an end-to-end fine-tuning strategy to enhance the prompt understanding and following by a novel image-to-text matching mechanism. Concept Activation module is proposed to address the concept ignorance problem. Given the generated image $\hat{x}$ conditioning on the prompt $c$, we seek to model and maximize the posterior probability $p(c|\hat{x})$ using a pre-trained image-to-text model. In contrast to regarding the textual prompt merely as a condition, as performed in the pre-training phase of the diffusion model, our approach incorporates the condition as a supervisory signal during the training process. Thanks to the proficiency of the image-to-text model in concept matching, whenever a particular concept is absent from the generated image, the diffusion model is steered to incorporate it within the image generation process. The guidance forces the diffusion model to revisit the ignored conditions and attend more to them. As an illustrative example shown in Fig. 2, the ignored concept in the image (e.g., the gown) possesses low attention activation values. After applying our method, we observe increased attention activation of each key concept, contributing to the aligned image. In addition, considering the catastrophic forgetting issue arising from the new optimization objective, we also introduce a novel fidelity preservation module and mixed latent strategy to preserve the generation capability of the diffusion model. As for the concept mismap** problem, we find it especially prevails among the attributes of the objects. Hence, the Attribute Concentration module is introduced to promote both positive and negative map**. We match the concept of attribute tokens in the text prompt to the generated image, with the insight that the attribute tokens should only be activated within its entity’s area. Since the concept is a general term for a variety of features, our method can address both global structures and local details.

As an end-to-end method, no extra overhead is introduced during inference. We also show that our method is composable with methods leveraging external knowledge. Our contributions are summarized as follows:

• We propose CoMat, a text-to-image diffusion model fine-tuning strategy to effectively enhance the condition utilization capability by explicitly addressing the condition ignorance and incorrect condition map** problem.

• We introduce the concept activation module equipped with fidelity preservation and mixed latent strategy to facilitate concept generation and attribute concentration module to foster correct concept map** from text to image.

• Extensive quantitative and qualitative comparisons with baseline models indicate that our method significantly improves the text-image alignment in various scenarios, including object existence, attribute binding, relationship, and complex prompts.

2 Related Work

Text-to-image alignment is the problem of enhancing coherence between the prompts and the generated images, which involves multiple aspects including existence, attribute binding, relationship, etc. Recent methods address the problem mainly in three ways.

Attention-based methods [6, 42, 36, 54, 2, 31] aim to modify or add restrictions on the attention map in the attention module in the UNet. This type of method often requires a heuristic design for each misalignment problem.

Planning-based methods first obtain the image layouts, either from the input of the user [30, 8, 24, 58, 12] or the generation of the Large Language Models (LLM) [37, 61, 53], and then produce aligned images conditioned on the layout. In addition, a few works propose to further refine the image with other vision expert models [43, 56, 55, 61]. Although such method splits a compositional prompt into single objects, it does not resolve the inaccuracy of the downstream diffusion model and still suffers from incorrect attribute binding problems. Besides, it exerts nonnegligible costs during inference.

Moreover, some works aim to enhance the alignment using feedback from image understanding models. [22, 48] fine-tune the diffusion model with well-aligned generated images chosen by the VQA model [27] to strategically bias the generation distribution. Other works propose to optimize the diffusion models in an online manner. [13, 4] introduce RL fine-tuning for generic rewards. As for differentiable reward, [11, 59, 57] propose to backpropagate the reward function gradient through the denoising process. Similar to our work, [14] also proposes to leverage image captioning models. We discuss the difference between their method with ours in Appendix C

Figure 3: We showcase the results of our CoMat-SDXL compared with other state-of-the-art models. CoMat-SDXL consistently generates more faithful images.

3 Preliminaries

We implement our method on the leading text-to-image diffusion model, Stable Diffusion [44], which belongs to the family of latent diffusion models (LDM). In the training process, a normally distributed noise $\epsilon$ is added to the original latent code $z_{0}$ with a variable extent based on a timestep $t$ sampling from $\{1,...,T\}$. Then, a denoising function $\epsilon_{\theta}$, parameterized by a UNet backbone, is trained to predict the noise added to $z_{0}$ with the text prompt $\mathcal{P}$ and the current latent $z_{t}$ as the input. Specifically, the text prompt is first encoded by the CLIP [39] text encoder $W$, then incorporated into the denoising function $\epsilon_{\theta}$ by the cross-attention mechanism. Concretely, for each cross-attention layer, the latent and text embedding is linearly projected to query $Q$ and key $K$, respectively. The cross-attention map $A^{(i)}\in\mathbb{R}^{h\times w\times l}$ is calculated as $A^{(i)}=\text{Softmax}(\frac{Q^{(i)}(K^{(i)})^{T}}{\sqrt{d}})$, where $i$ is the index of head. $h$ and $w$ are the resolution of the latent, $l$ is the token length for the text embedding, and $d$ is the feature dimension. $A_{i,j}^{n}$ denotes the attention score of the token index $n$ at the position $(i,j)$. The denoising loss in diffusion models’ training is formally expressed as:

$$\mathcal{L}_{\text{LDM}}=\mathbb{E}_{z_{0},t,p,\epsilon\sim\mathcal{N}({0},{I})}\left[\left\|{\epsilon}-{\epsilon}_{{\theta}}\left(z_{t},t,W(\mathcal{P})\right)\right\|^{2}\right].$$ (1)

For inference, one draws a noise sample $z_{T}\sim\mathcal{N}(0,I)$, and then iteratively uses $\epsilon_{\theta}$ to estimate the noise and compute the next latent sample.

4 Method

The overall framework of our method is shown in Fig. 4. In Section 4.1, we first illustrate the concept activation module. Following this, we detail how we maintain the generation capability of the diffusion model by the fidelity preservation module and mixed latent strategy. Subsequently, in Section 4.2, we introduce the attribute concentration module for promoting attribute binding, and then we integrate the two components for joint learning.

4.1 Concept Activation

As noted in Section 1, the diffusion model occasionally exhibits little attention on certain concepts, and the corresponding concept is therefore missing in the image, which we termed as the condition ignorance problem. To address this, our key insight is to add supervision on the generated image to detect the missing concepts. We achieve this by leveraging the image understanding ability of an image-to-text model, which can accurately identify concepts not present in the generated image based on the given text prompt. With the image-to-text model’s supervision, the diffusion model is compelled to revisit text tokens to search for ignored condition information and assign more significance to the previously overlooked text concepts for better text-image alignment. Concretely, given a prompt $\mathcal{P}$ with word tokens $\{w_{1},w_{2},\dots,w_{L}\}$, we first generate an image $\mathcal{I}$ with the denoising function $\epsilon_{\theta}$ after $T$ denoising steps. Then, a frozen image-to-text model $\mathcal{C}$ is used to score the alignment between the prompt and the image in the form of log-likelihood. Therefore, our training objective aims to minimize the negative of the score, denoted as $\mathcal{L}_{i2t}$:

$$\mathcal{L}_{\text{i2t}}=-\log(p_{\mathcal{C}}(\mathcal{P}|\mathcal{I}(\mathcal{P};\epsilon_{\theta})))=-\sum_{i=1}^{L}\log(p_{\mathcal{C}}(w_{i}|\mathcal{I},w_{1:i-1})).$$ (2)

It is important to note that the concepts in the image include a broad field. This method provides a universal solution to various misalignment problems like object existence, complex relationships, etc. To conduct the gradient update through the whole iterative denoising process, we follow [57] to fine-tune the denoising network $\epsilon_{\theta}$, which ensures the training effectiveness and efficiency by simply stop** the gradient of the denoising network input.

Figure 4: Overview of CoMat. The text-to-image diffusion model (T2I-Model) first generates an image according to the text prompt. Then the image is sent to the concept activation module and attribute concentration module to compute the loss for fine-tuning the online T2I-Model.

However, since this fine-tuning process is purely piloted by the knowledge from the image-to-text model, the diffusion model could quickly overfit to the image-to-text model, lose its original capability, and produce deteriorated images, as shown in Fig. 10. To address this hacking issue, we introduce a novel fidelity preservation module and a mixed latent training strategy to preserve the generation ability of the diffusion model and guide the learning process.

Fidelity Preservation. We propose a novel adversarial loss that uses a discriminator to differentiate between images generated by pre-trained and fine-tuned diffusion models. Instead of using real-world images as the real data input for the discriminator, we use images generated by the original pre-trained diffusion model. This choice is based on the significant gap that still exists between the images generated by the original diffusion model and real-world images. Simply aligning the distribution of images generated by the fine-tuned diffusion model with that of real-world images would pose an undesired challenge for the learning process. For the discriminator $\mathcal{D}_{\phi}$, we initialize it with the pre-trained UNet in the Stable Diffusion model. The choice is motivated by the fact that the pre-trained UNet shares similar knowledge with the online training model and fits well with the input domain. In our practice, this also enables the adversarial loss to be directly calculated in the latent space instead of the image space. Concretely, given a single text prompt, we employ the original diffusion model and the online training model to respectively generate image latent $\hat{z}_{0}$ and $\hat{z}_{0}^{\prime}$. The adversarial loss is then computed as follows:

$$\mathcal{L}_{adv}=\log\left(\mathcal{D}_{\phi}\left(\hat{z}_{0}\right)\right)+\log\left(1-\mathcal{D}_{\phi}\left(\hat{z}_{0}^{\prime}\right)\right).$$ (3)

We aim to fine-tune the online model to minimize this adversarial loss, while concurrently training the discriminator to maximize it.

Mixed Latent Strategy. Besides, we inject information from real-world images to guide the learning process. Specifically, in addition to the latents starting from pure noise (marked as black in Fig. 4), we obtain the noisy real latents by adding noise on a real-world image at a random timestep $\tau$ (marked as ‘Noisy GT’). We jointly denoise these two types of latents and calculate the loss given by the image-to-text model. The intuition is that, since the noisy real latent is a perturbed version of the real-world image, which is well aligned with its prompt, this provides a shortcut for the diffusion model to directly reconstruct the original image. This guidance can not only smooth the optimization process, but also prohibits the gradient from simply hacking the image-to-text model and encourages the diffusion model to generate an image both aligned with the prompt and of high fidelity.

4.2 Attribute Concentration

Except for paying enough attention to the concept, the diffusion model must also map the concepts correctly on the image. As we dive into the generation process by visualizing the token attention activation map, we find that, for the attribute token, even though it is activated, it fails to attend to the correct area in the image and still causes the misalignment, e.g., ‘yellow’ in Fig. 5. Hence, we introduce the attribute concentration module to encourage the positive and discourage the negative concept map** of attributes.

Specifically, we first extract all the entities $\{e_{1},..,e_{N}\}$ in the prompts. An entity can be defined as a tuple of a noun $n_{i}$ and its attributes $a_{i}$, i.e., $e_{i}=(n_{i},a_{i})$, where both $n_{i}$ and $a_{i}$ are the sets of one or multiple tokens. We employ spaCy’s transformer-based dependency parser [18] to parse the prompt to find all entity nouns, and then collect all attributes for each noun. A predefined set of nouns is established for filtering, including nouns that are abstract (e.g., scene, atmosphere, language), difficult to identify their area (e.g., sunlight, noise, place), or describe the background (e.g., morning, bathroom, party). Given all the selected nouns, we use them to prompt an open vocabulary segmentation model, Grounded-SAM [43], to find their corresponding regions as a binary mask $\{M^{1},...,M^{N}\}$. It is worth emphasizing that, to guarantee the segmentation accuracy, we only use the nouns of entities, excluding their associated attributes, as prompts for segmentation, considering the diffusion model could likely ignore the attribute or assign a wrong one to the object. Taking the ‘suitcase’ object in Fig 5 as an example, the model ignored the ‘purple’ attribute. Consequently, if the prompt ’purple suitcase’ is given to the segmentor, it will fail to identify the entity’s region. These inaccuracies can lead to a cascade of errors in the following process.

We add supervision to promote the diffusion model to map the entity tokens to the positive area, i.e., the entity area, and not to attend to the negative area, i.e., the other area:

$$\mathcal{L}_{\text{pos}} = -\frac{1}{N}\sum_{i=1}^{N}\sum_{M^{i}_{u,v}=1}\left(\alpha\sum_{k\in n_{i}\cup a_{i}}\left(\frac{A^{k}_{u,v}}{\sum_{x,y}A^{k}_{x,y}}\right)+\beta\frac{\log(\sum_{k\in n_{i}\cup a_{i}}A_{u,v}^{k})}{|A|}\right),$$ (4)

$$\mathcal{L}_{\text{neg}} = -\frac{1}{N}\sum_{i=1}^{N}\sum_{M^{i}_{u,v}=0}\left(\alpha\underbrace{\sum_{k\in n_{i}\cup a_{i}}\left(\frac{-A^{k}_{u,v}}{\sum_{x,y}A^{k}_{x,y}}\right)}_{\text{region-level}}+\beta\underbrace{\frac{\log(1-\sum_{k\in n_{i}\cup a_{i}}A_{u,v}^{k})}{|A|}}_{\text{pixel-level}}\right),$$ (5)

Figure 5: Overview of Attribute Concentration. Given a prompt, we first generate an image and record the cross-attention map for each token. We then identify regions of each entities in the prompt using the segmentation model. Finally, we optimize for the consistency between the entity attention map and its respective area in the image by encouraging positive and discouraging negative map**.

where $|A|$ is the number of pixels on the attention map, $\alpha$ and $\beta$ are two scaling factors, and $M^{i}$ should be resized to the resolution for each attention map $A$. The loss function covers the level of regions and pixels. Take the $\mathcal{L}_{\text{pos}}$ for example. We restrict the attention of each entity tokens $e_{i}$ only activated inside the positive region by the region-level loss. We further restrict each pixel in the positive region to only attend to entity tokens by the pixel-level loss. We take into account the scenario where certain objects in the prompt do not appear in the generated image due to misalignment. In this case, the negative loss of pixels is still valid. When the mask is entirely zero, it signifies that none of the pixels should attend to the missing entity tokens in the current image.

Finally, we combine the image-to-text model loss, adversarial loss and attribute concentration loss to build up our training objectives for the online diffusion model as follows:

$$\mathcal{L}=\mathcal{L}_{\text{i2t}}+\mathcal{L}_{\text{pos}}+\mathcal{L}_{\text{neg}}+\lambda\mathcal{L}_{\text{adv}},$$ (6)

where $\lambda$ are scaling factors to balance the loss. We provide the pseudocode for the loss computation process in Algorithm 1.

5 Experiment

5.1 Experimental Setup

Table 1: T2I-CompBench result. The best score is in blue, with the second-best score in green.

Model	Attribute Binding			Object Relationship		Complex$\uparrow$
	Color $\uparrow$	Shape$\uparrow$	Texture$\uparrow$	Spatial$\uparrow$	Non-Spatial$\uparrow$
PixArt-$\alpha$ [7]	0.6690	0.4927	0.6477	0.2064	0.3197	0.3433
Playground-v2 [25]	0.6208	0.5087	0.6125	0.2372	0.3098	0.3613
SD1.5 [44]	0.3758	0.3713	0.4186	0.1165	0.3112	0.3047
CoMat-SD1.5 (Ours)	0.6734	0.5064	0.6243	0.2073	0.3166	0.3575
	(+0.2976)	(+0.1351)	(+0.2057)	(+0.0908)	(+0.0054)	(+0.0528)
SDXL [38]	0.5879	0.4687	0.5299	0.2131	0.3119	0.3237
CoMat-SDXL (Ours)	0.7827	0.5329	0.6468	0.2428	0.3187	0.3680
	(+0.1948)	(+0.0642)	(+0.1169)	(+0.0297)	(+0.0068)	(+0.0443)

Base Model Settings. We mainly implement our method on SDXL [44] for all experiments, and we also evaluate our method on Stable Diffusion v1.5 [44] (SD1.5). For the captioning model, we choose BLIP [27] fine-tuned on COCO [33] image-caption data. We adopt the pre-trained UNet of SD1.5 as the discriminator in the fidelity preservation module. More training details are in Appendix E.1.

Dataset. Since the prompt to the diffusion model needs to be challenging enough to lead to missing concepts, we directly utilize the training data or text prompts provided in existing text-to-image alignment benchmarks. Specifically, the training data includes the training set provided in T2I-CompBench [22], all the data from HRS-Bench [3], and 5,000 prompts randomly chosen from ABC-6K [15]. Altogether, these amount to around 20,000 text prompts. Note that the training set composition can be freely adjusted according to the ability targeted to improve. The text-image pairs used in the mixed latent strategy are from the training set of COCO [33].

Benchmarks. We evaluate our method on three text-image alignment benchmarks and follow their default settings. T2I-CompBench [22] comprises 6,000 compositional text prompts evaluating 3 categories (attribute binding, object relationships, and complex compositions) and 6 sub-categories (color binding, shape binding, texture binding, spatial relationships, non-spatial relationships, and complex compositions). TIFA [21] uses pre-generated question-answer pairs and a VQA model to evaluate the generation results with 4,000 diverse text prompts and 25,000 questions across 12 categories. DPG-Bench [20] composes 1065 dense prompts with an average token length of 83.91. The prompt depicts a much more complex scenario with diverse objects and adjectives.

Table 2: TIFA and DPG-Bench results.

Model	TIFA$\uparrow$	DPG$\uparrow$
PixArt-$\alpha$ [7]	82.9	71.11
Playground-v2 [25]	86.2	74.54
SD1.5 [44]	78.4	63.18
CoMat-SD1.5 (Ours)	85.8	73.32
	(+7.4)	(+10.14)
SDXL [38]	85.9	74.65
CoMat-SDXL (Ours)	87.5	77.13
	(+1.6)	(+2.48)

Table 3: FID-10K result.

Model	$\mathcal{D}_{\phi}$	$\mathcal{D}_{\phi}$ input	ML	FID-10K$\downarrow$
SD1.5 [44]	N/A	N/A	N/A	16.69
CoMat-SD1.5	✗	N/A	✗	19.02
CoMat-SD1.5	UNet [45]	real-world latent	✗	17.99
CoMat-SD1.5	UNet [45]	generated latent	✗	16.69
CoMat-SD1.5	DINO [5]	generated image	✗	23.86
CoMat-SD1.5	UNet [45]	generated latent	✔	15.43

5.2 Quantitative Results

We compare our methods with our baseline models: SD1.5 and SDXL, and two state-of-the-art open-sourced text-to-image models: PixArt-$\alpha$ [7] and Playground-v2 [25].

T2I-CompBench. The evaluation result is shown in Table 1. Note that we cannot reproduce results reported in some relevant works [7, 22] due to the evolution of the evaluation code. All our shown results are based on the latest code released in GitHub. We observe significant gains in all six sub-categories compared with our baseline models. With our methods, SD1.5 can even achieve better or comparable results compared with PixArt-$\alpha$ and Playground-v2. Our CoMat-SDXL demonstrates the best performance regarding attribute binding, spatial relationships, and complex compositions.

TIFA. We show the results in TIFA in Table 3. Our CoMat-SDXL achieves the best performance with an improvement of 1.6 scores compared to SDXL. Besides, CoMat significantly enhances SD1.5 by 7.4 scores, which largely surpasses PixArt-$\alpha$.

DPG-Bench. The results in DPG-Bench is shown in Table 3. Although we do not train our model on dense prompts and can only accept 77 tokens, similar to Stable Diffusion, our method successfully generalizes to this more complex scenario and brings significant improvement to the baseline model.

5.3 Qualitative Results

Fig. 3 presents a side-by-side comparison between CoMat-SDXL and other state-of-the-art diffusion models. We observe these models exhibit inferior condition utilization ability compared with CoMat-SDXL. Prompts in Fig. 3 all possess concepts that are contradictory to real-world phenomena. All the three compared models stick to the original bias and choose to ignore the unrealistic content (e.g., waterfall cascading from a teapot, transparent violin, robot penguin, and waterfall of liquid gold), which causes misalignment. However, by training to faithfully align with the conditions in the prompt, CoMat-SDXL follows the unrealistic conditions and provides well-aligned images. The user study result and more visualization result is detailed in Appendix B.1 and F.2.

Table 4: Impact of concept activation and attribute concentration. ‘CA’ and ‘AC’ denote concept activation and attribute concentration respectively.

Model	CA	AC	Attribute Binding			Object Relationship		Complex$\uparrow$
			Color $\uparrow$	Shape$\uparrow$	Texture$\uparrow$	Spatial$\uparrow$	Non-Spatial$\uparrow$
SDXL			0.5879	0.4687	0.5299	0.2131	0.3119	0.3237
SDXL	✓		0.7455	0.5043	0.6252	0.2321	0.3171	0.3660
SDXL	✓	✓	0.7827	0.5329	0.6468	0.2428	0.3187	0.3680

Table 5: The impact of different image-to-text models.

Image-to-text Model	Attribute Binding			Object Relationship		Complex$\uparrow$
	Color $\uparrow$	Shape$\uparrow$	Texture$\uparrow$	Spatial$\uparrow$	Non-Spatial$\uparrow$
BLIP [27]	0.7827	0.5329	0.6468	0.2428	0.3187	0.3680
GIT [51]	0.6916	0.5146	0.5971	0.2404	0.3149	0.3413
LLaVA [34]	0.6338	0.4722	0.5518	0.1963	0.3117	0.3286
N/A	0.5879	0.4687	0.5299	0.2131	0.3119	0.3237

5.4 Ablation Study

Effectiveness of Concept Activation and Attribute Concentration. In Table 4, we show the T2I-CompBench result aiming to identify the effectiveness of the concept activation and attribute concentration modules. We find that the concept activation module accounts for major gains to the baseline model. On top of that, the attribute concentration module brings further improvement to all six sub-categories in T2I-CompBench. We show the qualitative effectiveness in Fig. 6.

Design of Fidelity Preservation and Mixed Latent. We examine the photorealism of generated images to evaluate the generation capability. We calculate the FID [16] score using 10K data from the COCO validation set. As shown in Table 3 and Fig. 10, without any preservation method, the diffusion model only tries to hack the image-to-text model and loses its original generation ability with an increase of FID score from 16.69 to 19.02. Besides, inputting the latent generated by the original diffusion model performs better than the latent of real-world images. As for the discriminator architecture, the UNet is superior to a pre-trained DINO [5] which even interferes the training process. Finally, the Mixed Latent (ML) strategy further enhances the generated image quality.

Figure 6: Visualization of the effectiveness of the proposed modules. CA contributes to the existence of objects mentioned in the prompts. AC further guides the attention of the attributes to focus on their corresponding objects.

Different Image-to-text Models. We show the T2I-CompBench results with different image captioning models in Table 5. We find that all three image-to-text models can boost the performance of the diffusion model with our framework, where BLIP achieves the best performance. We provide more analysis on the choice of the image-to-text models in Appendix B.3.

6 Conclusion

In this paper, we propose CoMat, an end-to-end diffusion model fine-tuning strategy equipped with image-to-text concept matching. We identify the two causes of the misalignment problem and propose two key components to explicitly address them. The concept activation module leverages an image-to-text model to supervise the generated image and find out the ignored condition information. It also integrates the fidelity preservation module and mixed latent strategy to maintain the generation capability. Besides, we introduce the attribute concentration module to address the attribute mismap** issue. Through extensive experiments, we have demonstrated that CoMat largely outperforms its baseline model and even surpasses commercial products in multiple aspects. We hope our work can inspire future work on the cause of the misalignment and the solution to it.

Appendix A Algorithm

The pseudocode for a single loss computation step for the online T2I-Model is described in  Algorithm 1.

Algorithm 1 A single loss computation step for the online T2I-Model during fine-tuning

Input: Text prompt $\mathcal{P}$, GT Image $\mathcal{I}_{g}$, GT Prompt $\mathcal{P}_{g}$, original T2I-Model $\epsilon_{pre}$, online T2I-Model $\epsilon_{\theta}$, pre-trained I2T-Model $\mathcal{C}$, discriminator $\mathcal{D}_{\phi}$, segmentation model $\mathcal{S}$, timestep range $[T_{1},T_{2}]$, timestep $\tau$, attention map $A$, scaler $\lambda$; $\left[;\right]$ denotes concatenate

1:  $x_{T},\xi\sim\mathcal{N}(0,I)$

2:  $\tau\sim\text{Uniform}[T_{1},T_{2}]$

3:  $x_{\tau}=\text{AddNoise}(\mathcal{I}_{g},\xi,\tau)$

4:  $\hat{\mathcal{I}},A=\text{GenerateImage}(\epsilon_{\theta},x_{T},\mathcal{P})$

5:  $\hat{\mathcal{I}_{g}},\underline{\phantom{A}}=\text{GenerateImage}(\epsilon_{\theta},x_{\tau},\mathcal{P}_{g})$

6:  $\mathcal{L}_{i2t}=\text{ComputeI2TLoss}(\mathcal{C},\left[\hat{\mathcal{I}};\hat{\mathcal{I}_{g}}\right],\left[\mathcal{P};\mathcal{P}_{g}\right])$

7:  $\hat{\mathcal{I}}_{pre},\underline{\phantom{A}}=\text{GenerateImage}(\epsilon_{pre},x_{T},\mathcal{P})$

8:  $\mathcal{L}_{adv}=\text{ComputeAdvLoss}(\mathcal{D}_{\phi},\hat{\mathcal{I}},\hat{\mathcal{I}}_{pre})$

9:  $\mathcal{L}_{pos},\mathcal{L}_{neg}=\text{ComputeAttrLoss}(\mathcal{S},\hat{\mathcal{I}},\mathcal{P},A)$

10:  $\mathcal{L}=\mathcal{L}_{i2t}+\mathcal{L}_{pos}+\mathcal{L}_{neg}+\lambda\mathcal{L}_{adv}$

Output: Training loss for the online T2I-Model $\mathcal{L}$

Appendix B Additional Results and Analysis

B.1 User preference study

Figure 7: User preference study results.

We randomly select 100 prompts from DSG1K [10] and use them to generate images with SDXL [44] and our method (CoMat-SDXL). We ask 5 participants to assess both the image quality and text-image alignment. Human raters are asked to select the superior respectively from the given two synthesized images, one from SDXL, and another from our CoMat-SDXL. For fairness, we use the same random seed for generating both images. The voting results are summarised in Fig. 7. Our CoMat-SDXL greatly enhances the alignment between the prompt and the image without sacrificing the image quality.

B.2 Composability with planning-based methods

Since our method is an end-to-end fine-tuning strategy, we demonstrate its flexibility in the integration with other planning-based methods, where combining our method also yields superior performance. RPG [61] is a planning-based method utilizing Large Language Model (LLM) to generate the description and subregion for each object in the prompt. We refer the reader to the original paper for details. We employ SDXL and our CoMat-SDXL as the base model used in [61] respectively. As shown in Fig. 8, even though the layout for the generated image is designed by LLM, SDXL still fails to faithfully generate the single object aligned with its description, e.g., the wrong mat color and the missing candle. Although the planning-based method generates the layout for each object, it is still bounded by the base model’s condition following capability. Combining our method can therefore perfectly address this issue and further enhance alignment.

Figure 8: Pipeline for integrating CoMat-SDXL with planning-based method. CoMat-SDXL correctly generates the green mat in the upper row and the tall white candle in the bottom row.

B.3 How to choose an image-to-text model?

We provide a further analysis of the varied performance improvements observed with different image-to-text models, as shown in Table 5 of the main text.

Figure 9: Examples for the three core sensitivities.

For an image-to-text model to be valid for the concept activation module, it should be able to tell whether each concept in the prompt appears and appears correctly. We construct a test set to evaluate this capability of the image-to-text model. Intuitively, given an image, a qualified image-to-text model should be sensitive enough to the prompts that faithfully describe it against those that are incorrect in certain aspects. We study three core demands for an image-to-text model:

• Attribute sensitivity. The image-to-text model should distinguish the noun and its corresponding attribute. The corrupted caption is constructed by switching the attributes of the two nouns in the prompt.

• Relation sensitivity. The image-to-text model should distinguish the subject and object of a relation. The corrupted caption is constructed by switching the subject and object.

• Quantity sensitivity. The image-to-text model should distinguish the quantity of an object. Here we only evaluate the model’s ability to tell one from many. The corrupted caption is constructed by turning singular nouns into plural or otherwise.

We assume that they are the basic requirements for an image-to-text model model to provide valid guidance for the diffusion model. Besides, we also choose images from two domains: real-world images and synthetic images. For real-world images, we randomly sample 100 images from the ARO benchmark [63]. As for the synthetic images, we use the pre-trained SD1.5 [44] and SDXL [38] to generate 100 images according to the prompts in T2ICompBench [22]. These selections make up for the 200 images in our test data. We show the examples in Fig. 9.

For the sensitivity score, we compare the difference between the alignment score (i.e., log-likelihood) of the correct and corrupted captions for an image. Given the correct caption $\mathcal{P}$ and corrupted caption $\mathcal{P}^{\prime}$ corresponding to image $\mathcal{I}$, we compute the sensitivity score $\mathcal{S}$ as follows:

$$\mathcal{S}=\frac{\log(p_{\mathcal{C}}(\mathcal{P}|\mathcal{I}))-\log(p_{\mathcal{C}}(\mathcal{P}^{\prime}|\mathcal{I}))}{\left|\log(p_{\mathcal{C}}(\mathcal{P}|\mathcal{I}))\right|}.$$ (7)

Then we take the mean value of all the images in the test set. The result is shown in Table 6. The rank of the sensitivity score aligns with the rank of the gains brought by the image-to-text model model shown in the main text. Hence, except for the parameters, we argue that sensitivity is also a must for an image-to-text model to function in the concept activation module.

Table 6: Statistics of image-to-text models.

Image-to-text Model	Parameters	Sensitivity Score
BLIP [27]	469M	0.1987
GIT [51]	394M	0.1728
LLaVA [34]	7.2B	0.1483

Appendix C More Related Work

The image-to-text model in the main text refers to the models capable of image captioning. Previous image captioning models are pre-trained on various vision and language tasks (e.g., image-text matching, (masked) language modeling) [29, 35, 23, 49], then fine-tuned with image captioning tasks [9]. Various model architectures have been proposed [52, 62, 28, 27, 51]. BLIP [27] takes a fused encoder architecture, while GIT [51] adopts a unified transformer architecture. Recently, multimodel large language models have been flourishing [34, 68, 64, 1]. For example, LLaVA [34] leverages LLM as the text decoder and obtains impressive results. When prompted properly, these models are also able to provide a detailed image caption.

Similar to our work, [14] proposes to caption the generated images and optimize the coherence between the produced captions and text prompts. Although an image-to-text model is also involved, they fail to provide detailed guidance. It has been shown that the generated captions are prone to omit key concepts and involve undesired added features [21]. Besides, the method leverages a pre-trained text encoder to compute the similarity between the prompt and generated caption, which further causes information to be missed during text encoding. All these designs lead the optimization target to be vague and sub-optimal.

C.1 vs. Differentiable Reward Method

Similarity: Our method is inspired by the technique introduced in the differentiable reward method to perform gradient update.

Difference: (1) Reward Model. Our method is the first to leverage an image-to-text model to perform image captioning on the generated image and compute the loss on the caption. (2) No fidelity preservation. The current differentiable reward method ignores the aspect of preserving the generation capability if not training against a reward of image quality. Our method introduces a novel fidelity preservation module, which utilizes a discriminator with similar knowledge to preserve the generation capability. This greatly alleviates the reward hacking problem introduced by only training with the differentiable reward method. (3) No guidance from real-world image. The current differentiable reward method all starts from pure noise. Since our method is optimizing for alignment, we can incorporate real-world image-text pairs to guide the optimization process. With our mixed latent strategy, the latent starting from the noise is conditioned on the difficult prompt to promote alignment, while the latent starting from the noisy GT image is used to prohibit the diffusion model from overfitting to the image-to-text model.

C.2 vs. TokenCompose [54]

Similarity: Both [54] and our method incorporates the object mask to guide the attention of the diffusion model.

Difference: (1) Limited and inferior optimizing target. [54] merely focuses on optimizing the consistency between the noun mask and the object mask. However, as shown in Fig. 5, the attention mask of the noun (the ‘bear’ token) has already aligned well with the object mask. Optimizing for this consistency is inferior. On the other hand, our method focuses on a much broader area, i.e., entity tokens, which consist of nouns and their various associated attributes. We also find that the consistency between the attributes and the object mask bears very little similarity, which should be paid more attention. (2) No negative concept map**. Since the training data of [54] is the real-world image-text pairs, all the nouns in the prompt show up in the image. However, this prohibits the model from learning in a negative way, i.e., if the entity is not on the image, none of the pixel should be activated by this token. Our method leverages images generated by the diffusion model. The entity missing is common. The model obtains the chance to learn in a negative way. (3) No difficult training data. Another issue caused by training with image-text pairs is that the training data may be of a common scenario, which is easier to learn. Since our method does not need real-world images and only starts from the noise and text prompt, this enables a more efficient training process.

Appendix D Limitation

How to more effectively incorporate Multimodal Large Language Models (MLLMs) into text-to-image diffusion models by our proposed method requires more exploration. MLLM possesses state-of-the-art image-text understanding capability in addition to image captioning. We will focus on leveraging MLLMs to provide finer-grained guidance to the diffusion model in our future work. In addition, we also observe the potential of CoMat for adapting to 3D domains, promoting the text-to-3D generation with stronger alignment. As for the training cost, since our method needs the diffusion model to perform the whole inference process, the training time is extended. Our future direction will be to accelerate the training process.

Appendix E Experimental Setup

E.1 Implementation Details

Training Details. In our method, we inject LoRA [19] layers into the UNet of the online training model and discriminator and keep all other components frozen. For both SDXL and SD1.5, we train 2,000 iters on 8 NVIDIA A100 GPUS. We use a local batch size of 6 for SDXL and 4 for SD1.5. We choose Grounded-SAM [43] from other open-vocabulary segmentation models [66, 69]. The DDPM [17] sampler with 50 steps is used to generate the image for both the online training model and the original model. In particular, we follow [57] and only enable gradients in 5 steps out of those 50 steps, where the attribute concentration module would also be operated. Besides, to speed up training, we use training prompts to generate and save the generated latents of the pre-trained model in advance, which are later input to the discriminator during fine-tuning. Training Resolutions. We observe that training SDXL is very slow due to the large memory overhead at $1024\times 1024$. However, SDXL is known to generate low-quality images at resolution $512\times 512$. This largely affects the image understanding of the image-to-text model. So we first equip the training model with better image generation capability at $512\times 512$. We use our training prompts to generate $1024\times 1024$ images with pre-trained SDXL. Then we resize these images to $512\times 512$ and use them to fine-tune the UNet of SDXL for 100 steps, after which the model can already generate high-quality $512\times 512$ images. We continue to implement our method on the fine-tuned UNet.

Training Layers for Attribute Concentration. Only cross-attention maps in the middle blocks and decoder blocks are used to compute the loss.

Hyperparameters Settings. We provide the detailed training hyperparameters in Table 7.

Table 7: CoMat training hyperparameters for SD1.5 and SDXL.

Name	SD1.5	SDXL
Online training model
Learning rate	5e-5	2e-5
Learning rate scheduler	Constant	Constant
LR warmup steps	0	0
Optimizer	AdamW	AdamW
AdamW - $\beta_{1}$	0.9	0.9
AdamW - $\beta_{2}$	0.999	0.999
Gradient clip**	$0.1$	$0.1$
Discriminator
Learning rate	5e-5	5e-5
Optimizer	AdamW	AdamW
AdamW - $\beta_{1}$	$0$	$0$
AdamW - $\beta_{2}$	$0.999$	$0.999$
Gradient clip**	$1.0$	$1.0$
Token loss weight $\alpha$	1e-3	1e-3
Pixel loss weight $\beta$	5e-5	5e-5
Adversarial loss weight $\lambda$	1	5e-1
Gradient enable steps	$5$	$5$
Attribute concentration steps $r$	$2$	$2$
LoRA rank	128	128
Classifier-free guidance scale	7.5	7.5
Resolution	$512\times 512$	$512\times 512$
Training steps	2,000	2,000
Local batch size	$4$	$6$
Local GT batch size	$2$	$2$
Mixed Precision	FP16	FP16
GPUs for Training	8 $\times$ NVIDIA A100	8 $\times$ NVIDIA A100
Training Time	$\sim$ 10 Hours	$\sim$ 24 Hours

Appendix F More Qualitative Results

Figure 10: Visualization result of the effectiveness of the Fidelity Preservation module (FP) and Mixed Latent (ML) strategy.

F.1 Effectiveness of the Fidelity Preservation module (FP) and Mixed Latent (ML) strategy

We visualize the effectiveness of how we preserve the generation capability of the diffusion model in Fig. 10. As shown in the figure, without any preservation technique, the diffusion model generates misshaped envelopes and swans. With the FP and ML applied, the diffusion model generates images aligned with the prompt and without artifacts.

F.2 Comparison with the baseline model

We showcase more comparison results between our method with the baseline model in Fig. 11 to 14. Fig. 11 shows the generation results with long and complex prompts. Fig. 12 to 14 shows that our method solves various problems of misalignment, including object missing, incorrect attribute binding, incorrect relationship, inferior prompt understanding.

Figure 11: More Comparisons between SDXL and CoMat-SDXL on complex prompts. All pairs are generated with the same random seed.

Figure 12: More Comparisons between SDXL and CoMat-SDXL. All pairs are generated with the same random seed.

Figure 13: More Comparisons between SDXL and CoMat-SDXL. All pairs are generated with the same random seed.

Figure 14: More Comparisons between SDXL and CoMat-SDXL. All pairs are generated with the same random seed.