AnyControl: Create Your Artwork with Versatile Control on Text-to-Image Generation

Abstract

The field of text-to-image (T2I) generation has made significant progress in recent years, largely driven by advancements in diffusion models. Linguistic control enables effective content creation, but struggles with fine-grained control over image generation. This challenge has been explored, to a great extent, by incorporating additional user-supplied spatial conditions, such as depth maps and edge maps, into pre-trained T2I models through extra encoding. However, multi-control image synthesis still faces several challenges. Specifically, current approaches are limited in handling free combinations of diverse input control signals, overlook the complex relationships among multiple spatial conditions, and often fail to maintain semantic alignment with provided textual prompts. This can lead to suboptimal user experiences. To address these challenges, we propose AnyControl, a multi-control image synthesis framework that supports arbitrary combinations of diverse control signals. AnyControl develops a novel Multi-Control Encoder that extracts a unified multi-modal embedding to guide the generation process. This approach enables a holistic understanding of user inputs, and produces high-quality, faithful results under versatile control signals, as demonstrated by extensive quantitative and qualitative evaluations. Our project page is available in https://any-control.github.io.

1 Introduction

In recent years, the field of text-to-image (T2I) generation has experienced significant advancements, leading to unprecedented improvements in generated image quality and diversity, primarily attributed to the introduction of diffusion models [13, 38, 14, 33]. While linguistic control allows for effective and engaging content creation, it also presents challenges in achieving fine-grained control over image generation. This challenge is extensively explored in [44, 25], where an additional network is employed to encode and inject the user-supplied control signal into the pre-trained T2I model such as Stable Diffusion [33], so that to exert influence over the image generation process. Built upon [44], subsequent approaches [28, 45, 15] presents unified architecture designs for managing multiple spatial conditions.

However, the task of multi-control image synthesis remains challenging in the following aspects: (1) accommodating free combinations of input conditions, (2) modeling complex relationships among multiple spatial conditions, and (3) maintaining compatibility with textual prompts. We refer to these three challenges as input flexibility, spatial compatibility, and textual compatibility, respectively.

Input flexibility. The first challenge comes from the any combination of available control signals based on user requirements. The amount and modality of control signals provided by users are varying, placing high demands on the input flexibility of the model. However, existing methods [45, 15] typically employ fixed-length input channels, limiting their ability to accommodate diverse inputs. Other approaches [44, 25, 28] adopt MoE design to solve varying-number conditions, which can result in unforeseen artifacts when processing unseen combinations of inputs.

Spatial compatibility. Secondly, control signals are not isolated; instead, they collectively influence the composition of a complete image. It is crucial to consider the relationships among these control signals, especially when managing occlusions among multiple spatial conditions. Unfortunately, current algorithms commonly combine multiple conditions through weighted summation with hand-crafted weights, easily leading to undesired blending results, or even causing low-response control signals to disappear when addressing occlusions.

Textual compatibility. Ultimately, textual compatibility emerges as an important factor influencing user experience. Typically, the textual descriptions govern the content of generated images, whereas spatial conditions compensate the structural information. Nevertheless, a lack of communication between the textual and spatial conditions often leads current algorithms to prioritize accommodating the spatial conditions, thereby disregarding the impact of textual prompts.

In summary, generating comprehensive and harmonious results that satisfy both textual prompts and multiple spatial conditions presents a significant challenge for multi-control image synthesis. To tackle the challenges of input flexibility, spatial compatibility, and textual compatibility, we propose AnyControl, a controllable image synthesis framework that supports arbitrary combinations of diverse control signals.

At the core of AnyControl is the Multi-Control Encoder, which plays a crucial role in ensuring coherent, spatially and semantically aligned multi-modal embeddings. This novel component allows AnyControl to extract a unified representation from various control signals, enabling a truly versatile and high-performing multi-control image synthesis framework.

Specifically, Multi-Control Encoder is driven by multi-control fusion block and multi-control alignment block in turns, with a set of query tokens to unite the two seamlessly.

Multi-control fusion block is employed to aggregate compatible information from multiple spatial conditions through the query tokens. A cross-attention transformer block is employed on the query tokens and the visual tokens of spatial conditions extracted from a pre-trained visual encoder. Therefore, the rich spatial controllable information is passed to the query tokens, which will be utilized in the multi-control alignment block.

Multi-control alignment is used to guarantee the compatibility between all forms of control signals by aligning all other signals to the textual signal. A self-attention transformer block is employed on the query tokens and textual tokens. The query tokens contain spatial controllable information, while the textual tokens carry semantic information. Through information exchange between the query and textual tokens, both types of tokens are able to represent compatible multi-modal information.

With alternating multi-control fusion and alignment blocks in several turns, the query tokens achieve a comprehensive understanding with highly aligned and compatible information from versatile user inputs. This capability empowers our method to handle complex relationships among conditions and uphold strong compatibility with textual prompts. Consequently, this approach fosters a more smooth and harmonious control over the generated images. Furthermore, transformer blocks with attention mechanisms inherently excel in accommodating a variety of control signals, and thus enable free combinations of user inputs.

In summary, our contributions are manifold in the following:

1. AnyControl proposes a novel Multi-Control Encoder comprising a sequence of alternative multi-control fusion and alignment blocks to achieve comprehensive understanding of complex multi-modal user inputs.

2. AnyControl supports flexible combinations of user inputs, regardless of the amount and modality of different control signals.

3. AnyControl produces more harmonious and natural high-quality outcomes, demonstrating state-of-the-art performance in multi-control image synthesis.

2 Related Work

2.1 Text-to-image Generation

T2I diffusion models [33, 31, 36, 26] have emerged as a promising approach for generating high-quality images from textual prompts. Diffusion models [13, 38], originally developed for image generation, have been adapted to the T2I domain, offering a novel perspective on the problem. These models leverage the concept of iterative denoising, where the generation process unfolds step-by-step, progressively refining the image quality. The diffusion process allows for better control over the generated images by conditioning on both the text input and intermediate image representations at each diffusion step. Recent advances in T2I diffusion models have explored various techniques to enhance the generation process, such as introducing attention mechanisms to better align textual and visual features [33] and operating in latent space [33, 31] to achieve complexity reduction and detail preservation. While T2I diffusion models have shown promising results, there is still ongoing research to address challenges such as controllability, also the focus discussed in this paper, in the context of diffusion-based T2I generation.

2.2 Controllable Image Synthesis

Text descriptions guide the diffusion model to generate user-desired images but are insufficient in fine-grained control over the generated results. The fine-grained control signals are diverse in modality, for instance, layout constraint is introduced to arrange the location of the objects given; a bundle of works [6, 46, 17, 41, 29, 40, 22, 42] thoroughly explore to synthesis images with high layout alignment given the semantic-aware boxes. Besides, segmentation map [7, 8, 1, 2] is another popular control signal for controlling the layout and object shape of generated images. InstanceFusion [39] proposes a method to support location control in more free-form such as point, scribble, box and segmentation map. Highly detailed control can be achieved by structure signal such as sketch [3]. Depth map [20] can provide the control of the depth of field for the generated images. As layout, structure and depth control all outline the generated image in spatial alignment, content control [9, 35, 43] enables the personalization of the generated appearance in semantic level through an additional image input.

Studies [25, 44, 24] propose general framework designs to process diverse spatial conditions instead of control-specific design, while both spatial and content control are jointly taken into consideration in works [16, 18]. Specifically, considering the powerful generation ability of T2I model, ControlNet [44] proposes to utilize the trainable copy of the UNet encoder in the T2I diffusion model to encode extra condition signals into latent representations and then apply zero convolution to inject into the backbone of the UNet in diffusion modal. The simple but effective design shows generalized and stable performance in spatial control, and thus are widely adopted in various downstream applications. However, ControlNet is a single-modality framework and requires separate model for each modality. To address this, unified ControlNet-like models [28, 45, 15] are proposed to handle diverse control signals with only one multi-modality model. Another advantage of these methods is that they can support multi-control image synthesis. They adopt fixed length input channels or MoE design with hand-crafted weighted summation to aggregate conditions. Nevertheless, these methods are short in handling conditions with complex relations and hard to generate harmonious, natural results under various control signals.

Figure 2: AnyControl and Multi-Control Encoder. Left shows the overall framework of our AnyControl, which develops a Multi-Control Encoder for extracting comprehensive multi-control embeddings based on the textual prompts and multiple spatial conditions. The multi-control embeddings are then utilized to guide the generation process. Right shows the detailed design of our Multi-Control Encoder driven by alternating multi-control fusion and alignment blocks, with query tokens defined to distill the compatible information from textual tokens and visual tokens of the spatial conditions.

3 Method

In this section, we first give a preliminary overview of Stable Diffusion [33] and ControlNet [44]. Subsequently, we introduce AnyControl, featuring a pioneering Multi-Control Encoder crafted for extracting a unified representation with compatible information for multiple control signals. Finally, we expound on our training dataset and strategy. Figure 2 depicts the architecture of AnyControl and the Multi-Control Encoder.

3.1 Preliminary

Stable Diffusion. T2I generation introduces text as conditions in diffusion models. In the forward pass, Gaussian noises are gradually added to the sample over a series of steps; while the backward process learns to recover the image by estimating and eliminating the noise with the text guidance. In this paper, we base Stable Diffusion [33], one of the most popular T2I diffusion model, to develop AnyControl for multi-control image synthesis.

Stable Diffusion model operates the diffusion and denoising process in latent space rather than pixels to reduce computation cost. It adopts UNet-like [34] structure as its backbone, comprising downsampling blocks, middle block and upsampling blocks. The text guidance are encoded through CLIP [30] text encoder and integrated into the UNet through a CrossAttention block after each ResBlock [10]. If we use $Z$ to denote the noise features derived from the last ResBlock and $Y$ to denote the embeddings encoded by the text encoder, the output noise features $\tilde{Z}$ from CrossAttention block can be obtained by

$$Q=W_{q}(Z),K=W_{k}(Y),V=W_{v}(Y),$$ (1)

$$\tilde{Z}=Softmax(\frac{QK^{T}}{\sqrt{d}})V,$$ (2)

where $W_{q}$, $W_{k}$ and $W_{v}$ are projection layer and $d$ is the dimension of the embedding space.

ControlNet. ControlNet [44] is developed to adapt Stable Diffusion model for spatial conditions. To be specific, it locks the parameters of Stable Diffusion, and makes a trainable copy of the encoding layers in the UNet. The two parts are connected by zero convolution layers with zero-initialized weights to progressively increase spatial control influence as the training goes. This design empowers ControlNet to achieve robust controllable image generation while preserve the quality and capabilities of Stable Diffusion model.

3.2 AnyControl

Multi-Control Encoder. Similar to ControlNet, in our AnyControl, we also lock the pre-trained stable diffusion model, and instead design a Multi-Control Encoder for understanding complex control signals. We first obtain three types of tokens, i.e., textual tokens $\mathcal{T}$, visual tokens $\mathcal{V}$ and query tokens $\mathcal{Q}$. The textual tokens are extracted from CLIP text encoder on textual prompts, while the visual tokens are obtained from a pre-trained visual encoder (e.g., CLIP image encoder) on all of the user-provided spatial conditions in image form. The query tokens are defined as a set of learnable parameters. To address the three challenges discussed in the introduction, i.e., input flexibility, spatial compatibility and textual compatibility, we develop the multi-control encoder via alternating multi-control fusion blocks and multi-control alignment blocks united by the query tokens.

Multi-Control Fusion. Multi-control fusion block aims to extract compatible information from various spatial conditions. This is accomplished by utilizing a cross-attention transformer block to facilitate interactions between the query tokens and the visual tokens of all spatial conditions.

Specifically, suppose that there are $n$ spatial conditions in image form of various modalities including depth, segmentation, etc. We can obtain the visual tokens $\mathcal{V}_{i,j}$ for condition $C_{i}$ from the $j$-th block of the pre-trained visual encoder. Here, we use $[\mathcal{V}_{1,j},\mathcal{V}_{2,j},\dots,\mathcal{V}_{n,j}]$ to represent the visual tokens for all the spatial conditions from the $j$-th block. Then the interactions in the multi-control fusion block can be formulated as

$$\mathcal{Q}_{j}=CrossAttention(\mathcal{Q}_{j},[\mathcal{V}_{1,j}+P,\mathcal{V}_{2,j}+P,\dots,\mathcal{V}_{n,j}+P]),$$ (3)

where $P$ denotes a shared learnable positional embedding additive to each $\mathcal{V}_{i,j}$ for better alignment between the query tokens and the visual tokens.

After this process, the spatial controllable information encoded in the visual tokens is passed on to the query tokens.

Multi-Control Alignment. Although the various controllable information is integrated into the query tokens, it is challenging to infer the priority of spatial control signals within the overlap** region due to the absence of a global condition that indicates the relationships among spatial conditions. Fortunately, textual prompts can serve as a global control that regulates the content of generated image. Therefore, in the multi-control alignment block, we facilitate the interactions between the query tokens and the textual tokens with a self-attention transformer block. Before we encode the textual prompts to tokens, we append a textual task prompt at the tail of the user-provided text to solve the modality discrepancy among diverse spatial conditions. Then we concatenate the query tokens $\mathcal{Q}$ and textual tokens $\mathcal{T}$ together and perform the self-attention as

$$[\mathcal{Q}_{j+1},\mathcal{T}_{j+1}]=SelfAttention([\mathcal{Q}_{j},\mathcal{T}_{j}]).$$ (4)

With self-attention, the query tokens, which carry the mixed controllable information, will exchange information with textual tokens and thus can achieve semantic alignment with user prompts.

Alternating Fusion and Alignment. To ensure the information aligned and compatible of all the control signals, we employ the multi-control fusion and alignment blocks alternately for multiple turns. Notably, we utilize multi-level visual tokens for fine-grained spatial control. Specifically, in each turn, the visual tokens consumed in the cross-attention transformer block are extracted from different levels of the pre-trained visual encoder, considering the spatial conditions are diverse in controlling level, i.e., layout control such as segmentation map and structural control such as edge map. Therefore, multi-level visual tokens are necessary for multi-control fusion blocks in different depth.

Advantages of AnyControl. The query tokens work as a bridge, uniting the two types of blocks seamlessly. After several turns, the query tokens retain well-aligned compositional information, served as a unified multi-modal representations for user inputs. This design empowers AnyControl in multi-control image synthesis even with occlusion, generating high-quality harmonious results with high spatial and textual compatibility. Our multi-control encoder shares a similar idea to Q-Former [21], however, AnyControl incorporates many dedicated design for the multi-control image synthesis such as the appended textual task prompt, additional shared position embeddings across all the conditions and the usage of multi-level visual tokens. In implementation, to save computation cost, we insert the cross-attention block after every two self-attention blocks.

Another natural advantage of our AnyControl lies in the input flexibility. AnyControl, utilizing the transformer blocks with attention mechanism, has natural advantage in accommodating free combinations of user inputs. Previous methods either adopt the design of fixed-length input channels or MoE structure as illustrated in Figure 3. The former limits the freedom of user inputs, while the latter, MoE design, supports combining flexible inputs with hand-crafted weighted summation, leading to laborious adjustments to the combination weights.

Figure 3: Three types of multi-control methods. Square in different color denotes different condition type while dotted square denotes zero tensor. (a) Some methods [45, 15] adopt fixed-length channels of the input convolution layer followed by several convolution blocks to serve as the Multi-Control Encoder. (b) Other methods [28, 44, 25] utilize the MoE design, that is, construct separate encoder for each type of control signal and then obtain the embeddings through weighted sum. (c) Different from them, AnyControl adopts attention mechanism to accommodate varying-number and varying-modality of conditions. “SAB” and “CAB” denotes self- and cross-attention block, respectively.

3.3 Training

Datasets.

Figure 4: Visualization of aligned and unaligned conditions. The first row shows the aligned case where pixels at the same location of all the control signals describe the same object. Conditions in the second and third rows describe the foreground and background respectively, contributing to a complete image together, constructing the unaligned case.

We adopt the training dataset, MultiGen, for multi-control image synthesis presented in [28]. This dataset is built from LAION [37] with aesthetics score above 6. Low-resolution images are removed and finally 2.8M images are kept. Different methods are utilized to extract the control signals. Unfortunately, there is a domain gap between the combinations of the spatial conditions during training and inference time, i.e., during training, all the spatial conditions extracted from the same image are fully aligned while the multiple spatial conditions accepted from users are not the case. User-provided conditions usually have multiple image sources, thus the extracted spatial conditions are not always aligned and sometimes have occlusion in the overlap** region, which requires the model to handle the spatial conditions in right arrangement according to the depth of the target scene. To relieve the discrepancy, we collect a subset of unaligned data as shown in Figure 4. To be specific, we utilize the images in Open Images dataset [19] and MSCOCO [23] dataset which are rich in objects to make the synthetic data. Given an image and the mask of a foreground object, we recover the background image with the masked region using the inpainting tool [47]. We discard images with too small or too large objects, and finally produce 0.44M images as supplementary unaligned training data.

Training Strategy. When utilizing the unaligned data for training, we take the combination of spatial conditions for the foreground object and the inpainted background image together while treating the original image as target. During training, for the data with fully aligned spatial conditions, we randomly pick two conditions for each training sample; for the synthetic unaligned data, we randomly pick a condition for the foreground object and the background inpainted image respectively. We randomly drop all the conditions at a rate of 0.05 to enable classifier free guidance, and also randomly drop the textual prompts at a rate of 0.05 to let the model learn from pure spatial conditions only.

4 Experiments

We validate the effectiveness of AnyControl with Stable Diffusion [33] of version 1.5 on four types of conditions, including Edge [4], Depth [32], Segmentation [27], and Human Pose [5]. We compare our AnyControl with state-of-the-art methods including Multi-ControlNet and Multi-Adapter, the versions of ControlNet [44] and T2I-Adapter [25] which support multiple spatial conditions, as well as Uni-ControlNet [45], Cocktail [15], UniControl [28], DiffBlender [18] and CnC [20] with extensive qualitative and quantitative results. Implementation details including network structure, hyper-parameters of training and inference can be found in the supplementary material.

Figure 5: Comparison on multi-control image synthesis. Multi-ControlNet adopts MoE design to process diverse conditions while Cocktail adopts the composition design by combining multiple conditions of the same type into one.

4.1 Qualitative Results

In this section, we analyze input flexibility, spatial compatibility, and also the compatibility with text, style and color control.

Input Flexibility. There are three ways to process free combinations of spatial conditions from users: 1) MoE design (i.e., UniControl[28], Multi-ControlNet [44]); 2) Attention design (AnyControl); 3) Composition design, which merges spatial conditions of the same type into one image so that methods with fixed-length input channels can work smoothly. In MoE-based methods, the composition of different conditions is achieved by hand-crafted weighted summation. Instead, our AnyControl adopts attention mechanism to learn the composition weights dynamically, achieving superior performance on multi-control image synthesis as shown in Figure 5 and Figure 6. In addition,“sticker” artifact is observed in the results of Cocktail [15] with composition design.

Figure 6: Varied-number and varied-type of input spatial conditions.

Figure 7: Spatial compatibility. AnyControl is capable of inferring the relationships not only between conditions but also between the generated objects and the environment.

Spatial Compatibility. In Figure 5, we provide comparisons given various conditions with occlusion, demonstrating the superiority of AnyConrol in handling complex multi-control synthesis. Blending issues are difficult to avoid in previous methods [44, 45, 15] with trivial design on multi-control combination.

Figure 7 further shows examples on AnyControl in dealing with relative spatial positions of two conditions, which generates high-quality results with correct spatial relation between conditions. For example, given different layout arrangements, the generated cat and teddy bear from AnyControl are always positioned in a natural way, rather than mixed within the occluded region. Another notable thing is that AnyControl shows a surprising ability of dealing with the interaction of generated objects and the corresponding environment. When placed at the same horizontal line, the cat and teddy bear are seated at the same plane, while the teddy bear sitting on a stage when its vertical position axis is raised up. These advantages contribute to the introduction of the Multi-Control Encoder, which strengthens the interactions between all control signals and consequently achieves a comprehensive understanding of complex user inputs.

Text Compatibility. Textual prompts play an important role in controllable image synthesis as text is typically the primary communication means between human and T2I models, while spatial conditions work as auxiliary roles in providing the fine-grained information. Therefore, while guided by spatial conditions, maintaining the compatibility with textual prompts is essential. However, existing methods commonly prioritize the response to spatial conditions and miss important message in textual prompts. For instance, in the third row of Figure 5, other methods either totally neglect the “Transformers Robot” information so that fail to bind the concept with the input human pose, or only partially respond to “Robot” but drop “Transformers” information. On the contrary, our method responds to all important information in multi-modality user inputs and produces harmonious results.

Compatibility with Style and Color Control. As a plug-and-play model,

Figure 8: AnyControl with style and color controls. The first two cases take style, depth and edge controls, while the last further takes color control.

AnyControl can be integrated with existing conditional generation methods in a convenient way. We take style and color controls as examples to demonstrate the effectiveness of AnyControl in the collaboration with other plug-and-play modules for wider applications. Specifically, we enhance AnyControl with decoupled cross-attention [43] in UNet to employ style and color controls. The compositional outcomes are visually depicted in Figure 8, revealing the generation of high-quality results that adhere to style, color and spatial constraints.

4.2 Multi-Control Benchmark: COCO-UM

Most existing methods evaluate multi-control image synthesis on MSCOCO validation set with totally spatio-aligned conditions extracted from different algorithms. However, we argue that evaluation on the dataset with well-aligned multi-control conditions cannot reflect the ability of methods to handle occluded multiple conditions in practical applications, given that the user provided conditions are typically collected from diverse sources which are not aligned.

Therefore, we construct an Unaligned Multi-control benchmark based on MSCOCO validation set, short for COCO-UM, for a more effective evaluation on multi-control image synthesis. The construction pipeline is similar to that used in unaligned data synthesis described in Section 3.3. That is, we decompose an image into the background image and foreground image outlined by an object mask, and then recover the background image through inpainting tools [47].

Additionally, after obtaining the recovered background image, we remove the bad cases with low image quality, such as cases with new generated object within the hole region rather than filled by background scene. Finally, we construct an occluded multi-control dataset with 1480 samples.

Table 1: Comparisons of multi-control image synthesis on COCO-UM with existing multi-control methods. Results with the best and the second best performance are highlighted in red and blue, respectively.

Methods	FID$\downarrow$	CLIP$\uparrow$	Depth (RMSE$\downarrow$)	Seg. (mPA$\uparrow$)	Edge (RMSE$\downarrow$)	Pose (mAP$\uparrow$)
Multi-ControlNet [44]	55.95	24.80	17.81	42.78	47.35	15.69
Multi-Adapter [25]	51.67	25.73	24.08	32.62	44.86	6.28
Uni-ControlNet [45]	55.28	24.48	20.57	41.10	43.28	18.40
UniConrol [28]	57.46	24.72	23.69	34.58	45.48	4.06
Cocktail [15]	47.39	25.33	-	31.74	-	12.16
AnyControl (Ours)	44.28	26.41	18.00	43.34	45.25	18.81

Table 2: Comparisons of single control with existing single- and multi-control methods on COCO-5K. Results with the best and the second best performance are highlighted in red and blue, respectively.

Methods	Depth			Segmentation			Edge		Pose
	FID $\downarrow$	CLIP $\uparrow$	RMSE $\downarrow$	FID $\downarrow$	CLIP $\uparrow$	mPA $\uparrow$	FID $\downarrow$	CLIP $\uparrow$	FID $\downarrow$	CLIP $\uparrow$
Single-Control Methods
ControlNet [44]	19.80	25.30	13.86	20.39	25.46	45.68	16.16	25.34	26.15	25.39
T2I-Adapter [25]	20.08	25.67	15.62	20.95	24.91	33.67	18.76	25.25	24.94	25.76
Multi-Control Methods
Uni-ControlNet [45]	20.09	25.25	15.93	22.96	25.11	35.56	17.51	25.18	26.61	24.86
UniControl [28]	20.67	25.51	14.07	19.73	26.11	45.77	16.69	25.15	27.90	25.27
Cocktail [15]	-	-	-	26.02	25.28	33.10	15.70	24.91	28.36	25.27
DiffBlender [18]	21.32	25.59	18.07	-	-	-	21.48	25.57	30.33	25.30
CnC [20]	20.42	26.95	17.57	-	-	-	-	-	-	-
AnyControl (Ours)	18.04	25.98	15.48	18.89	26.02	48.73	18.89	25.88	24.12	25.99

4.3 Quantitative Evaluation

For thorough quantitative evaluation, we employ various evaluation metrics, including FID [12] for the restructured image quality, CLIP-Score [11] for the alignment with textual prompt. For the condition fidelity, we adopt RMSE for depth map and edge map, mPA and mAP for segmentation map and pose map, respectively.

Multi-Control Synthesis Evaluation. We evaluate multi-control synthesis on COCO-UM. As depicted in Table 1, our method outperforms other multi-control methods on FID and CLIP-Score by a large margin. This remarkable achievement signifies that AnyControl is capable of processing complex combinations of multiple spatial conditions, and generates high-quality harmonious results well aligned with the textual prompts and spatial conditions.

Single-Control Synthesis Evaluation. For comprehensive evaluation, we also conduct the comparisons on each single condition as tabulated in Table 2. To be specific, we evaluate the single control synthesis on the full validation set of MSCOCO, that is, COCO-5K since there is no occlusion in single control scenario. Overall, AnyControl outperforms existing single- and multi-control methods on most metrics, illustrating the superiority of our method to existing methods.

Ablation Study on Unaligned Data. Unaligned data is provided to solve the gap between the alignment of input conditions during training and inference. That is, in training, conditions are totally aligned; while, in testing, the control

Table 3: Ablation study of training with and without unaligned data on COCO-UM.

Metric	FID$\downarrow$	CLIP$\uparrow$
w/o unaligned data	52.10	25.62
w unaligned data	44.28	26.40

signals from users contributing to a whole image are almost not aligned at all. In Table 3, we provide comparisons on FID and CLIP-Score of AnyControl trained with and without unaligned data. As illustrated, a large improvement on FID and CLIP-Score is observed through the data expansion on occluded cases. The introduction of unaligned data during training strengthens AnyControl in modeling complex multi-control synthesis, especially the occluded cases.

4.4 Discussion

Although the input number of spatial conditions is not limited in AnyControl, we

Figure 9: Miss-blending issue under too many spatial conditions.

observe the miss-blending issue as shown in Figure 9, when the number of spatial conditions is overlarge, such as 8 in this case. The possible reasons are as follows: 1) the limited ability of CLIP text encoder in understanding complex textual prompts with numerous concepts; 2) Too many visual tokens in cross-attention transformer block results in a decrease in the accuracy of softmax, and thus weaken AnyControl in precise multi-control understanding. We leave this issue as future work.

5 Conclusion

In conclusion, we propose a multi-control image synthesis framework based on the public T2I model to address the limitations of existing methods in accommodating diverse inputs, handling relationships among spatial conditions, and maintaining the compatibility with textual prompts. AnyControl supports free combination of versatile control signals, which develops a Multi-Control Encoder that enables holistic understanding of multi-modal user inputs. We achieve this through employing alternating multi-control fusion and alignment blocks united by a set of query tokens. This approach enables AnyControl model complex relationships among diverse control signals and extract a multi-control embedding with compatible information. Our method produces high-quality natural outcomes, positioning it as a state-of-the-art solution for multi-condition image generation. The advancements introduced by AnyControl contribute to the broader goal of enhancing controllable image synthesis and pushing the boundaries of T2I generation.

Appendix

Figure 10: Details of AnyControl and Multi-Control Encoder.

Appendix 0.A Implementation Details

Figure 11: More visualizations of unaligned data.

Network. The detailed structure of our AnyControl is depicted in Figure 10. We base Stable Diffusion of version 1.5 to build our AnyControl. Similar to ControlNet [44], we make a trainable copy of the UNet encoding blocks for adapting to controlling information while freezing the pre-trained weights of Stable Diffusion model totally. In our Multi-Control Encoder, the number of query tokens is set to 256 enabling detailed controllable information extraction. The additional position embedding, with the same length as the query tokens, are shared by all input spatial conditions. We take the pre-trained weights of Q-Former [21] as the initialization for Multi-Control Encoder except for the query tokens and the additional position embedding, which are randomly initialized.

Hyper Parameters. We train AnyControl on 8 A100 GPU cards with a batch size of 8 on each GPU. We train the model for totally 90K iterations with a initial learning rate of 1e-5. During inference, we set the classifier-free guidance scale to 7.5. In all the experiments, we adopt DDIM [38] sampler with 50 timesteps for all the compared methods.

Appendix 0.B Unaligned Data

During producing the synthetic unaligned dataset, we utilize the groudtruth object masks with the area ratio in $[0.1,0.4]$ to outline the foreground object, while oversmall or overlarge objects will lead to undesired recovered background image. PowerPaint [47] is a multi-task inpainting model supporting text-guided object inpainting, context-aware image inpainting as well as object removal. Here, we adopt the “object removal” mode for the unaligned data construction. More visualizations for synthetic unaligned data are in Figure 11.

Appendix 0.C Hand-crafted Weight Adjustment

As shown in Figure 12, multi-control methods with hand-crafted weights, i.e., Multi-ControlNet [44], usually require a series of laborious weight adjustments according to the synthesized results while ours can automatically infer the combination weights and extracts unified multi-control embedding, thus producing harmonious results.

Figure 12: Given prompt “cartoon style, a car parking in the canyon, a lion walking pass the car” and the edge conditions for a car as well as a lion, left shows hand-crafted weights adjustment for Multi-ControlNet [44]. X-axis and Y-axis represents the weight for lion and car conditions respectively. Right shows the results of AnyControl from three random seeds.

Appendix 0.D Multi-level Visual Tokens

Although the visual tokens from the last transformer block of the pre-trained visual encoder have already aggregated rich information, they are not sufficient

Table 4: Multi-level visual tokens. We gradually enable the visual tokens from the deepest level to the shallowest level.

Levels	1	2	3	4	5	6
FID $\downarrow$	45.64	43.73	43.69	43.67	43.74	44.28
CLIP $\uparrow$	26.35	26.40	26.39	26.39	26.38	26.40

to convey fine-grained controllable information. We conduct ablation experiments on the levels of used visual tokens from the visual encoder to the multi-control encoder. Table 4 demonstrate that integrating more visual tokens from middle layers increase FID and encounter performance saturation at 4-th level.

Appendix 0.E More Qualitative Results

More qualitative results on multi-control synthesis are shown in Figure 13. Results of single-control synthesis including depth map, edge map, segmentation map and human pose are shown in Figure 14 to Figure 17 respectively.

Figure 13: More visual results from AnyControl on multi-control image synthesis.

Figure 14: Visual results on depth controlled image synthesis.

Figure 15: Visual results on edge controlled image synthesis.

Figure 16: Visual results on segmentation controlled image synthesis.

Figure 17: Visual results on human pose controlled image synthesis.