HIVE: Harnessing Human Feedback for Instructional Visual Editing

Abstract

Incorporating human feedback has been shown to be crucial to align text generated by large language models to human preferences. We hypothesize that state-of-the-art instructional image editing models, where outputs are generated based on an input image and an editing instruction, could similarly benefit from human feedback, as their outputs may not adhere to the correct instructions and preferences of users. In this paper, we present a novel framework to harness human feedback for instructional visual editing (HIVE). Specifically, we collect human feedback on the edited images and learn a reward function to capture the underlying user preferences. We then introduce scalable diffusion model fine-tuning methods that can incorporate human preferences based on the estimated reward.

Besides, to mitigate the bias brought by the limitation of data, we contribute a new 1.1M training dataset, a 3.6K reward dataset for rewards learning, and a 1K evaluation dataset to boost the performance of instructional image editing. We conduct extensive empirical experiments quantitatively and qualitatively, showing that HIVE is favored over previous state-of-the-art instructional image editing approaches by a large margin.

1 Introduction

Figure 2: Overall architecture of HIVE. The first step is to train a baseline HIVE without human feedback. In the second step, we collect human feedback to rank variant outputs for each image-instruction pair, and train a reward model to learn the rewards. In the third step, we fine-tune diffusion models by integrating the estimated rewards.

State-of-the-art (SOTA) text-to-image generative models have shown impressive performance in terms of both image quality and alignment between output images and captions [1, 46, 44]. Thanks to the impressive generation abilities of these models, instructional image editing has emerged as one of the most promising application scenarios for content generation [7]. Different from traditional image editing [3, 16, 55, 30, 16, 55], where both the input and the edited caption are needed, instructional image editing only requires human-readable instructions. For instance, classic image editing approaches require an input caption “a dog is playing a ball”, and an edited caption “a cat is playing a ball”. In contrast, instructional image editing only needs editing instruction such as “change the dog to a cat”. This experience mimics how humans naturally perform image editing.

Instructional image editing was first proposed in InstructPix2Pix [7], which fine-tunes a pre-trained stable diffusion [46] by curating a triplet of the original image, instruction, and edited image, with the help of GPT-3 [8] and Prompt-to-Prompt image editing [16]. Though achieving promising results, the training data generation process of InstructPix2Pix lacks explicit alignment between editing instructions and edited images.

Consequently, the modified images may only align to a certain extent with the editing instructions, as shown in the second column of Fig. 4. Furthermore, since these editing instructions are provided by human users, it’s crucial that the final edited images accurately reflect the users’ true intentions and preferences. Typically, humans prefer to make selective changes to the original images, which are usually not factored into the training data or objectives of InstructPix2Pix [7]. Considering this observation and the recent successes of ChatGPT [35], we propose to refine the stable diffusion process with human feedback. This adjustment aims to ensure that the edited images more closely correspond to editing instructions provided by humans.

For large language models (LLMs) such as InstructGPT [35, 37], we often first learn a reward function to reflect what humans care about or prefer on the generated text output, and then leverage reinforcement learning (RL) algorithms such as proximal policy optimization (PPO) [50] to fine-tune the models. This process is often referred to as reinforcement learning with human feedback (RLHF). Leveraging RLHF to fine-tune diffusion-based generative models, however, remains challenging. Applying on-policy algorithms (e.g.,PPO) to maximize rewards during the fine-tuning process can be prohibitively expensive due to the hundreds or thousands of denoising steps required for each sampled image. Moreover, even with fast sampling methods [52, 57, 21, 31], it is still challenging to back-propagate the gradient signal to the parameters of the U-Net.

To address the technical issues described above, we propose Harnessing Human Feedback for Instructional Visual Editing (HIVE), which allows us to fine-tune diffusion-based generative models with human feedback. As shown in Fig. 2, HIVE consists of three steps:

1)  We perform instructional supervised fine-tuning on the dataset that combines our newly collected 1.1M training data and the data from InstructPix2Pix. Since observing failure cases and suspecting the grounding visual components from image to instruction is still a challenging problem, we collect 1.1M training data.

2)  For each input image and editing instruction pair, we ask human annotators to rank variant outputs of the fine-tuned model from step 1, which gives us a reward learning dataset. Using the collected dataset, we then train a reward model (RM) that reflects human preferences.

3)  We estimate the reward for each training data used in step 1, and integrate the reward to perform human feedback diffusion model finetuning using our proposed objectives presented in Sec. 3.4.

Our main contributions are summarized as follows:

$\bullet$  To tackle the technical challenge of fine-tuning diffusion models using human feedback, we introduce two scalable fine-tuning approaches in Sec. 3.4, which are computationally efficient and offer similar costs compared with supervised fine-tuning. Moreover, we empirically show that human feedback is an essential component to boost the performance of instructional image editing models.

$\bullet$  To explore the fundamental ability of instructional editing, we create a new dataset for HIVE including three sub-datasets: a new 1.1M training dataset, a 3.6K reward dataset for rewards learning, and a 1K evaluation dataset.

$\bullet$  To increase the diversity of the data for training, we introduce cycle consistency augmentation based on the inversion of editing instruction. Our dataset has been enriched with one pair of data for bi-directional editing.

2 Related Work

Text-To-Image Generation.   Text-to-image generative models have achieved tremendous success in the past decade. Generative adversarial nets (GANs) [15] is one of the fundamental methods that dominated the early-stage works [45, 61, 59]. Recently, diffusion models [51, 17, 53, 52] have achieved state-of-the-art text-to-image generation performance. [12, 34, 43, 44, 47, 60, 46, 29]. As a result, instead of training a text-to-image model from scratch, our work focuses on fine-tuning existing stable diffusion model [46], by leveraging additional human feedback.

Image Editing.   Similarly, diffusion models based image editing methods, e.g. SDEdit [32], BlendedDiffusion [3], BlendedLatentDiffusion [2], DiffusionClip [22], EDICT [55] or MagicMix [30], have garnered significant attention in recent years. To leverage a pre-trained image-text representation (e.g., CLIP [42], BLIP [28]) and text-to-image diffusion based pre-trained models [44, 47, 46], most existing works focus on text-based localized editing [5, 33, 16]. Prompt-to-Prompt [16] edits the cross-attention layer in Imagen and stable diffusion to control the similarity of image and text prompt. ControlNet [62] and UniControl [41] adopt controllable conditions to control image editings. Recently, InstructPix2Pix [7] tackle the problem via a different approach, requiring only human-readable editing instruction to perform image editing. Our work follows the same direction as InstructPix2Pix[7] and leverages human feedback to address the misalignment between editing instructions and resulting edited images.

Learning with Human Feedback.   Incorporating human feedback into the learning process can be a highly effective way to enhance performance across various tasks such as fine-tuning LLMs [35, 4, 48, 37, 54], robotic simulation [10, 18], computer vision [40], and to name a few. Many existing works leverage PPO [50] to align to human feedback, however on-policy RL algorithms are not suitable for diffusion-based model fine-tuning (See more discussion in Appendix C.1).

Simultaneously, several concurrent works [56, 58, 25] study the text-to-image generation problem using human feedback. ReFL [58] investigates how to back-propagate the reward signal to random latter denoising step in the diffusion process, while [56] explores how to design fine-grained human preference score to improve the generation quality. [25] leverages human feedback to align text-to-image generation, where they naively view reward as weights to perform maximum likelihood training. Different from the above works, our work tackles the problem of instructional image editing, where there are little or even no ground truth data for the alignment between human-readable editing instructions and edited images. In addition, the conditions on both image input and instructions make the human feedback more valuable than standard text-to-image tasks, since the conditions make the training harder than standard text-to-image tasks.

3 Methodology

In this section, we introduce the new datasets we collected in Sec. 3.1, and explain the three major steps of HIVE in the rest of the section. Concretely, we introduce the instructional supervised training in Sec. 3.2, and describe how to train a reward model to score edited images in Sec. 3.3, then present two scalable fine-tuning methods to align diffusion models with human feedback in Sec. 3.4.

3.1 Dataset

Instructional Edit Training Dataset.   We follow the same method of [7] to generate the training dataset. We collect 1K images and their corresponding captions. We ask three annotators to write three instructions and corresponding edited captions based on the collected input captions. Therefore, we obtain 9K prompt triplets: input caption, instruction, and edited caption. We fine-tune GPT-3 [8] with OpenAI API v0.25.0 [36] with them. We use the fine-tuned GPT-3 to generate five instructions and edited captions per input image-caption pair in Laion-Aesthetics V2 [49]. We observe that the captions from Laion are not always visually descriptive, so we use BLIP [28] to generate more diverse types of image captions. Later stable diffusion based Prompt-to-Prompt [16] is adopted to generate paired images. In addition, we design a cycle-consistent augmentation method (Sec. 3.2.1) to generate additional training data. We generate 1.17M training triplets in total. Combining the 281K training data from [7], we obtain 1.45M training image pairs along with instructions.

Reward Fine-tuning Dataset.   We collect 3.6K image-instruction pairs for the task of reward fine-tuning. Among them, 1.6K image-instruction pairs are manually collected, and the rest are from Laion-Aesthetics V2 with GPT-3 generated instructions. We use this dataset to ask annotators to rank various model outputs.

Evaluation Dataset.   We use two evaluation datasets: the test dataset in [7] for quantitative evaluation and a new 1K dataset collected for the user study. The quantitative evaluation dataset is generated following the same method as the training dataset, which means that the dataset does not contain real images. Our collected 1K dataset contains 200 real images, and each image is annotated with five human-written instructions. More details of annotation tooling, guidelines, and analysis are in Appendix A.

3.2 Instructional Supervised Training

We follow the instructional fine-tuning method in [7] with two major upgrades on dataset curation (Sec. 3.1) and cycle consistency augmentation (Sec. 3.2.1). A pre-trained stable diffusion model [46] is adopted as the backbone architecture. In instructional supervised training, the stable diffusion model has two conditions $c=\left[c_{I},c_{E}\right]$, where $c_{E}$ is the editing instruction, and $c_{I}$ is the latent space of the original input image. In the training process, a pre-trained auto-encoder [23] with encoder $\mathcal{E}$ and decoder $\mathcal{D}$ is used to convert between edited image $\tilde{\boldsymbol{x}}$ and its latent representation $z=\mathcal{E}(\tilde{\boldsymbol{x}})$. The diffusion process is composed of an equally weighted sequence of denoising autoencoders $\epsilon_{\theta}(z_{t},t,c)$, $t=1,\cdots,T$, which are trained to predict a denoised variant of their input $z_{t}$, a noisy version of $z$. The objective of instructional supervised training is:

$$L=\mathbb{E}_{\mathcal{E}(\tilde{\boldsymbol{x}}),c,\epsilon\sim\mathcal{N}(0,1),t}\Big{[}\|\epsilon-\epsilon_{\theta}(z_{t},t,c))\|_{2}^{2}\Big{]}\,.\vspace{-.5em}$$

3.2.1 Cycle Consistency Augmentation

Cycle consistency is a powerful technique that has been widely applied in image-to-image generation [63, 19]. It involves coupling and inverting bi-directional map**s of two variables $X$ and $Y$, $G:X\rightarrow Y$ and $F:Y\rightarrow X$, such that $F(G(X))\approx X$ and vice versa. This approach has been shown to enhance generative map** in both directions.

While Instructpix2pix [7] considers instructional image editing as a single-direction map**, we propose adding cycle consistency. Our approach involves a forward-pass editing step, $F:x\stackrel{{\scriptstyle inst}}{{\longrightarrow}}\tilde{\boldsymbol{x}}$. We then introduce instruction reversion to enable a reverse-pass map**, $R:\tilde{\boldsymbol{x}}\stackrel{{\scriptstyle\sim inst}}{{\longrightarrow}}x$. In this way, we could close the loop of image editing as: $x\stackrel{{\scriptstyle inst}}{{\longrightarrow}}\tilde{\boldsymbol{x}}\stackrel{{\scriptstyle\sim inst}}{{\longrightarrow}}x$, e.g. “add a dog” to “remove the dog”.

To ensure the effectiveness of this technique, we need to separate invertible and non-invertible instructions from the dataset. We devised a rule-based method that combines speech tagging and template matching. We found that most instructions adhere to a particular structure, with the verb appearing at the start, followed by objects and prepositions. Thus, we grammatically tagged all instructions using the Natural Language Toolkit (NLTK) . We identified all invertible verbs and pairing verbs, and also analyzed the semantics of the objects and the prepositions used. By summarizing invertible instructions in predefined templates, we matched desired instructions. Our analysis revealed that 29.1% of the instructions in the dataset were invertible. We augmented this data to create more comprehensive training data, which facilitated cycle consistency. For more information, see Appendix B.1.

3.3 Human Feedback Reward Learning

The second step of HIVE is to learn a reward function $\mathcal{R}_{\phi}(\tilde{\boldsymbol{x}},c)$, which takes the original input image, the text instruction condition $c=\left[c_{I},c_{E}\right]$, and the edited image $\tilde{\boldsymbol{x}}$ that is generated by the fine-tuned stable diffusion as input, and outputs a scalar that reflects human preference.

Unlike InstructGPT which only takes text as input, our reward model $\mathcal{R}_{\phi}(\tilde{\boldsymbol{x}},c)$ needs to measure the alignment between instructions and the edited images. To address the challenge, we present a reward model architecture in Fig. 3, which leverages pre-trained vision-language models such as BLIP [28]. More specifically, the reward model employs an image-grounded text encoder as the multi-modal encoder to take the joint image embedding and the text instruction as input and produce a multi-modal embedding. A linear layer is then applied to the multi-modal embedding to map it to a scalar value. More details are in Appendix B.2.

With the specifically designed network architecture, we train the reward function $\mathcal{R}_{\phi}(\tilde{\boldsymbol{x}},c)$ with our collected reward fine-tuning dataset $\mathcal{D}_{\mathrm{human}}$ induced in Sec. 3.1. For each input image $c_{I}$ and instruction $c_{E}$ pair, we have $K$ edited images $\{\tilde{\boldsymbol{x}}\}_{k=1}^{K}$ ranked by human annotators, and denote the human preference of edited image $\tilde{\boldsymbol{x}}_{i}$ over $\tilde{\boldsymbol{x}}_{j}$ by $\tilde{\boldsymbol{x}}_{i}\succ\tilde{\boldsymbol{x}}_{j}$. Then we can follow the Bradley-Terry model of preferences [6, 37] to define the pairwise loss function:

where $(i,j)\in[1\ldots K]$ and we can get $K\choose 2$ pairs of comparison for each condition $c$. Similar to [37], we put all the $K\choose 2$ pairs for each condition $c$ in a single batch to learn the reward functions. We provide a detailed reward model training discussion in Appendix B.2.

Figure 3: Model architecture for reward $R(\tilde{\boldsymbol{x}},c)$. Here the reward model evaluates human preference for an edited image of a hand selecting an orange compared to the original input image of the hand selecting an apple. The input to the reward model includes both images and a text instruction. The output is a score indicating the degree of preference for the edited image based on the input image and instruction.

3.4 Human Feedback based Model Fine-tuning

With the learned reward function $\mathcal{R}_{\phi}(c,\tilde{\boldsymbol{x}})$, the next step is to improve the instructional supervised training model by reward maximization. As a result, we can obtain an instructional diffusion model that aligns with human preferences.

The RL fine-tuning techniques we present are built upon recent offline RL techniques [27, 38, 9, 20] With an input image and editing instruction condition $c=[c_{I},c_{E}]$, we define the edited image data distribution generated by the instructional supervised diffusion model as $p(\tilde{\boldsymbol{x}}|~{}c)$, and the edited image data distribution generated by the current diffusion model we want to optimize as $\rho(\tilde{\boldsymbol{x}}|~{}c)$ , then under the pessimistic principle of offline RL, we can optimize $\rho$ by the following objectives:

$$\textstyle J(\rho):=\max_{\rho}\mathbb{E}_{c}\big{[} \mathbb{E}_{\tilde{\boldsymbol{x}}\sim\rho(\cdot|c)}[\mathcal{R}_{\phi}(\tilde{\boldsymbol{x}},c)]-$$

$$~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}\eta\mathrm{KL}(\rho(\tilde{\boldsymbol{x}}|c)||p(\tilde{\boldsymbol{x}}|c))\big{]}\,,$$ (1)

where $\eta$ is a hyper-parameter. The first term in Eq. (1) is the standard reward maximization in RL. The second term is a regularization to stabilize learning, which is a widely used technique in offline RL [24], and is also adopted for PPO fine-tuning of InstructGPT (a.k.a “PPO-ptx”) [37].

To avoid using sampling-based methods to optimize $\rho$, we can differentiate $J(\rho)$ w.r.t $\rho(\tilde{\boldsymbol{x}}|c)$ and solve for the optimal $\rho^{*}(\tilde{\boldsymbol{x}}|c)$, resulting the following expression for the optimal solution of Eq. (1):

$$\rho^{*}(\tilde{\boldsymbol{x}}|c)\propto p(\tilde{\boldsymbol{x}}|c)\exp\left(\mathcal{R}_{\phi}(\tilde{\boldsymbol{x}},c)/\eta\right)\,,$$ (2)

or $\rho^{*}(\tilde{\boldsymbol{x}}|c)=\frac{1}{Z(c)}p(\tilde{\boldsymbol{x}}|c)\exp\left(\mathcal{R}_{\phi}(\tilde{\boldsymbol{x}},c)/\eta\right)$, with $Z(c)=\int p(\tilde{\boldsymbol{x}}|c)\exp\left(\mathcal{R}_{\phi}(\tilde{\boldsymbol{x}},c)/\eta\right)d\tilde{\boldsymbol{x}}$ being the partition function. A detailed derivation is in Appendix C.2.

Weighted Reward Loss.   The optimal target distribution $\rho^{*}(\tilde{\boldsymbol{x}}|c)$ in Eq. (2) can be viewed as an exponential reward-weighted distribution for $p(\tilde{\boldsymbol{x}}|c)$. Moreover, we have already obtained the empirical edited image data drawn from $p(\tilde{\boldsymbol{x}}|c)$ when constructing the instructional editing dataset, and we can view the exponential reward weighted edited image $\tilde{\boldsymbol{x}}$ from the instructional editing dataset as an empirical approximation of samples drawn from $\rho^{*}(\tilde{\boldsymbol{x}}|c)$. Formally, we can fine-tune a diffusion model thus it generates data from $\rho^{*}(\tilde{\boldsymbol{x}}|c)$, resulting in the weighted reward loss:

with $\omega(\tilde{\boldsymbol{x}},c)=\exp\left(\mathcal{R}_{\phi}(\tilde{\boldsymbol{x}},c)/\eta\right)$ being the exponential reward weight for edited image $\tilde{\boldsymbol{x}}$ and condition $c$. Different from RL literature [39, 38] using exponential reward or advantage weights to learn a policy function, our weighted reward loss is derived for fine-tuning stable diffusion.

Condition Reward Loss.   We can also leverage the control-as-inference perspective of RL [26] to transform Eq. (2) to a conditional reward expression, thus we can directly view the reward as a conditional label to fine-tune diffusion models. Similar to [26], we introduce a new binary variable $R^{*}$ indicating whether human prefers the edited image or not, where $R^{*}=1$ denotes that human prefers the edited image, and $R^{*}=0$ denotes that human does not prefer, thus we have $p(R^{*}=1~{}|~{}\tilde{\boldsymbol{x}},c)\propto\exp\left(\mathcal{R}_{\phi}(\tilde{\boldsymbol{x}},c)\right)$. Together with Eq. (2), and applying Bayes rules gives us the following derivation:

$$\textstyle p(\tilde{\boldsymbol{x}}|c) \exp\left(\mathcal{R}_{\phi}(\tilde{\boldsymbol{x}},c)/\eta\right):=q(\tilde{\boldsymbol{x}}|c)\left(p(R^{*}=1~{}|~{}\tilde{\boldsymbol{x}},c)\right)^{1/\eta}$$

$$=p(\tilde{\boldsymbol{x}}|c)\left(\frac{p(\tilde{\boldsymbol{x}}|~{}R^{*}=1,c)p(R^{*}=1|~{}c)}{p(\tilde{\boldsymbol{x}}|c)}\right)^{1/\eta}$$

$$\propto p(\tilde{\boldsymbol{x}}|c)^{1-1/\eta}p(\tilde{\boldsymbol{x}}|~{}R^{*}=1,c)^{1/\eta}\,,$$

where we drop $p(R^{*}=1|~{}c)$ since it is a constant w.r.t $\tilde{\boldsymbol{x}}$. We can now view the reward for each edited image as an additional condition. Define the new condition $\tilde{c}=[c_{I},c_{E},c_{R}]$, with $c_{R}$ as the reward label, we can fine-tune the diffusion model with the condition reward loss:

$$\textstyle\ell_{\mathrm{CR}}(\theta)=\mathbb{E}_{\mathcal{E}(x),\tilde{c},\epsilon\sim\mathcal{N}(0,1),t}\Big{[}\|\epsilon-\epsilon_{\theta}(z_{t},t,\tilde{c})\|_{2}^{2}\Big{]}\,.$$

Figure 4: Comparisons between IP2P-Official (InstructPix2Pix official model), IP2P-Ours (InstructPix2Pix using our data) and HIVE. HIVE can boost performance by understanding the instruction correctly.

We quantize the reward into five categories, based on the quantile of the empirical reward distribution of the training dataset, and convert the reward value into a text prompt. For instance, if the reward value of a training pair lies in the bottom 20% of the reward distribution of the dataset, then we convert the reward value as a text prompt condition $c_{R}:=$“The image quality is one out of five”. And during the inference time to generate edited images, we fix the text prompt as $c_{R}:=$“The image quality is five out of five”, indicating we want the generated edited images with the highest reward. We empirically find this technique improves the stability of fine-tuning.

4 Experiments

This section presents the experimental results and ablation studies of HIVE’s technical choices, demonstrating the effectiveness of our method. We adopt the default guidance scale parameters in InstrcutPix2Pix for a fair comparison. Through our experiments, we discovered that the conditional reward loss performs slightly better than the weighted reward loss, and therefore, we present our results based on the conditional reward loss. The detailed comparisons can be found in Sec. 4.2 and Appendix D.3.

Figure 5: Comparisons between IP2P-Official, IP2P-Ours, and HIVE. It plots tradeoffs between consistency with the input image and consistency with the edit. The higher the better. For all methods, we adopt the same parameters as that in [7].

We evaluate our method using two datasets: a synthetic evaluation dataset with 15,652 image pairs from [7] and a self-collected 1K evaluation dataset with real image-instruction pairs. For the synthetic dataset, we follow InstructPix2Pix’s quantitative evaluation metric and plot the trade-offs between CLIP image similarity and directional CLIP similarity [14]. For the 1K dataset, we conduct a user study where for each instruction, the images generated by competing methods are reviewed and voted by three human annotators, and the winner is determined by majority votes.

IP2P-Official vs IP2P-Ours	IP2P-Ours vs HIVE

Figure 6: User study of comparison between (a) IP2P-Official vs IP2P-Ours and (b) IP2P-Ours and HIVE. IP2P-Ours obtains 30% more votes than IP2P-Official. HIVE obtains 25% more votes than that IP2P-Ours.

4.1 Baseline Comparisons

We perform experiments with the same setup as InstructPix2Pix, where stable diffusion (SD) v1.5 is adopted. We compare three models: InstructPix2Pix official model (IP2P-Official), InstructPix2Pix using our data (IP2P-Ours) , and HIVE. We report the quantitative results on the synthetic evaluation dataset in Fig. 5. We observe that IP2P-Ours improves notably over IP2P-Official (blue curve vs. green curve). Moreover, human feedback further boosts the performance of HIVE (red curve vs blue curve) over IP2P-Ours by a large margin. In other words, with the same directional similarity value, HIVE obtains better image consistency than InstructPix2Pix.

To test the effectiveness of HIVE on real-world images, we report the user study results on the 1K evaluation dataset. We use “Tie” to represent that users think results are equally good or equally bad. As shown in Fig. 6(a), IP2P-Ours gets around 30% more votes than the IP2P-Official. The result is consistent with the user study on the synthetic dataset. We also demonstrate the user study outcome between HIVE and IP2P-Ours in Fig. 6(b). The user study indicates similar conclusions to the consistency plot, where HIVE gets around 25% more favorites than IP2P-Ours.

In Fig. 4, we present representative edits that demonstrate the effectiveness of HIVE. The results show that while using more data can partially improve editing instructions without human feedback, the reward model leads to better alignment between instruction and the edited image. For example, in the second row, IP2P-Ours generates a door-like object, but with the guidance of human feedback, the generated door matches human perception better. In the fourth row, the example of which is from the failure examples in [7], HIVE can locate the tie and change its color correctly.

Additionally, our visual analysis of the results (Fig. 7) indicates that the HIVE model tends to preserve the remaining part of the original image that is not instructed to be edited, while IP2P-Ours leads to excessive image editing more often. For instance, in the first example of Fig. 7, HIVE blends a pond naturally into the original image. The two InstructPix2Pix models fulfill the same instruction, however, at the same time, alter the uninstructed part of the original background.

Figure 7: Human feedback tends to help HIVE avoid unwanted excessive image modifications.

4.2 Ablation Study

Weighted Reward and Condition Reward Loss.   We perform user study on HIVE with these two losses individually. As shown in Fig. 8, these two losses obtain similar human preferences on the evaluation dataset. More comparisons are in Appendix D.

HIVE with weighted reward loss	HIVE with condition reward loss

Figure 8: User study of pairwise comparison between (a) HIVE with weighted reward loss and (b) HIVE with condition reward loss. The human preferences are close to each other.

Cycle Consistency   We analyze the impact of it which is introduced in Sec. 3.2.1. The top five augmentations in the cycle consistency are demonstrated in Fig. 9(a). We perform evaluation on both synthetic dataset and the 1K evaluation dataset. The user study in Fig. 9(b) shows that the cycle consistency augmentation improves the performance of HIVE by a notable margin.

Top five augmentations.	User study of cycle consistency.

Figure 9: Cycle consistency analysis.

Success Rate on Verbs   It is observed that five verbs take around 85% of all verbs, where details can be found in Sec. A. We compare HIVE with IP2P-Ours on these five verbs, and report the success rate of these two methods on these verbs. It is seen in Fig. 10 that HIVE improves the most on “add” from 23.5% to 28.7%.

IP2P-Ours	HIVE

Figure 10: Success rate of IP2P-Ours and HIVE on top five verbs.

Other Baselines.   To test the effectiveness of HIVE, we experiment two additional baselines. In Fig. 11(a), we upgrade the backbone of stable diffusion from v1.5 to v2.1. We observe that the upgraded backbone slightly improves the results. In Fig. 11(b), we directly use the reward scalar instead of the reward prompt as the condition for training, and the condition on the highest reward scalar for generating the image. We adopt the user study to compare it (named HIVE-reward) with HIVE. HIVE obtains 25.8 % more votes than the baseline model conditioned on the reward score. This is mainly because directly conditioning on the highest reward might cause overfiting.

HIVE SD v1.5 and v2.1	HIVE-Reward vs. HIVE

Figure 11: User study of pairwise comparison between (a) HIVE with SD v1.5 and v2.1 and (b) HIVE conditioning on reward score and HIVE. The human preferences are very close to each other.

Failure Cases and Limitations.   We summarize representative failure cases in Fig. 12. First, some instructions cannot be understood. In the upper left example in Fig. 12, the prompt “zoom in” or similar instructions can rarely be successful. We believe the root cause is current training data generation method fails to generate image pairs with this type of instruction. Second, counting and spatial reasoning are common failure cases (see the upper right example in Fig. 12). We find that the instruction “one”, “two”, or “on the right” can lead to many undesired results. Third, the object understanding sometimes is wrong. In the bottom left example, the red color is changed on the wrong object. This is a common error in HIVE, where instructed edited objects are wrongly recognized.

We find some other limitations as well. One limitation of HIVE is that it cannot bring benefits to the cases where all outputs by the model without human feedback obtain the same wrong results. In such cases, user preferences cannot always be beneficial to the results. We believe that improving the data as well as the base model is an important step in the future. Another limitation is that compared to Prompt-to-Prompt [16], which is used to generate our training data, HIVE sometimes leads to some unstructured change in the image. We think that it is because of the limitation of the current training data. Instructed editing can have more diverse and ambiguous scenarios than traditional image editing problems. Using GPT-3 to finetune prompts to generate the training data is limited by the model and the labeled data. More ablation studies are in Appendix D.

Figure 12: Failure examples.

5 Conclusion and Discussion

In our paper, we introduce a novel framework called HIVE that enables instructional image editing with human feedback. Our framework integrates human feedback, which is quantified as reward values, into the diffusion model fine-tuning process. We design two variants of the approach and both of them improve performance over previous state-of-the-art instructional image editing methods. Our work demonstrates instructional image editing with human feedback is a variable approach to align image generation with human preference, thus unlocking new opportunities and potential to scale up the model capabilities towards more powerful applications such as conversational image editing. While our method demonstrates impressive performance, we have also identified failure scenarios, as discussed in Sec. 4.2. In addition, it is possible that our trained model inherits bias and suffers from harmful content from pre-trained foundation models such as Stable Diffusion, GPT3 and BLIP. These limitations would be considered when interpreting our results, and we expect red teaming with human feedback to mitigate some of the risks in future work.

Appendix

Appendix A Data Collection and User Study

In the evaluation steps, we collect real-world images with instructions using Amazon Mechanical Turk (Mturk) . We randomly collect 200 real-world images. Then we ask Mturk annotators to write five instructions for each image, and encourage them to have wild imaginations and diversify the instruction types. We encourage annotators to not be limited to making the image realistic. For example, annotators can write “add a horse in the sky”. A screenshot of the interface is illustrated in Fig. 13. We analyze the top five verbs and nouns in the evaluation dataset. It is shown in Fig. 15(a) that the verbs “add”, “change”, “make”, “remove” and “put” make up around 85% of all verbs, which means that the editing instruction verbs have a long-tail distribution. In contrast, the distribution of nouns in Fig. 15(b) is close to uniform, where the top five nouns represent only around 20% of all nouns.

Figure 13: Mturk writing editing instructions interface: write five instructions per image.

In user studies, we use Mturk to ask annotators to evaluate edited images. A screenshot of the interface is shown in Fig. 14. The annotators are provided with the original image, two edited images, and the editing instruction. They are asked to select the better edited image. The third option indicates that the edited images are equally good or equally bad. We ask three annotators to label one data sample, and use the majority votes to determine the results. We shuffle the edited images to avoid choosing the left image over the right and vice versa.

Figure 14: Mturk labeling interface: select the better edited image.

Top five verbs	Top five nouns

Figure 15: Top five verbs and nouns in the evaluation dataset.

Appendix B Implementation Details

B.1 Instructional Supervised Training

We use pre-trained stable diffusion models as the initial checkpoint to start instructional supervised training. We train HIVE on 40GB NVIDIA A100 GPUs for 500 epochs. We use the learning rate of $10^{-4}$ and the image size of 256. In the inference, we use 512 as the default image resolution.

B.2 Human Feedback Rewards Learning

Figure 16: Overall architecture of HIVE. Different from Fig. 2, in the third step, we use weighted reward loss instead of condition reward loss to fine-tune the diffusion model.

As shown in Fig. 3, the reward model takes in an input image $c_{I}$, a text instruction $c_{E}$, and an edited image $\tilde{x}$ and outputs a scalar value. Inspired by the recent work on the vision-language model, especially BLIP [28], we employ a visual transformer [13] as our image encoder and an image-grounded text encoder as the multimodal encoder for images and text. Finally, we set a linear layer on top of the image-grounded text encoder to map the multimodal embedding to a scalar value.

(1) Visual transformer. We encode both the input image $c_{I}$ and edited image $\tilde{x}$ with the same visual transformer. Then we obtain the joint image embedding by concatenating the two image embeddings $vit(c_{I})$, $vit(\tilde{x})$.

(2) Image-grounded text encoder. The image-grounded text encoder is a multimodal encoder that inserts one additional cross-attention layer between the self-attention layer and the feed-forward network for each transformer block of BERT [11]. The additional cross-attention layer incorporates visual information into the text model. The output embedding of the image-grounded text encoder is used as the multimodal representation of the ($c_{I}$, $c_{E}$, $\tilde{x}$) triplet.

We gather a dataset comprising 3,634 images for the purpose of ranking. For each image, we generate five variant edited images, and ask an annotator to rank images from best to worst. Additionally, we ask annotators to indicate if any of the following scenarios apply: (1) all edited images are edited but none of them follow the instruction; (2) all edited images are visually the same as the original image; (3) all images are edited beyond the scope of instruction; (4) edited images have harmful content containing sex, violence, porn, etc; and (5) all edited images look similar to each other. We compare training reward models by filtering some/all of these options.

We note that a considerable portion of the collected data falls under at least one of the aforementioned categories, indicating that even for humans, ranking these images is challenging. As a result, we only use the data that did not include any non-rankable options in the reward model training. From a pool of 1,412 images, we select 1,285 for the training set, while the remaining images were used for the validation set. The reward model is trained on a dataset of comparisons between multiple model outputs on the same input. Each comparison sample contains an input image, an instruction, five edited versions of the image, and the corresponding rankings. We divide the dataset into training and validation sets based on the distribution of the corresponding instructions.

We apply the method in Sec. 3.3 on the reward data to develop a reward model. We initialize the reward model from the pre-trained BLIP, which was trained on paired images and captions using three objectives: image-text contrastive learning, image-text matching, and masked language modeling. Although there is a domain gap between BLIP’s pre-training data and our reward data, where the captions in BLIP’s data describe a single image, and the instructions in our data refer to the difference between image pairs. We hypothesized that leveraging the learned alignment between text and image in BLIP could enhance the reward model’s ability to comprehend the relationship between the instruction and the image pairs.

The reward model is trained using 4 A100 GPUs for 10 epochs, employing a learning rate of $10^{-4}$ and weight decay of 0.05. The image encoder’s and multimodal encoder’s last layer outputs are utilized as image and multimodal representations, respectively. The encoders’ final layer is the only fine-tuned component.

We use the trained reward model to generate a reward score on our training data. We perform two experiments. The first experiment takes the exponential rewards as weights and fine-tunes the diffusion model with weighted reward loss as described in Sec. 3.4. See Fig. 16 for the visualization of the method. The second experiment transforms the rewards to text prompts and fine-tunes the diffusion model with the condition reward loss as described in Sec. 3.4. The method is introduced in Fig. 2. We compare those two experiment settings, and results can be found in Sec. D.3.

Appendix C Reward Maximization for Diffusion-Based Generative Models

C.1 Discussion on On-Policy based Reward Maximization for Diffusion Models

Directly adapting on-policy RL methods to the current training pipeline might be computationally expensive, but we do not conclude that sampling-based approaches are not doable for diffusion models. We consider develo** more scalable sampling-based methods as future work.

We start the sampling methods derivation with the following objective:

$$J(\theta):=\max_{\pi_{\theta}}\mathbb{E}_{c\sim p_{c}}\Big{[}\mathbb{E}_{\tilde{\boldsymbol{x}}\sim\pi_{\theta}(\cdot|c)}\left[\mathcal{R}_{\phi}(\tilde{\boldsymbol{x}},c)\right]-\eta\mathrm{KL}[p_{\mathcal{D}}(\tilde{\boldsymbol{x}}|c)||\pi_{\theta}(\tilde{\boldsymbol{x}}|c)]\Big{]}\,,$$ (3)

where $p_{c}(c)p_{\mathcal{D}}(\tilde{\boldsymbol{x}}|c)$ is the joint distribution of the condition and edited images pair, and $\pi_{\theta}$ denotes the policy or the diffusion model we want to optimize. Note that $p_{\mathcal{D}}(\tilde{\boldsymbol{x}}|c)$ and $\pi(\tilde{\boldsymbol{x}}|c)$ are swaped compared with the objective in Eq. (1). The second term in Eq. (3), is the KL Minimization formula for maximum likelihood estimation, equivalent to the loss of diffusion models. We represent the policy $\pi_{\theta}$ via the reverse process of a conditional diffusion model:

$$\pi_{\theta}(\tilde{\boldsymbol{x}}|c):=p_{\theta}(\tilde{\boldsymbol{x}}^{0:T}|~{}c)=p_{0}(\tilde{\boldsymbol{x}}^{T})\prod_{t=1}^{T}p_{\theta}(\tilde{\boldsymbol{x}}^{t-1}|\tilde{\boldsymbol{x}}^{t};c)\,,$$

where $p_{0}(\tilde{\boldsymbol{x}}^{T}):=\mathcal{N}(\tilde{\boldsymbol{x}}^{T},\bf{0};\bf{{I}})$, and $p_{\theta}(\tilde{\boldsymbol{x}}^{t-1}|\tilde{\boldsymbol{x}}^{t};c):=\mathcal{N}(\tilde{\boldsymbol{x}}^{t}|\mu_{\theta}(\tilde{\boldsymbol{x}}_{t},t),\sigma_{t}^{2})\bf{\mathrm{I}}$ is a Gaussian distribution, whose parameters are defined by score function $\epsilon_{\theta}$ and stepsize of noise scalings. So we can get a edited image sample $\tilde{\boldsymbol{x}}^{0}$ by running a reverse diffusion chain:

$$\tilde{\boldsymbol{x}}^{t-1}|\tilde{\boldsymbol{x}}^{t}=\frac{1}{\sqrt{\alpha_{t}}}\left(\tilde{\boldsymbol{x}}^{t}-\frac{1-\alpha_{t}}{\sqrt{1-\bar{\alpha}_{t}}}\epsilon_{\theta}(\tilde{\boldsymbol{x}}^{t},c,t)\right)+\sigma_{t}\boldsymbol{z}_{t},~{}~{}\boldsymbol{z}\sim\mathcal{N}(\textbf{0},\textbf{I}),\text{for}~{}~{}t=T,\ldots,1\,,$$

and $\tilde{\boldsymbol{x}}^{T}\sim\mathcal{N}(\textbf{0},\textbf{I})$.

As a result, the reverse diffusion process can be viewed as a black box function defined by $\epsilon_{\theta}$ and noises $\boldsymbol{\epsilon}:=(\boldsymbol{z}_{T},\ldots,\boldsymbol{z}_{1},\tilde{\boldsymbol{x}}^{T})$, which we can view as a shared parameter network with noises. And for each layer, we can view the parameter is the score function $\epsilon_{\theta}$. Define the network as

$$\tilde{\boldsymbol{x}}^{0}:=f(c,\boldsymbol{\epsilon};\theta)\,,~{}~{}\boldsymbol{\epsilon}\sim p_{\mathrm{noise}}(\cdot),c\sim p_{c}(\cdot)\,,$$

where we can rewrite the first term as

$$\mathbb{E}_{c\sim\mathcal{D},\boldsymbol{\epsilon}\sim p_{\mathrm{noise}}(\cdot)}[\mathcal{R}_{\phi}(f(c,\boldsymbol{\epsilon};\theta),c)]\,,$$

and we can optimize the parameter $\theta$ with path gradient if $\mathcal{R}_{\cdot}$ is differentiable with path gradient. Similarly, suppose we want to optimize the first term via PPO. In that case, the main technical difficulty is to estimate $\nabla_{\theta}\log\pi_{\theta}(\tilde{\boldsymbol{x}}|c)$, which can be estimated with the following derivation:

$$\nabla_{\theta}\log\pi_{\theta}(\tilde{\boldsymbol{x}}|c)=\nabla_{\theta}\log p_{\theta}(\tilde{\boldsymbol{x}}^{0:T}|~{}c)=\sum_{t=1}^{T}\nabla_{\theta}\log p_{\theta}(\tilde{\boldsymbol{x}}^{t-1}|\tilde{\boldsymbol{x}}^{t};c)\,.$$

Note that for both the end-to-end path gradient method and PPO we require to sample the reverse chain from $\tilde{\boldsymbol{x}}^{T}$ to $\tilde{\boldsymbol{x}}^{0}$, thus we can estimate $\nabla_{\theta}\log\pi(\tilde{\boldsymbol{x}}|c)$ using the empirical samples $\tilde{\boldsymbol{x}}^{0:T}$.

For the above two methods, to perform one step policy gradient update, we need to run the whole reverse chain to get an edited image sample $\tilde{\boldsymbol{x}}^{0}$ to estimate the parameter gradient for the first term. As a result, the computational cost is the number of diffusion steps more extensive than the supervised fine-tuning cost. Now we need more than two days to fine-tune the stable diffusion model, so for standard LDM, where the number of steps is 1000, we can not finish the training within an acceptable training time. Even if we can use some fast sampling methods such as DDIM or variance preserve (VP) based noise scaling, the diffusion steps are still more than 5 or 10. Further, we haven’t seen any previous work using such noise scaling to fine-tune stable diffusion. As a result, we think naive sampling methods might have high risk to obtain similar performance, compared with our current offline RL based approaches.

C.2 Derivation for Eq. (2)

Take a functional view of Eq. (2), and differentiate $J(\rho)$ w.r.t $\rho$, we get

$$\frac{\partial J(\rho)}{\partial\rho}=\mathcal{R}_{\phi}(\tilde{\boldsymbol{x}}|c)-\eta\left(\log\rho(\tilde{\boldsymbol{x}}|c)+1-\log p(\tilde{\boldsymbol{x}}|c)\right)\,.$$

Setting $\frac{\partial J(\rho)}{\partial\rho}=0$ gives us

$$\log\rho(\tilde{\boldsymbol{x}}|c) =\frac{1}{\eta}\mathcal{R}_{\phi}(\tilde{\boldsymbol{x}}|c)+\log p(\tilde{\boldsymbol{x}}|c)-1\,,$$

$$\rho(\tilde{\boldsymbol{x}}|c) \propto p(\tilde{\boldsymbol{x}}|c)\exp\left(\mathcal{R}_{\phi}(\tilde{\boldsymbol{x}},c)/\eta\right)\,.$$

Thus we can get the optimal $\rho^{*}(\tilde{\boldsymbol{x}}|c)$.

Appendix D Additional Ablation Study

D.1 SD v1.5 and v2.1.

In Sec. 4.2, we upgrade the backbone of stable diffusion from v1.5 to v2.1, where OpenCLIP text encoder [49] replaces the CLIP text encoder [42]. In this section, we demonstrate the quantitative consistency plot in Fig. 17(a) on the synthetic evaluation dataset, which shows similar conclusions to the user study in Fig. 11(a). We compare IP2P-Ours v1.5 with v2.1 as well. An interesting observation is that we train IP2P-Ours with SD v2.1 and show in Fig. 17(b) that its improvement over SD v1.5 is larger than HIVE in Fig. 17(a).

IP2P-Ours with SD v1.5 and v2.1	InstructPix2Pix with SD v1.5 and v2.1

Figure 17: HIVE and IP2P-Ours with SD v1.5 and v2.1.

D.2 Model Adaptation

We demonstrate that HIVE is able to adapt the reward model that is trained on a different backbone from the backbone in Step. 3. We use the SD v1.5 generated data to train the reward model, and process the rest steps using SD v2.1. We report user study results in Fig. 18. It is observed that the users vote similarly between the reward models that are trained on two SD backbones. In other words, the reward model is able to adapt from one backbone to another.

Figure 18: SD v1.5 trained vs. SD v2.1 trained reward model

D.3 Weighted Reward and Conditional Reward Losses

We compare the weighted reward loss and conditional reward loss on the synthetic evaluation dataset. As shown in Fig. 19, the performances of these two losses are close to each other, while the conditional reward loss is slightly better. Therefore we adopt the conditional reward loss in all our experiments.

Figure 19: HIVE with weighted reward loss and conditional reward loss.

D.4 Training with Less Data

We analyze the effect of the training data size. We compare HIVE with SD v1.5 at four training dataset size ratios: 100%, 50%, 30% and 10%. As shown in Fig. 20, significantly decreasing the size of the dataset, e.g. 10% data, leads to worse ability to perform large image edits. On the other hand, reasonable decreasing dataset size can result in a similar yet slightly worse performance e.g. 50% data.

Figure 20: HIVE with different training data size.

D.5 Subcategory Analysis

We classify the editing into the following sub-categories: changing the global style, adjust attributes for the main object, add/remove objects, manipulate objects, and other challenging cases such as zooming and camera view changes. We use ChatGPT to determine which sub-category the instruction belongs to. Specifically, the numbers of instructions in each sub-category are as follows: changing global style (133), adjust attributes for the main object (134), add/remove objects (508), manipulate objects (219), and others (6). We analyze user study results for each sub-category. It is shown in Fig. 21 that the most improvement comes from the sub-categories ”Add/remove objects” and ”Manipulate objects”.

Change global style	Adjust attributes	Add/remove

Manipulate objects	Others

Figure 21: Subcategory analysis between IP2P and HIVE.

D.6 Additional Visualized Results

We illustrate additional visualized results in Fig. 22, 23, 24, 25, 26, where each row illustrates three instructional editing examples.

Figure 22: Additional editing results.

Figure 23: Additional editing results.

Figure 24: Additional editing results.

Figure 25: Additional editing results.

Figure 26: Additional editing results.