DiffuseHigh: Training-free Progressive High-Resolution Image Synthesis through Structure Guidance

Abstract

Recent surge in large-scale generative models has spurred the development of vast fields in computer vision. In particular, text-to-image diffusion models have garnered widespread adoption across diverse domain due to their potential for high-fidelity image generation. Nonetheless, existing large-scale diffusion models are confined to generate images of up to 1K resolution, which is far from meeting the demands of contemporary commercial applications. Directly sampling higher-resolution images often yields results marred by artifacts such as object repetition and distorted shapes. Addressing the aforementioned issues typically necessitates training or fine-tuning models on higher resolution datasets. However, this undertaking poses a formidable challenge due to the difficulty in collecting large-scale high-resolution contents and substantial computational resources. While several preceding works have proposed alternatives, they often fail to produce convincing results. In this work, we probe the generative ability of diffusion models at higher resolution beyond its original capability and propose a novel progressive approach that fully utilizes generated low-resolution image to guide the generation of higher resolution image. Our method obviates the need for additional training or fine-tuning which significantly lowers the burden of computational costs. Extensive experiments and results validate the efficiency and efficacy of our method.

1 Introduction

Following the establishment of diffusion models as a cornerstone in the realm of generative modeling, there has been a rapid advancements across various domains within machine learning modalities. These advancements span areas such as audio synthesis [32, 13, 33, 26, 37], image synthesis [23, 62, 15, 50, 54, 48, 16, 4, 45], video generation [20, 22, 8, 61, 71, 7, 12], and 3D generation [46, 72, 36, 14, 59, 68, 75]. Notably, text-to-image diffusion models [4, 45, 50, 54, 48] have attracted considerable attention due to their ability to generate visually captivating images using intuitive, human-friendly natural language descriptions. Stable Diffusion (SD), an open-source text-to-image diffusion model trained on a large-scale online dataset [57], has emerged as a prominent choice for a diverse range of generative tasks and inverse problems. These tasks include but are not limited to image editing [1, 2, 21, 69, 30], inpainting [50, 53, 40], super-resolution [50, 55, 17], and image-to-image translation [10, 42, 76, 77].

Figure 1: Various baselines. Each images has $2048^{2}$ size, generated from SDXL 1.0. We used ‘A group of playful monkeys swinging through the branches of a dense jungle.’ and ‘A line of taxis queued up outside a busy train station.’ as a textual prompt for each rows, respectively.

Despite the promising performance exhibited by SD, it encounters limitations when generating images at higher resolutions beyond its training resolution. The direct inference of unseen high-resolution samples often reveals repetitive patterns and irregular structures, particularly noticeable in object-centric samples, as discussed in prior works [19, 79] (see Fig. 1). While a straightforward approach might involve training or fine-tuning diffusion models on higher-resolution images, several challenges impede this approach. First, collecting text-image pairs of higher resolution is not readily feasible. Second, training on large-resolution images demands substantial computational resources due to the increased size of the intermediate features. Furthermore, capturing and learning the features from high-dimensional data often requires a greater model capacity (more model parameters), leading to further computational strain on the training process.

Several tuning-free [6, 34, 19] methods proposed various approaches to adapt pre-trained SD on higher-resolutions beyond its original settings. MultiDiffusion [6] and SyncDiffusion [34] employs multiple diffusion process with overlap** windows, each corresponding to different regions within the generating image. These joint diffusion models can produce images of arbitrary shape, but the resulting image involves object repetition problem since the same textual prompt is fed into each windows. Attn-SF [27] associates inference resolution with attention entropy and introduces scaling factor to alleviate entropy fluctuations during sampling of variable-sized images. However, their work does not consider adapting SD on much higher-resolutions, e.g., 2K and 4K. ScaleCrafter [19], on the other hand, extends the receptive field of the diffusion model by dilating the pre-trained convolution weights of the denoising UNet [51]. While it effectively addresses repetition issues in certain instances, its success heavily depends on the extensive search of the hyperparameters.

Figure 2: Toy experiment on 2K image. (a) Original 2K image. (b) We slightly add the noise that does not perturb the global sturcture severely. (c) We denoise the image with SD 2.1 to restore the original contents.

In this work, we investigate the SD’s capability of generating previously unseen high-resolution images and introduce a novel approach that does not involve any training (or fine-tuning) and additional modules. We posit that SD innately possesses the potential to generate images at resolutions higher than its training resolution thanks to its convolutional architecture [50] and broad data distribution coverage. To substantiate our claim, we generate 2K images using SD from noisy images at different intermediate diffusion timesteps. Note that Gaussian noise is added to the latent space. Fig. 2 demonstrates that from noisy images but whose global structures are preserved, SD seamlessly restores clean, highly detailed images.

Building upon this observation, we introduce a novel progressive high-resolution image generation pipeline, dubbed DiffuseHigh, where a relatively low-resolution image (sampled from SD) serves as a guide for generating higher-resolution images. Inspired by the recent literature [45, 41], we suggest the noising-denoising technique to synthesize higher-resolution images. First, we generate the low-resolution image using SD and upsample it by bilinear interpolation. Then, we add sufficient noise to obfuscate the fine details of the interpolated images. Finally, we perform the reverse diffusion process to denoise those images to infuse the high-frequency details to synthesize higher-resolution images, and we can repeat this process until we obtain the desired resolution images. This approach leverages the overall structure from the low-resolution image, effectively addressing repetition issues observed in the prior methods.

However, the ‘adding noise to damage the images’ approach poses several challenges. If we add too much noise, then we lose most of the structure in the low-resolution images, resulting in repetitive outcomes similar to those we generate from scratch. On the other hand, if we introduce a minimal amount of noise, the generated higher-resolution images do not show notable differences from the interpolated images, losing the opportunity to synthesize high-frequency details. In addition, finding adequate noise relies on both the content of the image and the pre-trained models, which makes it challenging to offer precise suggestions to users.

To resolve the issues above, we propose a principled way of preserving the overall structure from the low-resolution image for the suggested progressive pipeline. We employ a frequency-domain representation to extract the global structure as well as detailed contents from the low-resolution images. More specifically, we adopt the Discrete Wavelet Transform (DWT) to obtain essential contents, e.g., the $LL$ component, which we then incorporate into the denoising procedure to ensure that the resulting image remains consistent and does not deviate excessively.

Fig. 3 provides an overview of the overall pipeline of our method. We validate the proposed pipeline on the LAION-400M dataset [58] and demonstrate the superior performance of DiffuseHigh compared to other baseline methods. Additionally, we extend our method to diffusion-based video generation [71] to showcase the versatility of DiffuseHigh. The contributions of our work are summarized as follows:

• Our observation indicates that SD has the innate ability to synthesize images with higher resolution than those it was trained on.

• We suggest a novel training-free progressive high-resolution image synthesis pipeline called DiffuseHigh, in which a lower-resolution image acts as a guide for generating higher-resolution images.

• We further propose Discrete Wavelet Transform (DWT)-based structure guidance during the denoising process, which enhances the structural properties and fine details of the generated samples.

• We conduct comprehensive experiments both on image and video synthesis, demonstrating the superiority and versatility of our method.

2 Related Work

2.0.1 Diffusion Models

Diffusion models (DMs) [23, 63] represent a novel paradigm within the generative modeling framework, employing numerical methods [39, 78, 5] to solve reverse-time stochastic differential equations (SDEs) for simulating the generative trajectories [64]. Under this rigorous theoretical framework, DMs enable to achieve state-of-the-art (SoTA) [28, 44] image quality and comparable model likelihood [31, 38].

2.0.2 Text-to-Image Generation

Text-driven image generation can be traced back to the use of GANs [9, 29, 56], often combined with image-text representations such as CLIP [47], achieving significant performance. However, generating semantically consistent images with text guidance remains challenging for GANs [52]. Recently, DMs have gained popularity for their ability to produce high-quality images [44], showcasing great potential in text-to-image generation [15, 25]. Especially the pioneering work, Stable Diffusion [50], which introduces text representations in latent space iteratively, with further advancements occurring rapidly. Moreover, thanks to the large-scale training of Stable Diffusion, it is applied to various text-to-image tasks [35, 43, 11] by fine-tuning [52] or using training-free [49] method. While significant progress has been made in the field of text-to-image generation, one limitation of DMs is their capacity to generate images only at fixed resolutions [52, 35, 54], attributed to their training on specific image sizes. To remedy this, in this paper, we employ text prompts to generate much higher-resolution images than those present in their training datasets in a training-free manner.

2.0.3 Noising-Denoising

Based on the stochastic differential equation (SDE) reflecting the generative diffusion process, SDEdit [41] proposed a unified framework for image editing and image synthesis. Given images with low-level details, e.g., stroke painting, they add an adequate amount of noise to the image. Subsequently, they restore a clean, natural image from the noisy image through an iterative reverse SDE.

This ‘noising-denoising’ strategy, which performs a reverse diffusion process from the intermediate noised image, has been widely adopted in various domains. AnoDDPM [73] proposed reconstruction-based anomaly detection with partial Markov chain, where the data sample is slightly noised with small timesteps and reconstructed. SDXL [45] employs an optional refinement network, in which the network refines the low-quality part of the image through a noising-denoising process. Similarily,   adopts this algorithm in a post-processing stage in order to rectify the imperfect video frames.

2.0.4 High-resolution Image Synthesis

Despite the progress made by current diffusion model-based synthesis methods, achieving high-resolution image generation remains elusive. Previous studies have tackled these challenges through methods such as training from scratch and fine-tuning [74, 80]. However, training from scratch and fine-tuning often require significant computational resources and a substantial amount of high-resolution training data. Consequently, there has been a recent trend towards training-free methods [79, 27] for generating arbitrary-size or high-resolution images. ScaleCrafter [19] utilized dilated convolution to adjust the convolutional receptive field, enabling adaptation to high-resolution image generation without any training.

Recently, Make-a-Cheap-Scaling [18] has also adopted the noising-denoising technique to synthesize higher-resolution images. To further boost the image quality, they propose to tune a lightweight upsampler module, which can provide proper semantic guidance during the generation process. Different from theirs, we propose to obtain explicit structural guidance from the low resolution image, which can effectively address object repetition problem and irregular structure issues. Our proposed DiffuseHigh can be directly applied to any prevalent pretrained diffusion models in a completely training-free manner, providing both efficacy and efficiency.

3 Preliminary

In this section, we briefly present preliminaries relevant to our method, including Stable Diffusion (SD) [50] and Discrete Wavelet Transform (DWT).

3.1 Stable Diffusion

Stable Diffusion is a text-to-image latent diffusion model where the diffusion process is performed on a low-dimensional latent space. Given a data sample $x$ from the unknown data distribution $p_{\text{data}}(x)$, stable diffusion encodes $x$ into a latent representation $z_{0}=\mathcal{E}(x)$, where $\mathcal{E}(\cdot)$ is an autoencoder that compresses the high-dimensional data into a compact latent space. Then, the model gradually adds isotropic gaussian noise $\epsilon\sim\mathcal{N}(0,I)$ to a clean sample $z_{0}$ with pre-defined noise schedule $\alpha_{t}\in(0,1)$:

$$z_{t}=\sqrt{\bar{\alpha_{t}}}z_{0}+\sqrt{1-\bar{\alpha_{t}}}\epsilon,$$ (1)

where $t\in[1,...,T]$ denotes the timesteps of the diffusion process and $\bar{\alpha}=\Pi_{s=1}^{t}\alpha_{s}$. The denoising network $\epsilon_{\phi}(z_{t};t,y)$ parametrized by $\phi$ learns to predict the amount of noise added, given noisy latent $z_{t}$ and text prompt $y$, with the following denoising score matching objective:

$$\mathcal{L}:=\mathbb{E}_{t,\epsilon}\left[||w(t)(\epsilon_{\phi}(z_{t};t,y)-\epsilon)||_{2}^{2}\right].$$ (2)

$w(t)$ is a weighting function applied to each loss term at timestep $t$.

Initiating from $z_{T}\sim\mathcal{N}(0,I)$, the reverse procces is formulated as $q_{\phi}(z_{t-1}|z_{t},z_{0})$ with $q_{\phi}(\cdot|\cdot)$ parametrized as a Gaussian distribution. For efficiency, DDIM [62] sampling strategy is generally adopted, where unknown $z_{0}$ is replaced with predicted clean latent $\hat{z_{0}}$ at timestep ‘$t$’:

$$\hat{z}_{0,t}=\frac{z_{t}-\sqrt{1-\bar{\alpha}_{t}}\epsilon_{\phi}(z_{t};t,y)}{\sqrt{\bar{\alpha}_{t}}}$$ (3)

Finally the clean image $\hat{x}$ is reconstructed from a decoder $\mathcal{D}(\cdot)$ of the stable diffusion, i.e., $\hat{x}=\mathcal{D}(z_{0})$.

3.2 Discrete Wavelet Transform

Frequency-based methods, including the Discrete Fourier Transform (DFT), Discrete Cosine Transform (DCT), and Discrete Wavelet Transform (DWT) play a pivotal role in discrete signal processing. Such frequency-based approaches transform the given signal into the frequency domain, enabling the analysis and manipulation of the individual frequency bands.

Among them, utilizing wavelets, DWT decomposes images into different components that are localized both in time and frequency. Specifically, at each DWT level, the decomposed components consist of an approximation coefficient denoted as $LL_{l}$ and a detail coefficient denoted as $LH_{l},HL_{l},HH_{l}$, where $l$ represents the level of the DWT. Leveraging the low-pass filter and high-pass filter in both vertical and horizontal directions, $LL_{l}$ represents the low-frequency details of the image, encompassing global structures, uniformly-colored regions, and smooth textures. On the other hand, $LH_{l},HL_{l},HH_{l}$ encapsulates the high-frequency details, such as edges, boundaries, and rough textures.

We adopt DWT as a tool for the guidance of overall structures and contents of the low-resolution image for generating a higher-resolution image. The details of applying DWT-based guidance on our pipeline are described in Sec. 4.3.

4 Method

Figure 3: Progressive High-Resolution Diffusion Pipeline. For simplicity, we did not depict the transformation between latent space and pixel space. The text prompt: ‘A group of playful dolphins lea** gracefully out of the sparkling ocean waves.’.

4.1 Problem Formulation

Our work aims to generate higher-resolution images over training size given textual prompts with a text-to-image diffusion model (stable diffusion) in a training-free manner. More formally, given text description $y$ and stable diffusion $\epsilon_{\phi}(\cdot)$ pretrained on fixed-size images $(h,w,3)$, our objective is to generate higher resolution image $(H,W,3)$ without training $\phi$, where $h\ll H,w\ll W$.

4.2 Progressive High-Resolution Diffusion Pipeline

We present progressive approach for generating high-resolution images using a pretrained stable diffusion model. Initially, our method generates a clean sample based on a given text description through stable diffusion. Assuming alignment between the generated image and the provided text, we then employ bilinear interpolation to upscale the image, thereby guiding the high-resolution image generation. Our method incorporates a noising-denoising technique [41], which gradually projects the sample onto the manifold of natural, highly detailed images that the diffusion model has learned. This iterative procedure can also be interpreted as a refinement stage [45], wherein the denoising process restores the missing high details on the low-resolution sample.

Let $x_{0}\in\mathbb{R}^{h\times w\times 3}$ be the generated sample of stable diffusion aligned with user-provided textual prompt $y$ and $p_{\phi}(x|y)$ be the text conditional data distribution the model has learned. In other words, $x_{0}\sim p_{\phi}(x|y)$ and $p_{\phi}(x|y)\approx p_{data}(x|y)$. We serve relatively low-resolution generated image $x_{0}$ as a guide for the higher resolution image generation and apply bilinear interpolation on the image $x_{0}$ to the desired size image $\tilde{x}_{0}\in\mathbb{R}^{H\times W\times 3}$. Note that the details of the resulting image $\tilde{x}_{0}$ lack clarity due to the nature of the interpolation, which entails averaging neighboring pixel values to compose newly introduced pixels.

In order to infuse the appropriate details to the current high-resolution, we first add noise corresponding to the diffusion timestep $N<T$ to its latent code $\tilde{z}_{0}=\mathcal{E}(\tilde{x}_{0})$ according to Eq. 1:

$$\hat{z}_{N}=\sqrt{\bar{\alpha}_{N}}\tilde{z}_{0}+\sqrt{1-\bar{\alpha}_{N}}\epsilon,\quad\epsilon\sim\mathcal{N}(0,I).$$ (4)

Then the denoising network $\epsilon_{\phi}(\cdot)$ performs the reverse process on the noisy latent representation $\hat{z}_{N}$ to recover the clean latent $\hat{z}_{0}$. By employing the latent decoder $\mathcal{D}(\cdot)$, we finally obtain the desired high-resolution image.

This progressive pipeline bears a resemblance to cascade diffusion models [24, 54], albeit with inherent differences. Cascade diffusion models employ multiple networks for each participating low-resolution image generation and super-resolution. In contrast, the described method relies solely on a single pretrained diffusion model, obviating the need for training the separate models.

Figure 4: Experiment with varying noising step $\mathbf{N}$. The first row shows the noisy images where noise corresponding to different timesteps is added. The second row is the denoised clean images. Small $N$ fails to generate detailed features, resulting in a blurry image. On the other hand, large $N$ introduces the object repetition problem: repeated objects on the road for $N=25$, bikes for $N=45$. Textual prompt: ‘a bear riding a bike in New York City’ is used in this experiment.

Noising timestep $N$ is an important factor for the overall quality of the resulting image (see Fig. 4). Large $N$ significantly destroys the critical structural properties in the image, leading to object repetition problems and undesirable object shapes. However, small $N$ does not provide the diffusion model enough timesteps to perform the denoising process to restore the fine, high-frequency details of the image. Determining appropriate noise levels is contingent upon both the content of the image and the characteristics of pretrained diffusion models, thereby presenting a challenge in practical usage. Furthermore, we observed numerous instances where this approach degraded image quality across all noise levels. This leads us to develop a more principled way to uphold the overall structure and maintain the quality of the generated higher-resolution images.

4.3 Structure Guidance through Discrete Wavelet Transform

The progressive approach demonstrates proficiency in generating high-fidelity images; however, it frequently encounters challenges in effectively capturing certain structural properties and nuanced details from low-resolution inputs. Consequently, this can lead to discrepancies between the generated image and the actual data distribution. (See Fig. 5). This phenomenon is apparent, as structures and intricate details are susceptible to damage and distortion by a certain amount of noise.

Figure 5: Comparison between DiffuseHigh without DWT and DiffuseHigh. As shown in the figure, (a) DiffuseHigh without DWT fails to generate realistic image, and shows artifacts as observed in the orange box. (b) DiffuseHigh without DWT generates black dots on the body of a teddy, and also fails to generate mouth. In contrast, in case of DiffuseHigh, structural property of the low resolution image successfully guide the object to have clean shape textures.

We hereby introduce the method DiffuseHigh (Fig. 3), in which we incorporate a Discrete Wavelet Transform (DWT)-based structure guide into the proposed progressive pipeline. This method aims to enhance the fidelity of generated images by encouraging the preservation of crucial features from the low-resolution input. Given an interpolated image $\tilde{x}_{0}\in\mathbb{R}^{H\times W\times 3}$, we extract its low-frequency component utilizing the DWT, which encapsulates the overall structure and coarse details of the image. More formally, let us define $\texttt{DWT}(\cdot):\mathbb{R}^{H\times W\times 3}\rightarrow\mathbb{R}^{4\times\frac{H}{2}\times\frac{W}{2}\times 3}$ and $\texttt{DWT}(\tilde{x}_{0})_{LL},\texttt{DWT}(\tilde{x}_{0})_{LH},\texttt{DWT}(\tilde{x}_{0})_{HL},\texttt{DWT}(\tilde{x}_{0})_{HH}\in\mathbb{R}^{\frac{H}{2}\times\frac{W}{2}\times 3}$ are four decomposed components of the interpolated high-resolution image. Then, we define a DWT-guided denoising step at ‘$t$’ as follows.

$$\begin{split}\hat{z}_{t-1}=&\sqrt{\bar{\alpha}_{t-1}}\mathcal{E}(\mathtt{iDWT}(\mathtt{DWT}(\tilde{x}_{0})_{LL},\mathtt{DWT}(\hat{x}_{0,t})_{LH},\mathtt{DWT}(\hat{x}_{0,t})_{HL},\mathtt{DWT}(\hat{x}_{0,t})_{HH}))\\ &+\sqrt{1-\bar{\alpha}_{t-1}}\epsilon,\end{split}$$ (5)

where $\hat{x}_{0,t}=\mathcal{D}(\hat{z}_{0,t})$ is a predicted clean image at timestep ‘$t$’, using Eq. 3 and the decoder $\mathcal{D}(\cdot)$, $\texttt{iDWT}(\cdot)$ is the inverse DWT, and $\mathcal{E}(\cdot)$ is the encoder. Finally, we recover the denoised image $\hat{x}_{t-1}=\mathcal{D}(\hat{z}_{t-1})$ at ‘$t-1$’.

Since SD performs the diffusion process in the latent space, frequent transitions between the latent and pixel spaces pose a considerable computation burden. This is particularly pronounced when the image’s resolution. In our empirical observations, we found that restricting the denoising procedure to the initial 5 steps out of a total of 15 achieves a favorable balance between image fidelity and computational efficiency.

5 Experiments

5.1 Implementation Details

For high-resolution image generation, we conducted extensive experiments on two text-to-image diffusion models, Stable Diffusion 2.1 [65] and Stable Diffusion XL [45]. To ensure a fair comparison with baseline methods, we validate our method with inference resolutions of 4$\times$ and 16$\times$ of the model’s original training resolution. In detail, we generate the resolutions of $1024^{2}$, $2048^{2}$ for Stable Diffusion 2.1, and $2048^{2}$, $4096^{2}$ for Stable Diffusion XL. In the case of video generation, we conducted experiments on ModelScope [71]. We used 50 DDIM steps to generate both images and videos. As mentioned in Sec. 4.2 and Sec. 4.3, we fixed our hyperparameters to $N=15$ and DWT-guidance step as 5 steps.

5.2 Evaluation

We utilized the LAION-400M [57] dataset as a benchmark for image generation experiments, which comprises 400 million image-text pairs. We randomly sample captions from the benchmark dataset and generated images corresponding to the sampled captions. Due to substantial computational cost, we generated 20K images for $1024^{2}$, 5K images for $2048^{2}$, and 1K images for $4096^{2}$, and compared the performance of our method against baselines. We selected Frechet Inception Distance (FID) and Kernel Inception Distance (KID), denoted as $FID_{r}$ and $KID_{r}$, as our evaluation metrics. Following previous work [19], we additionally report the metrics between the generated samples of the increased resolutions and the base resolutions in order to estimate the degree of each method preserving the original model generation capability, denoted as $FID_{b}$ and $KID_{b}$.

Similarly to image experiment settings, for video generation, we sampled 2048 random captions from the WebVid-10M dataset [3] and measured the Frechet Video Distance (FVD) [70] with 16 frame videos. The resolution of the video generated with our method is $1024^{2}$ since the model is capable of generating $512^{2}$ resolution videos.

Figure 6: $4\times$ larger resolution results obtained with DiffuseHigh.

5.3 Image generation

We compare both of our methods, the progressive-only approach, denoted as ‘DiffuseHigh (w/o DWT)’ (Sec. 4.2) and ‘DiffuseHigh’ (Sec. 4.3) using DWT-based guidance against the existing training-free methods, e.g., direct inference of the stable diffusion models (D.I) and ScaleCrafter [19]. We excluded methods [6, 34] that are able to generate higher resolution image but poses object repetition problem from baselines. Moreover, we empirically found that utilizing multiple resolutions during the generation yields better results. We, therefore, added one more intermediate resolution in DiffuseHigh, i.e., [$512^{2}$, $768^{2}$, $1024^{2}$] for Stable Diffusion and [$1024^{2}$, $1536^{2}$, $2048^{2}$] for Stable Diffusion XL, on $4\times$ experiments.

	SD 2.1 (1K)				SDXL 1.0 (2K)
Method	$FID_{r}$	$KID_{r}$	$FID_{b}$	$KID_{b}$	$FID_{r}$	$KID_{r}$	$FID_{b}$	$KID_{b}$
D.I	58.77	0.0176	43.21	0.0094	69.37	0.025	47.01	0.128
ScaleCrafter	32.60	0.0117	18.43	0.0047	62.84	0.020	44.84	0.0104
DiffuseHigh (w/o DWT)	18.11	0.0066	4.09	0.0006	30.74	0.0081	17.21	0.0015
DiffuseHigh	18.72	0.0069	3.56	0.0003	26.08	0.0077	12.46	0.0001

Table 1: Evaluation of $4\times$ experiments on LAION-400M. We compare our proposed method with training-free baseline methods. The table shows the metric scores of each methods.

Figure 7: Visualization of DWT components corresponding to $S$-th DDIM timestep within DiffuseHigh. Since we replace the LL component of the estimated clean image during several DDIM reverse steps, a large amount of LL components are preserved through the denoising process.

We report our evaluation results of $4\times$ resolution inference experiment on Tab. 1. As observed, both of our approaches surpassed the given baselines by a large margin in terms of both FID and KID. In settings of SD 2.1, DiffuseHigh (w/o DWT) slightly preceded DiffuseHigh, while DiffuseHigh showed superior results compared to DiffuseHigh (w/o DWT) in SDXL settings. Qualitative samples are shown in Fig. 6.

We also report the qualitative evaluation metrics on $16\times$ settings on Tab. 2, mainly comparing DiffuseHigh (w/o DWT) with DiffuseHigh. Similar to the $4\times$ experiment, DiffuseHigh (w/o DWT) approach showed better results with SD 2.1, while DiffuseHigh achieved a better score with SDXL.

Figure 8: $16\times$ larger resolution results obtained with DiffuseHigh.

We conjecture that this consistent observation stems from the capacity of the diffusion model to generate correct shapes, structures, and details of the object. We observed that SD 2.1 is more likely to generate objects with undesirable appearance compared to SDXL. Since DiffuseHigh generates a high-resolution image guided by the low-resolution generated image, the object in the image is highly likely to preserve eccentric structures. Nonetheless, the progressive-only approach has the opportunity to amend the flawed shapes since it has a more flexible denoising process. In contrast, the outcomes diverge significantly with SDXL. Due to its proficiency in generating natural-looking images, leveraging guidance from the low-resolution image proves advantageous. This process ensures the accurate incorporation of structural attributes, leading to highly convincing and satisfactory results with DiffuseHigh. The quantitative images produced by DiffuseHigh are illustrated in Fig. 8.

	SD 2.1 (2K)				SDXL 1.0 (4K)
Method	$FID_{r}$	$KID_{r}$	$FID_{b}$	$KID_{b}$	$FID_{r}$	$KID_{r}$	$FID_{b}$	$KID_{b}$
DiffuseHigh (w/o DWT)	27.42	0.0065	14.05	0.0011	60.27	0.0081	44.06	0.0003
DiffuseHigh	30.96	0.0083	12.95	0.0005	59.62	0.0077	43.35	0.0002

Table 2: Evaluation of $16\times$ experiments on LAION-400M.

5.4 Video generation

To further validate the versatility of DiffuseHigh, we adapt our proposed method on ModelScope [71] to generate higher-resolution video over its original resolution. As observed in images, directly inferencing on pretrained video model also resulted in severe object repetition problem (See Fig. 9). We report the quantitative results of video experiments on Tab. 3. As observed, our proposed DiffuseHigh achieved a lower FVD [70] score compared to the direct inference with a large margin. It demonstrates the versatility of our method, which shows superior performance in adapting the video diffusion model (ModelScope [71]) on higher-resolution settings. Additional qualitative examples are provided in supplemantary matarials.

	ModelScope (1K)
Method	FVD
D.I	785.16
DiffuseHigh	607.99

Table 3: Video experiments results. We used Frechet Video Distance(FVD) with 16 frames as an evaluation metric on 2048 generated videos. Captions are randomly sampled from the WebVid-10M dataset [3].

Figure 9: Video adaptation experiment. We used ModelScope [71] model as a baseline. (a) Text prompt: ‘Darth Vader is surfing on waves.’ (b) Text prompt: ’A teddy bear is dancing in front of the building’

6 Limitation and Discussion

Since DiffuseHigh leverages generated low-resolution images as structural guidance, the generation ability of the diffusion model at its original resolution heavily affects the overall performance of our method. That is, several structural defects or flaws in low-resolution images are also likely to be guided to the resulting higher-resolution image. However, we believe that leveraging tuning-free enhancement methods such as FreeU [60], which improve the quality and fidelity of the sampled low-resolution image, would further improve the quality and fidelity of the resulting high-resolution image and leave it as a future work.

7 Conclusion

We present a training-free progressive high-resolution image synthesis pipeline using a pretrained diffusion model on low-resolution images. Inspired by the recent noising-denoising technique, our proposal involves leveraging generated low-resolution images as a guiding mechanism to effectively preserve the overall structure and intricate details of the contents. We also propose a principled way of incorporating structure information into the denoising process through frequency domain representation. This allows us to retain the essential information presented in the low-resolution images. The extensive experiments with the pretrained SD models have shown that the proposed DiffuseHigh generates higher-resolution images without commonly known issues in the existing approaches, such as repetitive patterns and irregular structures.

Supplementary material for DiffuseHigh

A Comparison with Stable Diffusion Upscaler

Figure 10: Comparison to SD + SR on $1024^{2}$ image generation setting. (a) results of SD + SR, (b) results of DiffuseHigh. We used SD2.1 for both methods. Ours compose more reliable textures and high-frequency details.

Leveraging a Super-Resolution (SR) model is also an appealing approach for generating images with large resolution. However, utilizing SR models is often challenging due to the difficulty of collecting higher-resolution image datasets and the computational cost of training the separate super-resolution model on higher-resolution images.

In order to comprehensively evaluate our method, we compare DiffuseHigh against pretrained SR models, namely Stable Diffusion Latent Upscaler [66] and Stable Diffusion Upscaler [67], which has the capacity to increase the resolution of the given image up to $4\times$ and $16\times$, respectively. We conducted experiments on Stable Diffusion 2.1 [65] and generated $1024^{2}$ and $2048^{2}$ resolution images for each method.

	SD 2.1 (1K)		SD 2.1 (2K)
Method	FID	KID	FID	KID
SD + SR	18.66	0.0070	30.05	0.0087
DiffuseHigh	18.72	0.0069	27.80	0.0069

Table 4: SDSR experiments results.

As observed in Tab. 4, for $4\times$ experiment, the SD + SR method showed a slightly lower FID score and higher KID score compared to DiffuseHigh, but the difference is negligible. In the case of the $16\times$ setting, DiffuseHigh surpassed SD + SR both on FID and KID scores. Also, as mentioned in  [19], SD + SR often fails to compose reliable texture and details of the image, as shown in  Fig. 10. These quantitative and qualitative results highlight the efficiency and efficacy of DiffuseHigh, where ours employ only the pretrained Stable Diffusion.

B Additional qualitative image examples

Figure 11: Additional generated samples with SD 2.1.

Figure 12: Additional generated samples with SD 2.1.

Figure 13: Additional generated samples with SD 2.1.

Figure 14: Additional generated samples with SDXL.

Figure 15: Additional generated samples with SDXL.

Figure 16: Additional generated samples with SDXL.

Figure 17: Additional generated samples with SDXL.

C Additional qualitative video examples

Figure 18: Text prompt: ‘A dolphin jum** out of the water’

Figure 19: Text prompt: ‘A deer is walking in the forest’

Figure 20: Text prompt: ‘A flickering big campfire in the woods’