ConsistI2V: Enhancing Visual Consistency for Image-to-Video Generation

Diffusion Models (DMs) (Sohl-Dickstein et al., 2015; Ho et al., 2020) are generative models that learn to model the data distribution by iteratively recovering perturbed inputs. Given a training sample $\mathbf{x_{0}}\sim q(\mathbf{x_{0}})$, DMs first obtain the corrupted input through a forward diffusion process $q(\mathbf{x}_{t}|\mathbf{x}_{0},t),t\in\{1,2,...,T\}$ by using the parameterization trick from Sohl-Dickstein et al. (2015) and gradually adds Gaussian noise $\mathbf{\epsilon}\in\mathcal{N}(\mathbf{0},\mathbf{I})$ to the input: $\mathbf{x}_{t}=\sqrt{\bar{\alpha}_{t}}\mathbf{x}_{0}+\sqrt{1-\bar{\alpha}_{t}}\mathbf{\mathbf{\epsilon}},\bar{\alpha}_{t}=\prod_{i=1}^{t}(1-\beta_{i}),$ where $0<\beta_{1}<\beta_{2}<...<\beta_{T}<1$ is a known variance schedule that controls the amount of noise added at each time step $t$. The diffusion model is then trained to approximate the backward process $p(\mathbf{x}_{t-1}|\mathbf{x}_{t})$ and recovers $\mathbf{x}_{t-1}$ from $\mathbf{x}_{t}$ using a denoising network $\epsilon_{\theta}(\mathbf{x}_{t},\mathbf{c},t)$, which can be learned by minimizing the mean squared error (MSE) between the predicted and target noise: $\min_{\theta}\mathbb{E}_{\mathbf{x},\mathbf{\epsilon\in\mathcal{N}(\mathbf{0},\mathbf{I})},\mathbf{c},t}[\|\mathbf{\epsilon}-\mathbf{\epsilon}_{\theta}(\mathbf{x}_{t},\mathbf{c},t)\|_{2}^{2}]$ ($\epsilon-$prediction). Here, $\mathbf{c}$ denotes the (optional) conditional signal that DMs can be conditioned on. For our model, $\mathbf{c}$ is a combination of an input first frame and a text prompt.

Latent Diffusion Models (LDMs) (Rombach et al., 2022) are variants of diffusion models that first use a pretrained encoder $\mathcal{E}$ to obtain a latent representation $\mathbf{z}_{0}=\mathcal{E}(\mathbf{x}_{0})$. LDMs then perform the forward process $q(\mathbf{z}_{t}|\mathbf{z}_{0},t)$ and the backward process $p_{\theta}(\mathbf{z}_{t-1}|\mathbf{z}_{t})$ in this compressed latent space. The generated sample $\mathbf{\hat{x}}$ can be obtained from the denoised latent using a pretrained decoder $\mathbf{\hat{x}}=\mathcal{D}(\mathbf{\hat{z}})$.

The spatial attention layer in the LDM U-Net contains a self-attention layer that operates on each frame independently and a cross-attention layer that operates between frames and the encoded text prompt. Given an intermediate hidden state $z^{i}$ of the $i^{\mathrm{th}}$ frame, the self-attention operation is formulated as the attention between different spatial positions of $z^{i}$:

$$Q_{s}=W_{s}^{Q}z^{i},K_{s}=W_{s}^{K}z^{i},V_{s}=W_{s}^{V}z^{i},$$

(1)

$$\mathrm{Attention}(Q_{s},K_{s},V_{s})=\mathrm{Softmax}(\frac{Q_{s}K_{s}^{\top}}{\sqrt{d}})V_{s},$$

(2)

where $W_{s}^{Q},W_{s}^{K}$ and $W_{s}^{V}$ are learnable projection matrices for creating query, key and value vectors from the input. $d$ is the dimension of the query and key vectors. To achieve better visual coherency in the video, we modify the key and value vectors in the self-attention layers to also include features from the first frame $z^{1}$ (cf. Figure 3):

$$Q_{s}=W_{s}^{Q}z^{i},K_{s}^{\prime}=W_{s}^{K}[z^{i},z^{1}],V_{s}^{\prime}=W_{s}^{V}[z^{i},z^{1}],$$

(3)

where $[\cdot]$ represents the concatenation operation such that the token sequence length in $K_{s}^{\prime}$ and $V_{s}^{\prime}$ are doubled compared to the original $K_{s}$ and $V_{s}$. In this way, each spatial position in all frames gets access to the complete information from the first frame, allowing fine-grained feature conditioning in the spatial attention layers.

The temporal self-attention operations in video diffusion models (Ho et al., 2022b; Blattmann et al., 2023; Wang et al., 2023d; Chen et al., 2023a) share a similar formulation to spatial self-attention (cf. Equation 1), with the exception that the intermediate hidden state being formulated as $\mathbf{\bar{z}}\in\mathbb{R}^{(H\times W)\times N\times C}$, where $N$ is the number of frames and $C,H,W$ correspond to the channel, height and width dimension of the hidden state tensor. Formally, given an input hidden state $\mathbf{z}\in\mathbb{R}^{N\times C\times H\times W}$, $\mathbf{\bar{z}}$ is obtained by resha** the height and width dimension of $\mathbf{z}$ to the batch dimension such that $\mathbf{\bar{z}}\leftarrow\texttt{rearrange}(\mathbf{z},\texttt{"N C H W -> (H W) N C"})$ (using einops (Rogozhnikov, 2021) notation). Intuitively, every $1\times N\times C$ matrices in $\mathbf{\bar{z}}$ represents features at the same $1\times 1$ spatial location across all $N$ frames. This limits the temporal attention layers to always look at a small spatial window when attending to features between different frames.

To effectively leverage the first frame features, we also augment the temporal self-attention layers to include a broader window of features from the first frame. We propose to compute the query, key and value from $\mathbf{\bar{z}}$ as:

$$Q_{t}=W_{t}^{Q}\mathbf{\bar{z}},K_{t}^{\prime}=W_{t}^{K}[\mathbf{\bar{z}},\tilde{z}^{1}],V_{t}^{\prime}=W_{t}^{V}[\mathbf{\bar{z}},\tilde{z}^{1}],$$

(4)

where $\tilde{z}^{1}\in\mathbb{R}^{(H\times W)\times(K\times K-1)\times C}$ is a tensor constructed in a way that its $h\times w$ position in the batch dimension corresponds to an $K\times K$ window of the first frame features, centred at the spatial position of $(h,w)$. The first frame feature vector at $(h,w)$ is not included in $\tilde{z}^{1}$ as it is already presented in $\mathbf{\bar{z}}$, making the second dimension of $\tilde{z}^{1}$ as $K\times K-1$. We pad $\mathbf{z}$ in its spatial dimensions by replicating the boundary values to ensure that all spatial positions in $\mathbf{z}$ will have a complete window around them. We then concatenate $\mathbf{\bar{z}}$ with $\tilde{z}^{1}$ to enlarge the sequence length of the key and value matrices. See Figure 3 for a visualization of the augmented temporal self-attention. Our main rationale is that the visual objects in a video may move to different spatial locations as the video progresses. The vanilla or common formulation of temporal self-attention in video diffusion models essentially assumes $K=1$, as shown by the blue tokens in Figure 3. Having an extra window of keys and values around the query location with the window size $K>1$ (green tokens in Figure 3) increases the probability of attending to the same entity in the first frame when performing temporal self-attention. In practice, we set $K=3$ to control the time complexity of the attention operations. As the input latent is downsampled 8$\times$ by the VAE, and the denoising U-Net further downsamples the feature map in its intermediate layers, our selected window size ($K=3$) can still create a large receptive field in deeper U-Net layers. For example, in the second last level of the U-Net (32$\times$ total downsampling), a $256\times 256$ video will be compressed into a $8\times 8$ feature map. Consequently, a $3\times 3$ window on this feature map covers a region of $96\times 96$ in the input first frame. This is a significant increase from previous methods where a $1\times 1$ window on the feature map corresponded to only a $24\times 24$ region in the original frame.

Existing literature (Lin et al., 2024) in image diffusion models has identified that there exists a noise initialization gap between training and inference, due to the fact that common diffusion noise schedules create an information leak to the diffusion noise during training, causing it to be inconsistent with the random Gaussian noise sampled during inference. In the domain of video generation, this initialization gap has been further explored by FreeInit (Wu et al., 2023c), showing that the information leak mainly comes from the low-frequency component of a video after spatiotemporal frequency decomposition and adding this low-frequency component to the initial inference noise greatly enhances the quality of the generated videos. Inspired by the significance of frequency bands in images and videos for human perception, prior work like Blurring Diffusion Models (Hoogeboom & Salimans, 2022) employs diffusion processes in the frequency domain to enhance generation results. Here, we also aim to study how to effectively leverage the video’s different frequency components to enhance I2V generation.

To better understand how spatiotemporal frequency bands and visual features are correlated in videos, we visualize the videos decoded from the VAE latents after spatiotemporal frequency decomposition, as shown in Figure 4. We observe that the video’s high-frequency component captures the fast-moving objects and the fine details in the video, whereas the low-frequency component corresponds to those slowly moving parts and represents an overall layout. Based on this observation, we propose FrameInit, which duplicates the input first frame into a static video and uses its low-frequency component as a coarse layout guidance during inference. Formally, given the latent representation $\mathbf{z}_{0}$ of a static video and an inference noise $\mathbf{\epsilon}$, we first add $\tau$ step inference noise to the static video to obtain $\mathbf{z}_{\tau}=\texttt{add\_noise}(\mathbf{z}_{0},\mathbf{\epsilon},\tau)$. We then extract the low-frequency component of $\mathbf{z}_{\tau}$ and mix it with $\mathbf{\epsilon}$:

	$\displaystyle\mathcal{F}^{low}_{\mathbf{z}_{\tau}}$	$\displaystyle=\texttt{FFT\_3D}(\mathbf{z}_{\tau})\odot\mathcal{G}(D_{0}),$		(5)
	$\displaystyle\mathcal{F}^{high}_{\mathbf{\epsilon}}$	$\displaystyle=\texttt{FFT\_3D}(\mathbf{\epsilon})\odot(1-\mathcal{G}(D_{0})),$		(6)
	$\displaystyle\mathbf{\epsilon}^{\prime}$	$\displaystyle=\texttt{IFFT\_3D}(\mathcal{F}^{low}_{\mathbf{z}_{\tau}}+\mathcal{F}^{high}_{\mathbf{\epsilon}}),$		(7)

where FFT_3D is the 3D discrete fast Fourier transformation operating on spatiotemporal dimensions and IFFT_3D is the inverse FFT operation. $\mathcal{G}$ is the Gaussian low-pass filter parameterized by the normalized space-time stop frequency $D_{0}$. The modified noise $\mathbf{\epsilon}^{\prime}$ containing the low-frequency information from the static video is then used for denoising.

By implementing FrameInit, our empirical analysis reveals a significant enhancement in the stabilization of generated videos, demonstrating improved video quality and consistency. FrameInit also enables our model with two additional applications: (1) autoregressive long video generation and (2) camera motion control. We showcase more results for each application in Section 5.5.

We use Stable Diffusion 2.1-base (Rombach et al., 2022) as the base T2I model to initialize ConsistI2V and train the model on the WebVid-10M (Bain et al., 2021) dataset, which contains $\sim$10M video-text pairs. For each video, we sample 16 frames with a spatial resolution of $256\times 256$ and a frame interval between $1\leq v\leq 5$, which is used as a conditional input to the model to enable FPS control. We use the first frame as the image input and learn to denoise the subsequent 15 frames during training. Our model is trained with the $\epsilon$ objective over all U-Net parameters using a batch size of 192 and a learning rate of 5e-5 for 170k steps. During training, we randomly drop input text prompts with a probability of 0.1 to enable classifier-free guidance (Ho & Salimans, 2022). During inference, we employ the DDIM sampler (Song et al., 2020) with 50 steps and classifier-free guidance with a guidance scale of $w=7.5$ to sample videos. We apply FrameInit with $\tau=850$ and $D_{0}=0.25$ for inference noise initialization.

We present a visual comparison of our model with DynamiCrafter, SEINE and AnimateAnything in Figure 5. We exclude I2VGen-XL in this section as its generated video cannot fully adhere to the visual details from the input first frame. As shown in the figure, current methods often struggle with maintaining appearance and motion consistency in video sequences. This can include (1) sudden changes in subject appearance mid-video, as demonstrated in the “ice cream” case for DynamiCrafter and SEINE; (2) background inconsistency, as observed in DynamiCrafter’s “wooden figurine walking” case; (3) unnatural object movements, evident in the “dog swimming” case (DynamiCrafter) and “tornado in a jar” case (SEINE) and (4) minimal or absent movement, as displayed in most of the generated videos by AnimateAnything. On the other hand, ConsistI2V produces videos with subjects that consistently align with the input first frame. Additionally, our generated videos exhibit more natural and logical motion, avoiding abrupt changes and thereby ensuring improved appearance and motion consistency. More visual results of our model can be found in Appendix H.

To verify the effectiveness of our design choices, we conduct an ablation study on UCF-101 by iteratively disabling FrameInit, temporal first frame conditioning and spatial first frame conditioning. We follow the same experiment setups in Section 5.2 and show the results in Table 3.

Effectiveness of FrameInit According to the results, we find that applying FrameInit greatly enhances the model performance for all metrics. Our empirical observation suggests that FrameInit can stabilize the output video and reduce the abrupt appearance and motion changes. As shown in Figure 6, while our model can still generate reasonable results without enabling FrameInit, the output videos also suffer from a higher chance of rendering sudden object movements and blurry frames (last frame in the second row). This highlights the effectiveness of FrameInit in producing more natural motion and higher-quality frames.

Hyperparameters of FrameInit To further investigate how the FrameInit hyperparameters $\tau$ (number of steps of inference noise added to the first frame) and $D_{0}$ (spatiotemporal stop frequency) affect the generated videos, we conduct an additional qualitative experiment by generating the same video using different FrameInit hyperparameters and show the results in Figure 7. Comparing (1) and (4) in Figure 7, we find that while using our default setting of FrameInit achieves more stable outputs, it also slightly restricts the motion magnitude of the generated video compared to not using FrameInit at all. From comparisons between (1), (5), and (1), (6), we observe that either increasing $D_{0}$ alone or reducing $\tau$ alone does not significantly impact the motion magnitude of the videos. However, comparisons between (1), (2), and (3) reveal that when both $D_{0}$ is increased and $\tau$ is decreased, there is a further restriction in video dynamics. This suggests that the static first frame video exerts a stronger layout guidance, intensifying its influence on video dynamics.

Spatiotemporal First Frame Conditioning Our ablation results on UCF-101 (c.f. Table 3) reflects a significant performance boost after applying the proposed spatial and temporal first frame conditioning mechanisms to our model. Although removing temporal first frame conditioning leads to an overall better performance for the three quantitative metrics, in practice we find that only using spatial conditioning often results in jittering motion and larger object distortions, evident in the third row of Figure 6. When both spatial and temporal first frame conditioning is removed, our model loses the capability of maintaining the appearance of the input first frame.

Runtime Efficiency To better understand how the proposed model components affect the inference cost, we also benchmark the runtime statistics of our ConsistI2V and its variants after disabling different components. We measure all models on a single Nvidia RTX 4090 with float32 and an inference batch size of 1. For TFLOPs computation, we only consider the denoising U-Net and exclude components such as the VAE and text encoder as these modules share the same computation across all model variants. As shown in the right part of Table 3, we find that the proposed spatiotemporal feature conditioning and FrameInit incur limited computational overhead during inference, introducing approximately 200Mb extra GPU memory and $\sim$1.3s additional inference time. These results demonstrate that our method is highly efficient during inference.