-
Slicedit: Zero-Shot Video Editing With Text-to-Image Diffusion Models Using Spatio-Temporal Slices
Authors:
Nathaniel Cohen,
Vladimir Kulikov,
Matan Kleiner,
Inbar Huberman-Spiegelglas,
Tomer Michaeli
Abstract:
Text-to-image (T2I) diffusion models achieve state-of-the-art results in image synthesis and editing. However, leveraging such pretrained models for video editing is considered a major challenge. Many existing works attempt to enforce temporal consistency in the edited video through explicit correspondence mechanisms, either in pixel space or between deep features. These methods, however, struggle…
▽ More
Text-to-image (T2I) diffusion models achieve state-of-the-art results in image synthesis and editing. However, leveraging such pretrained models for video editing is considered a major challenge. Many existing works attempt to enforce temporal consistency in the edited video through explicit correspondence mechanisms, either in pixel space or between deep features. These methods, however, struggle with strong nonrigid motion. In this paper, we introduce a fundamentally different approach, which is based on the observation that spatiotemporal slices of natural videos exhibit similar characteristics to natural images. Thus, the same T2I diffusion model that is normally used only as a prior on video frames, can also serve as a strong prior for enhancing temporal consistency by applying it on spatiotemporal slices. Based on this observation, we present Slicedit, a method for text-based video editing that utilizes a pretrained T2I diffusion model to process both spatial and spatiotemporal slices. Our method generates videos that retain the structure and motion of the original video while adhering to the target text. Through extensive experiments, we demonstrate Slicedit's ability to edit a wide range of real-world videos, confirming its clear advantages compared to existing competing methods. Webpage: https://matankleiner.github.io/slicedit/
△ Less
Submitted 20 May, 2024;
originally announced May 2024.
-
SinDDM: A Single Image Denoising Diffusion Model
Authors:
Vladimir Kulikov,
Shahar Yadin,
Matan Kleiner,
Tomer Michaeli
Abstract:
Denoising diffusion models (DDMs) have led to staggering performance leaps in image generation, editing and restoration. However, existing DDMs use very large datasets for training. Here, we introduce a framework for training a DDM on a single image. Our method, which we coin SinDDM, learns the internal statistics of the training image by using a multi-scale diffusion process. To drive the reverse…
▽ More
Denoising diffusion models (DDMs) have led to staggering performance leaps in image generation, editing and restoration. However, existing DDMs use very large datasets for training. Here, we introduce a framework for training a DDM on a single image. Our method, which we coin SinDDM, learns the internal statistics of the training image by using a multi-scale diffusion process. To drive the reverse diffusion process, we use a fully-convolutional light-weight denoiser, which is conditioned on both the noise level and the scale. This architecture allows generating samples of arbitrary dimensions, in a coarse-to-fine manner. As we illustrate, SinDDM generates diverse high-quality samples, and is applicable in a wide array of tasks, including style transfer and harmonization. Furthermore, it can be easily guided by external supervision. Particularly, we demonstrate text-guided generation from a single image using a pre-trained CLIP model.
△ Less
Submitted 6 June, 2023; v1 submitted 29 November, 2022;
originally announced November 2022.
-
A Tight Bound on the Performance of a Minimal-Delay Joint Source-Channel Coding Scheme
Authors:
Marius Kleiner,
Bixio Rimoldi
Abstract:
An analog source is to be transmitted across a Gaussian channel in more than one channel use per source symbol. This paper derives a lower bound on the asymptotic mean squared error for a strategy that consists of repeatedly quantizing the source, transmitting the quantizer outputs in the first channel uses, and sending the remaining quantization error uncoded in the last channel use. The bound…
▽ More
An analog source is to be transmitted across a Gaussian channel in more than one channel use per source symbol. This paper derives a lower bound on the asymptotic mean squared error for a strategy that consists of repeatedly quantizing the source, transmitting the quantizer outputs in the first channel uses, and sending the remaining quantization error uncoded in the last channel use. The bound coincides with the performance achieved by a suboptimal decoder studied by the authors in a previous paper, thereby establishing that the bound is tight.
△ Less
Submitted 14 January, 2010;
originally announced January 2010.
-
Asymptotically Optimal Joint Source-Channel Coding with Minimal Delay
Authors:
Marius Kleiner,
Bixio Rimoldi
Abstract:
We present and analyze a joint source-channel coding strategy for the transmission of a Gaussian source across a Gaussian channel in n channel uses per source symbol. Among all such strategies, our scheme has the following properties: i) the resulting mean-squared error scales optimally with the signal-to-noise ratio, and ii) the scheme is easy to implement and the incurred delay is minimal, in…
▽ More
We present and analyze a joint source-channel coding strategy for the transmission of a Gaussian source across a Gaussian channel in n channel uses per source symbol. Among all such strategies, our scheme has the following properties: i) the resulting mean-squared error scales optimally with the signal-to-noise ratio, and ii) the scheme is easy to implement and the incurred delay is minimal, in the sense that a single source symbol is encoded at a time.
△ Less
Submitted 24 September, 2009; v1 submitted 8 April, 2009;
originally announced April 2009.
-
A Constrained Object Model for Configuration Based Workflow Composition
Authors:
Patrick Albert,
Laurent Henocque,
Mathias Kleiner
Abstract:
Automatic or assisted workflow composition is a field of intense research for applications to the world wide web or to business process modeling. Workflow composition is traditionally addressed in various ways, generally via theorem proving techniques. Recent research observed that building a composite workflow bears strong relationships with finite model search, and that some workflow languages…
▽ More
Automatic or assisted workflow composition is a field of intense research for applications to the world wide web or to business process modeling. Workflow composition is traditionally addressed in various ways, generally via theorem proving techniques. Recent research observed that building a composite workflow bears strong relationships with finite model search, and that some workflow languages can be defined as constrained object metamodels . This lead to consider the viability of applying configuration techniques to this problem, which was proven feasible. Constrained based configuration expects a constrained object model as input. The purpose of this document is to formally specify the constrained object model involved in ongoing experiments and research using the Z specification language.
△ Less
Submitted 9 June, 2005;
originally announced June 2005.
