-
Video Diffusion Alignment via Reward Gradients
Authors:
Mihir Prabhudesai,
Russell Mendonca,
Zheyang Qin,
Katerina Fragkiadaki,
Deepak Pathak
Abstract:
We have made significant progress towards building foundational video diffusion models. As these models are trained using large-scale unsupervised data, it has become crucial to adapt these models to specific downstream tasks. Adapting these models via supervised fine-tuning requires collecting target datasets of videos, which is challenging and tedious. In this work, we utilize pre-trained reward…
▽ More
We have made significant progress towards building foundational video diffusion models. As these models are trained using large-scale unsupervised data, it has become crucial to adapt these models to specific downstream tasks. Adapting these models via supervised fine-tuning requires collecting target datasets of videos, which is challenging and tedious. In this work, we utilize pre-trained reward models that are learned via preferences on top of powerful vision discriminative models to adapt video diffusion models. These models contain dense gradient information with respect to generated RGB pixels, which is critical to efficient learning in complex search spaces, such as videos. We show that backpropagating gradients from these reward models to a video diffusion model can allow for compute and sample efficient alignment of the video diffusion model. We show results across a variety of reward models and video diffusion models, demonstrating that our approach can learn much more efficiently in terms of reward queries and computation than prior gradient-free approaches. Our code, model weights,and more visualization are available at https://vader-vid.github.io.
△ Less
Submitted 11 July, 2024;
originally announced July 2024.
-
Diffusion-TTA: Test-time Adaptation of Discriminative Models via Generative Feedback
Authors:
Mihir Prabhudesai,
Tsung-Wei Ke,
Alexander C. Li,
Deepak Pathak,
Katerina Fragkiadaki
Abstract:
The advancements in generative modeling, particularly the advent of diffusion models, have sparked a fundamental question: how can these models be effectively used for discriminative tasks? In this work, we find that generative models can be great test-time adapters for discriminative models. Our method, Diffusion-TTA, adapts pre-trained discriminative models such as image classifiers, segmenters…
▽ More
The advancements in generative modeling, particularly the advent of diffusion models, have sparked a fundamental question: how can these models be effectively used for discriminative tasks? In this work, we find that generative models can be great test-time adapters for discriminative models. Our method, Diffusion-TTA, adapts pre-trained discriminative models such as image classifiers, segmenters and depth predictors, to each unlabelled example in the test set using generative feedback from a diffusion model. We achieve this by modulating the conditioning of the diffusion model using the output of the discriminative model. We then maximize the image likelihood objective by backpropagating the gradients to discriminative model's parameters. We show Diffusion-TTA significantly enhances the accuracy of various large-scale pre-trained discriminative models, such as, ImageNet classifiers, CLIP models, image pixel labellers and image depth predictors. Diffusion-TTA outperforms existing test-time adaptation methods, including TTT-MAE and TENT, and particularly shines in online adaptation setups, where the discriminative model is continually adapted to each example in the test set. We provide access to code, results, and visualizations on our website: https://diffusion-tta.github.io/.
△ Less
Submitted 29 November, 2023; v1 submitted 27 November, 2023;
originally announced November 2023.
-
Aligning Text-to-Image Diffusion Models with Reward Backpropagation
Authors:
Mihir Prabhudesai,
Anirudh Goyal,
Deepak Pathak,
Katerina Fragkiadaki
Abstract:
Text-to-image diffusion models have recently emerged at the forefront of image generation, powered by very large-scale unsupervised or weakly supervised text-to-image training datasets. Due to their unsupervised training, controlling their behavior in downstream tasks, such as maximizing human-perceived image quality, image-text alignment, or ethical image generation, is difficult. Recent works fi…
▽ More
Text-to-image diffusion models have recently emerged at the forefront of image generation, powered by very large-scale unsupervised or weakly supervised text-to-image training datasets. Due to their unsupervised training, controlling their behavior in downstream tasks, such as maximizing human-perceived image quality, image-text alignment, or ethical image generation, is difficult. Recent works finetune diffusion models to downstream reward functions using vanilla reinforcement learning, notorious for the high variance of the gradient estimators. In this paper, we propose AlignProp, a method that aligns diffusion models to downstream reward functions using end-to-end backpropagation of the reward gradient through the denoising process. While naive implementation of such backpropagation would require prohibitive memory resources for storing the partial derivatives of modern text-to-image models, AlignProp finetunes low-rank adapter weight modules and uses gradient checkpointing, to render its memory usage viable. We test AlignProp in finetuning diffusion models to various objectives, such as image-text semantic alignment, aesthetics, compressibility and controllability of the number of objects present, as well as their combinations. We show AlignProp achieves higher rewards in fewer training steps than alternatives, while being conceptually simpler, making it a straightforward choice for optimizing diffusion models for differentiable reward functions of interest. Code and Visualization results are available at https://align-prop.github.io/.
△ Less
Submitted 22 June, 2024; v1 submitted 5 October, 2023;
originally announced October 2023.
-
Your Diffusion Model is Secretly a Zero-Shot Classifier
Authors:
Alexander C. Li,
Mihir Prabhudesai,
Shivam Duggal,
Ellis Brown,
Deepak Pathak
Abstract:
The recent wave of large-scale text-to-image diffusion models has dramatically increased our text-based image generation abilities. These models can generate realistic images for a staggering variety of prompts and exhibit impressive compositional generalization abilities. Almost all use cases thus far have solely focused on sampling; however, diffusion models can also provide conditional density…
▽ More
The recent wave of large-scale text-to-image diffusion models has dramatically increased our text-based image generation abilities. These models can generate realistic images for a staggering variety of prompts and exhibit impressive compositional generalization abilities. Almost all use cases thus far have solely focused on sampling; however, diffusion models can also provide conditional density estimates, which are useful for tasks beyond image generation. In this paper, we show that the density estimates from large-scale text-to-image diffusion models like Stable Diffusion can be leveraged to perform zero-shot classification without any additional training. Our generative approach to classification, which we call Diffusion Classifier, attains strong results on a variety of benchmarks and outperforms alternative methods of extracting knowledge from diffusion models. Although a gap remains between generative and discriminative approaches on zero-shot recognition tasks, our diffusion-based approach has significantly stronger multimodal compositional reasoning ability than competing discriminative approaches. Finally, we use Diffusion Classifier to extract standard classifiers from class-conditional diffusion models trained on ImageNet. Our models achieve strong classification performance using only weak augmentations and exhibit qualitatively better "effective robustness" to distribution shift. Overall, our results are a step toward using generative over discriminative models for downstream tasks. Results and visualizations at https://diffusion-classifier.github.io/
△ Less
Submitted 12 September, 2023; v1 submitted 28 March, 2023;
originally announced March 2023.
-
Test-time Adaptation with Slot-Centric Models
Authors:
Mihir Prabhudesai,
Anirudh Goyal,
Sujoy Paul,
Sjoerd van Steenkiste,
Mehdi S. M. Sajjadi,
Gaurav Aggarwal,
Thomas Kipf,
Deepak Pathak,
Katerina Fragkiadaki
Abstract:
Current visual detectors, though impressive within their training distribution, often fail to parse out-of-distribution scenes into their constituent entities. Recent test-time adaptation methods use auxiliary self-supervised losses to adapt the network parameters to each test example independently and have shown promising results towards generalization outside the training distribution for the ta…
▽ More
Current visual detectors, though impressive within their training distribution, often fail to parse out-of-distribution scenes into their constituent entities. Recent test-time adaptation methods use auxiliary self-supervised losses to adapt the network parameters to each test example independently and have shown promising results towards generalization outside the training distribution for the task of image classification. In our work, we find evidence that these losses are insufficient for the task of scene decomposition, without also considering architectural inductive biases. Recent slot-centric generative models attempt to decompose scenes into entities in a self-supervised manner by reconstructing pixels. Drawing upon these two lines of work, we propose Slot-TTA, a semi-supervised slot-centric scene decomposition model that at test time is adapted per scene through gradient descent on reconstruction or cross-view synthesis objectives. We evaluate Slot-TTA across multiple input modalities, images or 3D point clouds, and show substantial out-of-distribution performance improvements against state-of-the-art supervised feed-forward detectors, and alternative test-time adaptation methods.
△ Less
Submitted 27 June, 2023; v1 submitted 21 March, 2022;
originally announced March 2022.
-
CoCoNets: Continuous Contrastive 3D Scene Representations
Authors:
Shamit Lal,
Mihir Prabhudesai,
Ishita Mediratta,
Adam W. Harley,
Katerina Fragkiadaki
Abstract:
This paper explores self-supervised learning of amodal 3D feature representations from RGB and RGB-D posed images and videos, agnostic to object and scene semantic content, and evaluates the resulting scene representations in the downstream tasks of visual correspondence, object tracking, and object detection. The model infers a latent3D representation of the scene in the form of 3D feature points…
▽ More
This paper explores self-supervised learning of amodal 3D feature representations from RGB and RGB-D posed images and videos, agnostic to object and scene semantic content, and evaluates the resulting scene representations in the downstream tasks of visual correspondence, object tracking, and object detection. The model infers a latent3D representation of the scene in the form of 3D feature points, where each continuous world 3D point is mapped to its corresponding feature vector. The model is trained for contrastive view prediction by rendering 3D feature clouds in queried viewpoints and matching against the 3D feature point cloud predicted from the query view. Notably, the representation can be queried for any 3D location, even if it is not visible from the input view. Our model brings together three powerful ideas of recent exciting research work: 3D feature grids as a neural bottleneck for view prediction, implicit functions for handling resolution limitations of 3D grids, and contrastive learning for unsupervised training of feature representations. We show the resulting 3D visual feature representations effectively scale across objects and scenes, imagine information occluded or missing from the input viewpoints, track objects over time, align semantically related objects in 3D, and improve 3D object detection. We outperform many existing state-of-the-art methods for 3D feature learning and view prediction, which are either limited by 3D grid spatial resolution, do not attempt to build amodal 3D representations, or do not handle combinatorial scene variability due to their non-convolutional bottlenecks.
△ Less
Submitted 8 April, 2021;
originally announced April 2021.
-
3D-OES: Viewpoint-Invariant Object-Factorized Environment Simulators
Authors:
Hsiao-Yu Fish Tung,
Zhou Xian,
Mihir Prabhudesai,
Shamit Lal,
Katerina Fragkiadaki
Abstract:
We propose an action-conditioned dynamics model that predicts scene changes caused by object and agent interactions in a viewpoint-invariant 3D neural scene representation space, inferred from RGB-D videos. In this 3D feature space, objects do not interfere with one another and their appearance persists over time and across viewpoints. This permits our model to predict future scenes long in the fu…
▽ More
We propose an action-conditioned dynamics model that predicts scene changes caused by object and agent interactions in a viewpoint-invariant 3D neural scene representation space, inferred from RGB-D videos. In this 3D feature space, objects do not interfere with one another and their appearance persists over time and across viewpoints. This permits our model to predict future scenes long in the future by simply "moving" 3D object features based on cumulative object motion predictions. Object motion predictions are computed by a graph neural network that operates over the object features extracted from the 3D neural scene representation. Our model's simulations can be decoded by a neural renderer into2D image views from any desired viewpoint, which aids the interpretability of our latent 3D simulation space. We show our model generalizes well its predictions across varying number and appearances of interacting objects as well as across camera viewpoints, outperforming existing 2D and 3D dynamics models. We further demonstrate sim-to-real transfer of the learnt dynamics by applying our model trained solely in simulation to model-based control for pushing objects to desired locations under clutter on a real robotic setup
△ Less
Submitted 12 November, 2020;
originally announced November 2020.
-
Disentangling 3D Prototypical Networks For Few-Shot Concept Learning
Authors:
Mihir Prabhudesai,
Shamit Lal,
Darshan Patil,
Hsiao-Yu Tung,
Adam W Harley,
Katerina Fragkiadaki
Abstract:
We present neural architectures that disentangle RGB-D images into objects' shapes and styles and a map of the background scene, and explore their applications for few-shot 3D object detection and few-shot concept classification. Our networks incorporate architectural biases that reflect the image formation process, 3D geometry of the world scene, and shape-style interplay. They are trained end-to…
▽ More
We present neural architectures that disentangle RGB-D images into objects' shapes and styles and a map of the background scene, and explore their applications for few-shot 3D object detection and few-shot concept classification. Our networks incorporate architectural biases that reflect the image formation process, 3D geometry of the world scene, and shape-style interplay. They are trained end-to-end self-supervised by predicting views in static scenes, alongside a small number of 3D object boxes. Objects and scenes are represented in terms of 3D feature grids in the bottleneck of the network. We show that the proposed 3D neural representations are compositional: they can generate novel 3D scene feature maps by mixing object shapes and styles, resizing and adding the resulting object 3D feature maps over background scene feature maps. We show that classifiers for object categories, color, materials, and spatial relationships trained over the disentangled 3D feature sub-spaces generalize better with dramatically fewer examples than the current state-of-the-art, and enable a visual question answering system that uses them as its modules to generalize one-shot to novel objects in the scene.
△ Less
Submitted 20 July, 2021; v1 submitted 6 November, 2020;
originally announced November 2020.
-
3D Object Recognition By Corresponding and Quantizing Neural 3D Scene Representations
Authors:
Mihir Prabhudesai,
Shamit Lal,
Hsiao-Yu Fish Tung,
Adam W. Harley,
Shubhankar Potdar,
Katerina Fragkiadaki
Abstract:
We propose a system that learns to detect objects and infer their 3D poses in RGB-D images. Many existing systems can identify objects and infer 3D poses, but they heavily rely on human labels and 3D annotations. The challenge here is to achieve this without relying on strong supervision signals. To address this challenge, we propose a model that maps RGB-D images to a set of 3D visual feature map…
▽ More
We propose a system that learns to detect objects and infer their 3D poses in RGB-D images. Many existing systems can identify objects and infer 3D poses, but they heavily rely on human labels and 3D annotations. The challenge here is to achieve this without relying on strong supervision signals. To address this challenge, we propose a model that maps RGB-D images to a set of 3D visual feature maps in a differentiable fully-convolutional manner, supervised by predicting views. The 3D feature maps correspond to a featurization of the 3D world scene depicted in the images. The object 3D feature representations are invariant to camera viewpoint changes or zooms, which means feature matching can identify similar objects under different camera viewpoints. We can compare the 3D feature maps of two objects by searching alignment across scales and 3D rotations, and, as a result of the operation, we can estimate pose and scale changes without the need for 3D pose annotations. We cluster object feature maps into a set of 3D prototypes that represent familiar objects in canonical scales and orientations. We then parse images by inferring the prototype identity and 3D pose for each detected object. We compare our method to numerous baselines that do not learn 3D feature visual representations or do not attempt to correspond features across scenes, and outperform them by a large margin in the tasks of object retrieval and object pose estimation. Thanks to the 3D nature of the object-centric feature maps, the visual similarity cues are invariant to 3D pose changes or small scale changes, which gives our method an advantage over 2D and 1D methods.
△ Less
Submitted 30 October, 2020;
originally announced October 2020.
-
Embodied Language Grounding with 3D Visual Feature Representations
Authors:
Mihir Prabhudesai,
Hsiao-Yu Fish Tung,
Syed Ashar Javed,
Maximilian Sieb,
Adam W. Harley,
Katerina Fragkiadaki
Abstract:
We propose associating language utterances to 3D visual abstractions of the scene they describe. The 3D visual abstractions are encoded as 3-dimensional visual feature maps. We infer these 3D visual scene feature maps from RGB images of the scene via view prediction: when the generated 3D scene feature map is neurally projected from a camera viewpoint, it should match the corresponding RGB image.…
▽ More
We propose associating language utterances to 3D visual abstractions of the scene they describe. The 3D visual abstractions are encoded as 3-dimensional visual feature maps. We infer these 3D visual scene feature maps from RGB images of the scene via view prediction: when the generated 3D scene feature map is neurally projected from a camera viewpoint, it should match the corresponding RGB image. We present generative models that condition on the dependency tree of an utterance and generate a corresponding visual 3D feature map as well as reason about its plausibility, and detector models that condition on both the dependency tree of an utterance and a related image and localize the object referents in the 3D feature map inferred from the image. Our model outperforms models of language and vision that associate language with 2D CNN activations or 2D images by a large margin in a variety of tasks, such as, classifying plausibility of utterances, detecting referential expressions, and supplying rewards for trajectory optimization of object placement policies from language instructions. We perform numerous ablations and show the improved performance of our detectors is due to its better generalization across camera viewpoints and lack of object interferences in the inferred 3D feature space, and the improved performance of our generators is due to their ability to spatially reason about objects and their configurations in 3D when map** from language to scenes.
△ Less
Submitted 17 June, 2021; v1 submitted 2 October, 2019;
originally announced October 2019.
-
Studying the Imaging Characteristics of Ultra Violet Imaging Telescope (UVIT) through Numerical Simulations
Authors:
Mudit K. Srivastava,
Swapnil M. Prabhudesai,
Shyam N. Tandon
Abstract:
Ultra-Violet Imaging Telescope (UVIT) is one of the five payloads aboard the Indian Space Research Organization (ISRO)'s ASTROSAT space mission. The science objectives of UVIT are broad, extending from individual hot stars, star-forming regions to active galactic nuclei. Imaging performance of UVIT would depend on several factors in addition to the optics, e.g. resolution of the detectors, Satel…
▽ More
Ultra-Violet Imaging Telescope (UVIT) is one of the five payloads aboard the Indian Space Research Organization (ISRO)'s ASTROSAT space mission. The science objectives of UVIT are broad, extending from individual hot stars, star-forming regions to active galactic nuclei. Imaging performance of UVIT would depend on several factors in addition to the optics, e.g. resolution of the detectors, Satellite Drift and Jitter, image frame acquisition rate, sky background, source intensity etc. The use of intensified CMOS-imager based photon counting detectors in UVIT put their own complexity over reconstruction of the images. All these factors could lead to several systematic effects in the reconstructed images. A study has been done through numerical simulations with artificial point sources and archival image of a galaxy from GALEX data archive, to explore the effects of all the above mentioned parameters on the reconstructed images. In particular the issues of angular resolution, photometric accuracy and photometric-nonlinearity associated with the intensified CMOS-imager based photon counting detectors have been investigated. The photon events in image frames are detected by three different centroid algorithms with some energy thresholds. Our results show that in presence of bright sources, reconstructed images from UVIT would suffer from photometric distortion in a complex way and the presence of overlap** photon events could lead to complex patterns near the bright sources. Further the angular resolution, photometric accuracy and distortion would depend on the values of various thresholds chosen to detect photon events.
△ Less
Submitted 24 March, 2009;
originally announced March 2009.
