A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence
Authors:
Junyi Zhang,
Charles Herrmann,
Junhwa Hur,
Luisa Polania Cabrera,
Varun Jampani,
Deqing Sun,
Ming-Hsuan Yang
Abstract:
Text-to-image diffusion models have made significant advances in generating and editing high-quality images. As a result, numerous approaches have explored the ability of diffusion model features to understand and process single images for downstream tasks, e.g., classification, semantic segmentation, and stylization. However, significantly less is known about what these features reveal across mul…
▽ More
Text-to-image diffusion models have made significant advances in generating and editing high-quality images. As a result, numerous approaches have explored the ability of diffusion model features to understand and process single images for downstream tasks, e.g., classification, semantic segmentation, and stylization. However, significantly less is known about what these features reveal across multiple, different images and objects. In this work, we exploit Stable Diffusion (SD) features for semantic and dense correspondence and discover that with simple post-processing, SD features can perform quantitatively similar to SOTA representations. Interestingly, the qualitative analysis reveals that SD features have very different properties compared to existing representation learning features, such as the recently released DINOv2: while DINOv2 provides sparse but accurate matches, SD features provide high-quality spatial information but sometimes inaccurate semantic matches. We demonstrate that a simple fusion of these two features works surprisingly well, and a zero-shot evaluation using nearest neighbors on these fused features provides a significant performance gain over state-of-the-art methods on benchmark datasets, e.g., SPair-71k, PF-Pascal, and TSS. We also show that these correspondences can enable interesting applications such as instance swap** in two images.
△ Less
Submitted 28 November, 2023; v1 submitted 24 May, 2023;
originally announced May 2023.
Boosted Embeddings for Time Series Forecasting
Authors:
Sankeerth Rao Karingula,
Nandini Ramanan,
Rasool Tahmasbi,
Mehrnaz Amjadi,
Deokwoo Jung,
Ricky Si,
Charanraj Thimmisetty,
Luisa Polania Cabrera,
Marjorie Sayer,
Claudionor Nunes Coelho Jr
Abstract:
Time series forecasting is a fundamental task emerging from diverse data-driven applications. Many advanced autoregressive methods such as ARIMA were used to develop forecasting models. Recently, deep learning based methods such as DeepAr, NeuralProphet, Seq2Seq have been explored for time series forecasting problem. In this paper, we propose a novel time series forecast model, DeepGB. We formulat…
▽ More
Time series forecasting is a fundamental task emerging from diverse data-driven applications. Many advanced autoregressive methods such as ARIMA were used to develop forecasting models. Recently, deep learning based methods such as DeepAr, NeuralProphet, Seq2Seq have been explored for time series forecasting problem. In this paper, we propose a novel time series forecast model, DeepGB. We formulate and implement a variant of Gradient boosting wherein the weak learners are DNNs whose weights are incrementally found in a greedy manner over iterations. In particular, we develop a new embedding architecture that improves the performance of many deep learning models on time series using Gradient boosting variant. We demonstrate that our model outperforms existing comparable state-of-the-art models using real-world sensor data and public dataset.
△ Less
Submitted 11 July, 2021; v1 submitted 10 April, 2021;
originally announced April 2021.
