-
Rethinking Score Distillation as a Bridge Between Image Distributions
Authors:
David McAllister,
Songwei Ge,
Jia-Bin Huang,
David W. Jacobs,
Alexei A. Efros,
Aleksander Holynski,
Angjoo Kanazawa
Abstract:
Score distillation sampling (SDS) has proven to be an important tool, enabling the use of large-scale diffusion priors for tasks operating in data-poor domains. Unfortunately, SDS has a number of characteristic artifacts that limit its usefulness in general-purpose applications. In this paper, we make progress toward understanding the behavior of SDS and its variants by viewing them as solving an…
▽ More
Score distillation sampling (SDS) has proven to be an important tool, enabling the use of large-scale diffusion priors for tasks operating in data-poor domains. Unfortunately, SDS has a number of characteristic artifacts that limit its usefulness in general-purpose applications. In this paper, we make progress toward understanding the behavior of SDS and its variants by viewing them as solving an optimal-cost transport path from a source distribution to a target distribution. Under this new interpretation, these methods seek to transport corrupted images (source) to the natural image distribution (target). We argue that current methods' characteristic artifacts are caused by (1) linear approximation of the optimal path and (2) poor estimates of the source distribution. We show that calibrating the text conditioning of the source distribution can produce high-quality generation and translation results with little extra overhead. Our method can be easily applied across many domains, matching or beating the performance of specialized methods. We demonstrate its utility in text-to-2D, text-based NeRF optimization, translating paintings to real images, optical illusion generation, and 3D sketch-to-real. We compare our method to existing approaches for score distillation sampling and show that it can produce high-frequency details with realistic colors.
△ Less
Submitted 13 June, 2024;
originally announced June 2024.
-
AC4MPC: Actor-Critic Reinforcement Learning for Nonlinear Model Predictive Control
Authors:
Rudolf Reiter,
Andrea Ghezzi,
Katrin Baumgärtner,
Jasper Hoffmann,
Robert D. McAllister,
Moritz Diehl
Abstract:
\Ac{MPC} and \ac{RL} are two powerful control strategies with, arguably, complementary advantages. In this work, we show how actor-critic \ac{RL} techniques can be leveraged to improve the performance of \ac{MPC}. The \ac{RL} critic is used as an approximation of the optimal value function, and an actor roll-out provides an initial guess for primal variables of the \ac{MPC}. A parallel control arc…
▽ More
\Ac{MPC} and \ac{RL} are two powerful control strategies with, arguably, complementary advantages. In this work, we show how actor-critic \ac{RL} techniques can be leveraged to improve the performance of \ac{MPC}. The \ac{RL} critic is used as an approximation of the optimal value function, and an actor roll-out provides an initial guess for primal variables of the \ac{MPC}. A parallel control architecture is proposed where each \ac{MPC} instance is solved twice for different initial guesses. Besides the actor roll-out initialization, a shifted initialization from the previous solution is used. Thereafter, the actor and the critic are again used to approximately evaluate the infinite horizon cost of these trajectories. The control actions from the lowest-cost trajectory are applied to the system at each time step. We establish that the proposed algorithm is guaranteed to outperform the original \ac{RL} policy plus an error term that depends on the accuracy of the critic and decays with the horizon length of the \ac{MPC} formulation. Moreover, we do not require globally optimal solutions for these guarantees to hold. The approach is demonstrated on an illustrative toy example and an \ac{AD} overtaking scenario.
△ Less
Submitted 6 June, 2024;
originally announced June 2024.
-
Distributionally Robust Model Predictive Control: Closed-loop Guarantees and Scalable Algorithms
Authors:
Robert D. McAllister,
Peyman Mohajerin Esfahani
Abstract:
We establish a collection of closed-loop guarantees and propose a scalable, Newton-type optimization algorithm for distributionally robust model predictive control (DRMPC) applied to linear systems, zero-mean disturbances, convex constraints, and quadratic costs. Via standard assumptions for the terminal cost and constraint, we establish distribtionally robust long-term and stage-wise performance…
▽ More
We establish a collection of closed-loop guarantees and propose a scalable, Newton-type optimization algorithm for distributionally robust model predictive control (DRMPC) applied to linear systems, zero-mean disturbances, convex constraints, and quadratic costs. Via standard assumptions for the terminal cost and constraint, we establish distribtionally robust long-term and stage-wise performance guarantees for the closed-loop system. We further demonstrate that a common choice of the terminal cost, i.e., as the solution to the discrete-algebraic Riccati equation, renders the origin input-to-state stable for the closed-loop system. This choice of the terminal cost also ensures that the exact long-term performance of the closed-loop system is independent of the choice of ambiguity set the for DRMPC formulation. Thus, we establish conditions under which DRMPC does not provide a long-term performance benefit relative to stochastic MPC (SMPC). To solve the proposed DRMPC optimization problem, we propose a Newton-type algorithm that empirically achieves superlinear convergence by solving a quadratic program at each iteration and guarantees the feasibility of each iterate. We demonstrate the implications of the closed-loop guarantees and the scalability of the proposed algorithm via two examples.
△ Less
Submitted 22 September, 2023;
originally announced September 2023.
-
Nerfstudio: A Modular Framework for Neural Radiance Field Development
Authors:
Matthew Tancik,
Ethan Weber,
Evonne Ng,
Ruilong Li,
Brent Yi,
Justin Kerr,
Terrance Wang,
Alexander Kristoffersen,
Jake Austin,
Kamyar Salahi,
Abhik Ahuja,
David McAllister,
Angjoo Kanazawa
Abstract:
Neural Radiance Fields (NeRF) are a rapidly growing area of research with wide-ranging applications in computer vision, graphics, robotics, and more. In order to streamline the development and deployment of NeRF research, we propose a modular PyTorch framework, Nerfstudio. Our framework includes plug-and-play components for implementing NeRF-based methods, which make it easy for researchers and pr…
▽ More
Neural Radiance Fields (NeRF) are a rapidly growing area of research with wide-ranging applications in computer vision, graphics, robotics, and more. In order to streamline the development and deployment of NeRF research, we propose a modular PyTorch framework, Nerfstudio. Our framework includes plug-and-play components for implementing NeRF-based methods, which make it easy for researchers and practitioners to incorporate NeRF into their projects. Additionally, the modular design enables support for extensive real-time visualization tools, streamlined pipelines for importing captured in-the-wild data, and tools for exporting to video, point cloud and mesh representations. The modularity of Nerfstudio enables the development of Nerfacto, our method that combines components from recent papers to achieve a balance between speed and quality, while also remaining flexible to future modifications. To promote community-driven development, all associated code and data are made publicly available with open-source licensing at https://nerf.studio.
△ Less
Submitted 16 October, 2023; v1 submitted 8 February, 2023;
originally announced February 2023.
-
Ring Deconvolution Microscopy: An Exact Solution for Spatially-Varying Aberration Correction
Authors:
Amit Kohli,
Anastasios N. Angelopoulos,
David McAllister,
Esther Whang,
Sixian You,
Kyrollos Yanny,
Federico M. Gasparoli,
Laura Waller
Abstract:
The most ubiquitous form of computational aberration correction for microscopy is deconvolution. However, deconvolution relies on the assumption that the point spread function is the same across the entire field-of-view. It is well recognized that this assumption is often inadequate, but space-variant deblurring techniques generally require impractical amounts of calibration and computation. We pr…
▽ More
The most ubiquitous form of computational aberration correction for microscopy is deconvolution. However, deconvolution relies on the assumption that the point spread function is the same across the entire field-of-view. It is well recognized that this assumption is often inadequate, but space-variant deblurring techniques generally require impractical amounts of calibration and computation. We present a new imaging pipeline, ring deconvolution microscopy (RDM), that leverages the rotational symmetry of most optical systems to provide simple and fast spatially-varying aberration correction. We formally derive theory and algorithms for exact image recovery and additionally propose a neural network based on Seidel coefficients as a fast alternative. We showcase significant enhancements both visually and quantitatively compared to standard deconvolution and existing spatially-varying deconvolution across a diverse range of microscope modalities, including miniature microscopy, multicolor fluorescence microscopy, and point-scanning multimode fiber micro-endoscopy. Our approach enables near-isotropic, subcellular resolution in each of these applications.
△ Less
Submitted 14 March, 2024; v1 submitted 17 June, 2022;
originally announced June 2022.
-
Shearing Mechanisms of Co-Precipitates in IN718
Authors:
Christopher H. Zenk,
Longsheng Feng,
Donald McAllister,
Yunzhi Wang,
Michael J. Mills
Abstract:
The Ni-base superalloy 718 is the most widely used material for turbomachinery in the aerospace industry and land-based turbines. Although the relationship between processing and the resulting properties is well known, an understanding of the specific deformation mechanisms activated across its application temperature range is required to create more mechanistically accurate property models. Direc…
▽ More
The Ni-base superalloy 718 is the most widely used material for turbomachinery in the aerospace industry and land-based turbines. Although the relationship between processing and the resulting properties is well known, an understanding of the specific deformation mechanisms activated across its application temperature range is required to create more mechanistically accurate property models. Direct atomic-scale imaging observations with high angle annular dark-field scanning transmission electron microscopy, complemented by phase-field modeling informed by generalized stacking fault surface calculations using density functional theory, were employed to understand the shear process of $γ''$ and $γ'/γ''$ co-precipitates after 1 \% macroscopic strain at lower temperature (ambient and $427 °C$). Experimentally, intrinsic stacking faults were observed in the $γ''$, whereas the $γ'$ was found to exhibit anti-phase boundaries or superlattice intrinsic stacking faults. Additionally, the atomically flat $γ'/γ''$ interfaces in the co-precipitates were found to exhibit offsets after shearing, which can be used as tracers for the deformation events. Phase-field modeling shows that the develo** fault-structure is dependent on the direction of the Burgers vector of the $a/2 \langle110\rangle$ matrix dislocation (or dislocation group) due to the lower crystal symmetry of the ${γ''}$ phase. The interplay between $γ'$ and $γ''$ phases results in unique deformation pathways of the co-precipitate and increases the shear resistance. Consistent with the experimental observations, the simulation results indicate that complex shearing mechanisms are active in the low-temperature deformation regime and that multiple $a/2 \langle110\rangle$ dislocations of non-parallel Burgers vectors may be active on the same slip plane.
△ Less
Submitted 29 July, 2021;
originally announced July 2021.
-
Cryogenic Memory Architecture Integrating Spin Hall Effect based Magnetic Memory and Superconductive Cryotron Devices
Authors:
Minh-Hai Nguyen,
Guilhem J. Ribeill,
Martin Gustafsson,
Shengjie Shi,
Sriharsha V. Aradhya,
Andrew P. Wagner,
Leonardo M. Ranzani,
Lijun Zhu,
Reza Baghdadi3 Brenden Butters,
Emily Toomey,
Marco Colangelo,
Patrick A. Truitt,
Amir Jafari-Salim,
David McAllister,
Daniel Yohannes,
Sean R. Cheng,
Rich Lazarus,
Oleg Mukhanov,
Karl K. Berggren,
Robert A. Buhrman,
Graham E. Rowlands,
Thomas A. Ohki
Abstract:
One of the most challenging obstacles to realizing exascale computing is minimizing the energy consumption of L2 cache, main memory, and interconnects to that memory. For promising cryogenic computing schemes utilizing Josephson junction superconducting logic, this obstacle is exacerbated by the cryogenic system requirements that expose the technology's lack of high-density, high-speed and power-e…
▽ More
One of the most challenging obstacles to realizing exascale computing is minimizing the energy consumption of L2 cache, main memory, and interconnects to that memory. For promising cryogenic computing schemes utilizing Josephson junction superconducting logic, this obstacle is exacerbated by the cryogenic system requirements that expose the technology's lack of high-density, high-speed and power-efficient memory. Here we demonstrate an array of cryogenic memory cells consisting of a non-volatile three-terminal magnetic tunnel junction element driven by the spin Hall effect, combined with a superconducting heater-cryotron bit-select element. The write energy of these memory elements is roughly 8 pJ with a bit-select element, designed to achieve a minimum overhead power consumption of about 30%. Individual magnetic memory cells measured at 4 K show reliable switching with write error rates below $10^{-6}$, and a 4x4 array can be fully addressed with bit select error rates of $10^{-6}$. This demonstration is a first step towards a full cryogenic memory architecture targeting energy and performance specifications appropriate for applications in superconducting high performance and quantum computing control systems, which require significant memory resources operating at 4 K.
△ Less
Submitted 1 July, 2019;
originally announced July 2019.
