-
LogoMotion: Visually Grounded Code Generation for Content-Aware Animation
Authors:
Vivian Liu,
Rubaiat Habib Kazi,
Li-Yi Wei,
Matthew Fisher,
Timothy Langlois,
Seth Walker,
Lydia Chilton
Abstract:
Animated logos are a compelling and ubiquitous way individuals and brands represent themselves online. Manually authoring these logos can require significant artistic skill and effort. To help novice designers animate logos, design tools currently offer templates and animation presets. However, these solutions can be limited in their expressive range. Large language models have the potential to he…
▽ More
Animated logos are a compelling and ubiquitous way individuals and brands represent themselves online. Manually authoring these logos can require significant artistic skill and effort. To help novice designers animate logos, design tools currently offer templates and animation presets. However, these solutions can be limited in their expressive range. Large language models have the potential to help novice designers create animated logos by generating animation code that is tailored to their content. In this paper, we introduce LogoMotion, an LLM-based system that takes in a layered document and generates animated logos through visually-grounded program synthesis. We introduce techniques to create an HTML representation of a canvas, identify primary and secondary elements, synthesize animation code, and visually debug animation errors. When compared with an industry standard tool, we find that LogoMotion produces animations that are more content-aware and are on par in terms of quality. We conclude with a discussion of the implications of LLM-generated animation for motion design.
△ Less
Submitted 11 May, 2024;
originally announced May 2024.
-
Convergent Incremental Potential Contact
Authors:
Minchen Li,
Zachary Ferguson,
Teseo Schneider,
Timothy Langlois,
Denis Zorin,
Daniele Panozzo,
Chenfanfu Jiang,
Danny M. Kaufman
Abstract:
Recent advances in the simulation of frictionally contacting elastodynamics with the Incremental Potential Contact (IPC) model have enabled inversion and intersection-free simulation via the application of mollified barriers, filtered line-search, and optimization-based solvers for time integration. In its current formulation the IPC model is constructed via a discrete constraint model, replacing…
▽ More
Recent advances in the simulation of frictionally contacting elastodynamics with the Incremental Potential Contact (IPC) model have enabled inversion and intersection-free simulation via the application of mollified barriers, filtered line-search, and optimization-based solvers for time integration. In its current formulation the IPC model is constructed via a discrete constraint model, replacing non-interpenetration constraints with barrier potentials on an already spatially discretized domain. However, while effective, this purely discrete formulation prohibits convergence under refinement. To enable a convergent IPC model we reformulate IPC potentials in the continuous setting and provide a first, convergent discretization thereof. We demonstrate and analyze the convergence behavior of this new model and discretization on a range of elastostatic and dynamic contact problems, and evaluate its accuracy on both analytical benchmarks and application-driven examples.
△ Less
Submitted 29 July, 2023;
originally announced July 2023.
-
Analyzing Diffusion as Serial Reproduction
Authors:
Raja Marjieh,
Ilia Sucholutsky,
Thomas A. Langlois,
Nori Jacoby,
Thomas L. Griffiths
Abstract:
Diffusion models are a class of generative models that learn to synthesize samples by inverting a diffusion process that gradually maps data into noise. While these models have enjoyed great success recently, a full theoretical understanding of their observed properties is still lacking, in particular, their weak sensitivity to the choice of noise family and the role of adequate scheduling of nois…
▽ More
Diffusion models are a class of generative models that learn to synthesize samples by inverting a diffusion process that gradually maps data into noise. While these models have enjoyed great success recently, a full theoretical understanding of their observed properties is still lacking, in particular, their weak sensitivity to the choice of noise family and the role of adequate scheduling of noise levels for good synthesis. By identifying a correspondence between diffusion models and a well-known paradigm in cognitive science known as serial reproduction, whereby human agents iteratively observe and reproduce stimuli from memory, we show how the aforementioned properties of diffusion models can be explained as a natural consequence of this correspondence. We then complement our theoretical analysis with simulations that exhibit these key features. Our work highlights how classic paradigms in cognitive science can shed light on state-of-the-art machine learning problems.
△ Less
Submitted 29 September, 2022;
originally announced September 2022.
-
Passive Attention in Artificial Neural Networks Predicts Human Visual Selectivity
Authors:
Thomas A. Langlois,
H. Charles Zhao,
Erin Grant,
Ishita Dasgupta,
Thomas L. Griffiths,
Nori Jacoby
Abstract:
Developments in machine learning interpretability techniques over the past decade have provided new tools to observe the image regions that are most informative for classification and localization in artificial neural networks (ANNs). Are the same regions similarly informative to human observers? Using data from 79 new experiments and 7,810 participants, we show that passive attention techniques r…
▽ More
Developments in machine learning interpretability techniques over the past decade have provided new tools to observe the image regions that are most informative for classification and localization in artificial neural networks (ANNs). Are the same regions similarly informative to human observers? Using data from 79 new experiments and 7,810 participants, we show that passive attention techniques reveal a significant overlap with human visual selectivity estimates derived from 6 distinct behavioral tasks including visual discrimination, spatial localization, recognizability, free-viewing, cued-object search, and saliency search fixations. We find that input visualizations derived from relatively simple ANN architectures probed using guided backpropagation methods are the best predictors of a shared component in the joint variability of the human measures. We validate these correlational results with causal manipulations using recognition experiments. We show that images masked with ANN attention maps were easier for humans to classify than control masks in a speeded recognition experiment. Similarly, we find that recognition performance in the same ANN models was likewise influenced by masking input images using human visual selectivity maps. This work contributes a new approach to evaluating the biological and psychological validity of leading ANNs as models of human vision: by examining their similarities and differences in terms of their visual selectivity to the information contained in images.
△ Less
Submitted 31 October, 2021; v1 submitted 14 July, 2021;
originally announced July 2021.
-
Scene-Aware Audio Rendering via Deep Acoustic Analysis
Authors:
Zhenyu Tang,
Nicholas J. Bryan,
Dingzeyu Li,
Timothy R. Langlois,
Dinesh Manocha
Abstract:
We present a new method to capture the acoustic characteristics of real-world rooms using commodity devices, and use the captured characteristics to generate similar sounding sources with virtual models. Given the captured audio and an approximate geometric model of a real-world room, we present a novel learning-based method to estimate its acoustic material properties. Our approach is based on de…
▽ More
We present a new method to capture the acoustic characteristics of real-world rooms using commodity devices, and use the captured characteristics to generate similar sounding sources with virtual models. Given the captured audio and an approximate geometric model of a real-world room, we present a novel learning-based method to estimate its acoustic material properties. Our approach is based on deep neural networks that estimate the reverberation time and equalization of the room from recorded audio. These estimates are used to compute material properties related to room reverberation using a novel material optimization objective. We use the estimated acoustic material characteristics for audio rendering using interactive geometric sound propagation and highlight the performance on many real-world scenarios. We also perform a user study to evaluate the perceptual similarity between the recorded sounds and our rendered audio.
△ Less
Submitted 9 February, 2020; v1 submitted 14 November, 2019;
originally announced November 2019.
-
Designing Volumetric Truss Structures
Authors:
Rahul Arora,
Alec Jacobson,
Timothy R. Langlois,
Yijiang Huang,
Caitlin Mueller,
Wojciech Matusik,
Ariel Shamir,
Karan Singh,
David I. W. Levin
Abstract:
We present the first algorithm for designing volumetric Michell Trusses. Our method uses a parametrization approach to generate trusses made of structural elements aligned with the primary direction of an object's stress field. Such trusses exhibit high strength-to-weight ratios. We demonstrate the structural robustness of our designs via a posteriori physical simulation. We believe our algorithm…
▽ More
We present the first algorithm for designing volumetric Michell Trusses. Our method uses a parametrization approach to generate trusses made of structural elements aligned with the primary direction of an object's stress field. Such trusses exhibit high strength-to-weight ratios. We demonstrate the structural robustness of our designs via a posteriori physical simulation. We believe our algorithm serves as an important complement to existing structural optimization tools and as a novel standalone design tool itself.
△ Less
Submitted 28 October, 2018; v1 submitted 1 October, 2018;
originally announced October 2018.
-
Self-Supervised Generation of Spatial Audio for 360 Video
Authors:
Pedro Morgado,
Nuno Vasconcelos,
Timothy Langlois,
Oliver Wang
Abstract:
We introduce an approach to convert mono audio recorded by a 360 video camera into spatial audio, a representation of the distribution of sound over the full viewing sphere. Spatial audio is an important component of immersive 360 video viewing, but spatial audio microphones are still rare in current 360 video production. Our system consists of end-to-end trainable neural networks that separate in…
▽ More
We introduce an approach to convert mono audio recorded by a 360 video camera into spatial audio, a representation of the distribution of sound over the full viewing sphere. Spatial audio is an important component of immersive 360 video viewing, but spatial audio microphones are still rare in current 360 video production. Our system consists of end-to-end trainable neural networks that separate individual sound sources and localize them on the viewing sphere, conditioned on multi-modal analysis of audio and 360 video frames. We introduce several datasets, including one filmed ourselves, and one collected in-the-wild from YouTube, consisting of 360 videos uploaded with spatial audio. During training, ground-truth spatial audio serves as self-supervision and a mixed down mono track forms the input to our network. Using our approach, we show that it is possible to infer the spatial location of sound sources based only on 360 video and a mono audio track.
△ Less
Submitted 7 September, 2018;
originally announced September 2018.
-
Scene-Aware Audio for 360\textdegree{} Videos
Authors:
Dingzeyu Li,
Timothy R. Langlois,
Changxi Zheng
Abstract:
Although 360\textdegree{} cameras ease the capture of panoramic footage, it remains challenging to add realistic 360\textdegree{} audio that blends into the captured scene and is synchronized with the camera motion. We present a method for adding scene-aware spatial audio to 360\textdegree{} videos in typical indoor scenes, using only a conventional mono-channel microphone and a speaker. We observ…
▽ More
Although 360\textdegree{} cameras ease the capture of panoramic footage, it remains challenging to add realistic 360\textdegree{} audio that blends into the captured scene and is synchronized with the camera motion. We present a method for adding scene-aware spatial audio to 360\textdegree{} videos in typical indoor scenes, using only a conventional mono-channel microphone and a speaker. We observe that the late reverberation of a room's impulse response is usually diffuse spatially and directionally. Exploiting this fact, we propose a method that synthesizes the directional impulse response between any source and listening locations by combining a synthesized early reverberation part and a measured late reverberation tail. The early reverberation is simulated using a geometric acoustic simulation and then enhanced using a frequency modulation method to capture room resonances. The late reverberation is extracted from a recorded impulse response, with a carefully chosen time duration that separates out the late reverberation from the early reverberation. In our validations, we show that our synthesized spatial audio matches closely with recordings using ambisonic microphones. Lastly, we demonstrate the strength of our method in several applications.
△ Less
Submitted 12 May, 2018;
originally announced May 2018.
-
On Evaluation Validity in Music Autotagging
Authors:
Fabien Gouyon,
Bob L. Sturm,
Joao Lobato Oliveira,
Nuno Hespanhol,
Thibault Langlois
Abstract:
Music autotagging, an established problem in Music Information Retrieval, aims to alleviate the human cost required to manually annotate collections of recorded music with textual labels by automating the process. Many autotagging systems have been proposed and evaluated by procedures and datasets that are now standard (used in MIREX, for instance). Very little work, however, has been dedicated to…
▽ More
Music autotagging, an established problem in Music Information Retrieval, aims to alleviate the human cost required to manually annotate collections of recorded music with textual labels by automating the process. Many autotagging systems have been proposed and evaluated by procedures and datasets that are now standard (used in MIREX, for instance). Very little work, however, has been dedicated to determine what these evaluations really mean about an autotagging system, or the comparison of two systems, for the problem of annotating music in the real world. In this article, we are concerned with explaining the figure of merit of an autotagging system evaluated with a standard approach. Specifically, does the figure of merit, or a comparison of figures of merit, warrant a conclusion about how well autotagging systems have learned to describe music with a specific vocabulary? The main contributions of this paper are a formalization of the notion of validity in autotagging evaluation, and a method to test it in general. We demonstrate the practical use of our method in experiments with three specific state-of-the-art autotagging systems --all of which are reproducible using the linked code and data. Our experiments show for these specific systems in a simple and objective two-class task that the standard evaluation approach does not provide valid indicators of their performance.
△ Less
Submitted 30 September, 2014;
originally announced October 2014.
