Showing 1–13 of 13 results for author: Lewis, J P

TrailBlazer: Trajectory Control for Diffusion-Based Video Generation

Authors: Wan-Duo Kurt Ma, J. P. Lewis, W. Bastiaan Kleijn

Abstract: Within recent approaches to text-to-video (T2V) generation, achieving controllability in the synthesized video is often a challenge. Typically, this issue is addressed by providing low-level per-frame guidance in the form of edge maps, depth maps, or an existing video to be altered. However, the process of obtaining such guidance can be labor-intensive. This paper focuses on enhancing controllabil… ▽ More Within recent approaches to text-to-video (T2V) generation, achieving controllability in the synthesized video is often a challenge. Typically, this issue is addressed by providing low-level per-frame guidance in the form of edge maps, depth maps, or an existing video to be altered. However, the process of obtaining such guidance can be labor-intensive. This paper focuses on enhancing controllability in video synthesis by employing straightforward bounding boxes to guide the subject in various ways, all without the need for neural network training, finetuning, optimization at inference time, or the use of pre-existing videos. Our algorithm, TrailBlazer, is constructed upon a pre-trained (T2V) model, and easy to implement. The subject is directed by a bounding box through the proposed spatial and temporal attention map editing. Moreover, we introduce the concept of keyframing, allowing the subject trajectory and overall appearance to be guided by both a moving bounding box and corresponding prompts, without the need to provide a detailed mask. The method is efficient, with negligible additional computation relative to the underlying pre-trained model. Despite the simplicity of the bounding box guidance, the resulting motion is surprisingly natural, with emergent effects including perspective and movement toward the virtual camera as the box size increases. △ Less

Submitted 8 April, 2024; v1 submitted 31 December, 2023; originally announced January 2024.

Comments: 14 pages, 18 figures, Project Page: https://hohonu-vicml.github.io/Trailblazer.Page/

FDLS: A Deep Learning Approach to Production Quality, Controllable, and Retargetable Facial Performances

Authors: Wan-Duo Kurt Ma, Muhammad Ghifary, J. P. Lewis, Byungkuk Choi, Haekwang Eom

Abstract: Visual effects commonly requires both the creation of realistic synthetic humans as well as retargeting actors' performances to humanoid characters such as aliens and monsters. Achieving the expressive performances demanded in entertainment requires manipulating complex models with hundreds of parameters. Full creative control requires the freedom to make edits at any stage of the production, whic… ▽ More Visual effects commonly requires both the creation of realistic synthetic humans as well as retargeting actors' performances to humanoid characters such as aliens and monsters. Achieving the expressive performances demanded in entertainment requires manipulating complex models with hundreds of parameters. Full creative control requires the freedom to make edits at any stage of the production, which prohibits the use of a fully automatic ``black box'' solution with uninterpretable parameters. On the other hand, producing realistic animation with these sophisticated models is difficult and laborious. This paper describes FDLS (Facial Deep Learning Solver), which is Weta Digital's solution to these challenges. FDLS adopts a coarse-to-fine and human-in-the-loop strategy, allowing a solved performance to be verified and edited at several stages in the solving process. To train FDLS, we first transform the raw motion-captured data into robust graph features. Secondly, based on the observation that the artists typically finalize the jaw pass animation before proceeding to finer detail, we solve for the jaw motion first and predict fine expressions with region-based networks conditioned on the jaw position. Finally, artists can optionally invoke a non-linear finetuning process on top of the FDLS solution to follow the motion-captured virtual markers as closely as possible. FDLS supports editing if needed to improve the results of the deep learning solution and it can handle small daily changes in the actor's face shape. FDLS permits reliable and production-quality performance solving with minimal training and little or no manual effort in many cases, while also allowing the solve to be guided and edited in unusual and difficult cases. The system has been under development for several years and has been used in major movies. △ Less

Submitted 26 September, 2023; originally announced September 2023.

Comments: DigiPro '22: The Digital Production Symposium

Exact Diffusion Inversion via Bi-directional Integration Approximation

Authors: Guoqiang Zhang, J. P. Lewis, W. Bastiaan Kleijn

Abstract: Recently, various methods have been proposed to address the inconsistency issue of DDIM inversion to enable image editing, such as EDICT [36] and Null-text inversion [22]. However, the above methods introduce considerable computational overhead. In this paper, we propose a new technique, named \emph{bi-directional integration approximation} (BDIA), to perform exact diffusion inversion with neglibl… ▽ More Recently, various methods have been proposed to address the inconsistency issue of DDIM inversion to enable image editing, such as EDICT [36] and Null-text inversion [22]. However, the above methods introduce considerable computational overhead. In this paper, we propose a new technique, named \emph{bi-directional integration approximation} (BDIA), to perform exact diffusion inversion with neglible computational overhead. Suppose we would like to estimate the next diffusion state $\boldsymbol{z}_{i-1}$ at timestep $t_i$ with the historical information $(i,\boldsymbol{z}_i)$ and $(i+1,\boldsymbol{z}_{i+1})$. We first obtain the estimated Gaussian noise $\hat{\boldsymbolε}(\boldsymbol{z}_i,i)$, and then apply the DDIM update procedure twice for approximating the ODE integration over the next time-slot $[t_i, t_{i-1}]$ in the forward manner and the previous time-slot $[t_i, t_{t+1}]$ in the backward manner. The DDIM step for the previous time-slot is used to refine the integration approximation made earlier when computing $\boldsymbol{z}_i$. A nice property of BDIA-DDIM is that the update expression for $\boldsymbol{z}_{i-1}$ is a linear combination of $(\boldsymbol{z}_{i+1}, \boldsymbol{z}_i, \hat{\boldsymbolε}(\boldsymbol{z}_i,i))$. This allows for exact backward computation of $\boldsymbol{z}_{i+1}$ given $(\boldsymbol{z}_i, \boldsymbol{z}_{i-1})$, thus leading to exact diffusion inversion. It is demonstrated with experiments that (round-trip) BDIA-DDIM is particularly effective for image editing. Our experiments further show that BDIA-DDIM produces markedly better image sampling qualities than DDIM for text-to-image generation. BDIA can also be applied to improve the performance of other ODE solvers in addition to DDIM. In our work, it is found that applying BDIA to the EDM sampling procedure produces consistently better performance over four pre-trained models. △ Less

Submitted 26 November, 2023; v1 submitted 10 July, 2023; originally announced July 2023.

Comments: arXiv admin note: text overlap with arXiv:2304.11328. Our code is available at https://github.com/guoqiang-zhang-x/BDIA

Training-Free Neural Matte Extraction for Visual Effects

Authors: Sharif Elcott, J. P. Lewis, Nori Kanazawa, Christoph Bregler

Abstract: Alpha matting is widely used in video conferencing as well as in movies, television, and social media sites. Deep learning approaches to the matte extraction problem are well suited to video conferencing due to the consistent subject matter (front-facing humans), however training-based approaches are somewhat pointless for entertainment videos where varied subjects (spaceships, monsters, etc.) may… ▽ More Alpha matting is widely used in video conferencing as well as in movies, television, and social media sites. Deep learning approaches to the matte extraction problem are well suited to video conferencing due to the consistent subject matter (front-facing humans), however training-based approaches are somewhat pointless for entertainment videos where varied subjects (spaceships, monsters, etc.) may appear only a few times in a single movie -- if a method of creating ground truth for training exists, just use that method to produce the desired mattes. We introduce a training-free high quality neural matte extraction approach that specifically targets the assumptions of visual effects production. Our approach is based on the deep image prior, which optimizes a deep neural network to fit a single image, thereby providing a deep encoding of the particular image. We make use of the representations in the penultimate layer to interpolate coarse and incomplete "trimap" constraints. Videos processed with this approach are temporally consistent. The algorithm is both very simple and surprisingly effective. △ Less

Submitted 29 June, 2023; originally announced June 2023.

ACM Class: I.4.6

Journal ref: SIGGRAPH Asia 2022 Technical Communications

Directed Diffusion: Direct Control of Object Placement through Attention Guidance

Authors: Wan-Duo Kurt Ma, J. P. Lewis, Avisek Lahiri, Thomas Leung, W. Bastiaan Kleijn

Abstract: Text-guided diffusion models such as DALLE-2, Imagen, eDiff-I, and Stable Diffusion are able to generate an effectively endless variety of images given only a short text prompt describing the desired image content. In many cases the images are of very high quality. However, these models often struggle to compose scenes containing several key objects such as characters in specified positional relat… ▽ More Text-guided diffusion models such as DALLE-2, Imagen, eDiff-I, and Stable Diffusion are able to generate an effectively endless variety of images given only a short text prompt describing the desired image content. In many cases the images are of very high quality. However, these models often struggle to compose scenes containing several key objects such as characters in specified positional relationships. The missing capability to ``direct'' the placement of characters and objects both within and across images is crucial in storytelling, as recognized in the literature on film and animation theory. In this work, we take a particularly straightforward approach to providing the needed direction. Drawing on the observation that the cross-attention maps for prompt words reflect the spatial layout of objects denoted by those words, we introduce an optimization objective that produces ``activation'' at desired positions in these cross-attention maps. The resulting approach is a step toward generalizing the applicability of text-guided diffusion models beyond single images to collections of related images, as in storybooks. Directed Diffusion provides easy high-level positional control over multiple objects, while making use of an existing pre-trained model and maintaining a coherent blend between the positioned objects and the background. Moreover, it requires only a few lines to implement. △ Less

Submitted 26 September, 2023; v1 submitted 25 February, 2023; originally announced February 2023.

Comments: Our project page: https://hohonu-vicml.github.io/DirectedDiffusion.Page

Calculated Effects of Vacancy and Ti-do** in 2D Janus MoSSe for Photocatalysis

Authors: Yi-Ming Zhao, Pengju Ren, Xing-Yu Ma, James P. Lewis, Qing-Bo Yan, Gang Su

Abstract: Two-dimensional (2D) Janus transitional metal dichalcogenides (TMDCs) have great potential for photocatalytic water splitting due to their novel properties induced by the unique out-of-plane asymmetric structures. Here, we systematically investigate the geometric, electronic and optical properties of 2D Janus MoSSe with titanium do** and vacancies to explore their synergistic effects on photocat… ▽ More Two-dimensional (2D) Janus transitional metal dichalcogenides (TMDCs) have great potential for photocatalytic water splitting due to their novel properties induced by the unique out-of-plane asymmetric structures. Here, we systematically investigate the geometric, electronic and optical properties of 2D Janus MoSSe with titanium do** and vacancies to explore their synergistic effects on photocatalytic activity. We find that there are effective attractions between the substituted or adsorbed Ti atoms and S/Se vacancies. The Ti adatoms dramatically extend the light absorption range to infrared region. The S/Se vacancies coexisting with Ti adatoms will modulate the transition of photo-excited electrons, thereby enhancing the sunlight absorption. The Ti adatoms either existing alone or coexisting with vacancies introduce smaller lattice distortion compared to substituted Ti atoms and these Ti adatoms induce smaller effective mass of charge carriers. The configuration of S vacancy coexisting with Ti adatoms on Se-surface exhibits the most significant synergistic effects and best overall photocatalytic performance. Our work reveals the mechanism and effects induced by do** and vacancies coexisting in 2D Janus TMDCs, also propose a new practical strategy to improve the performance of 2D photocatalysts. △ Less

Submitted 10 March, 2021; originally announced March 2021.

Journal ref: The Journal of Physical Chemistry C 2021

Accelerated Discovery of Two-Dimensional Optoelectronic Octahedral Oxyhalides via High-Throughput Ab Initio Calculations and Machine Learning

Authors: Xing-Yu Ma, James P. Lewis, Qing-Bo Yan, Gang Su

Abstract: Traditional trial-and-error methods are obstacles for large-scale searching of new optoelectronic materials. Here, we introduce a method combining high-throughput ab initio calculations and machine-learning approaches to predict two-dimensional octahedral oxyhalides with improved optoelectronic properties. We develop an effective machine-learning model based on an expansive dataset generated from… ▽ More Traditional trial-and-error methods are obstacles for large-scale searching of new optoelectronic materials. Here, we introduce a method combining high-throughput ab initio calculations and machine-learning approaches to predict two-dimensional octahedral oxyhalides with improved optoelectronic properties. We develop an effective machine-learning model based on an expansive dataset generated from density functional calculations including the geometric and electronic properties of 300 two-dimensional octahedral oxyhalides. Our model accelerates the screening of potential optoelectronic materials of 5,000 two-dimensional octahedral oxyhalides. The distorted stacked octahedral factors proposed in our model play essential roles in the machine-learning prediction. Several potential two-dimensional optoelectronic octahedral oxyhalides with moderate band gaps, high electron mobilities, and ultrahigh absorbance coefficients are successfully hypothesized. △ Less

Submitted 23 September, 2019; originally announced September 2019.

Comments: The Journal of Physical Chemistry Letters accepted

The HSIC Bottleneck: Deep Learning without Back-Propagation

Authors: Wan-Duo Kurt Ma, J. P. Lewis, W. Bastiaan Kleijn

Abstract: We introduce the HSIC (Hilbert-Schmidt independence criterion) bottleneck for training deep neural networks. The HSIC bottleneck is an alternative to the conventional cross-entropy loss and backpropagation that has a number of distinct advantages. It mitigates exploding and vanishing gradients, resulting in the ability to learn very deep networks without skip connections. There is no requirement f… ▽ More We introduce the HSIC (Hilbert-Schmidt independence criterion) bottleneck for training deep neural networks. The HSIC bottleneck is an alternative to the conventional cross-entropy loss and backpropagation that has a number of distinct advantages. It mitigates exploding and vanishing gradients, resulting in the ability to learn very deep networks without skip connections. There is no requirement for symmetric feedback or update locking. We find that the HSIC bottleneck provides performance on MNIST/FashionMNIST/CIFAR10 classification comparable to backpropagation with a cross-entropy target, even when the system is not encouraged to make the output resemble the classification labels. Appending a single layer trained with SGD (without backpropagation) to reformat the information further improves performance. △ Less

Submitted 5 December, 2019; v1 submitted 5 August, 2019; originally announced August 2019.

Photo-induced reactions from efficient molecular dynamics with electronic transitions using the FIREBALL local-orbital density functional theory formalism

Authors: Vladmír Zobač, James P. Lewis, Enrique Abad, Jesús I. Mendieta-Moreno, Prokop Hapala, Pavel Jelínek, José Ortega

Abstract: The computational simulation of photo-induced processes in large molecular systems is a very challenging problem. Here, we present a detailed description of our implementation of a molecular dynamics with electronic transitions algorithm within the local-orbital density functional theory code Fireball, suitable for the computational study of these problems. Our methodology enables simulating photo… ▽ More The computational simulation of photo-induced processes in large molecular systems is a very challenging problem. Here, we present a detailed description of our implementation of a molecular dynamics with electronic transitions algorithm within the local-orbital density functional theory code Fireball, suitable for the computational study of these problems. Our methodology enables simulating photo-induced reaction mechanisms over hundreds of trajectories; therefore, large statistically significant ensembles can be calculated to accurately represent a reaction profile. As an example of the application of this approach, we report results on the [2+2] cycloaddition of ethylene with maleic anhydride and on the [2+2] photo-induced polymerization reaction of two C60 molecules. We identify different deactivation channels of the initial electron excitation, depending on the time of the electronic transition from LUMO to HOMO, and the character of the HOMO after the transition. △ Less

Submitted 26 August, 2014; originally announced August 2014.

Hidden structure in amorphous solids

Authors: F. Inam, James P. Lewis, D. A. Drabold

Abstract: Recent theoretical studies of amorphous silicon [Y. Pan et al. Phys. Rev. Lett. 100 206403 (2008)] have revealed subtle but significant structural correlations in network topology: the tendency for short (long) bonds to be spatially correlated with other short (long) bonds). These structures were linked to the electronic band tails in the optical gap. In this paper, we further examine these issu… ▽ More Recent theoretical studies of amorphous silicon [Y. Pan et al. Phys. Rev. Lett. 100 206403 (2008)] have revealed subtle but significant structural correlations in network topology: the tendency for short (long) bonds to be spatially correlated with other short (long) bonds). These structures were linked to the electronic band tails in the optical gap. In this paper, we further examine these issues for amorphous silicon, and demonstrate that analogous correlations exist in amorphous SiO2, and in the organic molecule, b-carotene. We conclude with a discussion of the origin of the effects and its possible generality. △ Less

Submitted 6 August, 2009; originally announced August 2009.

Electronic transport and localization in short and long DNA

Authors: H. Wang, R. Marsh, J. P. Lewis, R. A. Roemer

Abstract: The question of whether DNA conducts electric charges is intriguing to physicists and biologists alike. The suggestion that electron transfer/transport in DNA might be biologically important has triggered a series of experimental and theoretical investigations. Here, we review recent theoretical progress by concentrating on quantum-chemical, molecular dynamics-based approaches to short DNA stran… ▽ More The question of whether DNA conducts electric charges is intriguing to physicists and biologists alike. The suggestion that electron transfer/transport in DNA might be biologically important has triggered a series of experimental and theoretical investigations. Here, we review recent theoretical progress by concentrating on quantum-chemical, molecular dynamics-based approaches to short DNA strands and physics-motivated tight-binding transport studies of long or even complete DNA sequences. In both cases, we observe small, but significant differences between specific DNA sequences such as periodic repetitions and aperiodic sequences of AT bases, lambda-DNA, centromeric DNA, promoter sequences as well as random-ATGC DNA. △ Less

Submitted 4 November, 2005; originally announced November 2005.

Comments: 26 pages, 12 .eps figures using the elsevier style files (enclosed); to appear in an Elsevier Review Book on Biopolymers

Report number: CSC-127

Journal ref: Modern Methods for Theoretical Physical Chemistry of Biopolymers (pp. 407-427), 2006

Photoemission study of poly(dA)-poly(dT) DNA : Experimental and theoretical approach to the electronic density of states

Authors: Hiroki Wadati, Kozo Okazaki, Yasuhiro Niimi, Atsushi Fujimori, Hitoshi Tabata, Jared Pikus, James P. Lewis

Abstract: We present results of an ultraviolet photoemission spectroscopy study of artificially synthesized poly(dA)-poly(dT) DNA molecules on $p$-type Si substrates. For comparison, we also present the electronic density of states (DOS) calculated using an \emph{ab initio} tight-binding method based on density-functional theory (DFT). Good agreement was obtained between experiment and theory. The spectra… ▽ More We present results of an ultraviolet photoemission spectroscopy study of artificially synthesized poly(dA)-poly(dT) DNA molecules on $p$-type Si substrates. For comparison, we also present the electronic density of states (DOS) calculated using an \emph{ab initio} tight-binding method based on density-functional theory (DFT). Good agreement was obtained between experiment and theory. The spectra of DNA networks on the Si substrate showed that the Fermi level of the substrate is located in the middle of the band gap of DNA. The spectra of thick ($\sim 70$ nm) DNA films showed a downward shift of $\sim 2$ eV compared to the network samples. △ Less

Submitted 24 September, 2004; originally announced September 2004.

Comments: 4 pages, 4 figures

Journal ref: Appl. Phys. Lett. 86, 023901 (2005), Virtual Journal of Biological Physics Research, January 1, 2005

A new multi-center approach to the exchange-correlation interactions in ab initio tight-binding methods

Authors: P. Jelinek, H. Wang, J. P. Lewis, O. F. Sankey, J. Ortega

Abstract: A new approximate method to calculate exchange-correlation contributions in the framework of first-principles tight-binding molecular dynamics methods has been developed. In the proposed scheme on-site (off-site) exchange-correlation matrix elements are expressed as a one-center (two-center) term plus a {\it correction} due to the rest of the atoms. The one-center (two-center) term is evaluated… ▽ More A new approximate method to calculate exchange-correlation contributions in the framework of first-principles tight-binding molecular dynamics methods has been developed. In the proposed scheme on-site (off-site) exchange-correlation matrix elements are expressed as a one-center (two-center) term plus a {\it correction} due to the rest of the atoms. The one-center (two-center) term is evaluated directly, while the {\it correction} is calculated using a variation of the Sankey-Niklewski \cite{Sankey89} approach generalized for arbitrary atomic-like basis sets. The proposed scheme for exchange-correlation part permits the accurate and computationally efficient calculation of corresponding tight-binding matrices and atomic forces for complex systems. We calculate bulk properties of selected transition (W,Pd), noble (Au) or simple (Al) metals, a semiconductor (Si) and the transition metal oxide Ti$O_2$ with the new method to demonstrate its flexibility and good accuracy. △ Less

Submitted 21 September, 2004; v1 submitted 20 September, 2004; originally announced September 2004.

Comments: 17 pages, 5 figures