-
LipSync3D: Data-Efficient Learning of Personalized 3D Talking Faces from Video using Pose and Lighting Normalization
Authors:
Avisek Lahiri,
Vivek Kwatra,
Christian Frueh,
John Lewis,
Chris Bregler
Abstract:
In this paper, we present a video-based learning framework for animating personalized 3D talking faces from audio. We introduce two training-time data normalizations that significantly improve data sample efficiency. First, we isolate and represent faces in a normalized space that decouples 3D geometry, head pose, and texture. This decomposes the prediction problem into regressions over the 3D fac…
▽ More
In this paper, we present a video-based learning framework for animating personalized 3D talking faces from audio. We introduce two training-time data normalizations that significantly improve data sample efficiency. First, we isolate and represent faces in a normalized space that decouples 3D geometry, head pose, and texture. This decomposes the prediction problem into regressions over the 3D face shape and the corresponding 2D texture atlas. Second, we leverage facial symmetry and approximate albedo constancy of skin to isolate and remove spatio-temporal lighting variations. Together, these normalizations allow simple networks to generate high fidelity lip-sync videos under novel ambient illumination while training with just a single speaker-specific video. Further, to stabilize temporal dynamics, we introduce an auto-regressive approach that conditions the model on its previous visual state. Human ratings and objective metrics demonstrate that our method outperforms contemporary state-of-the-art audio-driven video reenactment benchmarks in terms of realism, lip-sync and visual quality scores. We illustrate several applications enabled by our framework.
△ Less
Submitted 8 June, 2021;
originally announced June 2021.
-
The observing campaign on the deep-space debris WT1190F as a test case for short-warning NEO impacts
Authors:
Marco Micheli,
Alberto Buzzoni,
Detlef Koschny,
Gerhard Drolshagen,
Ettore Perozzi,
Olivier Hainaut,
Stijn Lemmens,
Giuseppe Altavilla,
Italo Foppiani,
Jaime Nomen,
Noelia Sánchez-Ortiz,
Wladimiro Marinello,
Gianpaolo Pizzetti,
Andrea Soffiantini,
Siwei Fan,
Carolin Frueh
Abstract:
On 2015 November 13, the small artificial object designated WT1190F entered the Earth atmosphere above the Indian Ocean offshore Sri Lanka after being discovered as a possible new asteroid only a few weeks earlier. At ESA's SSA-NEO Coordination Centre we took advantage of this opportunity to organize a ground-based observational campaign, using WT1190F as a test case for a possible similar future…
▽ More
On 2015 November 13, the small artificial object designated WT1190F entered the Earth atmosphere above the Indian Ocean offshore Sri Lanka after being discovered as a possible new asteroid only a few weeks earlier. At ESA's SSA-NEO Coordination Centre we took advantage of this opportunity to organize a ground-based observational campaign, using WT1190F as a test case for a possible similar future event involving a natural asteroidal body.
△ Less
Submitted 20 October, 2017;
originally announced October 2017.
-
Coupled Orbit-Attitude Dynamics of High Area-to-Mass Ratio (HAMR) Objects: Influence of Solar Radiation Pressure, Earth's Shadow and the Visibility in Light Curves
Authors:
Carolin Frueh,
Moriba Jah,
Thomas Kelecy
Abstract:
The orbital and attitude dynamics of uncontrolled Earth orbiting objects are perturbed by a variety of sources. In research, emphasis has been put on operational space vehicles. Operational satellites typically have a relatively compact shape, and hence, a low area-to-mass ratio (AMR), and are in most cases actively or passively attitude stabilized. This enables one to treat the orbit and attitude…
▽ More
The orbital and attitude dynamics of uncontrolled Earth orbiting objects are perturbed by a variety of sources. In research, emphasis has been put on operational space vehicles. Operational satellites typically have a relatively compact shape, and hence, a low area-to-mass ratio (AMR), and are in most cases actively or passively attitude stabilized. This enables one to treat the orbit and attitude propagation as decoupled problems, and in many cases the attitude dynamics can be neglected completely. The situation is different for space debris objects, which are in an uncontrolled attitude state. Furthermore, the assumption that a steady-state attitude motion can be averaged over data reduction intervals may no longer be valid. Additionally, a subset of the debris objects have significantly high area-to-mass ratio values, resulting in highly perturbed orbits, e.g. by solar radiation pressure, even if a stable AMR value is assumed. This assumption implies a steady-state attitude such that the average cross-sectional area exposed to the sun is close to constant. Time-varying solar radiation pressure accelerations due to attitude variations will result in un-modeled errors in the state propagation. This work investigates the evolution of the coupled attitude and orbit motion of HAMR objects. Standardized pieces of multilayer insulation are simulated in near geosynchronous orbits. It is assumed that the objects are rigid bodies and are in uncontrolled attitude states. The integrated effects of the Earth gravitational field and solar radiation pressure on the attitude motion are investigated. The light curves that represent the observed brightness variations over time in a specific viewing direction are extracted. A sensor model is utilized to generate light curves with visibility constraints and magnitude uncertainties as observed by a standard ground based telescope.
△ Less
Submitted 30 November, 2013;
originally announced December 2013.
-
Object Image Linking of Earth Orbiting Objects in the Presence of Cosmics
Authors:
Carolin Früh,
Thomas Schildknecht
Abstract:
In survey series of unknown Earth orbiting objects, no a priori orbital elements are available. In surveys of wide field telescopes possibly many nonresolved object images are present on the single frames of the series. Reliable methods have to be found to associate the object images stemming from the same object with each other, so-called linking. The presence of cosmic ray events, so-called Cosm…
▽ More
In survey series of unknown Earth orbiting objects, no a priori orbital elements are available. In surveys of wide field telescopes possibly many nonresolved object images are present on the single frames of the series. Reliable methods have to be found to associate the object images stemming from the same object with each other, so-called linking. The presence of cosmic ray events, so-called Cosmics, complicates reliable linking of non-resolved images. The tracklets of object images allow to extract exact positions for a first orbit determination. A two step method is used and tested on observation frames of space debris surveys of the ESA Space Debris Telescope, located on Tenerife, Spain: In a first step a cosmic filter is applied in the single observation frames. Four different filter approaches are compared and tested in performance. In a second step, the detected object images are linked on observation series based on the assumption of a linear accelerated movement of the objects over the frame during the series, which is updated with every object image, that could be successfully linked.
△ Less
Submitted 10 November, 2011;
originally announced November 2011.
-
Variation of Area-to-Mass-Ratio of HAMR Space Debris Objects
Authors:
Carolin Früh,
Thomas Schildknecht
Abstract:
An unexpected space debris population has been detected in 2004 Schildknecht et al. (2003, 2004) with the unique properties of a very high area-to-mass ratio (HAMR) Schildknecht et al. (2005a). Ever since it has been tried to investigate the dynamical properties of those objects further. The orbits of those objects are heavily perturbed by the effect of direct radiation pressure. Unknown attitude…
▽ More
An unexpected space debris population has been detected in 2004 Schildknecht et al. (2003, 2004) with the unique properties of a very high area-to-mass ratio (HAMR) Schildknecht et al. (2005a). Ever since it has been tried to investigate the dynamical properties of those objects further. The orbits of those objects are heavily perturbed by the effect of direct radiation pressure. Unknown attitude motion complicates orbit prediction. The area-to-mass ratio of the objects seems to be not stable over time. Only sparse optical data is available for those objects in drift orbits. The current work uses optical observations of five HAMR objects, observed over several years and investigates the variation of their area-to-mass ratio and orbital parameters. A normalized orbit determination setup has been established and validated with two low and two of the high ratio objects, to ensure, that comparable orbits over longer time spans are determined even with sparse optical data.
△ Less
Submitted 18 October, 2011;
originally announced October 2011.