-
Inverse Neural Rendering for Explainable Multi-Object Tracking
Authors:
Julian Ost,
Tanushree Banerjee,
Mario Bijelic,
Felix Heide
Abstract:
Today, most methods for image understanding tasks rely on feed-forward neural networks. While this approach has allowed for empirical accuracy, efficiency, and task adaptation via fine-tuning, it also comes with fundamental disadvantages. Existing networks often struggle to generalize across different datasets, even on the same task. By design, these networks ultimately reason about high-dimension…
▽ More
Today, most methods for image understanding tasks rely on feed-forward neural networks. While this approach has allowed for empirical accuracy, efficiency, and task adaptation via fine-tuning, it also comes with fundamental disadvantages. Existing networks often struggle to generalize across different datasets, even on the same task. By design, these networks ultimately reason about high-dimensional scene features, which are challenging to analyze. This is true especially when attempting to predict 3D information based on 2D images. We propose to recast 3D multi-object tracking from RGB cameras as an \emph{Inverse Rendering (IR)} problem, by optimizing via a differentiable rendering pipeline over the latent space of pre-trained 3D object representations and retrieve the latents that best represent object instances in a given input image. To this end, we optimize an image loss over generative latent spaces that inherently disentangle shape and appearance properties. We investigate not only an alternate take on tracking but our method also enables examining the generated objects, reasoning about failure situations, and resolving ambiguous cases. We validate the generalization and scaling capabilities of our method by learning the generative prior exclusively from synthetic data and assessing camera-based 3D tracking on the nuScenes and Waymo datasets. Both these datasets are completely unseen to our method and do not require fine-tuning. Videos and code are available at https://light.princeton.edu/inverse-rendering-tracking/.
△ Less
Submitted 18 April, 2024;
originally announced April 2024.
-
Machine Learning: Algorithms, Models, and Applications
Authors:
Jaydip Sen,
Sidra Mehtab,
Rajdeep Sen,
Abhishek Dutta,
Pooja Kherwa,
Saheel Ahmed,
Pranay Berry,
Sahil Khurana,
Sonali Singh,
David W. W Cadotte,
David W. Anderson,
Kalum J. Ost,
Racheal S. Akinbo,
Oladunni A. Daramola,
Bongs Lainjo
Abstract:
Recent times are witnessing rapid development in machine learning algorithm systems, especially in reinforcement learning, natural language processing, computer and robot vision, image processing, speech, and emotional processing and understanding. In tune with the increasing importance and relevance of machine learning models, algorithms, and their applications, and with the emergence of more inn…
▽ More
Recent times are witnessing rapid development in machine learning algorithm systems, especially in reinforcement learning, natural language processing, computer and robot vision, image processing, speech, and emotional processing and understanding. In tune with the increasing importance and relevance of machine learning models, algorithms, and their applications, and with the emergence of more innovative uses cases of deep learning and artificial intelligence, the current volume presents a few innovative research works and their applications in real world, such as stock trading, medical and healthcare systems, and software automation. The chapters in the book illustrate how machine learning and deep learning algorithms and models are designed, optimized, and deployed. The volume will be useful for advanced graduate and doctoral students, researchers, faculty members of universities, practicing data scientists and data engineers, professionals, and consultants working on the broad areas of machine learning, deep learning, and artificial intelligence.
△ Less
Submitted 6 January, 2022;
originally announced January 2022.
-
Neural Point Light Fields
Authors:
Julian Ost,
Issam Laradji,
Alejandro Newell,
Yuval Bahat,
Felix Heide
Abstract:
We introduce Neural Point Light Fields that represent scenes implicitly with a light field living on a sparse point cloud. Combining differentiable volume rendering with learned implicit density representations has made it possible to synthesize photo-realistic images for novel views of small scenes. As neural volumetric rendering methods require dense sampling of the underlying functional scene r…
▽ More
We introduce Neural Point Light Fields that represent scenes implicitly with a light field living on a sparse point cloud. Combining differentiable volume rendering with learned implicit density representations has made it possible to synthesize photo-realistic images for novel views of small scenes. As neural volumetric rendering methods require dense sampling of the underlying functional scene representation, at hundreds of samples along a ray cast through the volume, they are fundamentally limited to small scenes with the same objects projected to hundreds of training views. Promoting sparse point clouds to neural implicit light fields allows us to represent large scenes effectively with only a single radiance evaluation per ray. These point light fields are a function of the ray direction, and local point feature neighborhood, allowing us to interpolate the light field conditioned training images without dense object coverage and parallax. We assess the proposed method for novel view synthesis on large driving scenarios, where we synthesize realistic unseen views that existing implicit approaches fail to represent. We validate that Neural Point Light Fields make it possible to predict videos along unseen trajectories previously only feasible to generate by explicitly modeling the scene.
△ Less
Submitted 7 June, 2022; v1 submitted 2 December, 2021;
originally announced December 2021.
-
Spherically symmetric exact vacuum solutions in Einstein-aether theory
Authors:
Jacob Oost,
Shinji Mukohyama,
Anzhong Wang
Abstract:
We study spherically symmetric spacetimes in Einstein-aether theory in three different coordinate systems, the isotropic, Painlevè-Gullstrand, and Schwarzschild coordinates, in which the aether is always comoving, and present both time-dependent and time-independent exact vacuum solutions. In particular, in the isotropic coordinates we find a class of exact static solutions characterized by a sing…
▽ More
We study spherically symmetric spacetimes in Einstein-aether theory in three different coordinate systems, the isotropic, Painlevè-Gullstrand, and Schwarzschild coordinates, in which the aether is always comoving, and present both time-dependent and time-independent exact vacuum solutions. In particular, in the isotropic coordinates we find a class of exact static solutions characterized by a single parameter $c_{14}$ in closed forms, which satisfies all the current observational constraints of the theory, and reduces to the Schwarzschild vacuum black hole solution in the decoupling limit ($c_{14} = 0$). However, as long as $c_{14} \not= 0$, a marginally trapped throat with a finite non-zero radius always exists, and in one side of it the spacetime is asymptotically flat, while in the other side the spacetime becomes singular within a finite proper distance from the throat, although the geometric area is infinitely large at the singularity. Moreover, the singularity is a strong and spacetime curvature singularity, at which both of the Ricci and Kretschmann scalars become infinitely large.
△ Less
Submitted 28 July, 2021; v1 submitted 16 June, 2021;
originally announced June 2021.
-
Neural Scene Graphs for Dynamic Scenes
Authors:
Julian Ost,
Fahim Mannan,
Nils Thuerey,
Julian Knodt,
Felix Heide
Abstract:
Recent implicit neural rendering methods have demonstrated that it is possible to learn accurate view synthesis for complex scenes by predicting their volumetric density and color supervised solely by a set of RGB images. However, existing methods are restricted to learning efficient representations of static scenes that encode all scene objects into a single neural network, and lack the ability t…
▽ More
Recent implicit neural rendering methods have demonstrated that it is possible to learn accurate view synthesis for complex scenes by predicting their volumetric density and color supervised solely by a set of RGB images. However, existing methods are restricted to learning efficient representations of static scenes that encode all scene objects into a single neural network, and lack the ability to represent dynamic scenes and decompositions into individual scene objects. In this work, we present the first neural rendering method that decomposes dynamic scenes into scene graphs. We propose a learned scene graph representation, which encodes object transformation and radiance, to efficiently render novel arrangements and views of the scene. To this end, we learn implicitly encoded scenes, combined with a jointly learned latent representation to describe objects with a single implicit function. We assess the proposed method on synthetic and real automotive data, validating that our approach learns dynamic scenes -- only by observing a video of this scene -- and allows for rendering novel photo-realistic views of novel scene compositions with unseen sets of objects at unseen poses.
△ Less
Submitted 5 March, 2021; v1 submitted 20 November, 2020;
originally announced November 2020.
-
Gravitational plane waves in Einstein-aether theory
Authors:
Jacob Oost,
Madhurima Bhattacharjee,
Anzhong Wang
Abstract:
In this paper, we systematically study spacetimes of gravitational plane waves in Einstein-aether theory. Due to the presence of the timelike aether vector field, now the problem in general becomes overdetermined. In particular, for the linearly polarized plane waves, there are five independent vacuum Einstein-aether field equations for three unknown functions. Therefore, solutions exist only for…
▽ More
In this paper, we systematically study spacetimes of gravitational plane waves in Einstein-aether theory. Due to the presence of the timelike aether vector field, now the problem in general becomes overdetermined. In particular, for the linearly polarized plane waves, there are five independent vacuum Einstein-aether field equations for three unknown functions. Therefore, solutions exist only for particular choices of the four free parameters $c_{i}$'s of the theory. We find that there exist eight cases, in two of which any form of gravitational plane waves can exist, similar to that in general relativity, while in the other six cases, gravitational plane waves exist only in particular forms. Beyond these eight cases, solutions either do not exist or are trivial (simply representing a Minkowski spacetime with a constant or dynamical aether field.).
△ Less
Submitted 23 October, 2018; v1 submitted 3 April, 2018;
originally announced April 2018.
-
Constraints on Einstein-aether theory after GW170817
Authors:
Jacob Oost,
Shinji Mukohyama,
Anzhong Wang
Abstract:
In this paper, we carry out a systematic analysis of the theoretical and observational constraints on the dimensionless coupling constants $c_i$ ($i=1,2,3,4$) of the Einstein-aether theory, taking into account the events GW170817 and GRB 170817A. The combination of these events restricts the deviation of the speed $c_T$ of the spin-2 graviton to the range,…
▽ More
In this paper, we carry out a systematic analysis of the theoretical and observational constraints on the dimensionless coupling constants $c_i$ ($i=1,2,3,4$) of the Einstein-aether theory, taking into account the events GW170817 and GRB 170817A. The combination of these events restricts the deviation of the speed $c_T$ of the spin-2 graviton to the range, $- 3\times 10^{-15} < c_T -1 < 7\times 10^{-16}$, which for the Einstein-aether theory implies $\left|c_{13}\right| \le 10^{-15}$ with $c_{ij} \equiv c_{i} + c_{j}$. The rest of the constraints are divided into two groups: those on the ($c_1, c_{14}$)-plane and those on the ($c_2, c_{14}$)-plane, except the strong-field constraints. The latter depend on the sensitivities $σ_æ$ of neutron stars, which are not known at present in the new ranges of the parameters found in this paper.
△ Less
Submitted 15 June, 2018; v1 submitted 12 February, 2018;
originally announced February 2018.
