-
Self-Calibrating, Fully Differentiable NLOS Inverse Rendering
Authors:
Kiseok Choi,
Inchul Kim,
Dongyoung Choi,
Julio Marco,
Diego Gutierrez,
Min H. Kim
Abstract:
Existing time-resolved non-line-of-sight (NLOS) imaging methods reconstruct hidden scenes by inverting the optical paths of indirect illumination measured at visible relay surfaces. These methods are prone to reconstruction artifacts due to inversion ambiguities and capture noise, which are typically mitigated through the manual selection of filtering functions and parameters. We introduce a fully…
▽ More
Existing time-resolved non-line-of-sight (NLOS) imaging methods reconstruct hidden scenes by inverting the optical paths of indirect illumination measured at visible relay surfaces. These methods are prone to reconstruction artifacts due to inversion ambiguities and capture noise, which are typically mitigated through the manual selection of filtering functions and parameters. We introduce a fully-differentiable end-to-end NLOS inverse rendering pipeline that self-calibrates the imaging parameters during the reconstruction of hidden scenes, using as input only the measured illumination while working both in the time and frequency domains. Our pipeline extracts a geometric representation of the hidden scene from NLOS volumetric intensities and estimates the time-resolved illumination at the relay wall produced by such geometric information using differentiable transient rendering. We then use gradient descent to optimize imaging parameters by minimizing the error between our simulated time-resolved illumination and the measured illumination. Our end-to-end differentiable pipeline couples diffraction-based volumetric NLOS reconstruction with path-space light transport and a simple ray marching technique to extract detailed, dense sets of surface points and normals of hidden scenes. We demonstrate the robustness of our method to consistently reconstruct geometry and albedo, even under significant noise levels.
△ Less
Submitted 25 September, 2023; v1 submitted 21 September, 2023;
originally announced September 2023.
-
Structure-Aware Parametric Representations for Time-Resolved Light Transport
Authors:
Diego Royo,
Zesheng Huang,
Yun Liang,
Boyan Song,
Adolfo Muñoz,
Diego Gutierrez,
Julio Marco
Abstract:
Time-resolved illumination provides rich spatio-temporal information for applications such as accurate depth sensing or hidden geometry reconstruction, becoming a useful asset for prototy** and as input for data-driven approaches. However, time-resolved illumination measurements are high-dimensional and have a low signal-to-noise ratio, hampering their applicability in real scenarios. We propose…
▽ More
Time-resolved illumination provides rich spatio-temporal information for applications such as accurate depth sensing or hidden geometry reconstruction, becoming a useful asset for prototy** and as input for data-driven approaches. However, time-resolved illumination measurements are high-dimensional and have a low signal-to-noise ratio, hampering their applicability in real scenarios. We propose a novel method to compactly represent time-resolved illumination using mixtures of exponentially-modified Gaussians that are robust to noise and preserve structural information. Our method yields representations two orders of magnitude smaller than discretized data, providing consistent results in applications such as hidden scene reconstruction and depth estimation, and quantitative improvements over previous approaches.
△ Less
Submitted 30 August, 2023;
originally announced August 2023.
-
Virtual Light Transport Matrices for Non-Line-Of-Sight Imaging
Authors:
Julio Marco,
Adrian Jarabo,
Ji Hyun Nam,
Xiaochun Liu,
Miguel Ángel Cosculluela,
Andreas Velten,
Diego Gutierrez
Abstract:
The light transport matrix (LTM) is an instrumental tool in line-of-sight (LOS) imaging, describing how light interacts with the scene and enabling applications such as relighting or separation of illumination components. We introduce a framework to estimate the LTM of non-line-of-sight (NLOS) scenarios, coupling recent virtual forward light propagation models for NLOS imaging with the LOS light t…
▽ More
The light transport matrix (LTM) is an instrumental tool in line-of-sight (LOS) imaging, describing how light interacts with the scene and enabling applications such as relighting or separation of illumination components. We introduce a framework to estimate the LTM of non-line-of-sight (NLOS) scenarios, coupling recent virtual forward light propagation models for NLOS imaging with the LOS light transport equation. We design computational projector-camera setups, and use these virtual imaging systems to estimate the transport matrix of hidden scenes. We introduce the specific illumination functions to compute the different elements of the matrix, overcoming the challenging wide-aperture conditions of NLOS setups. Our NLOS light transport matrix allows us to (re)illuminate specific locations of a hidden scene, and separate direct, first-order indirect, and higher-order indirect illumination of complex cluttered hidden scenes, similar to existing LOS techniques.
△ Less
Submitted 5 October, 2021; v1 submitted 23 March, 2021;
originally announced March 2021.
-
On the assesment of functional connectivity in an immersive brain-computer interface during motor imagery
Authors:
Myriam Alanis-Espinosa,
David Gutiérrez
Abstract:
New trends on brain-computer interface (BCI) design are aiming to combine this technology with immersive virtual reality in order to provide a sense of realism to its users. In this study, we propose an experimental BCI to control an immersive telepresence system using motor imagery (MI). The system is immersive in the sense that the users can control the movement of a NAO humanoid robot in a firs…
▽ More
New trends on brain-computer interface (BCI) design are aiming to combine this technology with immersive virtual reality in order to provide a sense of realism to its users. In this study, we propose an experimental BCI to control an immersive telepresence system using motor imagery (MI). The system is immersive in the sense that the users can control the movement of a NAO humanoid robot in a first person perspective (1PP), i.e., as if the movement of the robot was his/her own. We analyze functional brain connectivity between 1PP and 3PP during the control of our BCI using graph theory properties such as degree, betweenness centrality, and efficiency. Changes in these metrics are obtained for the case of the 1PP, as well as for the traditional third person perspective (3PP) in which the user can see the movement of the robot as feedback. As proof-of-concept, electroencephalography (EEG) signals were recorded from two subjects while they performed MI to control the movement of the robot. The graph theoretical analysis was applied to the binary directed networks obtained through the partial directed coherence (PDC). In our preliminary assessment we found that the efficiency in the alpha brain rhythm is greater in 1PP condition in comparison to the 3PP at the prefrontal cortex. Also, a stronger influence of signals measured at EEG channel C3 (primary motor cortex) to other regions was found in 1PP condition. Furthermore, our preliminary results seem to indicate that alpha and beta brain rhythms have a high indegree at prefrontal cortex in 1PP condition, and this could be possibly related to the experience of sense of agency. Therefore, using the PDC combined with graph theory while controlling a telepresence robot in an immersive system may contribute to understand the organization and behavior of brain networks in these environments.
△ Less
Submitted 5 December, 2019;
originally announced December 2019.
-
Evaluating a Semi-Autonomous Brain-Computer Interface Based on Conformal Geometric Algebra and Artificial Vision
Authors:
M. A. Ramirez-Moreno,
D. Gutiérrez
Abstract:
In this paper, we evaluate a semi-autonomous brain-computer interface (BCI) for manipulation tasks. In such system, the user controls a robotic arm through motor imagery commands. In traditional process-control BCI systems, the user has to provide those commands continuously in order manipulate the effector of the robot step-by-step, which results in a tiresome process for simple tasks such as pic…
▽ More
In this paper, we evaluate a semi-autonomous brain-computer interface (BCI) for manipulation tasks. In such system, the user controls a robotic arm through motor imagery commands. In traditional process-control BCI systems, the user has to provide those commands continuously in order manipulate the effector of the robot step-by-step, which results in a tiresome process for simple tasks such as pick and replace an item from a surface. Here, we take a semi-autonomous approach based on a conformal geometric algebra model that solves the inverse kinematics of the robot on the fly, then the user only has to decide on the start of the movement and the final position of the effector (goal-selection approach). Under these conditions, we implemented pick-and-place tasks with a disk as an item and two target areas placed on the table at arbitrary positions. An artificial vision (AV) algorithm was used to obtain the positions of the items expressed in the robot frame through images captured with a webcam. Then, the AV algorithm is integrated to the inverse kinematics model to perform the manipulation tasks. As proof-of-concept, different users were trained to control the pick-and-place tasks through the process-control and semi-autonomous goal-selection approaches, so that the performance of both schemes could be compared. Our results show the superiority in performance of the semi-autonomous approach, as well as evidence of less mental fatigue with it.
△ Less
Submitted 30 October, 2019;
originally announced October 2019.
-
Convolutional Sparse Coding for High Dynamic Range Imaging
Authors:
Ana Serrano,
Felix Heide,
Diego Gutierrez,
Gordon Wetzstein,
Belen Masia
Abstract:
Current HDR acquisition techniques are based on either (i) fusing multibracketed, low dynamic range (LDR) images, (ii) modifying existing hardware and capturing different exposures simultaneously with multiple sensors, or (iii) reconstructing a single image with spatially-varying pixel exposures. In this paper, we propose a novel algorithm to recover high-quality HDRI images from a single, coded e…
▽ More
Current HDR acquisition techniques are based on either (i) fusing multibracketed, low dynamic range (LDR) images, (ii) modifying existing hardware and capturing different exposures simultaneously with multiple sensors, or (iii) reconstructing a single image with spatially-varying pixel exposures. In this paper, we propose a novel algorithm to recover high-quality HDRI images from a single, coded exposure. The proposed reconstruction method builds on recently-introduced ideas of convolutional sparse coding (CSC); this paper demonstrates how to make CSC practical for HDR imaging. We demonstrate that the proposed algorithm achieves higher-quality reconstructions than alternative methods, we evaluate optical coding schemes, analyze algorithmic parameters, and build a prototype coded HDR camera that demonstrates the utility of convolutional sparse HDRI coding with a custom hardware platform.
△ Less
Submitted 13 June, 2018;
originally announced June 2018.
-
Convolutional sparse coding for capturing high speed video content
Authors:
Ana Serrano,
Elena Garces,
Diego Gutierrez,
Belen Masia
Abstract:
Video capture is limited by the trade-off between spatial and temporal resolution: when capturing videos of high temporal resolution, the spatial resolution decreases due to bandwidth limitations in the capture system. Achieving both high spatial and temporal resolution is only possible with highly specialized and very expensive hardware, and even then the same basic trade-off remains. The recent…
▽ More
Video capture is limited by the trade-off between spatial and temporal resolution: when capturing videos of high temporal resolution, the spatial resolution decreases due to bandwidth limitations in the capture system. Achieving both high spatial and temporal resolution is only possible with highly specialized and very expensive hardware, and even then the same basic trade-off remains. The recent introduction of compressive sensing and sparse reconstruction techniques allows for the capture of single-shot high-speed video, by coding the temporal information in a single frame, and then reconstructing the full video sequence from this single coded image and a trained dictionary of image patches. In this paper, we first analyze this approach, and find insights that help improve the quality of the reconstructed videos. We then introduce a novel technique, based on convolutional sparse coding (CSC), and show how it outperforms the state-of-the-art, patch-based approach in terms of flexibility and efficiency, due to the convolutional nature of its filter banks. The key idea for CSC high-speed video acquisition is extending the basic formulation by imposing an additional constraint in the temporal dimension, which enforces sparsity of the first-order derivatives over time.
△ Less
Submitted 13 June, 2018;
originally announced June 2018.
-
Reconstruction of Brain Activity from EEG/MEG Using MV-PURE Framework
Authors:
Tomasz Piotrowski,
Jan Nikadon,
David Gutierrez
Abstract:
We consider the problem of reconstruction of brain activity from electroencephalography (EEG) or magnetoencephalography (MEG) using spatial filtering (beamforming). We propose spatial filters which are based on the minimum-variance pseudo-unbiased reduced-rank estimation (MV-PURE) framework. They come in two flavours, depending whether the EEG/MEG forward model considers explicitly "interfering ac…
▽ More
We consider the problem of reconstruction of brain activity from electroencephalography (EEG) or magnetoencephalography (MEG) using spatial filtering (beamforming). We propose spatial filters which are based on the minimum-variance pseudo-unbiased reduced-rank estimation (MV-PURE) framework. They come in two flavours, depending whether the EEG/MEG forward model considers explicitly "interfering activity", understood as brain's electrical activity originating from brain areas other than regions of interest which is recorded at EEG/MEG sensors as a signal correlated with activity of interest. In both cases, the proposed filters are equipped with a rank-selection criterion minimizing the mean-square-error (MSE) of the filter output. Therefore, we consider them as novel nontrivial generalizations of well-known linearly constrained minimum-variance (LCMV) and nulling filters. The proposed filters have equally wide area of applications, which include in particular evaluation of directed connectivity measures based on the reconstructed activity of sources of interest, considered in this paper as a sample application. Moreover, in order to facilitate reproducibility of our research, we provide (jointly with this paper) comprehensive simulation framework that allows for estimation of error of signal reconstruction for a number of spatial filters applied to MEG or EEG signals. Based on this framework, chief properties of proposed filters are verified in a set of detailed simulations.
△ Less
Submitted 8 December, 2017;
originally announced December 2017.
-
On the linear quadratic problem for systems with time reversed Markov jump parameters and the duality with filtering of Markov jump linear systems
Authors:
Daniel Gutierrez,
Eduardo F. Costa
Abstract:
We study a class of systems whose parameters are driven by a Markov chain in reverse time. A recursive characterization for the second moment matrix, a spectral radius test for mean square stability and the formulas for optimal control are given. Our results are determining for the question: is it possible to extend the classical duality between filtering and control of linear systems (whose matri…
▽ More
We study a class of systems whose parameters are driven by a Markov chain in reverse time. A recursive characterization for the second moment matrix, a spectral radius test for mean square stability and the formulas for optimal control are given. Our results are determining for the question: is it possible to extend the classical duality between filtering and control of linear systems (whose matrices are transposed in the dual problem) by simply adding the jump variable of a Markov jump linear system. The answer is positive provided the jump process is reversed in time.
△ Less
Submitted 22 November, 2016;
originally announced November 2016.
