-
Z-Splat: Z-Axis Gaussian Splatting for Camera-Sonar Fusion
Authors:
Ziyuan Qu,
Omkar Vengurlekar,
Mohamad Qadri,
Kevin Zhang,
Michael Kaess,
Christopher Metzler,
Suren Jayasuriya,
Adithya Pediredla
Abstract:
Differentiable 3D-Gaussian splatting (GS) is emerging as a prominent technique in computer vision and graphics for reconstructing 3D scenes. GS represents a scene as a set of 3D Gaussians with varying opacities and employs a computationally efficient splatting operation along with analytical derivatives to compute the 3D Gaussian parameters given scene images captured from various viewpoints. Unfo…
▽ More
Differentiable 3D-Gaussian splatting (GS) is emerging as a prominent technique in computer vision and graphics for reconstructing 3D scenes. GS represents a scene as a set of 3D Gaussians with varying opacities and employs a computationally efficient splatting operation along with analytical derivatives to compute the 3D Gaussian parameters given scene images captured from various viewpoints. Unfortunately, capturing surround view ($360^{\circ}$ viewpoint) images is impossible or impractical in many real-world imaging scenarios, including underwater imaging, rooms inside a building, and autonomous navigation. In these restricted baseline imaging scenarios, the GS algorithm suffers from a well-known 'missing cone' problem, which results in poor reconstruction along the depth axis. In this manuscript, we demonstrate that using transient data (from sonars) allows us to address the missing cone problem by sampling high-frequency data along the depth axis. We extend the Gaussian splatting algorithms for two commonly used sonars and propose fusion algorithms that simultaneously utilize RGB camera data and sonar data. Through simulations, emulations, and hardware experiments across various imaging scenarios, we show that the proposed fusion algorithms lead to significantly better novel view synthesis (5 dB improvement in PSNR) and 3D geometry reconstruction (60% lower Chamfer distance).
△ Less
Submitted 5 July, 2024; v1 submitted 6 April, 2024;
originally announced April 2024.
-
AONeuS: A Neural Rendering Framework for Acoustic-Optical Sensor Fusion
Authors:
Mohamad Qadri,
Kevin Zhang,
Akshay Hinduja,
Michael Kaess,
Adithya Pediredla,
Christopher A. Metzler
Abstract:
Underwater perception and 3D surface reconstruction are challenging problems with broad applications in construction, security, marine archaeology, and environmental monitoring. Treacherous operating conditions, fragile surroundings, and limited navigation control often dictate that submersibles restrict their range of motion and, thus, the baseline over which they can capture measurements. In the…
▽ More
Underwater perception and 3D surface reconstruction are challenging problems with broad applications in construction, security, marine archaeology, and environmental monitoring. Treacherous operating conditions, fragile surroundings, and limited navigation control often dictate that submersibles restrict their range of motion and, thus, the baseline over which they can capture measurements. In the context of 3D scene reconstruction, it is well-known that smaller baselines make reconstruction more challenging. Our work develops a physics-based multimodal acoustic-optical neural surface reconstruction framework (AONeuS) capable of effectively integrating high-resolution RGB measurements with low-resolution depth-resolved imaging sonar measurements. By fusing these complementary modalities, our framework can reconstruct accurate high-resolution 3D surfaces from measurements captured over heavily-restricted baselines. Through extensive simulations and in-lab experiments, we demonstrate that AONeuS dramatically outperforms recent RGB-only and sonar-only inverse-differentiable-rendering--based surface reconstruction methods. A website visualizing the results of our paper is located at this address: https://aoneus.github.io/
△ Less
Submitted 21 May, 2024; v1 submitted 5 February, 2024;
originally announced February 2024.
-
Scalable underwater assembly with reconfigurable visual fiducials
Authors:
Samuel Lensgraf,
Ankita Sarkar,
Adithya Pediredla,
Devin Balkcom,
Alberto Quattrini Li
Abstract:
We present a scalable combined localization infrastructure deployment and task planning algorithm for underwater assembly. Infrastructure is autonomously modified to suit the needs of manipulation tasks based on an uncertainty model on the infrastructure's positional accuracy. Our uncertainty model can be combined with the noise characteristics from multiple devices. For the task planning problem,…
▽ More
We present a scalable combined localization infrastructure deployment and task planning algorithm for underwater assembly. Infrastructure is autonomously modified to suit the needs of manipulation tasks based on an uncertainty model on the infrastructure's positional accuracy. Our uncertainty model can be combined with the noise characteristics from multiple devices. For the task planning problem, we propose a layer-based clustering approach that completes the manipulation tasks one cluster at a time. We employ movable visual fiducial markers as infrastructure and an autonomous underwater vehicle (AUV) for manipulation tasks. The proposed task planning algorithm is computationally simple, and we implement it on AUV without any offline computation requirements. Combined hardware experiments and simulations over large datasets show that the proposed technique is scalable to large areas.
△ Less
Submitted 30 October, 2023;
originally announced October 2023.
-
Doppler Time-of-Flight Rendering
Authors:
Juhyeon Kim,
Wojciech Jarosz,
Ioannis Gkioulekas,
Adithya Pediredla
Abstract:
We introduce Doppler time-of-flight (D-ToF) rendering, an extension of ToF rendering for dynamic scenes, with applications in simulating D-ToF cameras. D-ToF cameras use high-frequency modulation of illumination and exposure, and measure the Doppler frequency shift to compute the radial velocity of dynamic objects. The time-varying scene geometry and high-frequency modulation functions used in suc…
▽ More
We introduce Doppler time-of-flight (D-ToF) rendering, an extension of ToF rendering for dynamic scenes, with applications in simulating D-ToF cameras. D-ToF cameras use high-frequency modulation of illumination and exposure, and measure the Doppler frequency shift to compute the radial velocity of dynamic objects. The time-varying scene geometry and high-frequency modulation functions used in such cameras make it challenging to accurately and efficiently simulate their measurements with existing ToF rendering algorithms. We overcome these challenges in a twofold manner: To achieve accuracy, we derive path integral expressions for D-ToF measurements under global illumination and form unbiased Monte Carlo estimates of these integrals. To achieve efficiency, we develop a tailored time-path sampling technique that combines antithetic time sampling with correlated path sampling. We show experimentally that our sampling technique achieves up to two orders of magnitude lower variance compared to naive time-path sampling. We provide an open-source simulator that serves as a digital twin for D-ToF imaging systems, allowing imaging researchers, for the first time, to investigate the impact of modulation functions, material properties, and global illumination on D-ToF imaging performance.
△ Less
Submitted 5 October, 2023; v1 submitted 28 September, 2023;
originally announced September 2023.
-
Neural Volumetric Reconstruction for Coherent Synthetic Aperture Sonar
Authors:
Albert W. Reed,
Juhyeon Kim,
Thomas Blanford,
Adithya Pediredla,
Daniel C. Brown,
Suren Jayasuriya
Abstract:
Synthetic aperture sonar (SAS) measures a scene from multiple views in order to increase the resolution of reconstructed imagery. Image reconstruction methods for SAS coherently combine measurements to focus acoustic energy onto the scene. However, image formation is typically under-constrained due to a limited number of measurements and bandlimited hardware, which limits the capabilities of exist…
▽ More
Synthetic aperture sonar (SAS) measures a scene from multiple views in order to increase the resolution of reconstructed imagery. Image reconstruction methods for SAS coherently combine measurements to focus acoustic energy onto the scene. However, image formation is typically under-constrained due to a limited number of measurements and bandlimited hardware, which limits the capabilities of existing reconstruction methods. To help meet these challenges, we design an analysis-by-synthesis optimization that leverages recent advances in neural rendering to perform coherent SAS imaging. Our optimization enables us to incorporate physics-based constraints and scene priors into the image formation process. We validate our method on simulation and experimental results captured in both air and water. We demonstrate both quantitatively and qualitatively that our method typically produces superior reconstructions than existing approaches. We share code and data for reproducibility.
△ Less
Submitted 16 June, 2023;
originally announced June 2023.
-
Adaptive Gating for Single-Photon 3D Imaging
Authors:
Ryan Po,
Adithya Pediredla,
Ioannis Gkioulekas
Abstract:
Single-photon avalanche diodes (SPADs) are growing in popularity for depth sensing tasks. However, SPADs still struggle in the presence of high ambient light due to the effects of pile-up. Conventional techniques leverage fixed or asynchronous gating to minimize pile-up effects, but these gating schemes are all non-adaptive, as they are unable to incorporate factors such as scene priors and previo…
▽ More
Single-photon avalanche diodes (SPADs) are growing in popularity for depth sensing tasks. However, SPADs still struggle in the presence of high ambient light due to the effects of pile-up. Conventional techniques leverage fixed or asynchronous gating to minimize pile-up effects, but these gating schemes are all non-adaptive, as they are unable to incorporate factors such as scene priors and previous photon detections into their gating strategy. We propose an adaptive gating scheme built upon Thompson sampling. Adaptive gating periodically updates the gate position based on prior photon observations in order to minimize depth errors. Our experiments show that our gating strategy results in significantly reduced depth reconstruction error and acquisition time, even when operating outdoors under strong sunlight conditions.
△ Less
Submitted 11 April, 2022; v1 submitted 29 November, 2021;
originally announced November 2021.
-
Overcoming the tradeoff between confinement and focal distance using virtual ultrasonic optical waveguides
Authors:
Matteo Giuseppe Scopelliti,
Hengji Huang,
Adithya Pediredla,
Srinivasa Narasimhan,
Ioannis Gkioulekas,
Maysamreza Chamanzar
Abstract:
Conventional optical lenses have been used to focus light from outside without disturbing the medium. The focused spot size is proportional to the focal distance in a conventional lens, resulting in a tradeoff between depth of penetration in the target medium and spatial resolution. We have shown that virtual ultrasonically sculpted gradient-index (GRIN) optical waveguides can be formed in the tar…
▽ More
Conventional optical lenses have been used to focus light from outside without disturbing the medium. The focused spot size is proportional to the focal distance in a conventional lens, resulting in a tradeoff between depth of penetration in the target medium and spatial resolution. We have shown that virtual ultrasonically sculpted gradient-index (GRIN) optical waveguides can be formed in the target medium to guide and steer light without disturbing the medium. Here, we demonstrate that such virtual waveguides can relay an externally focused beam of light through the medium beyond the focal distance of an external physical lens to extend the penetration depth without compromising the spot size. Moreover, the spot size can be tuned by reconfiguring the virtual waveguide. We show that these virtual GRIN waveguides can be formed in transparent as well as turbid media to enhance the confinement and contrast ratio of the focused beam of light at the target location. This method can be extended to realize complex optical systems of external physical lenses and in situ virtual waveguides to extend the reach and flexibility of optical methods.
△ Less
Submitted 18 June, 2020; v1 submitted 28 February, 2020;
originally announced March 2020.
-
Signal Processing Based Pile-up Compensation for Gated Single-Photon Avalanche Diodes
Authors:
Adithya K. Pediredla,
Aswin C. Sankaranarayanan,
Mauro Buttafava,
Alberto Tosi,
Ashok Veeraraghavan
Abstract:
Single-photon avalanche diode (SPAD) based transient imaging suffers from an aberration called pile-up. When multiple photons arrive within a single repetition period of the illuminating laser, the SPAD records only the arrival of the first photon; this leads to a bias in the recorded light transient wherein the transient response at later time-instants are under-estimated. An unfortunate conseque…
▽ More
Single-photon avalanche diode (SPAD) based transient imaging suffers from an aberration called pile-up. When multiple photons arrive within a single repetition period of the illuminating laser, the SPAD records only the arrival of the first photon; this leads to a bias in the recorded light transient wherein the transient response at later time-instants are under-estimated. An unfortunate consequence of this is the need to operate the illumination at low-power levels to reduce the probability of multiple photons returning in a single period. Operating the laser at low power results in either low signal-to-noise ratio (SNR) in the measured transients or reduced frame rate due to longer exposure durations to achieve a high SNR. In this paper, we propose a signal processing-based approach to compensate pile-up in post-processing, thereby enabling high power operation of the illuminating laser. While increasing illumination does cause a fundamental information loss in the data captured by SPAD, we quantify this information loss using Cramer-Rao bound and show that the errors in our framework are only limited to this information loss. We experimentally validate our hypotheses using real data from a lab prototype.
△ Less
Submitted 14 June, 2018;
originally announced June 2018.
-
Deep Imaging in Scattering Media with Single Photon Selective Plane Illumination Microscopy (SPIM)
Authors:
Adithya Kumar Pediredla,
Shizheng Zhang,
Ben Avants,
Fan Ye,
Shin Nagayama,
Ziying Chen,
Caleb Kemere,
Jacob Robinson,
Ashok Veeraraghavan
Abstract:
In most biological tissues, light scattering due to small differences in refractive index limits the depth of optical imaging systems. Two-photon microscopy (2PM), which significantly reduces the scattering of the excitation light, has emerged as the most common method to image deep within scattering biological tissue. This technique, however, requires high-power pulsed lasers that are both expens…
▽ More
In most biological tissues, light scattering due to small differences in refractive index limits the depth of optical imaging systems. Two-photon microscopy (2PM), which significantly reduces the scattering of the excitation light, has emerged as the most common method to image deep within scattering biological tissue. This technique, however, requires high-power pulsed lasers that are both expensive and difficult to integrate into compact portable systems. In this paper, using a combination of theoretical and experimental techniques, we show that Selective Plane Illumination Microscopy (SPIM) can image nearly as deep as 2PM without the need for a high-powered pulsed laser. Compared to other single photon imaging techniques like epifluorescence and confocal microscopy, SPIM can image more than twice as deep in scattering media (approximately 10 times the mean scattering length). These results suggest that SPIM has the potential to provide deep imaging in scattering media in situations where 2PM systems would be too large or costly.
△ Less
Submitted 23 April, 2016;
originally announced April 2016.
-
Spatial Phase-Sweep: Increasing temporal resolution of transient imaging using a light source array
Authors:
Ryuichi Tadano,
Adithya Kumar Pediredla,
Kaushik Mitra,
Ashok Veeraraghavan
Abstract:
Transient imaging or light-in-flight techniques capture the propagation of an ultra-short pulse of light through a scene, which in effect captures the optical impulse response of the scene. Recently, it has been shown that we can capture transient images using commercially available Time-of-Flight (ToF) systems such as Photonic Mixer Devices (PMD). In this paper, we propose `spatial phase-sweep',…
▽ More
Transient imaging or light-in-flight techniques capture the propagation of an ultra-short pulse of light through a scene, which in effect captures the optical impulse response of the scene. Recently, it has been shown that we can capture transient images using commercially available Time-of-Flight (ToF) systems such as Photonic Mixer Devices (PMD). In this paper, we propose `spatial phase-sweep', a technique that exploits the speed of light to increase the temporal resolution beyond the 100 picosecond limit imposed by current electronics. Spatial phase-sweep uses a linear array of light sources with spatial separation of about 3 mm between them, thereby resulting in a time shift of about 10 picoseconds, which translates into 100 Gfps of transient imaging in theory. We demonstrate a prototype and transient imaging results using spatial phase-sweep.
△ Less
Submitted 21 December, 2015;
originally announced December 2015.
-
Depth Fields: Extending Light Field Techniques to Time-of-Flight Imaging
Authors:
Suren Jayasuriya,
Adithya Pediredla,
Sriram Sivaramakrishnan,
Alyosha Molnar,
Ashok Veeraraghavan
Abstract:
A variety of techniques such as light field, structured illumination, and time-of-flight (TOF) are commonly used for depth acquisition in consumer imaging, robotics and many other applications. Unfortunately, each technique suffers from its individual limitations preventing robust depth sensing. In this paper, we explore the strengths and weaknesses of combining light field and time-of-flight imag…
▽ More
A variety of techniques such as light field, structured illumination, and time-of-flight (TOF) are commonly used for depth acquisition in consumer imaging, robotics and many other applications. Unfortunately, each technique suffers from its individual limitations preventing robust depth sensing. In this paper, we explore the strengths and weaknesses of combining light field and time-of-flight imaging, particularly the feasibility of an on-chip implementation as a single hybrid depth sensor. We refer to this combination as depth field imaging. Depth fields combine light field advantages such as synthetic aperture refocusing with TOF imaging advantages such as high depth resolution and coded signal processing to resolve multipath interference. We show applications including synthesizing virtual apertures for TOF imaging, improved depth map** through partial and scattering occluders, and single frequency TOF phase unwrap**. Utilizing space, angle, and temporal coding, depth fields can improve depth sensing in the wild and generate new insights into the dimensions of light's plenoptic function.
△ Less
Submitted 2 September, 2015;
originally announced September 2015.
-
Template-Based Active Contours
Authors:
Jayanth Krishna Mogali,
Adithya Kumar Pediredla,
Chandra Sekhar Seelamantula
Abstract:
We develop a generalized active contour formalism for image segmentation based on shape templates. The shape template is subjected to a restricted affine transformation (RAT) in order to segment the object of interest. RAT allows for translation, rotation, and scaling, which give a total of five degrees of freedom. The proposed active contour comprises an inner and outer contour pair, which are cl…
▽ More
We develop a generalized active contour formalism for image segmentation based on shape templates. The shape template is subjected to a restricted affine transformation (RAT) in order to segment the object of interest. RAT allows for translation, rotation, and scaling, which give a total of five degrees of freedom. The proposed active contour comprises an inner and outer contour pair, which are closed and concentric. The active contour energy is a contrast function defined based on the intensities of pixels that lie inside the inner contour and those that lie in the annulus between the inner and outer contours. We show that the contrast energy functional is optimal under certain conditions. The optimal RAT parameters are computed by maximizing the contrast function using a gradient descent optimizer. We show that the calculations are made efficient through use of Green's theorem. The proposed formalism is capable of handling a variety of shapes because for a chosen template, optimization is carried with respect to the RAT parameters only. The proposed formalism is validated on multiple images to show robustness to Gaussian and Poisson noise, to initialization, and to partial loss of structure in the object to be segmented.
△ Less
Submitted 3 December, 2013;
originally announced December 2013.
