Showing 1–12 of 12 results for author: Raskar, R

Event Cameras Meet SPADs for High-Speed, Low-Bandwidth Imaging

Authors: Manasi Muglikar, Siddharth Somasundaram, Akshat Dave, Edoardo Charbon, Ramesh Raskar, Davide Scaramuzza

Abstract: Traditional cameras face a trade-off between low-light performance and high-speed imaging: longer exposure times to capture sufficient light results in motion blur, whereas shorter exposures result in Poisson-corrupted noisy images. While burst photography techniques help mitigate this tradeoff, conventional cameras are fundamentally limited in their sensor noise characteristics. Event cameras and… ▽ More Traditional cameras face a trade-off between low-light performance and high-speed imaging: longer exposure times to capture sufficient light results in motion blur, whereas shorter exposures result in Poisson-corrupted noisy images. While burst photography techniques help mitigate this tradeoff, conventional cameras are fundamentally limited in their sensor noise characteristics. Event cameras and single-photon avalanche diode (SPAD) sensors have emerged as promising alternatives to conventional cameras due to their desirable properties. SPADs are capable of single-photon sensitivity with microsecond temporal resolution, and event cameras can measure brightness changes up to 1 MHz with low bandwidth requirements. We show that these properties are complementary, and can help achieve low-light, high-speed image reconstruction with low bandwidth requirements. We introduce a sensor fusion framework to combine SPADs with event cameras to improves the reconstruction of high-speed, low-light scenes while reducing the high bandwidth cost associated with using every SPAD frame. Our evaluation, on both synthetic and real sensor data, demonstrates significant enhancements ( > 5 dB PSNR) in reconstructing low-light scenes at high temporal resolution (100 kHz) compared to conventional cameras. Event-SPAD fusion shows great promise for real-world applications, such as robotics or medical imaging. △ Less

Submitted 17 April, 2024; originally announced April 2024.

PlatoNeRF: 3D Reconstruction in Plato's Cave via Single-View Two-Bounce Lidar

Authors: Tzofi Klinghoffer, Xiaoyu Xiang, Siddharth Somasundaram, Yuchen Fan, Christian Richardt, Ramesh Raskar, Rakesh Ranjan

Abstract: 3D reconstruction from a single-view is challenging because of the ambiguity from monocular cues and lack of information about occluded regions. Neural radiance fields (NeRF), while popular for view synthesis and 3D reconstruction, are typically reliant on multi-view images. Existing methods for single-view 3D reconstruction with NeRF rely on either data priors to hallucinate views of occluded reg… ▽ More 3D reconstruction from a single-view is challenging because of the ambiguity from monocular cues and lack of information about occluded regions. Neural radiance fields (NeRF), while popular for view synthesis and 3D reconstruction, are typically reliant on multi-view images. Existing methods for single-view 3D reconstruction with NeRF rely on either data priors to hallucinate views of occluded regions, which may not be physically accurate, or shadows observed by RGB cameras, which are difficult to detect in ambient light and low albedo backgrounds. We propose using time-of-flight data captured by a single-photon avalanche diode to overcome these limitations. Our method models two-bounce optical paths with NeRF, using lidar transient data for supervision. By leveraging the advantages of both NeRF and two-bounce light measured by lidar, we demonstrate that we can reconstruct visible and occluded geometry without data priors or reliance on controlled ambient lighting or scene albedo. In addition, we demonstrate improved generalization under practical constraints on sensor spatial- and temporal-resolution. We believe our method is a promising direction as single-photon lidars become ubiquitous on consumer devices, such as phones, tablets, and headsets. △ Less

Submitted 5 April, 2024; v1 submitted 21 December, 2023; originally announced December 2023.

Comments: CVPR 2024. Project Page: https://platonerf.github.io/

Role of Transients in Two-Bounce Non-Line-of-Sight Imaging

Authors: Siddharth Somasundaram, Akshat Dave, Connor Henley, Ashok Veeraraghavan, Ramesh Raskar

Abstract: The goal of non-line-of-sight (NLOS) imaging is to image objects occluded from the camera's field of view using multiply scattered light. Recent works have demonstrated the feasibility of two-bounce (2B) NLOS imaging by scanning a laser and measuring cast shadows of occluded objects in scenes with two relay surfaces. In this work, we study the role of time-of-flight (ToF) measurements, \ie transie… ▽ More The goal of non-line-of-sight (NLOS) imaging is to image objects occluded from the camera's field of view using multiply scattered light. Recent works have demonstrated the feasibility of two-bounce (2B) NLOS imaging by scanning a laser and measuring cast shadows of occluded objects in scenes with two relay surfaces. In this work, we study the role of time-of-flight (ToF) measurements, \ie transients, in 2B-NLOS under multiplexed illumination. Specifically, we study how ToF information can reduce the number of measurements and spatial resolution needed for shape reconstruction. We present our findings with respect to tradeoffs in (1) temporal resolution, (2) spatial resolution, and (3) number of image captures by studying SNR and recoverability as functions of system parameters. This leads to a formal definition of the mathematical constraints for 2B lidar. We believe that our work lays an analytical groundwork for design of future NLOS imaging systems, especially as ToF sensors become increasingly ubiquitous. △ Less

Submitted 3 April, 2023; originally announced April 2023.

Detection and Map** of Specular Surfaces Using Multibounce Lidar Returns

Authors: Connor Henley, Siddharth Somasundaram, Joseph Hollmann, Ramesh Raskar

Abstract: We propose methods that use specular, multibounce lidar returns to detect and map specular surfaces that might be invisible to conventional lidar systems that rely on direct, single-scatter returns. We derive expressions that relate the time- and angle-of-arrival of these multibounce returns to scattering points on the specular surface, and then use these expressions to formulate techniques for re… ▽ More We propose methods that use specular, multibounce lidar returns to detect and map specular surfaces that might be invisible to conventional lidar systems that rely on direct, single-scatter returns. We derive expressions that relate the time- and angle-of-arrival of these multibounce returns to scattering points on the specular surface, and then use these expressions to formulate techniques for retrieving specular surface geometry when the scene is scanned by a single beam or illuminated with a multi-beam flash. We also consider the special case of transparent specular surfaces, for which surface reflections can be mixed together with light that scatters off of objects lying behind the surface. △ Less

Submitted 7 September, 2022; originally announced September 2022.

Physics vs. Learned Priors: Rethinking Camera and Algorithm Design for Task-Specific Imaging

Authors: Tzofi Klinghoffer, Siddharth Somasundaram, Kushagra Tiwary, Ramesh Raskar

Abstract: Cameras were originally designed using physics-based heuristics to capture aesthetic images. In recent years, there has been a transformation in camera design from being purely physics-driven to increasingly data-driven and task-specific. In this paper, we present a framework to understand the building blocks of this nascent field of end-to-end design of camera hardware and algorithms. As part of… ▽ More Cameras were originally designed using physics-based heuristics to capture aesthetic images. In recent years, there has been a transformation in camera design from being purely physics-driven to increasingly data-driven and task-specific. In this paper, we present a framework to understand the building blocks of this nascent field of end-to-end design of camera hardware and algorithms. As part of this framework, we show how methods that exploit both physics and data have become prevalent in imaging and computer vision, underscoring a key trend that will continue to dominate the future of task-specific camera design. Finally, we share current barriers to progress in end-to-end design, and hypothesize how these barriers can be overcome. △ Less

Submitted 11 January, 2023; v1 submitted 21 April, 2022; originally announced April 2022.

Comments: Published at the International Conference on Computational Photography (ICCP), 2022

Automatic calibration of time of flight based non-line-of-sight reconstruction

Authors: Subhash Chandra Sadhu, Abhishek Singh, Tomohiro Maeda, Tristan Swedish, Ryan Kim, Lagnojita Sinha, Ramesh Raskar

Abstract: Time of flight based Non-line-of-sight (NLOS) imaging approaches require precise calibration of illumination and detector positions on the visible scene to produce reasonable results. If this calibration error is sufficiently high, reconstruction can fail entirely without any indication to the user. In this work, we highlight the necessity of building autocalibration into NLOS reconstruction in or… ▽ More Time of flight based Non-line-of-sight (NLOS) imaging approaches require precise calibration of illumination and detector positions on the visible scene to produce reasonable results. If this calibration error is sufficiently high, reconstruction can fail entirely without any indication to the user. In this work, we highlight the necessity of building autocalibration into NLOS reconstruction in order to handle mis-calibration. We propose a forward model of NLOS measurements that is differentiable with respect to both, the hidden scene albedo, and virtual illumination and detector positions. With only a mean squared error loss and no regularization, our model enables joint reconstruction and recovery of calibration parameters by minimizing the measurement residual using gradient descent. We demonstrate our method is able to produce robust reconstructions using simulated and real data where the calibration error applied causes other state of the art algorithms to fail. △ Less

Submitted 21 May, 2021; originally announced May 2021.

Proximity Inference with Wifi-Colocation during the COVID-19 Pandemic

Authors: Mikhail Dmitrienko, Abhishek Singh, Patrick Erichsen, Ramesh Raskar

Abstract: In this work we propose a WiFi colocation methodology for digital contact tracing. The approach works by having a device scan and store nearby access point information to perform proximity inference. We make our approach resilient to different practical scenarios by configuring a device to turn into a hotspot if access points are unavailable, which makes the approach feasible in both dense urban a… ▽ More In this work we propose a WiFi colocation methodology for digital contact tracing. The approach works by having a device scan and store nearby access point information to perform proximity inference. We make our approach resilient to different practical scenarios by configuring a device to turn into a hotspot if access points are unavailable, which makes the approach feasible in both dense urban areas and sparse rural places. We compare various shortcomings and advantages of this work over other conventional ways of doing digital contact tracing. Preliminary results indicate the feasibility of our approach for determining proximity between users, which is relevant for improving existing digital contact tracing and exposure notification implementations. △ Less

Submitted 19 October, 2020; v1 submitted 26 September, 2020; originally announced September 2020.

Comments: 5 pages, 7 figures

Proximity Sensing: Modeling and Understanding Noisy RSSI-BLE Signals and Other Mobile Sensor Data for Digital Contact Tracing

Authors: Sheshank Shankar, Rishank Kanaparti, Ayush Chopra, Rohan Sukumaran, Parth Patwa, Myungsun Kang, Abhishek Singh, Kevin P. McPherson, Ramesh Raskar

Abstract: As we await a vaccine, social-distancing via efficient contact tracing has emerged as the primary health strategy to dampen the spread of COVID-19. To enable efficient digital contact tracing, we present a novel system to estimate pair-wise individual proximity, via a joint model of Bluetooth Low Energy (BLE) signals with other on-device sensors (accelerometer, magnetometer, gyroscope). We explore… ▽ More As we await a vaccine, social-distancing via efficient contact tracing has emerged as the primary health strategy to dampen the spread of COVID-19. To enable efficient digital contact tracing, we present a novel system to estimate pair-wise individual proximity, via a joint model of Bluetooth Low Energy (BLE) signals with other on-device sensors (accelerometer, magnetometer, gyroscope). We explore multiple ways of interpreting the sensor data stream (time-series, histogram, etc) and use several statistical and deep learning methods to learn representations for sensing proximity. We report the normalized Decision Cost Function (nDCF) metric and analyze the differential impact of the various input signals, as well as discuss various challenges associated with this task. △ Less

Submitted 24 December, 2020; v1 submitted 3 September, 2020; originally announced September 2020.

Comments: Accepted to IEEE/ICACT' 2021: International Conference on Advanced Communication Technology. Also presented at the Machine Learning for Mobile Health workshop at NeurIPS 2020

Automatic Differentiation for All Photons Imaging to See Inside Volumetric Scattering Media

Authors: Tomohiro Maeda, Ankit Ranjan, Ramesh Raskar

Abstract: Imaging through dense scattering media - such as biological tissue, fog, and smoke - has applications in the medical and robotics fields. We propose a new framework using automatic differentiation for All Photons Imaging through homogeneous scattering media with unknown optical properties for non-invasive sensing and diagnostics. We overcome the need for the imaging target to be visible to the ill… ▽ More Imaging through dense scattering media - such as biological tissue, fog, and smoke - has applications in the medical and robotics fields. We propose a new framework using automatic differentiation for All Photons Imaging through homogeneous scattering media with unknown optical properties for non-invasive sensing and diagnostics. We overcome the need for the imaging target to be visible to the illumination source in All Photons Imaging, enabling practical and non-invasive imaging through turbid media with a simple optical setup. Our method does not require calibration to acquire the sensor position or optical properties of the media. △ Less

Submitted 2 June, 2020; originally announced June 2020.

Recent Advances in Imaging Around Corners

Authors: Tomohiro Maeda, Guy Satat, Tristan Swedish, Lagnojita Sinha, Ramesh Raskar

Abstract: Seeing around corners, also known as non-line-of-sight (NLOS) imaging is a computational method to resolve or recover objects hidden around corners. Recent advances in imaging around corners have gained significant interest. This paper reviews different types of existing NLOS imaging techniques and discusses the challenges that need to be addressed, especially for their applications outside of a c… ▽ More Seeing around corners, also known as non-line-of-sight (NLOS) imaging is a computational method to resolve or recover objects hidden around corners. Recent advances in imaging around corners have gained significant interest. This paper reviews different types of existing NLOS imaging techniques and discusses the challenges that need to be addressed, especially for their applications outside of a constrained laboratory environment. Our goal is to introduce this topic to broader research communities as well as provide insights that would lead to further developments in this research area. △ Less

Submitted 12 October, 2019; originally announced October 2019.

On Unlimited Sampling and Reconstruction

Authors: Ayush Bhandari, Felix Krahmer, Ramesh Raskar

Abstract: Shannon's sampling theorem is one of the cornerstone topics that is well understood and explored, both mathematically and algorithmically. That said, practical realization of this theorem still suffers from a severe bottleneck due to the fundamental assumption that the samples can span an arbitrary range of amplitudes. In practice, the theorem is realized using so-called analog-to-digital converte… ▽ More Shannon's sampling theorem is one of the cornerstone topics that is well understood and explored, both mathematically and algorithmically. That said, practical realization of this theorem still suffers from a severe bottleneck due to the fundamental assumption that the samples can span an arbitrary range of amplitudes. In practice, the theorem is realized using so-called analog-to-digital converters (ADCs) which clip or saturate whenever the signal amplitude exceeds the maximum recordable ADC voltage thus leading to a significant information loss. In this paper, we develop an alternative paradigm for sensing and recovery, called the Unlimited Sampling Framework. It is based on the observation that when a signal is mapped to an appropriate bounded interval via a modulo operation before entering the ADC, the saturation problem no longer exists, but one rather encounters a different type of information loss due to the modulo operation. Such an alternative setup can be implemented, for example, via so-called folding or self-reset ADCs, as they have been proposed in various contexts in the circuit design literature. The key task that we need to accomplish in order to cope with this new type of information loss is to recover a bandlimited signal from its modulo samples. In this paper we derive conditions when perfect recovery is possible and complement them with a stable recovery algorithm. The sampling density required to guarantee recovery is independent of the maximum recordable ADC voltage and depends on the signal bandwidth only. Our recovery guarantees extend to measurements affected by bounded noise, which includes the case of round-off quantization. Numerical experiments validate our approach. Applications of the unlimited sampling paradigm can be found in a number of fields such as signal processing, communication and imaging. △ Less

Submitted 30 November, 2020; v1 submitted 9 May, 2019; originally announced May 2019.

Comments: Accepted to IEEE Trans. on Sig. Proc. (10.1109/TSP.2020.3041955)

Light-Field for RF

Authors: Manikanta Kotaru, Guy Satat, Ramesh Raskar, Sachin Katti

Abstract: Most computer vision systems and computational photography systems are visible light based which is a small fraction of the electromagnetic (EM) spectrum. In recent years radio frequency (RF) hardware has become more widely available, for example, many cars are equipped with a RADAR, and almost every home has a WiFi device. In the context of imaging, RF spectrum holds many advantages compared to v… ▽ More Most computer vision systems and computational photography systems are visible light based which is a small fraction of the electromagnetic (EM) spectrum. In recent years radio frequency (RF) hardware has become more widely available, for example, many cars are equipped with a RADAR, and almost every home has a WiFi device. In the context of imaging, RF spectrum holds many advantages compared to visible light systems. In particular, in this regime, EM energy effectively interacts in different ways with matter. This property allows for many novel applications such as privacy preserving computer vision and imaging through absorbing and scattering materials in visible light such as walls. Here, we expand many of the concepts in computational photography in visible light to RF cameras. The main limitation of imaging with RF is the large wavelength that limits the imaging resolution when compared to visible light. However, the output of RF cameras is usually processed by computer vision and perception algorithms which would benefit from multi-modal sensing of the environment, and from sensing in situations in which visible light systems fail. To bridge the gap between computational photography and RF imaging, we expand the concept of light-field to RF. This work paves the way to novel computational sensing systems with RF. △ Less

Submitted 13 January, 2019; originally announced January 2019.