Showing 1–13 of 13 results for author: Sundar, V

Generalized Event Cameras

Authors: Varun Sundar, Matthew Dutson, Andrei Ardelean, Claudio Bruschini, Edoardo Charbon, Mohit Gupta

Abstract: Event cameras capture the world at high time resolution and with minimal bandwidth requirements. However, event streams, which only encode changes in brightness, do not contain sufficient scene information to support a wide variety of downstream tasks. In this work, we design generalized event cameras that inherently preserve scene intensity in a bandwidth-efficient manner. We generalize event cam… ▽ More Event cameras capture the world at high time resolution and with minimal bandwidth requirements. However, event streams, which only encode changes in brightness, do not contain sufficient scene information to support a wide variety of downstream tasks. In this work, we design generalized event cameras that inherently preserve scene intensity in a bandwidth-efficient manner. We generalize event cameras in terms of when an event is generated and what information is transmitted. To implement our designs, we turn to single-photon sensors that provide digital access to individual photon detections; this modality gives us the flexibility to realize a rich space of generalized event cameras. Our single-photon event cameras are capable of high-speed, high-fidelity imaging at low readout rates. Consequently, these event cameras can support plug-and-play downstream inference, without capturing new event datasets or designing specialized event-vision models. As a practical implication, our designs, which involve lightweight and near-sensor-compatible computations, provide a way to use single-photon sensors without exorbitant bandwidth costs. △ Less

Submitted 2 July, 2024; originally announced July 2024.

Comments: CVPR 2024

SoDaCam: Software-defined Cameras via Single-Photon Imaging

Authors: Varun Sundar, Andrei Ardelean, Tristan Swedish, Claudio Bruschini, Edoardo Charbon, Mohit Gupta

Abstract: Reinterpretable cameras are defined by their post-processing capabilities that exceed traditional imaging. We present "SoDaCam" that provides reinterpretable cameras at the granularity of photons, from photon-cubes acquired by single-photon devices. Photon-cubes represent the spatio-temporal detections of photons as a sequence of binary frames, at frame-rates as high as 100 kHz. We show that simpl… ▽ More Reinterpretable cameras are defined by their post-processing capabilities that exceed traditional imaging. We present "SoDaCam" that provides reinterpretable cameras at the granularity of photons, from photon-cubes acquired by single-photon devices. Photon-cubes represent the spatio-temporal detections of photons as a sequence of binary frames, at frame-rates as high as 100 kHz. We show that simple transformations of the photon-cube, or photon-cube projections, provide the functionality of numerous imaging systems including: exposure bracketing, flutter shutter cameras, video compressive systems, event cameras, and even cameras that move during exposure. Our photon-cube projections offer the flexibility of being software-defined constructs that are only limited by what is computable, and shot-noise. We exploit this flexibility to provide new capabilities for the emulated cameras. As an added benefit, our projections provide camera-dependent compression of photon-cubes, which we demonstrate using an implementation of our projections on a novel compute architecture that is designed for single-photon imaging. △ Less

Submitted 8 September, 2023; v1 submitted 31 August, 2023; originally announced September 2023.

Comments: Accepted at ICCV 2023 (oral). Project webpage can be found at https://wisionlab.com/project/sodacam/

TERMinator: A Neural Framework for Structure-Based Protein Design using Tertiary Repeating Motifs

Authors: Alex J. Li, Vikram Sundar, Gevorg Grigoryan, Amy E. Keating

Abstract: Computational protein design has the potential to deliver novel molecular structures, binders, and catalysts for myriad applications. Recent neural graph-based models that use backbone coordinate-derived features show exceptional performance on native sequence recovery tasks and are promising frameworks for design. A statistical framework for modeling protein sequence landscapes using Tertiary Mot… ▽ More Computational protein design has the potential to deliver novel molecular structures, binders, and catalysts for myriad applications. Recent neural graph-based models that use backbone coordinate-derived features show exceptional performance on native sequence recovery tasks and are promising frameworks for design. A statistical framework for modeling protein sequence landscapes using Tertiary Motifs (TERMs), compact units of recurring structure in proteins, has also demonstrated good performance on protein design tasks. In this work, we investigate the use of TERM-derived data as features in neural protein design frameworks. Our graph-based architecture, TERMinator, incorporates TERM-based and coordinate-based information and outputs a Potts model over sequence space. TERMinator outperforms state-of-the-art models on native sequence recovery tasks, suggesting that utilizing TERM-based and coordinate-based features together is beneficial for protein design. △ Less

Submitted 27 April, 2022; originally announced April 2022.

Comments: Machine Learning for Structural Biology, NeurIPS 2021

Single-Photon Structured Light

Authors: Varun Sundar, Sizhuo Ma, Aswin C. Sankaranarayanan, Mohit Gupta

Abstract: We present a novel structured light technique that uses Single Photon Avalanche Diode (SPAD) arrays to enable 3D scanning at high-frame rates and low-light levels. This technique, called "Single-Photon Structured Light", works by sensing binary images that indicates the presence or absence of photon arrivals during each exposure; the SPAD array is used in conjunction with a high-speed binary proje… ▽ More We present a novel structured light technique that uses Single Photon Avalanche Diode (SPAD) arrays to enable 3D scanning at high-frame rates and low-light levels. This technique, called "Single-Photon Structured Light", works by sensing binary images that indicates the presence or absence of photon arrivals during each exposure; the SPAD array is used in conjunction with a high-speed binary projector, with both devices operated at speeds as high as 20~kHz. The binary images that we acquire are heavily influenced by photon noise and are easily corrupted by ambient sources of light. To address this, we develop novel temporal sequences using error correction codes that are designed to be robust to short-range effects like projector and camera defocus as well as resolution mismatch between the two devices. Our lab prototype is capable of 3D imaging in challenging scenarios involving objects with extremely low albedo or undergoing fast motion, as well as scenes under strong ambient illumination. △ Less

Submitted 11 April, 2022; originally announced April 2022.

Comments: Accepted at CVPR 2022 (poster). 26 pages, 23 figures

[Reproducibility Report] Rigging the Lottery: Making All Tickets Winners

Authors: Varun Sundar, Rajat Vadiraj Dwaraknath

Abstract: $\textit{RigL}$, a sparse training algorithm, claims to directly train sparse networks that match or exceed the performance of existing dense-to-sparse training techniques (such as pruning) for a fixed parameter count and compute budget. We implement $\textit{RigL}… ▽ More $\textit{RigL}$, a sparse training algorithm, claims to directly train sparse networks that match or exceed the performance of existing dense-to-sparse training techniques (such as pruning) for a fixed parameter count and compute budget. We implement $\textit{RigL}$ from scratch in Pytorch and reproduce its performance on CIFAR-10 within 0.1% of the reported value. On both CIFAR-10/100, the central claim holds -- given a fixed training budget, $\textit{RigL}$ surpasses existing dynamic-sparse training methods over a range of target sparsities. By training longer, the performance can match or exceed iterative pruning, while consuming constant FLOPs throughout training. We also show that there is little benefit in tuning $\textit{RigL}$'s hyper-parameters for every sparsity, initialization pair -- the reference choice of hyperparameters is often close to optimal performance. Going beyond the original paper, we find that the optimal initialization scheme depends on the training constraint. While the Erdos-Renyi-Kernel distribution outperforms the Uniform distribution for a fixed parameter count, for a fixed FLOP count, the latter performs better. Finally, redistributing layer-wise sparsity while training can bridge the performance gap between the two initialization schemes, but increases computational cost. △ Less

Submitted 29 March, 2021; v1 submitted 29 March, 2021; originally announced March 2021.

Comments: Under review at ML Reproducibility Challenge 2020. Code available at https://github.com/varun19299/rigl-reproducibility. Training plots and other logs available at https://wandb.ai/ml-reprod-2020

FlatNet: Towards Photorealistic Scene Reconstruction from Lensless Measurements

Authors: Salman S. Khan, Varun Sundar, Vivek Boominathan, Ashok Veeraraghavan, Kaushik Mitra

Abstract: Lensless imaging has emerged as a potential solution towards realizing ultra-miniature cameras by eschewing the bulky lens in a traditional camera. Without a focusing lens, the lensless cameras rely on computational algorithms to recover the scenes from multiplexed measurements. However, the current iterative-optimization-based reconstruction algorithms produce noisier and perceptually poorer imag… ▽ More Lensless imaging has emerged as a potential solution towards realizing ultra-miniature cameras by eschewing the bulky lens in a traditional camera. Without a focusing lens, the lensless cameras rely on computational algorithms to recover the scenes from multiplexed measurements. However, the current iterative-optimization-based reconstruction algorithms produce noisier and perceptually poorer images. In this work, we propose a non-iterative deep learning based reconstruction approach that results in orders of magnitude improvement in image quality for lensless reconstructions. Our approach, called $\textit{FlatNet}$, lays down a framework for reconstructing high-quality photorealistic images from mask-based lensless cameras, where the camera's forward model formulation is known. FlatNet consists of two stages: (1) an inversion stage that maps the measurement into a space of intermediate reconstruction by learning parameters within the forward model formulation, and (2) a perceptual enhancement stage that improves the perceptual quality of this intermediate reconstruction. These stages are trained together in an end-to-end manner. We show high-quality reconstructions by performing extensive experiments on real and challenging scenes using two different types of lensless prototypes: one which uses a separable forward model and another, which uses a more general non-separable cropped-convolution model. Our end-to-end approach is fast, produces photorealistic reconstructions, and is easy to adopt for other mask-based lensless cameras. △ Less

Submitted 29 October, 2020; originally announced October 2020.

Comments: Accepted to IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 2020. Supplementary material attached. For project website, see https://siddiquesalman.github.io/flatnet/

UDC 2020 Challenge on Image Restoration of Under-Display Camera: Methods and Results

Authors: Yuqian Zhou, Michael Kwan, Kyle Tolentino, Neil Emerton, Sehoon Lim, Tim Large, Lijiang Fu, Zhihong Pan, Baopu Li, Qirui Yang, Yihao Liu, Jigang Tang, Tao Ku, Shibin Ma, Bingnan Hu, Jiarong Wang, Densen Puthussery, Hrishikesh P S, Melvin Kuriakose, Jiji C V, Varun Sundar, Sumanth Hegde, Divya Kothandaraman, Kaushik Mitra, Akashdeep Jassal , et al. (20 additional authors not shown)

Abstract: This paper is the report of the first Under-Display Camera (UDC) image restoration challenge in conjunction with the RLQ workshop at ECCV 2020. The challenge is based on a newly-collected database of Under-Display Camera. The challenge tracks correspond to two types of display: a 4k Transparent OLED (T-OLED) and a phone Pentile OLED (P-OLED). Along with about 150 teams registered the challenge, ei… ▽ More This paper is the report of the first Under-Display Camera (UDC) image restoration challenge in conjunction with the RLQ workshop at ECCV 2020. The challenge is based on a newly-collected database of Under-Display Camera. The challenge tracks correspond to two types of display: a 4k Transparent OLED (T-OLED) and a phone Pentile OLED (P-OLED). Along with about 150 teams registered the challenge, eight and nine teams submitted the results during the testing phase for each track. The results in the paper are state-of-the-art restoration performance of Under-Display Camera Restoration. Datasets and paper are available at https://yzhouas.github.io/projects/UDC/udc.html. △ Less

Submitted 18 August, 2020; originally announced August 2020.

Comments: 15 pages

Deep Atrous Guided Filter for Image Restoration in Under Display Cameras

Authors: Varun Sundar, Sumanth Hegde, Divya Kothandaraman, Kaushik Mitra

Abstract: Under Display Cameras present a promising opportunity for phone manufacturers to achieve bezel-free displays by positioning the camera behind semi-transparent OLED screens. Unfortunately, such imaging systems suffer from severe image degradation due to light attenuation and diffraction effects. In this work, we present Deep Atrous Guided Filter (DAGF), a two-stage, end-to-end approach for image re… ▽ More Under Display Cameras present a promising opportunity for phone manufacturers to achieve bezel-free displays by positioning the camera behind semi-transparent OLED screens. Unfortunately, such imaging systems suffer from severe image degradation due to light attenuation and diffraction effects. In this work, we present Deep Atrous Guided Filter (DAGF), a two-stage, end-to-end approach for image restoration in UDC systems. A Low-Resolution Network first restores image quality at low-resolution, which is subsequently used by the Guided Filter Network as a filtering input to produce a high-resolution output. Besides the initial downsampling, our low-resolution network uses multiple, parallel atrous convolutions to preserve spatial resolution and emulates multi-scale processing. Our approach's ability to directly train on megapixel images results in significant performance improvement. We additionally propose a simple simulation scheme to pre-train our model and boost performance. Our overall framework ranks 2nd and 5th in the RLQ-TOD'20 UDC Challenge for POLED and TOLED displays, respectively. △ Less

Submitted 1 September, 2020; v1 submitted 14 August, 2020; originally announced August 2020.

Comments: To appear in ECCV 2020 RLQ Workshop. Supplementary material attached. For project website, see https://varun19299.github.io/deep-atrous-guided-filter/

Attribution Methods Reveal Flaws in Fingerprint-Based Virtual Screening

Authors: Vikram Sundar, Lucy Colwell

Abstract: Fingerprint-based models for protein-ligand binding have demonstrated outstanding success on benchmark datasets; however, these models may not learn the correct binding rules. To assess this concern, we use in silico datasets with known binding rules to develop a general framework for evaluating model attribution. This framework identifies fragments that a model considers necessary to achieve a pa… ▽ More Fingerprint-based models for protein-ligand binding have demonstrated outstanding success on benchmark datasets; however, these models may not learn the correct binding rules. To assess this concern, we use in silico datasets with known binding rules to develop a general framework for evaluating model attribution. This framework identifies fragments that a model considers necessary to achieve a particular score, sidestep** the need for a model to be differentiable. Our results confirm that high-performing models may not learn the correct binding rule, and suggest concrete steps that can remedy this situation. We show that adding fragment-matched inactive molecules (decoys) to the data reduces attribution false negatives, while attribution false positives largely arise from the background correlation structure of molecular data. Normalizing for these background correlations helps to reveal the true binding logic. Our work highlights the danger of trusting attributions from high-performing models and suggests that a closer examination of fingerprint correlation structure and better decoy selection may help reduce misattributions. △ Less

Submitted 8 July, 2020; v1 submitted 2 July, 2020; originally announced July 2020.

Comments: 4 pages, 5 figures. In proceedings for the 2020 ICML workshop on Machine Learning Interpretability for Scientific Discovery

Out-of-Distribution Detection in Multi-Label Datasets using Latent Space of $β$-VAE

Authors: Vijaya Kumar Sundar, Shreyas Ramakrishna, Zahra Rahiminasab, Arvind Easwaran, Abhishek Dubey

Abstract: Learning Enabled Components (LECs) are widely being used in a variety of perception based autonomy tasks like image segmentation, object detection, end-to-end driving, etc. These components are trained with large image datasets with multimodal factors like weather conditions, time-of-day, traffic-density, etc. The LECs learn from these factors during training, and while testing if there is variati… ▽ More Learning Enabled Components (LECs) are widely being used in a variety of perception based autonomy tasks like image segmentation, object detection, end-to-end driving, etc. These components are trained with large image datasets with multimodal factors like weather conditions, time-of-day, traffic-density, etc. The LECs learn from these factors during training, and while testing if there is variation in any of these factors, the components get confused resulting in low confidence predictions. The images with factors not seen during training is commonly referred to as Out-of-Distribution (OOD). For safe autonomy it is important to identify the OOD images, so that a suitable mitigation strategy can be performed. Classical one-class classifiers like SVM and SVDD are used to perform OOD detection. However, the multiple labels attached to the images in these datasets, restricts the direct application of these techniques. We address this problem using the latent space of the $β$-Variational Autoencoder ($β$-VAE). We use the fact that compact latent space generated by an appropriately selected $β$-VAE will encode the information about these factors in a few latent variables, and that can be used for computationally inexpensive detection. We evaluate our approach on the nuScenes dataset, and our results shows the latent space of $β$-VAE is sensitive to encode changes in the values of the generative factor. △ Less

Submitted 10 March, 2020; originally announced March 2020.

Comments: Workshop on Assured Autonomy (WAAS) -2020

Reproducing Quantum Probability Distributions at the Speed of Classical Dynamics: A New Approach for Develo** Force-Field Functors

Authors: Vikram Sundar, David Gelbwaser-Klimovsky, Alan Aspuru-Guzik

Abstract: Modeling nuclear quantum effects is required for accurate molecular dynamics (MD) simulations of molecules. The community has paid special attention to water and other biomolecules that show hydrogen bonding. Standard methods of modeling nuclear quantum effects like Ring Polymer Molecular Dynamics (RPMD) are computationally costlier than running classical trajectories. A force-field functor (FFF)… ▽ More Modeling nuclear quantum effects is required for accurate molecular dynamics (MD) simulations of molecules. The community has paid special attention to water and other biomolecules that show hydrogen bonding. Standard methods of modeling nuclear quantum effects like Ring Polymer Molecular Dynamics (RPMD) are computationally costlier than running classical trajectories. A force-field functor (FFF) is an alternative method that computes an effective force field which replicates quantum properties of the original force field. In this work, we propose an efficient method of computing FFF using the Wigner-Kirkwood expansion. As a test case, we calculate a range of thermodynamic properties of Neon, obtaining the same level of accuracy as RPMD, but with the shorter runtime of classical simulations. By modifying existing MD programs, the proposed method could be used in the future to increase the efficiency and accuracy of MD simulations involving water and proteins. △ Less

Submitted 18 December, 2017; originally announced December 2017.

Energetic molding of chiral magnetic bubbles

Authors: Derek Lau, Vignesh Sundar, Jian-Gang Zhu, Vincent Sokalski

Abstract: Topologically protected magnetic structures such as skyrmions and domain walls (DWs) have drawn a great deal of attention recently due to their thermal stability and potential for manipulation by spin current, which is the result of chiral magnetic configurations induced by the interfacial Dzyaloshinskii-Moriya Interaction (DMI). Designing devices that incorporate DMI necessitates a thorough under… ▽ More Topologically protected magnetic structures such as skyrmions and domain walls (DWs) have drawn a great deal of attention recently due to their thermal stability and potential for manipulation by spin current, which is the result of chiral magnetic configurations induced by the interfacial Dzyaloshinskii-Moriya Interaction (DMI). Designing devices that incorporate DMI necessitates a thorough understanding of how the interaction presents and can be measured. One approach is to measure growth asymmetry of chiral bubble domains in perpendicularly magnetized thin films, which has been described elsewhere by thermally activated DW motion. Here, we demonstrate that the anisotropic angular dependence of DW energy originating from the DMI is critical to understanding this behavior. Domains in Co/Ni multi-layers are observed to preferentially grow into non-elliptical teardrop shapes, which vary with the magnitude of an applied in-plane field. We model the domain profile using energetic calculations of equilibrium shape via the Wulff construction, which explains both the teardrop shape and the reversal of growth symmetry at large fields. △ Less

Submitted 25 May, 2016; v1 submitted 11 April, 2016; originally announced April 2016.

Journal ref: Phys. Rev. B 94, 060401 (2016)

Modeling and Analysis of the Wind-Waves Field Variability in the Indian Ocean During 1998-2009 Years

Authors: V. G. Polnikov, F. A. Pogarskii, S. A. Sannasiraj, V. Sundar

Abstract: To calculate the wind-waves in the Indian Ocean (IO), the wind field for the period from 1998 to 2009 was used, obtained from the NCEP/NOAA archive, and numerical model WAM (Cycle-4) was applied, modified by the new source function proposed in Polnikov (2005). Based on buoy data for the Indian Ocean, the fitting of the modified model WAM was done, which provides the win in accuracy of calculations… ▽ More To calculate the wind-waves in the Indian Ocean (IO), the wind field for the period from 1998 to 2009 was used, obtained from the NCEP/NOAA archive, and numerical model WAM (Cycle-4) was applied, modified by the new source function proposed in Polnikov (2005). Based on buoy data for the Indian Ocean, the fitting of the modified model WAM was done, which provides the win in accuracy of calculations on 35%, in comparison with the original model. All the further calculations of the wave fields in IO were made for these model settings. At the first stage, the analysis of the simulation results involves a) map** the fields of the significant wave height <Hs(x,y,T,R)> and the wave energy <Ea(x,y,T,R>, calculated with different scales of averaging in time T and space R; b) estimating the fields of seasonal, annual and long-term variability; and c) determining the 12-year trend of the annually averaged fields. The analysis was carried out taking into account the previously introduced zoning the ocean area, provided by the spatial inhomogeneity of the wind field [3]. Further analysis includes a) creation of time series for the averaged (over zones and across the ocean) wave height, <Hs(x,y,T,R)>, and wave energy, <Ea(x,y,T,R>; b) construction of the frequency spectra of these series; c) finding the extrema of wave field; d) making histograms of wave heights (in the zones and the whole ocean); and e) calculating the first four statistical moments for the waveheight field (in the zones and whole IO). The results obtained allow us to estimate the stored energy of the wave field in the Indian Ocean and the scales of its variability; to establish a positive 12-year trend of the averaged wave height (about 1% per year) and wave energy (2% per year); to determine features of the probability distribution; and to describe the statistical properties of the wave field in the zones of the Indian ocean. △ Less

Submitted 29 September, 2011; originally announced September 2011.

Comments: 38 pages, 10 figures, 4 tables, Appendix with 3 fugures

MSC Class: 76F55 ACM Class: G.3