Showing 51–100 of 127 results for author: Barron, J

NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis

Authors: Pratul P. Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben Mildenhall, Jonathan T. Barron

Abstract: We present a method that takes as input a set of images of a scene illuminated by unconstrained known lighting, and produces as output a 3D representation that can be rendered from novel viewpoints under arbitrary lighting conditions. Our method represents the scene as a continuous volumetric function parameterized as MLPs whose inputs are a 3D location and whose outputs are the following scene pr… ▽ More We present a method that takes as input a set of images of a scene illuminated by unconstrained known lighting, and produces as output a 3D representation that can be rendered from novel viewpoints under arbitrary lighting conditions. Our method represents the scene as a continuous volumetric function parameterized as MLPs whose inputs are a 3D location and whose outputs are the following scene properties at that input location: volume density, surface normal, material parameters, distance to the first surface intersection in any direction, and visibility of the external environment in any direction. Together, these allow us to render novel views of the object under arbitrary lighting, including indirect illumination effects. The predicted visibility and surface intersection fields are critical to our model's ability to simulate direct and indirect illumination during training, because the brute-force techniques used by prior work are intractable for lighting conditions outside of controlled setups with a single light. Our method outperforms alternative approaches for recovering relightable 3D scene representations, and performs well in complex lighting settings that have posed a significant challenge to prior work. △ Less

Submitted 7 December, 2020; originally announced December 2020.

Comments: Project page: https://people.eecs.berkeley.edu/~pratul/nerv

NeRD: Neural Reflectance Decomposition from Image Collections

Authors: Mark Boss, Raphael Braun, Varun Jampani, Jonathan T. Barron, Ce Liu, Hendrik P. A. Lensch

Abstract: Decomposing a scene into its shape, reflectance, and illumination is a challenging but important problem in computer vision and graphics. This problem is inherently more challenging when the illumination is not a single light source under laboratory conditions but is instead an unconstrained environmental illumination. Though recent work has shown that implicit representations can be used to model… ▽ More Decomposing a scene into its shape, reflectance, and illumination is a challenging but important problem in computer vision and graphics. This problem is inherently more challenging when the illumination is not a single light source under laboratory conditions but is instead an unconstrained environmental illumination. Though recent work has shown that implicit representations can be used to model the radiance field of an object, most of these techniques only enable view synthesis and not relighting. Additionally, evaluating these radiance fields is resource and time-intensive. We propose a neural reflectance decomposition (NeRD) technique that uses physically-based rendering to decompose the scene into spatially varying BRDF material properties. In contrast to existing techniques, our input images can be captured under different illumination conditions. In addition, we also propose techniques to convert the learned reflectance volume into a relightable textured mesh enabling fast real-time rendering with novel illuminations. We demonstrate the potential of the proposed approach with experiments on both synthetic and real datasets, where we are able to obtain high-quality relightable 3D assets from image collections. The datasets and code is available on the project page: https://markboss.me/publication/2021-nerd/ △ Less

Submitted 26 August, 2021; v1 submitted 7 December, 2020; originally announced December 2020.

Comments: Accepted at ICCV 2021

Learned Initializations for Optimizing Coordinate-Based Neural Representations

Authors: Matthew Tancik, Ben Mildenhall, Terrance Wang, Divi Schmidt, Pratul P. Srinivasan, Jonathan T. Barron, Ren Ng

Abstract: Coordinate-based neural representations have shown significant promise as an alternative to discrete, array-based representations for complex low dimensional signals. However, optimizing a coordinate-based network from randomly initialized weights for each new signal is inefficient. We propose applying standard meta-learning algorithms to learn the initial weight parameters for these fully-connect… ▽ More Coordinate-based neural representations have shown significant promise as an alternative to discrete, array-based representations for complex low dimensional signals. However, optimizing a coordinate-based network from randomly initialized weights for each new signal is inefficient. We propose applying standard meta-learning algorithms to learn the initial weight parameters for these fully-connected networks based on the underlying class of signals being represented (e.g., images of faces or 3D models of chairs). Despite requiring only a minor change in implementation, using these learned initial weights enables faster convergence during optimization and can serve as a strong prior over the signal class being modeled, resulting in better generalization when only partial observations of a given signal are available. We explore these benefits across a variety of tasks, including representing 2D images, reconstructing CT scans, and recovering 3D shapes and scenes from 2D image observations. △ Less

Submitted 23 March, 2021; v1 submitted 3 December, 2020; originally announced December 2020.

Comments: Project page: https://www.matthewtancik.com/learnit

Nerfies: Deformable Neural Radiance Fields

Authors: Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Seitz, Ricardo Martin-Brualla

Abstract: We present the first method capable of photorealistically reconstructing deformable scenes using photos/videos captured casually from mobile phones. Our approach augments neural radiance fields (NeRF) by optimizing an additional continuous volumetric deformation field that warps each observed point into a canonical 5D NeRF. We observe that these NeRF-like deformation fields are prone to local mini… ▽ More We present the first method capable of photorealistically reconstructing deformable scenes using photos/videos captured casually from mobile phones. Our approach augments neural radiance fields (NeRF) by optimizing an additional continuous volumetric deformation field that warps each observed point into a canonical 5D NeRF. We observe that these NeRF-like deformation fields are prone to local minima, and propose a coarse-to-fine optimization method for coordinate-based models that allows for more robust optimization. By adapting principles from geometry processing and physical simulation to NeRF-like models, we propose an elastic regularization of the deformation field that further improves robustness. We show that our method can turn casually captured selfie photos/videos into deformable NeRF models that allow for photorealistic renderings of the subject from arbitrary viewpoints, which we dub "nerfies." We evaluate our method by collecting time-synchronized data using a rig with two mobile phones, yielding train/validation images of the same pose at different viewpoints. We show that our method faithfully reconstructs non-rigidly deforming scenes and reproduces unseen views with high fidelity. △ Less

Submitted 9 September, 2021; v1 submitted 25 November, 2020; originally announced November 2020.

Comments: ICCV 2021, Project page with videos: https://nerfies.github.io/

How to Train Neural Networks for Flare Removal

Authors: Yicheng Wu, Qiurui He, Tianfan Xue, Rahul Garg, Jiawen Chen, Ashok Veeraraghavan, Jonathan T. Barron

Abstract: When a camera is pointed at a strong light source, the resulting photograph may contain lens flare artifacts. Flares appear in a wide variety of patterns (halos, streaks, color bleeding, haze, etc.) and this diversity in appearance makes flare removal challenging. Existing analytical solutions make strong assumptions about the artifact's geometry or brightness, and therefore only work well on a sm… ▽ More When a camera is pointed at a strong light source, the resulting photograph may contain lens flare artifacts. Flares appear in a wide variety of patterns (halos, streaks, color bleeding, haze, etc.) and this diversity in appearance makes flare removal challenging. Existing analytical solutions make strong assumptions about the artifact's geometry or brightness, and therefore only work well on a small subset of flares. Machine learning techniques have shown success in removing other types of artifacts, like reflections, but have not been widely applied to flare removal due to the lack of training data. To solve this problem, we explicitly model the optical causes of flare either empirically or using wave optics, and generate semi-synthetic pairs of flare-corrupted and clean images. This enables us to train neural networks to remove lens flare for the first time. Experiments show our data synthesis approach is critical for accurate flare removal, and that models trained with our technique generalize well to real lens flares across different scenes, lighting conditions, and cameras. △ Less

Submitted 7 October, 2021; v1 submitted 24 November, 2020; originally announced November 2020.

Comments: A new version paper is uploaded

Cross-Camera Convolutional Color Constancy

Authors: Mahmoud Afifi, Jonathan T. Barron, Chloe LeGendre, Yun-Ta Tsai, Francois Bleibel

Abstract: We present "Cross-Camera Convolutional Color Constancy" (C5), a learning-based method, trained on images from multiple cameras, that accurately estimates a scene's illuminant color from raw images captured by a new camera previously unseen during training. C5 is a hypernetwork-like extension of the convolutional color constancy (CCC) approach: C5 learns to generate the weights of a CCC model that… ▽ More We present "Cross-Camera Convolutional Color Constancy" (C5), a learning-based method, trained on images from multiple cameras, that accurately estimates a scene's illuminant color from raw images captured by a new camera previously unseen during training. C5 is a hypernetwork-like extension of the convolutional color constancy (CCC) approach: C5 learns to generate the weights of a CCC model that is then evaluated on the input image, with the CCC weights dynamically adapted to different input content. Unlike prior cross-camera color constancy models, which are usually designed to be agnostic to the spectral properties of test-set images from unobserved cameras, C5 approaches this problem through the lens of transductive inference: additional unlabeled images are provided as input to the model at test time, which allows the model to calibrate itself to the spectral properties of the test-set camera during inference. C5 achieves state-of-the-art accuracy for cross-camera color constancy on several datasets, is fast to evaluate (~7 and ~90 ms per image on a GPU or CPU, respectively), and requires little memory (~2 MB), and thus is a practical solution to the problem of calibration-free automatic white balance for mobile photography. △ Less

Submitted 10 February, 2022; v1 submitted 23 November, 2020; originally announced November 2020.

Journal ref: ICCV 2021

A Convenient Generalization of Schlick's Bias and Gain Functions

Authors: Jonathan T. Barron

Abstract: We present a generalization of Schlick's bias and gain functions -- simple parametric curve-shaped functions for inputs in [0, 1]. Our single function includes both bias and gain as special cases, and is able to describe other smooth and monotonic curves with variable degrees of asymmetry. We present a generalization of Schlick's bias and gain functions -- simple parametric curve-shaped functions for inputs in [0, 1]. Our single function includes both bias and gain as special cases, and is able to describe other smooth and monotonic curves with variable degrees of asymmetry. △ Less

Submitted 16 October, 2020; originally announced October 2020.

Light Stage Super-Resolution: Continuous High-Frequency Relighting

Authors: Tiancheng Sun, Zexiang Xu, Xiuming Zhang, Sean Fanello, Christoph Rhemann, Paul Debevec, Yun-Ta Tsai, Jonathan T. Barron, Ravi Ramamoorthi

Abstract: The light stage has been widely used in computer graphics for the past two decades, primarily to enable the relighting of human faces. By capturing the appearance of the human subject under different light sources, one obtains the light transport matrix of that subject, which enables image-based relighting in novel environments. However, due to the finite number of lights in the stage, the light t… ▽ More The light stage has been widely used in computer graphics for the past two decades, primarily to enable the relighting of human faces. By capturing the appearance of the human subject under different light sources, one obtains the light transport matrix of that subject, which enables image-based relighting in novel environments. However, due to the finite number of lights in the stage, the light transport matrix only represents a sparse sampling on the entire sphere. As a consequence, relighting the subject with a point light or a directional source that does not coincide exactly with one of the lights in the stage requires interpolation and resampling the images corresponding to nearby lights, and this leads to ghosting shadows, aliased specularities, and other artifacts. To ameliorate these artifacts and produce better results under arbitrary high-frequency lighting, this paper proposes a learning-based solution for the "super-resolution" of scans of human faces taken from a light stage. Given an arbitrary "query" light direction, our method aggregates the captured images corresponding to neighboring lights in the stage, and uses a neural network to synthesize a rendering of the face that appears to be illuminated by a "virtual" light source at the query location. This neural network must circumvent the inherent aliasing and regularity of the light stage data that was used for training, which we accomplish through the use of regularized traditional interpolation methods within our network. Our learned model is able to produce renderings for arbitrary light directions that exhibit realistic shadows and specular highlights, and is able to generalize across a wide variety of subjects. △ Less

Submitted 17 October, 2020; originally announced October 2020.

Comments: Siggraph Asia 2020

Shape, Illumination, and Reflectance from Shading

Authors: Jonathan T. Barron, Jitendra Malik

Abstract: A fundamental problem in computer vision is that of inferring the intrinsic, 3D structure of the world from flat, 2D images of that world. Traditional methods for recovering scene properties such as shape, reflectance, or illumination rely on multiple observations of the same scene to overconstrain the problem. Recovering these same properties from a single image seems almost impossible in compari… ▽ More A fundamental problem in computer vision is that of inferring the intrinsic, 3D structure of the world from flat, 2D images of that world. Traditional methods for recovering scene properties such as shape, reflectance, or illumination rely on multiple observations of the same scene to overconstrain the problem. Recovering these same properties from a single image seems almost impossible in comparison -- there are an infinite number of shapes, paint, and lights that exactly reproduce a single image. However, certain explanations are more likely than others: surfaces tend to be smooth, paint tends to be uniform, and illumination tends to be natural. We therefore pose this problem as one of statistical inference, and define an optimization problem that searches for the *most likely* explanation of a single image. Our technique can be viewed as a superset of several classic computer vision problems (shape-from-shading, intrinsic images, color constancy, illumination estimation, etc) and outperforms all previous solutions to those constituent problems. △ Less

Submitted 7 October, 2020; originally announced October 2020.

Journal ref: TPAMI 2015

Learned Dual-View Reflection Removal

Authors: Simon Niklaus, Xuaner Cecilia Zhang, Jonathan T. Barron, Neal Wadhwa, Rahul Garg, Feng Liu, Tianfan Xue

Abstract: Traditional reflection removal algorithms either use a single image as input, which suffers from intrinsic ambiguities, or use multiple images from a moving camera, which is inconvenient for users. We instead propose a learning-based dereflection algorithm that uses stereo images as input. This is an effective trade-off between the two extremes: the parallax between two views provides cues to remo… ▽ More Traditional reflection removal algorithms either use a single image as input, which suffers from intrinsic ambiguities, or use multiple images from a moving camera, which is inconvenient for users. We instead propose a learning-based dereflection algorithm that uses stereo images as input. This is an effective trade-off between the two extremes: the parallax between two views provides cues to remove reflections, and two views are easy to capture due to the adoption of stereo cameras in smartphones. Our model consists of a learning-based reflection-invariant flow model for dual-view registration, and a learned synthesis model for combining aligned image pairs. Because no dataset for dual-view reflection removal exists, we render a synthetic dataset of dual-views with and without reflections for use in training. Our evaluation on an additional real-world dataset of stereo pairs shows that our algorithm outperforms existing single-image and multi-image dereflection approaches. △ Less

Submitted 1 October, 2020; originally announced October 2020.

Comments: http://sniklaus.com/dualref

An Update to the Letter of Intent for MATHUSLA: Search for Long-Lived Particles at the HL-LHC

Authors: Cristiano Alpigiani, Juan Carlos Arteaga-Velázquez, Austin Ball, Liron Barak, Jared Barron, Brian Batell, James Beacham, Yan Benhammo, Karen Salomé Caballero-Mora, Paolo Camarri, Roberto Cardarelli, John Paul Chou, Wentao Cui, David Curtin, Miriam Diamond, Keith R. Dienes, Liam Andrew Dougherty, Giuseppe Di Sciascio, Marco Drewes, Erez Etzion, Rouven Essig, Jared Evans, Arturo Fernández Téllez, Oliver Fischer, Jim Freeman , et al. (58 additional authors not shown)

Abstract: We report on recent progress in the design of the proposed MATHUSLA Long Lived Particle (LLP) detector for the HL-LHC, updating the information in the original Letter of Intent (LoI), see CDS:LHCC-I-031, arXiv:1811.00927. A suitable site has been identified at LHC Point 5 that is closer to the CMS Interaction Point (IP) than assumed in the LoI. The decay volume has been increased from 20 m to 25 m… ▽ More We report on recent progress in the design of the proposed MATHUSLA Long Lived Particle (LLP) detector for the HL-LHC, updating the information in the original Letter of Intent (LoI), see CDS:LHCC-I-031, arXiv:1811.00927. A suitable site has been identified at LHC Point 5 that is closer to the CMS Interaction Point (IP) than assumed in the LoI. The decay volume has been increased from 20 m to 25 m in height. Engineering studies have been made in order to locate much of the decay volume below ground, bringing the detector even closer to the IP. With these changes, a 100 m x 100 m detector has the same physics reach for large c$τ$ as the 200 m x 200 m detector described in the LoI and other studies. The performance for small c$τ$ is improved because of the proximity to the IP. Detector technology has also evolved while retaining the strip-like sensor geometry in Resistive Plate Chambers (RPC) described in the LoI. The present design uses extruded scintillator bars read out using wavelength shifting fibers and silicon photomultipliers (SiPM). Operations will be simpler and more robust with much lower operating voltages and without the use of greenhouse gases. Manufacturing is straightforward and should result in cost savings. Understanding of backgrounds has also significantly advanced, thanks to new simulation studies and measurements taken at the MATHUSLA test stand operating above ATLAS in 2018. We discuss next steps for the MATHUSLA collaboration, and identify areas where new members can make particularly important contributions. △ Less

Submitted 3 September, 2020; originally announced September 2020.

Comments: 22 pages + references, 12 Figures

Report number: CERN-LHCC-2020-014, LHCC-I-031-ADD-1

Neural Light Transport for Relighting and View Synthesis

Authors: Xiuming Zhang, Sean Fanello, Yun-Ta Tsai, Tiancheng Sun, Tianfan Xue, Rohit Pandey, Sergio Orts-Escolano, Philip Davidson, Christoph Rhemann, Paul Debevec, Jonathan T. Barron, Ravi Ramamoorthi, William T. Freeman

Abstract: The light transport (LT) of a scene describes how it appears under different lighting and viewing directions, and complete knowledge of a scene's LT enables the synthesis of novel views under arbitrary lighting. In this paper, we focus on image-based LT acquisition, primarily for human bodies within a light stage setup. We propose a semi-parametric approach to learn a neural representation of LT t… ▽ More The light transport (LT) of a scene describes how it appears under different lighting and viewing directions, and complete knowledge of a scene's LT enables the synthesis of novel views under arbitrary lighting. In this paper, we focus on image-based LT acquisition, primarily for human bodies within a light stage setup. We propose a semi-parametric approach to learn a neural representation of LT that is embedded in the space of a texture atlas of known geometric properties, and model all non-diffuse and global LT as residuals added to a physically-accurate diffuse base rendering. In particular, we show how to fuse previously seen observations of illuminants and views to synthesize a new image of the same scene under a desired lighting condition from a chosen viewpoint. This strategy allows the network to learn complex material effects (such as subsurface scattering) and global illumination, while guaranteeing the physical correctness of the diffuse LT (such as hard shadows). With this learned LT, one can relight the scene photorealistically with a directional light or an HDRI map, synthesize novel views with view-dependent effects, or do both simultaneously, all in a unified framework using a set of sparse, previously seen observations. Qualitative and quantitative experiments demonstrate that our neural LT (NLT) outperforms state-of-the-art solutions for relighting and view synthesis, without separate treatment for both problems that prior work requires. △ Less

Submitted 20 January, 2021; v1 submitted 9 August, 2020; originally announced August 2020.

Comments: Camera-ready version for TOG 2021. Project Page: http://nlt.csail.mit.edu/

NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections

Authors: Ricardo Martin-Brualla, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, Daniel Duckworth

Abstract: We present a learning-based method for synthesizing novel views of complex scenes using only unstructured collections of in-the-wild photographs. We build on Neural Radiance Fields (NeRF), which uses the weights of a multilayer perceptron to model the density and color of a scene as a function of 3D coordinates. While NeRF works well on images of static subjects captured under controlled settings,… ▽ More We present a learning-based method for synthesizing novel views of complex scenes using only unstructured collections of in-the-wild photographs. We build on Neural Radiance Fields (NeRF), which uses the weights of a multilayer perceptron to model the density and color of a scene as a function of 3D coordinates. While NeRF works well on images of static subjects captured under controlled settings, it is incapable of modeling many ubiquitous, real-world phenomena in uncontrolled images, such as variable illumination or transient occluders. We introduce a series of extensions to NeRF to address these issues, thereby enabling accurate reconstructions from unstructured image collections taken from the internet. We apply our system, dubbed NeRF-W, to internet photo collections of famous landmarks, and demonstrate temporally consistent novel view renderings that are significantly closer to photorealism than the prior state of the art. △ Less

Submitted 6 January, 2021; v1 submitted 5 August, 2020; originally announced August 2020.

Comments: Project website: https://nerf-w.github.io. Ricardo Martin-Brualla, Noha Radwan, and Mehdi S. M. Sajjadi contributed equally to this work. Updated with results for three additional scenes

A Generalization of Otsu's Method and Minimum Error Thresholding

Authors: Jonathan T. Barron

Abstract: We present Generalized Histogram Thresholding (GHT), a simple, fast, and effective technique for histogram-based image thresholding. GHT works by performing approximate maximum a posteriori estimation of a mixture of Gaussians with appropriate priors. We demonstrate that GHT subsumes three classic thresholding techniques as special cases: Otsu's method, Minimum Error Thresholding (MET), and weight… ▽ More We present Generalized Histogram Thresholding (GHT), a simple, fast, and effective technique for histogram-based image thresholding. GHT works by performing approximate maximum a posteriori estimation of a mixture of Gaussians with appropriate priors. We demonstrate that GHT subsumes three classic thresholding techniques as special cases: Otsu's method, Minimum Error Thresholding (MET), and weighted percentile thresholding. GHT thereby enables the continuous interpolation between those three algorithms, which allows thresholding accuracy to be improved significantly. GHT also provides a clarifying interpretation of the common practice of coarsening a histogram's bin width during thresholding. We show that GHT outperforms or matches the performance of all algorithms on a recent challenge for handwritten document image binarization (including deep neural networks trained to produce per-pixel binarizations), and can be implemented in a dozen lines of code or as a trivial modification to Otsu's method or MET. △ Less

Submitted 18 August, 2020; v1 submitted 14 July, 2020; originally announced July 2020.

Comments: ECCV 2020

On the Origin of Long-Lived Particles

Authors: Jared Barron, David Curtin

Abstract: MATHUSLA is a proposed large-volume displaced vertex (DV) detector, situated on the surface above CMS and designed to search for long-lived particles (LLPs) produced at the HL-LHC. We show that a discovery of LLPs at MATHUSLA would not only prove the existence of BSM physics, it would also uncover the theoretical origin of the LLPs, despite the fact that MATHUSLA gathers no energy or momentum info… ▽ More MATHUSLA is a proposed large-volume displaced vertex (DV) detector, situated on the surface above CMS and designed to search for long-lived particles (LLPs) produced at the HL-LHC. We show that a discovery of LLPs at MATHUSLA would not only prove the existence of BSM physics, it would also uncover the theoretical origin of the LLPs, despite the fact that MATHUSLA gathers no energy or momentum information on the LLP decay products. Our analysis is simple and robust, making it easily generalizable to include more complex LLP scenarios, and our methods are applicable to LLP decays discovered in ATLAS, CMS, LHCb, or other external detectors. In the event of an LLP detection, MATHUSLA can act as a Level-1 trigger for the main detector, guaranteeing that the LLP production event is read out at CMS. We perform an LLP simplified model analysis to show that combining information from the MATHUSLA and CMS detectors would allow the LLP production mode topology to be determined with as few as $\sim 100$ observed LLP decays. Underlying theory parameters, like the LLP and parent particle masses, can also be measured with $\lesssim 10\%$ precision. Together with information on the LLP decay mode from the geometric properties of the observed DV, it is clear that MATHUSLA and CMS together will be able to characterize any newly discovered physics in great detail. △ Less

Submitted 1 November, 2021; v1 submitted 10 July, 2020; originally announced July 2020.

Comments: 35 pages, 14 figures plus references. v2: updated grant information

Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains

Authors: Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, Ren Ng

Abstract: We show that passing input points through a simple Fourier feature map** enables a multilayer perceptron (MLP) to learn high-frequency functions in low-dimensional problem domains. These results shed light on recent advances in computer vision and graphics that achieve state-of-the-art results by using MLPs to represent complex 3D objects and scenes. Using tools from the neural tangent kernel (N… ▽ More We show that passing input points through a simple Fourier feature map** enables a multilayer perceptron (MLP) to learn high-frequency functions in low-dimensional problem domains. These results shed light on recent advances in computer vision and graphics that achieve state-of-the-art results by using MLPs to represent complex 3D objects and scenes. Using tools from the neural tangent kernel (NTK) literature, we show that a standard MLP fails to learn high frequencies both in theory and in practice. To overcome this spectral bias, we use a Fourier feature map** to transform the effective NTK into a stationary kernel with a tunable bandwidth. We suggest an approach for selecting problem-specific Fourier features that greatly improves the performance of MLPs for low-dimensional regression tasks relevant to the computer vision and graphics communities. △ Less

Submitted 18 June, 2020; originally announced June 2020.

Comments: Project page: https://people.eecs.berkeley.edu/~bmild/fourfeat/

Sky Optimization: Semantically aware image processing of skies in low-light photography

Authors: Orly Liba, Longqi Cai, Yun-Ta Tsai, Elad Eban, Yair Movshovitz-Attias, Yael Pritch, Huizhong Chen, Jonathan T. Barron

Abstract: The sky is a major component of the appearance of a photograph, and its color and tone can strongly influence the mood of a picture. In nighttime photography, the sky can also suffer from noise and color artifacts. For this reason, there is a strong desire to process the sky in isolation from the rest of the scene to achieve an optimal look. In this work, we propose an automated method, which can… ▽ More The sky is a major component of the appearance of a photograph, and its color and tone can strongly influence the mood of a picture. In nighttime photography, the sky can also suffer from noise and color artifacts. For this reason, there is a strong desire to process the sky in isolation from the rest of the scene to achieve an optimal look. In this work, we propose an automated method, which can run as a part of a camera pipeline, for creating accurate sky alpha-masks and using them to improve the appearance of the sky. Our method performs end-to-end sky optimization in less than half a second per image on a mobile device. We introduce a method for creating an accurate sky-mask dataset that is based on partially annotated images that are inpainted and refined by our modified weighted guided filter. We use this dataset to train a neural network for semantic sky segmentation. Due to the compute and power constraints of mobile devices, sky segmentation is performed at a low image resolution. Our modified weighted guided filter is used for edge-aware upsampling to resize the alpha-mask to a higher resolution. With this detailed mask we automatically apply post-processing steps to the sky in isolation, such as automatic spatially varying white-balance, brightness adjustments, contrast enhancement, and noise reduction. △ Less

Submitted 15 June, 2020; originally announced June 2020.

Comments: Published in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 2020

What Matters in Unsupervised Optical Flow

Authors: Rico Jonschkowski, Austin Stone, Jonathan T. Barron, Ariel Gordon, Kurt Konolige, Anelia Angelova

Abstract: We systematically compare and analyze a set of key components in unsupervised optical flow to identify which photometric loss, occlusion handling, and smoothness regularization is most effective. Alongside this investigation we construct a number of novel improvements to unsupervised flow models, such as cost volume normalization, stop** the gradient at the occlusion mask, encouraging smoothness… ▽ More We systematically compare and analyze a set of key components in unsupervised optical flow to identify which photometric loss, occlusion handling, and smoothness regularization is most effective. Alongside this investigation we construct a number of novel improvements to unsupervised flow models, such as cost volume normalization, stop** the gradient at the occlusion mask, encouraging smoothness before upsampling the flow field, and continual self-supervision with image resizing. By combining the results of our investigation with our improved model components, we are able to present a new unsupervised flow technique that significantly outperforms the previous unsupervised state-of-the-art and performs on par with supervised FlowNet2 on the KITTI 2015 dataset, while also being significantly simpler than related approaches. △ Less

Submitted 14 August, 2020; v1 submitted 8 June, 2020; originally announced June 2020.

Comments: Accepted at ECCV 2020 (Oral). Source code is available at https://github.com/google-research/google-research/tree/master/uflow

Portrait Shadow Manipulation

Authors: Xuaner Cecilia Zhang, Jonathan T. Barron, Yun-Ta Tsai, Rohit Pandey, Xiuming Zhang, Ren Ng, David E. Jacobs

Abstract: Casually-taken portrait photographs often suffer from unflattering lighting and shadowing because of suboptimal conditions in the environment. Aesthetic qualities such as the position and softness of shadows and the lighting ratio between the bright and dark parts of the face are frequently determined by the constraints of the environment rather than by the photographer. Professionals address this… ▽ More Casually-taken portrait photographs often suffer from unflattering lighting and shadowing because of suboptimal conditions in the environment. Aesthetic qualities such as the position and softness of shadows and the lighting ratio between the bright and dark parts of the face are frequently determined by the constraints of the environment rather than by the photographer. Professionals address this issue by adding light sha** tools such as scrims, bounce cards, and flashes. In this paper, we present a computational approach that gives casual photographers some of this control, thereby allowing poorly-lit portraits to be relit post-capture in a realistic and easily-controllable way. Our approach relies on a pair of neural networks---one to remove foreign shadows cast by external objects, and another to soften facial shadows cast by the features of the subject and to add a synthetic fill light to improve the lighting ratio. To train our first network we construct a dataset of real-world portraits wherein synthetic foreign shadows are rendered onto the face, and we show that our network learns to remove those unwanted shadows. To train our second network we use a dataset of Light Stage scans of human subjects to construct input/output pairs of input images harshly lit by a small light source, and variably softened and fill-lit output images of each face. We propose a way to explicitly encode facial symmetry and show that our dataset and training procedure enable the model to generalize to images taken in the wild. Together, these networks enable the realistic and aesthetically pleasing enhancement of shadows and lights in real-world portrait images △ Less

Submitted 20 May, 2020; v1 submitted 18 May, 2020; originally announced May 2020.

Comments: (updated version); SIGGRAPH 2020;Project webpage: https://people.eecs.berkeley.edu/~cecilia77/project-pages/portrait Video: https://youtu.be/M_qYTXhzyac

Learning to Autofocus

Authors: Charles Herrmann, Richard Strong Bowen, Neal Wadhwa, Rahul Garg, Qiurui He, Jonathan T. Barron, Ramin Zabih

Abstract: Autofocus is an important task for digital cameras, yet current approaches often exhibit poor performance. We propose a learning-based approach to this problem, and provide a realistic dataset of sufficient size for effective learning. Our dataset is labeled with per-pixel depths obtained from multi-view stereo, following "Learning single camera depth estimation using dual-pixels". Using this data… ▽ More Autofocus is an important task for digital cameras, yet current approaches often exhibit poor performance. We propose a learning-based approach to this problem, and provide a realistic dataset of sufficient size for effective learning. Our dataset is labeled with per-pixel depths obtained from multi-view stereo, following "Learning single camera depth estimation using dual-pixels". Using this dataset, we apply modern deep classification models and an ordinal regression loss to obtain an efficient learning-based autofocus technique. We demonstrate that our approach provides a significant improvement compared with previous learned and non-learned methods: our model reduces the mean absolute error by a factor of 3.6 over the best comparable baseline algorithm. Our dataset and code are publicly available. △ Less

Submitted 2 May, 2020; v1 submitted 25 April, 2020; originally announced April 2020.

Comments: CVPR 2020

NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis

Authors: Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, Ren Ng

Abstract: We present a method that achieves state-of-the-art results for synthesizing novel views of complex scenes by optimizing an underlying continuous volumetric scene function using a sparse set of input views. Our algorithm represents a scene using a fully-connected (non-convolutional) deep network, whose input is a single continuous 5D coordinate (spatial location $(x,y,z)$ and viewing direction… ▽ More We present a method that achieves state-of-the-art results for synthesizing novel views of complex scenes by optimizing an underlying continuous volumetric scene function using a sparse set of input views. Our algorithm represents a scene using a fully-connected (non-convolutional) deep network, whose input is a single continuous 5D coordinate (spatial location $(x,y,z)$ and viewing direction $(θ, φ)$) and whose output is the volume density and view-dependent emitted radiance at that spatial location. We synthesize views by querying 5D coordinates along camera rays and use classic volume rendering techniques to project the output colors and densities into an image. Because volume rendering is naturally differentiable, the only input required to optimize our representation is a set of images with known camera poses. We describe how to effectively optimize neural radiance fields to render photorealistic novel views of scenes with complicated geometry and appearance, and demonstrate results that outperform prior work on neural rendering and view synthesis. View synthesis results are best viewed as videos, so we urge readers to view our supplementary video for convincing comparisons. △ Less

Submitted 3 August, 2020; v1 submitted 19 March, 2020; originally announced March 2020.

Comments: ECCV 2020 (oral). Project page with videos and code: http://tancik.com/nerf

Lighthouse: Predicting Lighting Volumes for Spatially-Coherent Illumination

Authors: Pratul P. Srinivasan, Ben Mildenhall, Matthew Tancik, Jonathan T. Barron, Richard Tucker, Noah Snavely

Abstract: We present a deep learning solution for estimating the incident illumination at any 3D location within a scene from an input narrow-baseline stereo image pair. Previous approaches for predicting global illumination from images either predict just a single illumination for the entire scene, or separately estimate the illumination at each 3D location without enforcing that the predictions are consis… ▽ More We present a deep learning solution for estimating the incident illumination at any 3D location within a scene from an input narrow-baseline stereo image pair. Previous approaches for predicting global illumination from images either predict just a single illumination for the entire scene, or separately estimate the illumination at each 3D location without enforcing that the predictions are consistent with the same 3D scene. Instead, we propose a deep learning model that estimates a 3D volumetric RGBA model of a scene, including content outside the observed field of view, and then uses standard volume rendering to estimate the incident illumination at any 3D location within that volume. Our model is trained without any ground truth 3D data and only requires a held-out perspective view near the input stereo pair and a spherical panorama taken within each scene as supervision, as opposed to prior methods for spatially-varying lighting estimation, which require ground truth scene geometry for training. We demonstrate that our method can predict consistent spatially-varying lighting that is convincing enough to plausibly relight and insert highly specular virtual objects into real images. △ Less

Submitted 13 May, 2020; v1 submitted 18 March, 2020; originally announced March 2020.

Comments: CVPR 2020. Project page: https://people.eecs.berkeley.edu/~pratul/lighthouse/ [Updates: typos corrected]

MOBSTER: Identifying Candidate Magnetic O Stars through Rotational Modulation of TESS Photometry

Authors: James Barron, Gregg A. Wade, Dominic M. Bowman, Alexandre David-Uraz, Melissa S. Munoz, Herbert Pablo, Sergio Simón-Díaz

Abstract: Being relatively rare, the properties of magnetic O stars are not fully understood. To date fewer than a dozen of these stars have been confirmed, making any inference of their global properties uncertain due to small number statistics. To better understand these objects it is necessary to increase the known sample. The MOBSTER collaboration aims to do this by identifying candidate magnetic O, B,… ▽ More Being relatively rare, the properties of magnetic O stars are not fully understood. To date fewer than a dozen of these stars have been confirmed, making any inference of their global properties uncertain due to small number statistics. To better understand these objects it is necessary to increase the known sample. The MOBSTER collaboration aims to do this by identifying candidate magnetic O, B, and A stars from the identification of rotational modulation in high-precision photometry from the Transiting Exoplanet Survey Satellite (TESS). Here we discuss the collaboration's efforts to detect rotational modulation in TESS targets to identify candidate magnetic O stars for future spectropolarimetric observations. △ Less

Submitted 13 January, 2020; originally announced January 2020.

Comments: 9 pages, 4 figures, Conference proceedings: "Stellar Magnetic Fields - A workshop in honour of the career and contributions of John Landstreet", London, Canada, July 2019 (to be published in the Proceedings of the Polish Astronomical Society)

Boundary Cues for 3D Object Shape Recovery

Authors: Kevin Karsch, Zicheng Liao, Jason Rock, Jonathan T. Barron, Derek Hoiem

Abstract: Early work in computer vision considered a host of geometric cues for both shape reconstruction and recognition. However, since then, the vision community has focused heavily on shading cues for reconstruction, and moved towards data-driven approaches for recognition. In this paper, we reconsider these perhaps overlooked "boundary" cues (such as self occlusions and folds in a surface), as well as… ▽ More Early work in computer vision considered a host of geometric cues for both shape reconstruction and recognition. However, since then, the vision community has focused heavily on shading cues for reconstruction, and moved towards data-driven approaches for recognition. In this paper, we reconsider these perhaps overlooked "boundary" cues (such as self occlusions and folds in a surface), as well as many other established constraints for shape reconstruction. In a variety of user studies and quantitative tasks, we evaluate how well these cues inform shape reconstruction (relative to each other) in terms of both shape quality and shape recognition. Our findings suggest many new directions for future research in shape reconstruction, such as automatic boundary cue detection and relaxing assumptions in shape from shading (e.g. orthographic projection, Lambertian surfaces). △ Less

Submitted 24 December, 2019; originally announced December 2019.

MOBSTER: Establishing a Picture of Magnetic Massive Stars as a Population

Authors: Alexandre David-Uraz, Coralie Neiner, James Sikora, James Barron, Dominic M. Bowman, Pınar Cerrahoğlu, David H. Cohen, Christiana Erba, Oleksandr Kobzar, Oleg Kochukhov, Véronique Petit, Matthew E. Shultz, Asif ud-Doula, Gregg A. Wade, the MOBSTER Collaboration

Abstract: Magnetic massive and intermediate-mass stars constitute a separate population whose properties are still not fully understood. Increasing the sample of known objects of this type would help answer fundamental questions regarding the origins and characteristics of their magnetic fields. The MOBSTER Collaboration seeks to identify candidate magnetic A, B and O stars and explore the incidence and ori… ▽ More Magnetic massive and intermediate-mass stars constitute a separate population whose properties are still not fully understood. Increasing the sample of known objects of this type would help answer fundamental questions regarding the origins and characteristics of their magnetic fields. The MOBSTER Collaboration seeks to identify candidate magnetic A, B and O stars and explore the incidence and origins of photometric rotational modulation using high-precision photometry from the Transiting Exoplanet Survey Satellite (\textit{TESS}) mission. In this contribution, we present an overview of our methods and planned targeted spectropolarimetric follow-up surveys. △ Less

Submitted 5 December, 2019; originally announced December 2019.

Comments: 7 pages, 1 figure, Conference proceedings: "Stellar Magnetic Fields - A workshop in honour of the career and contributions of John Landstreet", London, Canada, 2019 (to be published in the Proceedings of the Polish Astronomical Society)

Magnetic OB[A] Stars with TESS: probing their Evolutionary and Rotational properties -- The MOBSTER Collaboration

Authors: A. David-Uraz, C. Neiner, J. Sikora, J. Barron, D. M. Bowman, P. Cerrahoğlu, D. H. Cohen, C. Erba, V. Khalack, O. Kobzar, O. Kochukhov, H. Pablo, V. Petit, M. E. Shultz, A. ud-Doula, G. A. Wade, the MOBSTER Collaboration

Abstract: In this contribution, we present the MOBSTER Collaboration, a large community effort to leverage high-precision photometry from the Transiting Exoplanet Survey Satellite (\textit{TESS}) in order to characterize the variability of magnetic massive and intermediate-mass stars. These data can be used to probe the varying column density of magnetospheric plasma along the line of sight for OB stars, th… ▽ More In this contribution, we present the MOBSTER Collaboration, a large community effort to leverage high-precision photometry from the Transiting Exoplanet Survey Satellite (\textit{TESS}) in order to characterize the variability of magnetic massive and intermediate-mass stars. These data can be used to probe the varying column density of magnetospheric plasma along the line of sight for OB stars, thus improving our understanding of the interaction between surface magnetic fields and massive star winds. They can also be used to map out the brightness inhomogeneities present on the surfaces of Ap/Bp stars, informing present models of atomic diffusion in their atmospheres. Finally, we review our current and ongoing studies, which lead to new insights on this topic. △ Less

Submitted 4 December, 2019; v1 submitted 2 December, 2019; originally announced December 2019.

Comments: 4 pages, Conference proceedings: "Stars and their variability observed from space", Vienna, Austria, 2019

Handheld Mobile Photography in Very Low Light

Authors: Orly Liba, Kiran Murthy, Yun-Ta Tsai, Tim Brooks, Tianfan Xue, Nikhil Karnad, Qiurui He, Jonathan T. Barron, Dillon Sharlet, Ryan Geiss, Samuel W. Hasinoff, Yael Pritch, Marc Levoy

Abstract: Taking photographs in low light using a mobile phone is challenging and rarely produces pleasing results. Aside from the physical limits imposed by read noise and photon shot noise, these cameras are typically handheld, have small apertures and sensors, use mass-produced analog electronics that cannot easily be cooled, and are commonly used to photograph subjects that move, like children and pets.… ▽ More Taking photographs in low light using a mobile phone is challenging and rarely produces pleasing results. Aside from the physical limits imposed by read noise and photon shot noise, these cameras are typically handheld, have small apertures and sensors, use mass-produced analog electronics that cannot easily be cooled, and are commonly used to photograph subjects that move, like children and pets. In this paper we describe a system for capturing clean, sharp, colorful photographs in light as low as 0.3~lux, where human vision becomes monochromatic and indistinct. To permit handheld photography without flash illumination, we capture, align, and combine multiple frames. Our system employs "motion metering", which uses an estimate of motion magnitudes (whether due to handshake or moving objects) to identify the number of frames and the per-frame exposure times that together minimize both noise and motion blur in a captured burst. We combine these frames using robust alignment and merging techniques that are specialized for high-noise imagery. To ensure accurate colors in such low light, we employ a learning-based auto white balancing algorithm. To prevent the photographs from looking like they were shot in daylight, we use tone map** techniques inspired by illusionistic painting: increasing contrast, crushing shadows to black, and surrounding the scene with darkness. All of these processes are performed using the limited computational resources of a mobile device. Our system can be used by novice photographers to produce shareable pictures in a few seconds based on a single shutter press, even in environments so dim that humans cannot see clearly. △ Less

Submitted 23 October, 2019; originally announced October 2019.

Comments: 22 pages, 27 figures

Journal ref: ACM Trans. Graph.38, 6, Article 164 (November 2019)

A Deep Factorization of Style and Structure in Fonts

Authors: Nikita Srivatsan, Jonathan T. Barron, Dan Klein, Taylor Berg-Kirkpatrick

Abstract: We propose a deep factorization model for typographic analysis that disentangles content from style. Specifically, a variational inference procedure factors each training glyph into the combination of a character-specific content embedding and a latent font-specific style variable. The underlying generative model combines these factors through an asymmetric transpose convolutional process to gener… ▽ More We propose a deep factorization model for typographic analysis that disentangles content from style. Specifically, a variational inference procedure factors each training glyph into the combination of a character-specific content embedding and a latent font-specific style variable. The underlying generative model combines these factors through an asymmetric transpose convolutional process to generate the image of the glyph itself. When trained on corpora of fonts, our model learns a manifold over font styles that can be used to analyze or reconstruct new, unseen fonts. On the task of reconstructing missing glyphs from an unknown font given only a small number of observations, our model outperforms both a strong nearest neighbors baseline and a state-of-the-art discriminative model from prior work. △ Less

Submitted 15 May, 2020; v1 submitted 1 October, 2019; originally announced October 2019.

Comments: EMNLP 2019 (oral)

Single Image Portrait Relighting

Authors: Tiancheng Sun, Jonathan T. Barron, Yun-Ta Tsai, Zexiang Xu, Xueming Yu, Graham Fyffe, Christoph Rhemann, Jay Busch, Paul Debevec, Ravi Ramamoorthi

Abstract: Lighting plays a central role in conveying the essence and depth of the subject in a portrait photograph. Professional photographers will carefully control the lighting in their studio to manipulate the appearance of their subject, while consumer photographers are usually constrained to the illumination of their environment. Though prior works have explored techniques for relighting an image, thei… ▽ More Lighting plays a central role in conveying the essence and depth of the subject in a portrait photograph. Professional photographers will carefully control the lighting in their studio to manipulate the appearance of their subject, while consumer photographers are usually constrained to the illumination of their environment. Though prior works have explored techniques for relighting an image, their utility is usually limited due to requirements of specialized hardware, multiple images of the subject under controlled or known illuminations, or accurate models of geometry and reflectance. To this end, we present a system for portrait relighting: a neural network that takes as input a single RGB image of a portrait taken with a standard cellphone camera in an unconstrained environment, and from that image produces a relit image of that subject as though it were illuminated according to any provided environment map. Our method is trained on a small database of 18 individuals captured under different directional light sources in a controlled light stage setup consisting of a densely sampled sphere of lights. Our proposed technique produces quantitatively superior results on our dataset's validation set compared to prior works, and produces convincing qualitative relighting results on a dataset of hundreds of real-world cellphone portraits. Because our technique can produce a 640 $\times$ 640 image in only 160 milliseconds, it may enable interactive user-facing photographic applications in the future. △ Less

Submitted 2 May, 2019; originally announced May 2019.

Comments: SIGGRAPH 2019 Technical Paper accepted

Journal ref: ACM Transactions on Graphics (SIGGRAPH 2019) 38 (4)

Pushing the Boundaries of View Extrapolation with Multiplane Images

Authors: Pratul P. Srinivasan, Richard Tucker, Jonathan T. Barron, Ravi Ramamoorthi, Ren Ng, Noah Snavely

Abstract: We explore the problem of view synthesis from a narrow baseline pair of images, and focus on generating high-quality view extrapolations with plausible disocclusions. Our method builds upon prior work in predicting a multiplane image (MPI), which represents scene content as a set of RGB$α$ planes within a reference view frustum and renders novel views by projecting this content into the target vie… ▽ More We explore the problem of view synthesis from a narrow baseline pair of images, and focus on generating high-quality view extrapolations with plausible disocclusions. Our method builds upon prior work in predicting a multiplane image (MPI), which represents scene content as a set of RGB$α$ planes within a reference view frustum and renders novel views by projecting this content into the target viewpoints. We present a theoretical analysis showing how the range of views that can be rendered from an MPI increases linearly with the MPI disparity sampling frequency, as well as a novel MPI prediction procedure that theoretically enables view extrapolations of up to $4\times$ the lateral viewpoint movement allowed by prior work. Our method ameliorates two specific issues that limit the range of views renderable by prior methods: 1) We expand the range of novel views that can be rendered without depth discretization artifacts by using a 3D convolutional network architecture along with a randomized-resolution training procedure to allow our model to predict MPIs with increased disparity sampling frequency. 2) We reduce the repeated texture artifacts seen in disocclusions by enforcing a constraint that the appearance of hidden content at any depth must be drawn from visible content at or behind that depth. Please see our results video at: https://www.youtube.com/watch?v=aJqAaMNL2m4. △ Less

Submitted 1 May, 2019; originally announced May 2019.

Comments: Oral presentation at CVPR 2019

Learning Single Camera Depth Estimation using Dual-Pixels

Authors: Rahul Garg, Neal Wadhwa, Sameer Ansari, Jonathan T. Barron

Abstract: Deep learning techniques have enabled rapid progress in monocular depth estimation, but their quality is limited by the ill-posed nature of the problem and the scarcity of high quality datasets. We estimate depth from a single camera by leveraging the dual-pixel auto-focus hardware that is increasingly common on modern camera sensors. Classic stereo algorithms and prior learning-based depth estima… ▽ More Deep learning techniques have enabled rapid progress in monocular depth estimation, but their quality is limited by the ill-posed nature of the problem and the scarcity of high quality datasets. We estimate depth from a single camera by leveraging the dual-pixel auto-focus hardware that is increasingly common on modern camera sensors. Classic stereo algorithms and prior learning-based depth estimation techniques under-perform when applied on this dual-pixel data, the former due to too-strong assumptions about RGB image matching, and the latter due to not leveraging the understanding of optics of dual-pixel image formation. To allow learning based methods to work well on dual-pixel imagery, we identify an inherent ambiguity in the depth estimated from dual-pixel cues, and develop an approach to estimate depth up to this ambiguity. Using our approach, existing monocular depth estimation techniques can be effectively applied to dual-pixel data, and much smaller models can be constructed that still infer high quality depth. To demonstrate this, we capture a large dataset of in-the-wild 5-viewpoint RGB images paired with corresponding dual-pixel data, and show how view supervision with this data can be used to learn depth up to the unknown ambiguities. On our new task, our model is 30% more accurate than any prior work on learning-based monocular or stereoscopic depth estimation. △ Less

Submitted 14 August, 2019; v1 submitted 11 April, 2019; originally announced April 2019.

Comments: Accepted to ICCV 2019 (oral)

Stereoscopic Dark Flash for Low-light Photography

Authors: Jian Wang, Tianfan Xue, Jonathan T. Barron, Jiawen Chen

Abstract: In this work, we present a camera configuration for acquiring "stereoscopic dark flash" images: a simultaneous stereo pair in which one camera is a conventional RGB sensor, but the other camera is sensitive to near-infrared and near-ultraviolet instead of R and B. When paired with a "dark" flash (i.e., one having near-infrared and near-ultraviolet light, but no visible light) this camera allows us… ▽ More In this work, we present a camera configuration for acquiring "stereoscopic dark flash" images: a simultaneous stereo pair in which one camera is a conventional RGB sensor, but the other camera is sensitive to near-infrared and near-ultraviolet instead of R and B. When paired with a "dark" flash (i.e., one having near-infrared and near-ultraviolet light, but no visible light) this camera allows us to capture the two images in a flash/no-flash image pair at the same time, all while not disturbing any human subjects or onlookers with a dazzling visible flash. We present a hardware prototype of this camera that approximates an idealized camera, and we present an imaging procedure that let us acquire dark flash stereo pairs that closely resemble those we would get from that idealized camera. We then present a technique for fusing these stereo pairs, first by performing registration and war**, and then by using recent advances in hyperspectral image fusion and deep learning to produce a final image. Because our camera configuration and our data acquisition process allow us to capture true low-noise long exposure RGB images alongside our dark flash stereo pairs, our learned model can be trained end-to-end to produce a fused image that retains the color and tone of a real RGB image while having the low-noise properties of a flash image. △ Less

Submitted 8 January, 2019; v1 submitted 5 January, 2019; originally announced January 2019.

Comments: 9 pages, 8 figures

All-Sky Measurement of the Anisotropy of Cosmic Rays at 10 TeV and Map** of the Local Interstellar Magnetic Field

Authors: HAWC Collaboration, A. U. Abeysekara, R. Alfaro, C. Alvarez, J. D. Álvarez, R. Arceo, J. C. Arteaga-Velázquez, D. Avila Rojas, E. Belmont-Moreno, S. Y. BenZvi, C. Brisbois, T. Capistrán, A. Carramiñana, S. Casanova, U. Cotti, J. Cotzomi, J. C. Díaz-Vélez, C. De León, E. De la Fuente, S. Dichiara, M. A. DuVernois, C. Espinoza, D. W. Fiorino, H. Fleischhack, N. Fraija , et al. (382 additional authors not shown)

Abstract: We present the first full-sky analysis of the cosmic ray arrival direction distribution with data collected by the HAWC and IceCube observatories in the Northern and Southern hemispheres at the same median primary particle energy of 10 TeV. The combined sky map and angular power spectrum largely eliminate biases that result from partial sky coverage and holds a key to probe into the propagation pr… ▽ More We present the first full-sky analysis of the cosmic ray arrival direction distribution with data collected by the HAWC and IceCube observatories in the Northern and Southern hemispheres at the same median primary particle energy of 10 TeV. The combined sky map and angular power spectrum largely eliminate biases that result from partial sky coverage and holds a key to probe into the propagation properties of TeV cosmic rays through our local interstellar medium and the interaction between the interstellar and heliospheric magnetic fields. From the map we determine the horizontal dipole components of the anisotropy $δ_{0h} = 9.16 \times 10^{-4}$ and $δ_{6h} = 7.25 \times 10^{-4}~(\pm0.04 \times 10^{-4})$. In addition, we infer the direction ($229.2\pm 3.5^\circ$ RA , $11.4\pm 3.0^\circ$ Dec.) of the interstellar magnetic field from the boundary between large scale excess and deficit regions from which we estimate the missing corresponding vertical dipole component of the large scale anisotropy to be $δ_N \sim -3.97 ^{+1.0}_{-2.0} \times 10^{-4}$. △ Less

Submitted 24 January, 2019; v1 submitted 13 December, 2018; originally announced December 2018.

Comments: Accepted by The Astrophysical Journal

Journal ref: ApJ, 871, (2019) 96

Learning to Synthesize Motion Blur

Authors: Tim Brooks, Jonathan T. Barron

Abstract: We present a technique for synthesizing a motion blurred image from a pair of unblurred images captured in succession. To build this system we motivate and design a differentiable "line prediction" layer to be used as part of a neural network architecture, with which we can learn a system to regress from image pairs to motion blurred images that span the capture time of the input image pair. Train… ▽ More We present a technique for synthesizing a motion blurred image from a pair of unblurred images captured in succession. To build this system we motivate and design a differentiable "line prediction" layer to be used as part of a neural network architecture, with which we can learn a system to regress from image pairs to motion blurred images that span the capture time of the input image pair. Training this model requires an abundance of data, and so we design and execute a strategy for using frame interpolation techniques to generate a large-scale synthetic dataset of motion blurred images and their respective inputs. We additionally capture a high quality test set of real motion blurred images, synthesized from slow motion videos, with which we evaluate our model against several baseline techniques that can be used to synthesize motion blur. Our model produces higher accuracy output than our baselines, and is significantly faster than baselines with competitive accuracy. △ Less

Submitted 20 June, 2019; v1 submitted 27 November, 2018; originally announced November 2018.

Comments: http://timothybrooks.com/tech/motion-blur/ . IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019

Unprocessing Images for Learned Raw Denoising

Authors: Tim Brooks, Ben Mildenhall, Tianfan Xue, Jiawen Chen, Dillon Sharlet, Jonathan T. Barron

Abstract: Machine learning techniques work best when the data used for training resembles the data used for evaluation. This holds true for learned single-image denoising algorithms, which are applied to real raw camera sensor readings but, due to practical constraints, are often trained on synthetic image data. Though it is understood that generalizing from synthetic to real data requires careful considera… ▽ More Machine learning techniques work best when the data used for training resembles the data used for evaluation. This holds true for learned single-image denoising algorithms, which are applied to real raw camera sensor readings but, due to practical constraints, are often trained on synthetic image data. Though it is understood that generalizing from synthetic to real data requires careful consideration of the noise properties of image sensors, the other aspects of a camera's image processing pipeline (gain, color correction, tone map**, etc) are often overlooked, despite their significant effect on how raw measurements are transformed into finished images. To address this, we present a technique to "unprocess" images by inverting each step of an image processing pipeline, thereby allowing us to synthesize realistic raw sensor measurements from commonly available internet photos. We additionally model the relevant components of an image processing pipeline when evaluating our loss function, which allows training to be aware of all relevant photometric processing that will occur after denoising. By processing and unprocessing model outputs and training data in this way, we are able to train a simple convolutional neural network that has 14%-38% lower error rates and is 9x-18x faster than the previous state of the art on the Darmstadt Noise Dataset, and generalizes to sensors outside of that dataset as well. △ Less

Submitted 27 November, 2018; originally announced November 2018.

Comments: http://timothybrooks.com/tech/unprocessing/

Search for steady point-like sources in the astrophysical muon neutrino flux with 8 years of IceCube data

Authors: IceCube Collaboration, M. G. Aartsen, M. Ackermann, J. Adams, J. A. Aguilar, M. Ahlers, M. Ahrens, D. Altmann, K. Andeen, T. Anderson, I. Ansseau, G. Anton, C. Argüelles, J. Auffenberg, S. Axani, P. Backes, H. Bagherpour, X. Bai, A. Barbano, J. P. Barron, S. W. Barwick, V. Baum, R. Bay, J. J. Beatty, J. Becker Tjus , et al. (304 additional authors not shown)

Abstract: The IceCube Collaboration has observed a high-energy astrophysical neutrino flux and recently found evidence for neutrino emission from the blazar TXS 0506+056. These results open a new window into the high-energy universe. However, the source or sources of most of the observed flux of astrophysical neutrinos remains uncertain. Here, a search for steady point-like neutrino sources is performed usi… ▽ More The IceCube Collaboration has observed a high-energy astrophysical neutrino flux and recently found evidence for neutrino emission from the blazar TXS 0506+056. These results open a new window into the high-energy universe. However, the source or sources of most of the observed flux of astrophysical neutrinos remains uncertain. Here, a search for steady point-like neutrino sources is performed using an unbinned likelihood analysis. The method searches for a spatial accumulation of muon-neutrino events using the very high-statistics sample of about $497\,000$ neutrinos recorded by IceCube between 2009 and 2017. The median angular resolution is $\sim1^\circ$ at 1 TeV and improves to $\sim0.3^\circ$ for neutrinos with an energy of 1 PeV. Compared to previous analyses, this search is optimized for point-like neutrino emission with the same flux-characteristics as the observed astrophysical muon-neutrino flux and introduces an improved event-reconstruction and parametrization of the background. The result is an improvement in sensitivity to the muon-neutrino flux compared to the previous analysis of $\sim35\%$ assuming an $E^{-2}$ spectrum. The sensitivity on the muon-neutrino flux is at a level of $E^2 \mathrm{d} N /\mathrm{d} E = 3\cdot 10^{-13}\,\mathrm{TeV}\,\mathrm{cm}^{-2}\,\mathrm{s}^{-1}$. No new evidence for neutrino sources is found in a full sky scan and in an a priori candidate source list that is motivated by gamma-ray observations. Furthermore, no significant excesses above background are found from populations of sub-threshold sources. The implications of the non-observation for potential source classes are discussed. △ Less

Submitted 16 February, 2019; v1 submitted 19 November, 2018; originally announced November 2018.

Journal ref: Eur. Phys. J. C (2019) 79: 234

Detection of the Temporal Variation of the Sun's Cosmic Ray Shadow with the IceCube Detector

Authors: IceCube Collaboration, M. G. Aartsen, M. Ackermann, J. Adams, J. A. Aguilar, M. Ahlers, M. Ahrens, D. Altmann, K. Andeen, T. Anderson, I. Ansseau, G. Anton, C. Argüelles, J. Auffenberg, S. Axani, P. Backes, H. Bagherpour, X. Bai, A. Barbano, J. P. Barron, S. W. Barwick, V. Baum, R. Bay, J. J. Beatty, J. Becker Tjus , et al. (305 additional authors not shown)

Abstract: We report on the observation of a deficit in the cosmic ray flux from the directions of the Moon and Sun with five years of data taken by the IceCube Neutrino Observatory. Between May 2010 and May 2011 the IceCube detector operated with 79 strings deployed in the glacial ice at the South Pole, and with 86 strings between May 2011 and May 2015. A binned analysis is used to measure the relative defi… ▽ More We report on the observation of a deficit in the cosmic ray flux from the directions of the Moon and Sun with five years of data taken by the IceCube Neutrino Observatory. Between May 2010 and May 2011 the IceCube detector operated with 79 strings deployed in the glacial ice at the South Pole, and with 86 strings between May 2011 and May 2015. A binned analysis is used to measure the relative deficit and significance of the cosmic ray shadows. Both the cosmic ray Moon and Sun shadows are detected with high statistical significance ($>10σ$) for each year. The results for the Moon shadow are consistent with previous analyses and verify the stability of the IceCube detector over time. This work represents the first observation of the Sun shadow with the IceCube detector. We show that the cosmic ray shadow of the Sun varies with time. These results open the possibility to study cosmic ray transport near the Sun with future data from IceCube. △ Less

Submitted 21 February, 2019; v1 submitted 5 November, 2018; originally announced November 2018.

Comments: 14 pages, 23 figures; published in the Astrophysical Journal

Journal ref: ApJ, 872, (2019) 133

Search for Multi-messenger Sources of Gravitational Waves and High-energy Neutrinos with Advanced LIGO during its first Observing Run, ANTARES and IceCube

Authors: ANTARES, IceCube, LIGO, Virgo Collaborations, :, A. Albert, M. Andre, M. Anghinolfi, M. Ardid, J. -J. Aubert, J. Aublin, T. Avgitas, B. Baret, J. Barrios-Marti, S. Basa, B. Belhorma, V. Bertin, S. Biagi, R. Bormuth, J. Boumaaza, S. Bourret, M. C. Bouwhuis, H. Brânzas, R. Bruijn, J. Brunner , et al. (1570 additional authors not shown)

Abstract: Astrophysical sources of gravitational waves, such as binary neutron star and black hole mergers or core-collapse supernovae, can drive relativistic outflows, giving rise to non-thermal high-energy emission. High-energy neutrinos are signatures of such outflows. The detection of gravitational waves and high-energy neutrinos from common sources could help establish the connection between the dynami… ▽ More Astrophysical sources of gravitational waves, such as binary neutron star and black hole mergers or core-collapse supernovae, can drive relativistic outflows, giving rise to non-thermal high-energy emission. High-energy neutrinos are signatures of such outflows. The detection of gravitational waves and high-energy neutrinos from common sources could help establish the connection between the dynamics of the progenitor and the properties of the outflow. We searched for associated emission of gravitational waves and high-energy neutrinos from astrophysical transients with minimal assumptions using data from Advanced LIGO from its first observing run O1, and data from the ANTARES and IceCube neutrino observatories from the same time period. We focused on candidate events whose astrophysical origin could not be determined from a single messenger. We found no significant coincident candidate, which we used to constrain the rate density of astrophysical sources dependent on their gravitational wave and neutrino emission processes. △ Less

Submitted 15 November, 2018; v1 submitted 24 October, 2018; originally announced October 2018.

Comments: 20 pages, 2 figures

Journal ref: Astrophys.J. 870 (2019) 134

Measurements using the inelasticity distribution of multi-TeV neutrino interactions in IceCube

Authors: IceCube Collaboration, M. G. Aartsen, M. Ackermann, J. Adams, J. A. Aguilar, M. Ahlers, M. Ahrens, I. Al Samarai, D. Altmann, K. Andeen, T. Anderson, I. Ansseau, G. Anton, C. Argüelles, J. Auffenberg, S. Axani, P. Backes, H. Bagherpour, X. Bai, A. Barbano, J. P. Barron, S. W. Barwick, V. Baum, R. Bay, J. J. Beatty , et al. (309 additional authors not shown)

Abstract: Inelasticity--the fraction of a neutrino's energy transferred to hadrons--is a quantity of interest in the study of astrophysical and atmospheric neutrino interactions at multi-TeV energies with IceCube. In this work, a sample of contained neutrino interactions in IceCube is obtained from 5 years of data and classified as 2650 tracks and 965 cascades. Tracks arise predominantly from charged-curren… ▽ More Inelasticity--the fraction of a neutrino's energy transferred to hadrons--is a quantity of interest in the study of astrophysical and atmospheric neutrino interactions at multi-TeV energies with IceCube. In this work, a sample of contained neutrino interactions in IceCube is obtained from 5 years of data and classified as 2650 tracks and 965 cascades. Tracks arise predominantly from charged-current $ν_μ$ interactions, and we demonstrate that we can reconstruct their energy and inelasticity. The inelasticity distribution is found to be consistent with the calculation of Cooper-Sarkar et al. across the energy range from $\sim$ 1 TeV to $\sim$ 100 TeV. Along with cascades from neutrinos of all flavors, we also perform a fit over the energy, zenith angle, and inelasticity distribution to characterize the flux of astrophysical and atmospheric neutrinos. The energy spectrum of diffuse astrophysical neutrinos is well-described by a power-law in both track and cascade samples, and a best-fit index $γ=2.62\pm0.07$ is found in the energy range from 3.5 TeV to 2.6 PeV. Limits are set on the astrophysical flavor composition that are compatible with a ratio of $\left(\frac{1}{3}:\frac{1}{3}:\frac{1}{3}\right)_{\oplus}$. Exploiting the distinct inelasticity distribution of $ν_μ$ and $\barν_μ$ interactions, the atmospheric $ν_μ$ to $\barν_μ$ flux ratio in the energy range from 770 GeV to 21 TeV is found to be $0.77^{+0.44}_{-0.25}$ times the calculation by Honda et al. Lastly, the inelasticity distribution is also sensitive to neutrino charged-current charm production. The data are consistent with a leading-order calculation, with zero charm production excluded at $91\%$ confidence level. Future analyses of inelasticity distributions may probe new physics that affects neutrino interactions both in and beyond the Standard Model. △ Less

Submitted 24 February, 2019; v1 submitted 23 August, 2018; originally announced August 2018.

Comments: 20 pages, 13 figures

Journal ref: Phys. Rev. D 99, 032004 (2019)

Joint constraints on Galactic diffuse neutrino emission from ANTARES and IceCube

Authors: A. Albert, M. André, M. Anghinolfi, M. Ardid, J. -J. Aubert, J. Aublin, T. Avgitas, B. Baret, J. Barrios-Martí, S. Basa, B. Belhorma, V. Bertin, S. Biagi, R. Bormuth, J. Boumaaza, S. Bourret, M. C. Bouwhuis, H. Brânzaş, R. Bruijn, J. Brunner, J. Busto, A. Capone, L. Caramete, J. Carr, S. Celli , et al. (434 additional authors not shown)

Abstract: The existence of diffuse Galactic neutrino production is expected from cosmic ray interactions with Galactic gas and radiation fields. Thus, neutrinos are a unique messenger offering the opportunity to test the products of Galactic cosmic ray interactions up to energies of hundreds of TeV. Here we present a search for this production using ten years of ANTARES track and shower data, as well as sev… ▽ More The existence of diffuse Galactic neutrino production is expected from cosmic ray interactions with Galactic gas and radiation fields. Thus, neutrinos are a unique messenger offering the opportunity to test the products of Galactic cosmic ray interactions up to energies of hundreds of TeV. Here we present a search for this production using ten years of ANTARES track and shower data, as well as seven years of IceCube track data. The data are combined into a joint likelihood test for neutrino emission according to the KRA$_γ$ model assuming a 5 PeV per nucleon Galactic cosmic ray cutoff. No significant excess is found. As a consequence, the limits presented in this work start constraining the model parameter space for Galactic cosmic ray production and transport. △ Less

Submitted 14 November, 2018; v1 submitted 10 August, 2018; originally announced August 2018.

Comments: 5 figures

Journal ref: Astrophys.J. 868 (2018) no.2, L20

Constraints on Minute-Scale Transient Astrophysical Neutrino Sources

Authors: M. G. Aartsen, M. Ackermann, J. Adams, J. A. Aguilar, M. Ahlers, M. Ahrens, I. Al Samarai, D. Altmann, K. Andeen, T. Anderson, I. Ansseau, G. Anton, C. Argüelles, J. Auffenberg, S. Axani, P. Backes, H. Bagherpour, X. Bai, A. Barbano, J. P. Barron, S. W. Barwick, V. Baum, R. Bay, J. J. Beatty, J. Becker Tjus , et al. (309 additional authors not shown)

Abstract: High-energy neutrino emission has been predicted for several short-lived astrophysical transients including gamma-ray bursts (GRBs), core-collapse supernovae with choked jets and neutron star mergers. IceCube's optical and X-ray follow-up program searches for such transient sources by looking for two or more muon neutrino candidates in directional coincidence and arriving within 100s. The measured… ▽ More High-energy neutrino emission has been predicted for several short-lived astrophysical transients including gamma-ray bursts (GRBs), core-collapse supernovae with choked jets and neutron star mergers. IceCube's optical and X-ray follow-up program searches for such transient sources by looking for two or more muon neutrino candidates in directional coincidence and arriving within 100s. The measured rate of neutrino alerts is consistent with the expected rate of chance coincidences of atmospheric background events and no likely electromagnetic counterparts have been identified in Swift follow-up observations. Here, we calculate generic bounds on the neutrino flux of short-lived transient sources. Assuming an $E^{-2.5}$ neutrino spectrum, we find that the neutrino flux of rare sources, like long gamma-ray bursts, is constrained to <5% of the detected astrophysical flux and the energy released in neutrinos (100GeV to 10PeV) by a median bright GRB-like source is $<10^{52.5}$erg. For a harder $E^{-2.13}$ neutrino spectrum up to 30% of the flux could be produced by GRBs and the allowed median source energy is $< 10^{52}$erg. A hypothetical population of transient sources has to be more common than $10^{-5}\text{Mpc}^{-3}\text{yr}^{-1}$ ($5\times10^{-8}\text{Mpc}^{-3}\text{yr}^{-1}$ for the $E^{-2.13}$ spectrum) to account for the complete astrophysical neutrino flux. △ Less

Submitted 27 February, 2019; v1 submitted 30 July, 2018; originally announced July 2018.

Comments: published in PRL, table with neutrino alerts in ancillary file

Journal ref: Phys. Rev. Lett. 122, 051102 (2019)

Differential limit on the extremely-high-energy cosmic neutrino flux in the presence of astrophysical background from nine years of IceCube data

Authors: IceCube Collaboration, M. G. Aartsen, M. Ackermann, J. Adams, J. A. Aguilar, M. Ahlers, M. Ahrens, I. Al Samarai, D. Altmann, K. Andeen, T. Anderson, I. Ansseau, G. Anton, C. Argüelles, J. Auffenberg, S. Axani, P. Backes, H. Bagherpour, X. Bai, A. Barbano, J. P. Barron, S. W. Barwick, V. Baum, R. Bay, J. J. Beatty , et al. (309 additional authors not shown)

Abstract: We report a quasi-differential upper limit on the extremely-high-energy (EHE) neutrino flux above $5\times 10^{6}$ GeV based on an analysis of nine years of IceCube data. The astrophysical neutrino flux measured by IceCube extends to PeV energies, and it is a background flux when searching for an independent signal flux at higher energies, such as the cosmogenic neutrino signal. We have developed… ▽ More We report a quasi-differential upper limit on the extremely-high-energy (EHE) neutrino flux above $5\times 10^{6}$ GeV based on an analysis of nine years of IceCube data. The astrophysical neutrino flux measured by IceCube extends to PeV energies, and it is a background flux when searching for an independent signal flux at higher energies, such as the cosmogenic neutrino signal. We have developed a new method to place robust limits on the EHE neutrino flux in the presence of an astrophysical background, whose spectrum has yet to be understood with high precision at PeV energies. A distinct event with a deposited energy above $10^{6}$ GeV was found in the new two-year sample, in addition to the one event previously found in the seven-year EHE neutrino search. These two events represent a neutrino flux that is incompatible with predictions for a cosmogenic neutrino flux and are considered to be an astrophysical background in the current study. The obtained limit is the most stringent to date in the energy range between $5 \times 10^{6}$ and $5 \times 10^{10}$ GeV. This result constrains neutrino models predicting a three-flavor neutrino flux of $E_ν^2φ_{ν_e+ν_μ+ν_τ}\simeq2\times 10^{-8}\ {\rm GeV}/{\rm cm}^2\ \sec\ {\rm sr}$ at $10^9\ {\rm GeV}$. A significant part of the parameter-space for EHE neutrino production scenarios assuming a proton-dominated composition of ultra-high-energy cosmic rays is excluded. △ Less

Submitted 4 September, 2018; v1 submitted 4 July, 2018; originally announced July 2018.

Comments: The version accepted for publication in Physical Review D

Journal ref: Phys. Rev. D 98, 062003 (2018)

Synthetic Depth-of-Field with a Single-Camera Mobile Phone

Authors: Neal Wadhwa, Rahul Garg, David E. Jacobs, Bryan E. Feldman, Nori Kanazawa, Robert Carroll, Yair Movshovitz-Attias, Jonathan T. Barron, Yael Pritch, Marc Levoy

Abstract: Shallow depth-of-field is commonly used by photographers to isolate a subject from a distracting background. However, standard cell phone cameras cannot produce such images optically, as their short focal lengths and small apertures capture nearly all-in-focus images. We present a system to computationally synthesize shallow depth-of-field images with a single mobile camera and a single button pre… ▽ More Shallow depth-of-field is commonly used by photographers to isolate a subject from a distracting background. However, standard cell phone cameras cannot produce such images optically, as their short focal lengths and small apertures capture nearly all-in-focus images. We present a system to computationally synthesize shallow depth-of-field images with a single mobile camera and a single button press. If the image is of a person, we use a person segmentation network to separate the person and their accessories from the background. If available, we also use dense dual-pixel auto-focus hardware, effectively a 2-sample light field with an approximately 1 millimeter baseline, to compute a dense depth map. These two signals are combined and used to render a defocused image. Our system can process a 5.4 megapixel image in 4 seconds on a mobile phone, is fully automatic, and is robust enough to be used by non-experts. The modular nature of our system allows it to degrade naturally in the absence of a dual-pixel sensor or a human subject. △ Less

Submitted 11 June, 2018; originally announced June 2018.

Comments: Accepted to SIGGRAPH 2018. Basis for Portrait Mode on Google Pixel 2 and Pixel 2 XL

Search for neutrinos from decaying dark matter with IceCube

Authors: IceCube Collaboration, M. G. Aartsen, M. Ackermann, J. Adams, J. A. Aguilar, M. Ahlers, M. Ahrens, I. Al Samarai, D. Altmann, K. Andeen, T. Anderson, I. Ansseau, G. Anton, C. Argüelles, J. Auffenberg, S. Axani, P. Backes, H. Bagherpour, X. Bai, J. P. Barron, S. W. Barwick, V. Baum, R. Bay, J. J. Beatty, J. Becker Tjus , et al. (306 additional authors not shown)

Abstract: With the observation of high-energy astrophysical neutrinos by the IceCube Neutrino Observatory, interest has risen in models of PeV-mass decaying dark matter particles to explain the observed flux. We present two dedicated experimental analyses to test this hypothesis. One analysis uses six years of IceCube data focusing on muon neutrino 'track' events from the Northern Hemisphere, while the seco… ▽ More With the observation of high-energy astrophysical neutrinos by the IceCube Neutrino Observatory, interest has risen in models of PeV-mass decaying dark matter particles to explain the observed flux. We present two dedicated experimental analyses to test this hypothesis. One analysis uses six years of IceCube data focusing on muon neutrino 'track' events from the Northern Hemisphere, while the second analysis uses two years of 'cascade' events from the full sky. Known background components and the hypothetical flux from unstable dark matter are fitted to the experimental data. Since no significant excess is observed in either analysis, lower limits on the lifetime of dark matter particles are derived: We obtain the strongest constraint to date, excluding lifetimes shorter than $10^{28}\,$s at $90\%$ CL for dark matter masses above $10\,$TeV. △ Less

Submitted 18 October, 2018; v1 submitted 11 April, 2018; originally announced April 2018.

Comments: 8 pages, 8 fgures

Journal ref: Eur. Phys. J. C (2018) 78: 831

Computational Techniques for the Analysis of Small Signals in High-Statistics Neutrino Oscillation Experiments

Authors: IceCube Collaboration, M. G. Aartsen, M. Ackermann, J. Adams, J. A. Aguilar, M. Ahlers, M. Ahrens, I. Al Samarai, D. Altmann, K. Andeen, T. Anderson, I. Ansseau, G. Anton, C. Argüelles, T. C. Arlen, J. Auffenberg, S. Axani, H. Bagherpour, X. Bai, A. Balagopal V., J. P. Barron, I. Bartos, S. W. Barwick, V. Baum, R. Bay , et al. (347 additional authors not shown)

Abstract: The current and upcoming generation of Very Large Volume Neutrino Telescopes---collecting unprecedented quantities of neutrino events---can be used to explore subtle effects in oscillation physics, such as (but not restricted to) the neutrino mass ordering. The sensitivity of an experiment to these effects can be estimated from Monte Carlo simulations. With the high number of events that will be c… ▽ More The current and upcoming generation of Very Large Volume Neutrino Telescopes---collecting unprecedented quantities of neutrino events---can be used to explore subtle effects in oscillation physics, such as (but not restricted to) the neutrino mass ordering. The sensitivity of an experiment to these effects can be estimated from Monte Carlo simulations. With the high number of events that will be collected, there is a trade-off between the computational expense of running such simulations and the inherent statistical uncertainty in the determined values. In such a scenario, it becomes impractical to produce and use adequately-sized sets of simulated events with traditional methods, such as Monte Carlo weighting. In this work we present a staged approach to the generation of binned event distributions in order to overcome these challenges. By combining multiple integration and smoothing techniques which address limited statistics from simulation it arrives at reliable analysis results using modest computational resources. △ Less

Submitted 4 December, 2019; v1 submitted 14 March, 2018; originally announced March 2018.

A Search for Neutrino Emission from Fast Radio Bursts with Six Years of IceCube Data

Authors: IceCube Collaboration, M. G. Aartsen, M. Ackermann, J. Adams, J. A. Aguilar, M. Ahlers, M. Ahrens, I. Al Samarai, D. Altmann, K. Andeen, T. Anderson, I. Ansseau, G. Anton, C. Argüelles, J. Auffenberg, S. Axani, H. Bagherpour, X. Bai, J. P. Barron, S. W. Barwick, V. Baum, R. Bay, J. J. Beatty, J. Becker Tjus, K. -H. Becker , et al. (299 additional authors not shown)

Abstract: We present a search for coincidence between IceCube TeV neutrinos and fast radio bursts (FRBs). During the search period from 2010 May 31 to 2016 May 12, a total of 29 FRBs with 13 unique locations have been detected in the whole sky. An unbinned maximum likelihood method was used to search for spatial and temporal coincidence between neutrinos and FRBs in expanding time windows, in both the north… ▽ More We present a search for coincidence between IceCube TeV neutrinos and fast radio bursts (FRBs). During the search period from 2010 May 31 to 2016 May 12, a total of 29 FRBs with 13 unique locations have been detected in the whole sky. An unbinned maximum likelihood method was used to search for spatial and temporal coincidence between neutrinos and FRBs in expanding time windows, in both the northern and southern hemispheres. No significant correlation was found in six years of IceCube data. Therefore, we set upper limits on neutrino fluence emitted by FRBs as a function of time window duration. We set the most stringent limit obtained to date on neutrino fluence from FRBs with an $E^{-2}$ energy spectrum assumed, which is 0.0021 GeV cm$^{-2}$ per burst for emission timescales up to \textasciitilde10$^2$ seconds from the northern hemisphere stacking search. △ Less

Submitted 18 December, 2017; originally announced December 2017.

Comments: 15 pages, 9 figures

Journal ref: Astrophys.J. 857 (2018) 117

Burst Denoising with Kernel Prediction Networks

Authors: Ben Mildenhall, Jonathan T. Barron, Jiawen Chen, Dillon Sharlet, Ren Ng, Robert Carroll

Abstract: We present a technique for jointly denoising bursts of images taken from a handheld camera. In particular, we propose a convolutional neural network architecture for predicting spatially varying kernels that can both align and denoise frames, a synthetic data generation approach based on a realistic noise formation model, and an optimization guided by an annealed loss function to avoid undesirable… ▽ More We present a technique for jointly denoising bursts of images taken from a handheld camera. In particular, we propose a convolutional neural network architecture for predicting spatially varying kernels that can both align and denoise frames, a synthetic data generation approach based on a realistic noise formation model, and an optimization guided by an annealed loss function to avoid undesirable local minima. Our model matches or outperforms the state-of-the-art across a wide range of noise levels on both real and synthetic data. △ Less

Submitted 29 March, 2018; v1 submitted 6 December, 2017; originally announced December 2017.

Comments: To appear in CVPR 2018 (spotlight). Project page: http://people.eecs.berkeley.edu/~bmild/kpn/

Measurement of the multi-TeV neutrino cross section with IceCube using Earth absorption

Authors: IceCube Collaboration, M. G. Aartsen, M. Ackermann, J. Adams, J. A. Aguilar, M. Ahlers, M. Ahrens, I. Al Samarai, D. Altmann, K. Andeen, T. Anderson, I. Ansseau, G. Anton, C. Argüelles, J. Auffenberg, S. Axani, H. Bagherpour, X. Bai, J. P. Barron, S. W. Barwick, V. Baum, R. Bay, J. J. Beatty, J. Becker Tjus, K. -H. Becker , et al. (293 additional authors not shown)

Abstract: Neutrinos interact only very weakly, so they are extremely penetrating. However, the theoretical neutrino-nucleon interaction cross section rises with energy such that, at energies above 40 TeV, neutrinos are expected to be absorbed as they pass through the Earth. Experimentally, the cross section has been measured only at the relatively low energies (below 400 GeV) available at neutrino beams fro… ▽ More Neutrinos interact only very weakly, so they are extremely penetrating. However, the theoretical neutrino-nucleon interaction cross section rises with energy such that, at energies above 40 TeV, neutrinos are expected to be absorbed as they pass through the Earth. Experimentally, the cross section has been measured only at the relatively low energies (below 400 GeV) available at neutrino beams from accelerators \cite{Agashe:2014kda, Formaggio:2013kya}. Here we report the first measurement of neutrino absorption in the Earth, using a sample of 10,784 energetic upward-going neutrino-induced muons observed with the IceCube Neutrino Observatory. The flux of high-energy neutrinos transiting long paths through the Earth is attenuated compared to a reference sample that follows shorter trajectories through the Earth. Using a fit to the two-dimensional distribution of muon energy and zenith angle, we determine the cross section for neutrino energies between 6.3 TeV and 980 TeV, more than an order of magnitude higher in energy than previous measurements. The measured cross section is $1.30^{+0.21}_{-0.19}$ (stat.) $^{+0.39}_{-0.43}$ (syst.) times the prediction of the Standard Model \cite{CooperSarkar:2011pa}, consistent with the expectation for charged and neutral current interactions. We do not observe a dramatic increase in the cross section, expected in some speculative models, including those invoking new compact dimensions \cite{AlvarezMuniz:2002ga} or the production of leptoquarks \cite{Romero:2009vu}. △ Less

Submitted 21 November, 2017; originally announced November 2017.

Comments: Preprint version of Nature paper 10.1038/nature24459

Journal ref: Nature 551 (2017) 596-600

Aperture Supervision for Monocular Depth Estimation

Authors: Pratul P. Srinivasan, Rahul Garg, Neal Wadhwa, Ren Ng, Jonathan T. Barron

Abstract: We present a novel method to train machine learning algorithms to estimate scene depths from a single image, by using the information provided by a camera's aperture as supervision. Prior works use a depth sensor's outputs or images of the same scene from alternate viewpoints as supervision, while our method instead uses images from the same viewpoint taken with a varying camera aperture. To enabl… ▽ More We present a novel method to train machine learning algorithms to estimate scene depths from a single image, by using the information provided by a camera's aperture as supervision. Prior works use a depth sensor's outputs or images of the same scene from alternate viewpoints as supervision, while our method instead uses images from the same viewpoint taken with a varying camera aperture. To enable learning algorithms to use aperture effects as supervision, we introduce two differentiable aperture rendering functions that use the input image and predicted depths to simulate the depth-of-field effects caused by real camera apertures. We train a monocular depth estimation network end-to-end to predict the scene depths that best explain these finite aperture images as defocus-blurred renderings of the input all-in-focus image. △ Less

Submitted 29 March, 2018; v1 submitted 21 November, 2017; originally announced November 2017.

Comments: To appear at CVPR 2018 (updated to camera ready version)

Individuals, Institutions, and Innovation in the Debates of the French Revolution

Authors: Alexander T. J. Barron, Jenny Huang, Rebecca L. Spang, Simon DeDeo

Abstract: The French Revolution brought principles of "liberty, equality, and brotherhood" to bear on the day-to-day challenges of governing what was then the largest country in Europe. Its experiments provided a model for future revolutions and democracies across the globe, but this first modern revolution had no model to follow. Using reconstructed transcripts of debates held in the Revolution's first par… ▽ More The French Revolution brought principles of "liberty, equality, and brotherhood" to bear on the day-to-day challenges of governing what was then the largest country in Europe. Its experiments provided a model for future revolutions and democracies across the globe, but this first modern revolution had no model to follow. Using reconstructed transcripts of debates held in the Revolution's first parliament, we present a quantitative analysis of how this system managed innovation. We use information theory to track the creation, transmission, and destruction of patterns of word-use across over 40,000 speeches and more than one thousand speakers. The parliament as a whole was biased toward the adoption of new patterns, but speakers' individual qualities could break these overall trends. Speakers on the left innovated at higher rates while speakers on the right acted, often successfully, to preserve prior patterns. Key players such as Robespierre (on the left) and Abbé Maury (on the right) played information-processing roles emblematic of their politics. Newly-created organizational functions---such as the Assembly's President and committee chairs---had significant effects on debate outcomes, and a distinct transition appears mid-way through the parliament when committees, external to the debate process, gain new powers to "propose and dispose" to the body as a whole. Taken together, these quantitative results align with existing qualitative interpretations but also reveal crucial information-processing dynamics that have hitherto been overlooked. Great orators had the public's attention, but deputies (mostly on the political left) who mastered the committee system gained new powers to shape revolutionary legislation. △ Less

Submitted 18 October, 2017; originally announced October 2017.

Comments: 8 pages, 3 figures, 1 table. Comments solicited

Journal ref: Proceedings of the National Academy of Sciences, 201717729 (2018)