-
Long-Lived Accurate Keypoints in Event Streams
Authors:
Philippe Chiberre,
Etienne Perot,
Amos Sironi,
Vincent Lepetit
Abstract:
We present a novel end-to-end approach to keypoint detection and tracking in an event stream that provides better precision and much longer keypoint tracks than previous methods. This is made possible by two contributions working together.
First, we propose a simple procedure to generate stable keypoint labels, which we use to train a recurrent architecture. This training data results in detecti…
▽ More
We present a novel end-to-end approach to keypoint detection and tracking in an event stream that provides better precision and much longer keypoint tracks than previous methods. This is made possible by two contributions working together.
First, we propose a simple procedure to generate stable keypoint labels, which we use to train a recurrent architecture. This training data results in detections that are very consistent over time.
Moreover, we observe that previous methods for keypoint detection work on a representation (such as the time surface) that integrates events over a period of time. Since this integration is required, we claim it is better to predict the keypoints' trajectories for the time period rather than single locations, as done in previous approaches. We predict these trajectories in the form of a series of heatmaps for the integration time period. This improves the keypoint localization.
Our architecture can also be kept very simple, which results in very fast inference times. We demonstrate our approach on the HVGA ATIS Corner dataset as well as "The Event-Camera Dataset and Simulator" dataset, and show it results in keypoint tracks that are three times longer and nearly twice as accurate as the best previous state-of-the-art methods. We believe our approach can be generalized to other event-based camera problems, and we release our source code to encourage other authors to explore it.
△ Less
Submitted 7 October, 2022; v1 submitted 21 September, 2022;
originally announced September 2022.
-
Performance of the polarization leakage correction in the PILOT data
Authors:
J-Ph. Bernard,
A. Bernard,
H. Roussel,
I. Choubani,
D. Alina,
J. Aumont,
A. Hughes,
I. Ristorcelli,
S. Stever,
T. Matsumura S. Sugiyama,
K. Komatsu,
G. de Gasperis,
K. Ferriere,
V. Guillet,
N. Ysard,
P. Ade,
P. de Bernardis,
N. Bray,
B. Crane,
J. P. Dubois,
M. Griffin,
P. Hargrave,
Y. Longval,
S. Louvel,
B. Maffei
, et al. (11 additional authors not shown)
Abstract:
The Polarized Instrument for Long-wavelength Observation of the Tenuous interstellar medium (PILOT) is a balloon-borne experiment that aims to measure the polarized emission of thermal dust at a wavelength of 240 um (1.2 THz). The PILOT experiment flew from Timmins, Ontario, Canada in 2015 and 2019 and from Alice Springs, Australia in April 2017. The in-flight performance of the instrument during…
▽ More
The Polarized Instrument for Long-wavelength Observation of the Tenuous interstellar medium (PILOT) is a balloon-borne experiment that aims to measure the polarized emission of thermal dust at a wavelength of 240 um (1.2 THz). The PILOT experiment flew from Timmins, Ontario, Canada in 2015 and 2019 and from Alice Springs, Australia in April 2017. The in-flight performance of the instrument during the second flight was described in Mangilli et al. 2019. In this paper, we present data processing steps that were not presented in Mangilli et al. 2019 and that we have recently implemented to correct for several remaining instrumental effects. The additional data processing concerns corrections related to detector cross-talk and readout circuit memory effects, and leakage from total intensity to polarization. We illustrate the above effects and the performance of our corrections using data obtained during the third flight of PILOT, but the methods used to assess the impact of these effects on the final science-ready data, and our strategies for correcting them will be applied to all PILOT data. We show that the above corrections, and in particular that for the intensity to polarization leakage, which is most critical for accurate polarization measurements with PILOT, are accurate to better than 0.4 % as measured on Jupiter during flight#3.
△ Less
Submitted 7 May, 2022;
originally announced May 2022.
-
End-to-end ground calibration and in-flight performance of the FIREBall-2 instrument
Authors:
Vincent Picouet,
Bruno Milliard,
Gillian Kyne,
Didier Vibert,
David Schiminovich,
Christopher Martin,
Erika Hamden,
Keri Hoadley,
Johan Montel,
Nicole Melso,
Donal OSullivan,
Jean Evrard,
Etienne Perot,
Robert Grange,
Shouleh Nikzad,
Philippe Balard,
Patrick Blanchard,
Frederi Mirc,
Nicolas Bray,
April Jewell,
Samuel Quiret
Abstract:
The payload of the Faint Intergalactic Redshifted Emission Balloon (FIREBall-2), the second generation of the FIREBall instrument (PI: C. Martin, Caltech), has been calibrated and launched from the NASA Columbia Scientific Balloon Facility (CSBF) in Fort Sumner, NM. FIREBall-2 was launched for the first time on the 22nd September 2018, and the payload performed the very first multi-object acquisit…
▽ More
The payload of the Faint Intergalactic Redshifted Emission Balloon (FIREBall-2), the second generation of the FIREBall instrument (PI: C. Martin, Caltech), has been calibrated and launched from the NASA Columbia Scientific Balloon Facility (CSBF) in Fort Sumner, NM. FIREBall-2 was launched for the first time on the 22nd September 2018, and the payload performed the very first multi-object acquisition from space using a multi-object slit spectrograph (MOS). This performance-oriented paper presents the calibration and last ground adjustments of FIREBall-2, the in-flight performance assessed based on the flight data, and the predicted instrument's ultimate sensitivity. This analysis predicts that future flights of FIREBall-2 should be able to detect the HI Lyαresonance line in galaxies at z~0.67, but will find it challenging to spatially resolve the circumgalactic medium (CGM).
△ Less
Submitted 6 January, 2021;
originally announced January 2021.
-
Real-Time Face & Eye Tracking and Blink Detection using Event Cameras
Authors:
Cian Ryan,
Brian O Sullivan,
Amr Elrasad,
Joe Lemley,
Paul Kielty,
Christoph Posch,
Etienne Perot
Abstract:
Event cameras contain emerging, neuromorphic vision sensors that capture local light intensity changes at each pixel, generating a stream of asynchronous events. This way of acquiring visual information constitutes a departure from traditional frame based cameras and offers several significant advantages: low energy consumption, high temporal resolution, high dynamic range and low latency. Driver…
▽ More
Event cameras contain emerging, neuromorphic vision sensors that capture local light intensity changes at each pixel, generating a stream of asynchronous events. This way of acquiring visual information constitutes a departure from traditional frame based cameras and offers several significant advantages: low energy consumption, high temporal resolution, high dynamic range and low latency. Driver monitoring systems (DMS) are in-cabin safety systems designed to sense and understand a drivers physical and cognitive state. Event cameras are particularly suited to DMS due to their inherent advantages. This paper proposes a novel method to simultaneously detect and track faces and eyes for driver monitoring. A unique, fully convolutional recurrent neural network architecture is presented. To train this network, a synthetic event-based dataset is simulated with accurate bounding box annotations, called Neuromorphic HELEN. Additionally, a method to detect and analyse drivers eye blinks is proposed, exploiting the high temporal resolution of event cameras. Behaviour of blinking provides greater insights into a driver level of fatigue or drowsiness. We show that blinks have a unique temporal signature that can be better captured by event cameras.
△ Less
Submitted 16 October, 2020;
originally announced October 2020.
-
Learning to Detect Objects with a 1 Megapixel Event Camera
Authors:
Etienne Perot,
Pierre de Tournemire,
Davide Nitti,
Jonathan Masci,
Amos Sironi
Abstract:
Event cameras encode visual information with high temporal precision, low data-rate, and high-dynamic range. Thanks to these characteristics, event cameras are particularly suited for scenarios with high motion, challenging lighting conditions and requiring low latency. However, due to the novelty of the field, the performance of event-based systems on many vision tasks is still lower compared to…
▽ More
Event cameras encode visual information with high temporal precision, low data-rate, and high-dynamic range. Thanks to these characteristics, event cameras are particularly suited for scenarios with high motion, challenging lighting conditions and requiring low latency. However, due to the novelty of the field, the performance of event-based systems on many vision tasks is still lower compared to conventional frame-based solutions. The main reasons for this performance gap are: the lower spatial resolution of event sensors, compared to frame cameras; the lack of large-scale training datasets; the absence of well established deep learning architectures for event-based processing. In this paper, we address all these problems in the context of an event-based object detection task. First, we publicly release the first high-resolution large-scale dataset for object detection. The dataset contains more than 14 hours recordings of a 1 megapixel event camera, in automotive scenarios, together with 25M bounding boxes of cars, pedestrians, and two-wheelers, labeled at high frequency. Second, we introduce a novel recurrent architecture for event-based detection and a temporal consistency loss for better-behaved training. The ability to compactly represent the sequence of events into the internal memory of the model is essential to achieve high accuracy. Our model outperforms by a large margin feed-forward event-based architectures. Moreover, our method does not require any reconstruction of intensity images from events, showing that training directly from raw events is possible, more efficient, and more accurate than passing through an intermediate intensity image. Experiments on the dataset introduced in this work, for which events and gray level images are available, show performance on par with that of highly tuned and studied frame-based detectors.
△ Less
Submitted 9 December, 2020; v1 submitted 28 September, 2020;
originally announced September 2020.
-
A Large Scale Event-based Detection Dataset for Automotive
Authors:
Pierre de Tournemire,
Davide Nitti,
Etienne Perot,
Davide Migliore,
Amos Sironi
Abstract:
We introduce the first very large detection dataset for event cameras. The dataset is composed of more than 39 hours of automotive recordings acquired with a 304x240 ATIS sensor. It contains open roads and very diverse driving scenarios, ranging from urban, highway, suburbs and countryside scenes, as well as different weather and illumination conditions. Manual bounding box annotations of cars and…
▽ More
We introduce the first very large detection dataset for event cameras. The dataset is composed of more than 39 hours of automotive recordings acquired with a 304x240 ATIS sensor. It contains open roads and very diverse driving scenarios, ranging from urban, highway, suburbs and countryside scenes, as well as different weather and illumination conditions. Manual bounding box annotations of cars and pedestrians contained in the recordings are also provided at a frequency between 1 and 4Hz, yielding more than 255,000 labels in total. We believe that the availability of a labeled dataset of this size will contribute to major advances in event-based vision tasks such as object detection and classification. We also expect benefits in other tasks such as optical flow, structure from motion and tracking, where for example, the large amount of data can be leveraged by self-supervised learning methods.
△ Less
Submitted 31 January, 2020; v1 submitted 23 January, 2020;
originally announced January 2020.
-
FIREBall-2: advancing TRL while doing proof-of-concept astrophysics on a suborbital platform
Authors:
Erika T. Hamden,
Keri Hoadley,
D. Christopher Martin,
David Schiminovich,
Bruno Milliard,
Shouleh Nikzad,
Ramona Augustin,
Philippe Balard,
Patrick Blanchard,
Nicolas Bray,
Marty Crabill,
Jean Evrard,
Albert Gomes,
Robert Grange,
Julia Gross,
April Jewell,
Gillian Kyne,
Michele Limon,
Nicole Lingner,
Mateusz Matuszewski,
Nicole Melso,
Frederi Mirc,
Johan Montel,
Hwei Ru Ong,
Donal O'Sullivan
, et al. (10 additional authors not shown)
Abstract:
Here we discuss advances in UV technology over the last decade, with an emphasis on photon counting, low noise, high efficiency detectors in sub-orbital programs. We focus on the use of innovative UV detectors in a NASA astrophysics balloon telescope, FIREBall-2, which successfully flew in the Fall of 2018. The FIREBall-2 telescope is designed to make observations of distant galaxies to understand…
▽ More
Here we discuss advances in UV technology over the last decade, with an emphasis on photon counting, low noise, high efficiency detectors in sub-orbital programs. We focus on the use of innovative UV detectors in a NASA astrophysics balloon telescope, FIREBall-2, which successfully flew in the Fall of 2018. The FIREBall-2 telescope is designed to make observations of distant galaxies to understand more about how they evolve by looking for diffuse hydrogen in the galactic halo. The payload utilizes a 1.0-meter class telescope with an ultraviolet multi-object spectrograph and is a joint collaboration between Caltech, JPL, LAM, CNES, Columbia, the University of Arizona, and NASA. The improved detector technology that was tested on FIREBall-2 can be applied to any UV mission. We discuss the results of the flight and detector performance. We will also discuss the utility of sub-orbital platforms (both balloon payloads and rockets) for testing new technologies and proof-of-concept scientific ideas
△ Less
Submitted 3 May, 2019; v1 submitted 1 May, 2019;
originally announced May 2019.
-
End-to-End Race Driving with Deep Reinforcement Learning
Authors:
Maximilian Jaritz,
Raoul de Charette,
Marin Toromanoff,
Etienne Perot,
Fawzi Nashashibi
Abstract:
We present research using the latest reinforcement learning algorithm for end-to-end driving without any mediated perception (object recognition, scene understanding). The newly proposed reward and learning strategies lead together to faster convergence and more robust driving using only RGB image from a forward facing camera. An Asynchronous Actor Critic (A3C) framework is used to learn the car c…
▽ More
We present research using the latest reinforcement learning algorithm for end-to-end driving without any mediated perception (object recognition, scene understanding). The newly proposed reward and learning strategies lead together to faster convergence and more robust driving using only RGB image from a forward facing camera. An Asynchronous Actor Critic (A3C) framework is used to learn the car control in a physically and graphically realistic rally game, with the agents evolving simultaneously on tracks with a variety of road structures (turns, hills), graphics (seasons, location) and physics (road adherence). A thorough evaluation is conducted and generalization is proven on unseen tracks and using legal speed limits. Open loop tests on real sequences of images show some domain adaption capability of our method.
△ Less
Submitted 31 August, 2018; v1 submitted 6 July, 2018;
originally announced July 2018.
-
PILOT balloon-borne experiment in-flight performance
Authors:
A. Mangilli,
G. Foënard,
J. Aumont,
A. Hughes,
B. Mot,
J-Ph. Bernard,
A. Lacourt,
I. Ristorcelli,
Y. Longval,
P. Ade,
Y. André,
L. Bautista,
P. deBernardis,
O. Boulade,
F. Bousqet,
M. Bouzit,
V. Buttice,
M. Charra,
B. Crane,
E. Doumayrou,
J. P. Dubois,
C. Engel,
M. Griffin,
S. Grabarnik,
P. Hargrave
, et al. (26 additional authors not shown)
Abstract:
The Polarized Instrument for Long-wavelength Observation of the Tenuous interstellar medium (PILOT) is a balloon-borne experiment aiming at measuring the polarized emission of thermal dust at a wavelength of 240 mm (1.2 THz). A first PILOT flight (flight#1) of the experiment took place from Timmins, Ontario, Canada, in September 2015 and a second flight (flight#2) took place from Alice Springs, Au…
▽ More
The Polarized Instrument for Long-wavelength Observation of the Tenuous interstellar medium (PILOT) is a balloon-borne experiment aiming at measuring the polarized emission of thermal dust at a wavelength of 240 mm (1.2 THz). A first PILOT flight (flight#1) of the experiment took place from Timmins, Ontario, Canada, in September 2015 and a second flight (flight#2) took place from Alice Springs, Australia in april 2017. In this paper, we present the inflight performance of the instrument during these two flights. We concentrate on performances during flight#2, but allude to flight#1 performances if significantly different. We first present a short description of the instrument and the flights. We determine the time constants of our detectors combining inflight information from the signal decay following high energy particle impacts (glitches) and of our internal calibration source. We use these time constants to deconvolve the data timelines and analyse the optical quality of the instrument as measured on planets. We then analyse the structure and polarization of the instrumental background. We measure the detector response flat field and its time variations using the signal from the residual atmosphere and of our internal calibration source. Finally, we analyze the detector noise spectral and temporal properties. The in-flight performances are found to be satisfactory and globally in line with expectations from ground calibrations. We conclude by assessing the expected in-flight sensitivity of the instrument in light of the above in-flight performances.
△ Less
Submitted 7 January, 2019; v1 submitted 16 April, 2018;
originally announced April 2018.
-
Deep Reinforcement Learning framework for Autonomous Driving
Authors:
Ahmad El Sallab,
Mohammed Abdou,
Etienne Perot,
Senthil Yogamani
Abstract:
Reinforcement learning is considered to be a strong AI paradigm which can be used to teach machines through interaction with the environment and learning from their mistakes. Despite its perceived utility, it has not yet been successfully applied in automotive applications. Motivated by the successful demonstrations of learning of Atari games and Go by Google DeepMind, we propose a framework for a…
▽ More
Reinforcement learning is considered to be a strong AI paradigm which can be used to teach machines through interaction with the environment and learning from their mistakes. Despite its perceived utility, it has not yet been successfully applied in automotive applications. Motivated by the successful demonstrations of learning of Atari games and Go by Google DeepMind, we propose a framework for autonomous driving using deep reinforcement learning. This is of particular relevance as it is difficult to pose autonomous driving as a supervised learning problem due to strong interactions with the environment including other vehicles, pedestrians and roadworks. As it is a relatively new area of research for autonomous driving, we provide a short overview of deep reinforcement learning and then describe our proposed framework. It incorporates Recurrent Neural Networks for information integration, enabling the car to handle partially observable scenarios. It also integrates the recent work on attention models to focus on relevant information, thereby reducing the computational complexity for deployment on embedded hardware. The framework was tested in an open source 3D car racing simulator called TORCS. Our simulation results demonstrate learning of autonomous maneuvering in a scenario of complex road curvatures and simple interaction of other vehicles.
△ Less
Submitted 8 April, 2017;
originally announced April 2017.
-
End-to-End Deep Reinforcement Learning for Lane Kee** Assist
Authors:
Ahmad El Sallab,
Mohammed Abdou,
Etienne Perot,
Senthil Yogamani
Abstract:
Reinforcement learning is considered to be a strong AI paradigm which can be used to teach machines through interaction with the environment and learning from their mistakes, but it has not yet been successfully used for automotive applications. There has recently been a revival of interest in the topic, however, driven by the ability of deep learning algorithms to learn good representations of th…
▽ More
Reinforcement learning is considered to be a strong AI paradigm which can be used to teach machines through interaction with the environment and learning from their mistakes, but it has not yet been successfully used for automotive applications. There has recently been a revival of interest in the topic, however, driven by the ability of deep learning algorithms to learn good representations of the environment. Motivated by Google DeepMind's successful demonstrations of learning for games from Breakout to Go, we will propose different methods for autonomous driving using deep reinforcement learning. This is of particular interest as it is difficult to pose autonomous driving as a supervised learning problem as it has a strong interaction with the environment including other vehicles, pedestrians and roadworks. As this is a relatively new area of research for autonomous driving, we will formulate two main categories of algorithms: 1) Discrete actions category, and 2) Continuous actions category. For the discrete actions category, we will deal with Deep Q-Network Algorithm (DQN) while for the continuous actions category, we will deal with Deep Deterministic Actor Critic Algorithm (DDAC). In addition to that, We will also discover the performance of these two categories on an open source car simulator for Racing called (TORCS) which stands for The Open Racing car Simulator. Our simulation results demonstrate learning of autonomous maneuvering in a scenario of complex road curvatures and simple interaction with other vehicles. Finally, we explain the effect of some restricted conditions, put on the car during the learning phase, on the convergence time for finishing its learning phase.
△ Less
Submitted 13 December, 2016;
originally announced December 2016.
