-
Neuromorphic Optical Flow and Real-time Implementation with Event Cameras
Authors:
Yannick Schnider,
Stanislaw Wozniak,
Mathias Gehrig,
Jules Lecomte,
Axel von Arnim,
Luca Benini,
Davide Scaramuzza,
Angeliki Pantazi
Abstract:
Optical flow provides information on relative motion that is an important component in many computer vision pipelines. Neural networks provide high accuracy optical flow, yet their complexity is often prohibitive for application at the edge or in robots, where efficiency and latency play crucial role. To address this challenge, we build on the latest developments in event-based vision and spiking…
▽ More
Optical flow provides information on relative motion that is an important component in many computer vision pipelines. Neural networks provide high accuracy optical flow, yet their complexity is often prohibitive for application at the edge or in robots, where efficiency and latency play crucial role. To address this challenge, we build on the latest developments in event-based vision and spiking neural networks. We propose a new network architecture, inspired by Timelens, that improves the state-of-the-art self-supervised optical flow accuracy when operated both in spiking and non-spiking mode. To implement a real-time pipeline with a physical event camera, we propose a methodology for principled model simplification based on activity and latency analysis. We demonstrate high speed optical flow prediction with almost two orders of magnitude reduced complexity while maintaining the accuracy, opening the path for real-time deployments.
△ Less
Submitted 12 July, 2023; v1 submitted 14 April, 2023;
originally announced April 2023.
-
Dynamic Event-based Optical Identification and Communication
Authors:
Axel von Arnim,
Jules Lecomte,
Naima Elosegui Borras,
Stanislaw Wozniak,
Angeliki Pantazi
Abstract:
Optical identification is often done with spatial or temporal visual pattern recognition and localization. Temporal pattern recognition, depending on the technology, involves a trade-off between communication frequency, range and accurate tracking. We propose a solution with light-emitting beacons that improves this trade-off by exploiting fast event-based cameras and, for tracking, sparse neuromo…
▽ More
Optical identification is often done with spatial or temporal visual pattern recognition and localization. Temporal pattern recognition, depending on the technology, involves a trade-off between communication frequency, range and accurate tracking. We propose a solution with light-emitting beacons that improves this trade-off by exploiting fast event-based cameras and, for tracking, sparse neuromorphic optical flow computed with spiking neurons. The system is embedded in a simulated drone and evaluated in an asset monitoring use case. It is robust to relative movements and enables simultaneous communication with, and tracking of, multiple moving beacons. Finally, in a hardware lab prototype, we demonstrate for the first time beacon tracking performed simultaneously with state-of-the-art frequency communication in the kHz range.
△ Less
Submitted 7 May, 2024; v1 submitted 13 March, 2023;
originally announced March 2023.
-
Level set based particle filter driven by optical flow: an application to track the salt boundary from X-ray CT time-series
Authors:
Karim Makki,
Jean François Lecomte,
Lukas Fuchs,
Sylvie Schueller,
Etienne Mémin
Abstract:
Image-based computational fluid dynamics have long played an important role in leveraging knowledge and understanding of several physical phenomena. In particular, probabilistic computational methods have opened the way to modelling the complex dynamics of systems in purely random turbulent motion. In the field of structural geology, a better understanding of the deformation and stress state both…
▽ More
Image-based computational fluid dynamics have long played an important role in leveraging knowledge and understanding of several physical phenomena. In particular, probabilistic computational methods have opened the way to modelling the complex dynamics of systems in purely random turbulent motion. In the field of structural geology, a better understanding of the deformation and stress state both within the salt and the surrounding rocks is of great interest to characterize all kinds of subsurface long-terms energy-storage systems. The objective of this research is to determine the non-linear deformation of the salt boundary over time using a parallelized, stochastic filtering approach from x-ray computed tomography (CT) image time series depicting the evolution of salt structures triggered by gravity and under differential loading. This work represents a first step towards bringing together physical modeling and advanced stochastic image processing methods where model uncertainty is taken into account.
△ Less
Submitted 17 February, 2022;
originally announced February 2022.
-
Flexible and Scalable State Tracking Framework for Goal-Oriented Dialogue Systems
Authors:
Rahul Goel,
Shachi Paul,
Tagyoung Chung,
Jeremie Lecomte,
Arindam Mandal,
Dilek Hakkani-Tur
Abstract:
Goal-oriented dialogue systems typically rely on components specifically developed for a single task or domain. This limits such systems in two different ways: If there is an update in the task domain, the dialogue system usually needs to be updated or completely re-trained. It is also harder to extend such dialogue systems to different and multiple domains. The dialogue state tracker in conventio…
▽ More
Goal-oriented dialogue systems typically rely on components specifically developed for a single task or domain. This limits such systems in two different ways: If there is an update in the task domain, the dialogue system usually needs to be updated or completely re-trained. It is also harder to extend such dialogue systems to different and multiple domains. The dialogue state tracker in conventional dialogue systems is one such component - it is usually designed to fit a well-defined application domain. For example, it is common for a state variable to be a categorical distribution over a manually-predefined set of entities (Henderson et al., 2013), resulting in an inflexible and hard-to-extend dialogue system. In this paper, we propose a new approach for dialogue state tracking that can generalize well over multiple domains without incorporating any domain-specific knowledge. Under this framework, discrete dialogue state variables are learned independently and the information of a predefined set of possible values for dialogue state variables is not required. Furthermore, it enables adding arbitrary dialogue context as features and allows for multiple values to be associated with a single state variable. These characteristics make it much easier to expand the dialogue state space. We evaluate our framework using the widely used dialogue state tracking challenge data set (DSTC2) and show that our framework yields competitive results with other state-of-the-art results despite incorporating little domain knowledge. We also show that this framework can benefit from widely available external resources such as pre-trained word embeddings.
△ Less
Submitted 30 November, 2018;
originally announced November 2018.
-
Critical parameters for the partial coalescence of a droplet
Authors:
T. Gilet,
K. Mulleners,
J. P. Lecomte,
N. Vandewalle,
S. Dorbolo
Abstract:
The partial coalescence of a droplet onto a planar liquid/liquid interface is investigated experimentally by tuning the viscosities of both liquids. The problem mainly depends on four dimensionless parameters: the Bond number (gravity vs. surface tension), the Ohnesorge numbers (viscosity in both fluids vs. surface tension), and the density relative difference. The ratio between the daughter dro…
▽ More
The partial coalescence of a droplet onto a planar liquid/liquid interface is investigated experimentally by tuning the viscosities of both liquids. The problem mainly depends on four dimensionless parameters: the Bond number (gravity vs. surface tension), the Ohnesorge numbers (viscosity in both fluids vs. surface tension), and the density relative difference. The ratio between the daughter droplet size and the mother droplet size is investigated as a function of these dimensionless numbers. Global quantities such as the available surface energy of the droplet has been measured during the coalescence. The capillary waves propagation and dam** are studied in detail. The relation between these waves and the partial coalescence is discussed. Additional viscous mechanisms are proposed in order to explain the asymmetric role played by both viscosities.
△ Less
Submitted 15 December, 2006; v1 submitted 11 August, 2006;
originally announced August 2006.
