-
Blind as a bat: audible echolocation on small robots
Authors:
Frederike Dümbgen,
Adrien Hoffet,
Mihailo Kolundžija,
Adam Scholefield,
Martin Vetterli
Abstract:
For safe and efficient operation, mobile robots need to perceive their environment, and in particular, perform tasks such as obstacle detection, localization, and map**. Although robots are often equipped with microphones and speakers, the audio modality is rarely used for these tasks. Compared to the localization of sound sources, for which many practical solutions exist, algorithms for active…
▽ More
For safe and efficient operation, mobile robots need to perceive their environment, and in particular, perform tasks such as obstacle detection, localization, and map**. Although robots are often equipped with microphones and speakers, the audio modality is rarely used for these tasks. Compared to the localization of sound sources, for which many practical solutions exist, algorithms for active echolocation are less developed and often rely on hardware requirements that are out of reach for small robots. We propose an end-to-end pipeline for sound-based localization and map** that is targeted at, but not limited to, robots equipped with only simple buzzers and low-end microphones. The method is model-based, runs in real time, and requires no prior calibration or training. We successfully test the algorithm on the e-puck robot with its integrated audio hardware, and on the Crazyflie drone, for which we design a reproducible audio extension deck. We achieve centimeter-level wall localization on both platforms when the robots are static during the measurement process. Even in the more challenging setting of a flying drone, we can successfully localize walls, which we demonstrate in a proof-of-concept multi-wall localization and map** demo.
△ Less
Submitted 19 January, 2023;
originally announced January 2023.
-
Lippmann Photography: A Signal Processing Perspective
Authors:
Gilles Baechler,
Michalina Pacholska,
Arnaud Latty,
Adam Scholefield,
Martin Vetterli
Abstract:
Lippmann (or interferential) photography is the first and only analog photography method that can capture the full color spectrum of a scene in a single take. This technique, invented more than a hundred years ago, records the colors by creating interference patterns inside the photosensitive plate. Lippmann photography provides a great opportunity to demonstrate several fundamental concepts in si…
▽ More
Lippmann (or interferential) photography is the first and only analog photography method that can capture the full color spectrum of a scene in a single take. This technique, invented more than a hundred years ago, records the colors by creating interference patterns inside the photosensitive plate. Lippmann photography provides a great opportunity to demonstrate several fundamental concepts in signal processing. Conversely, a signal processing perspective enables us to shed new light on the technique. In our previous work, we analyzed the spectra of historical Lippmann plates using our own mathematical model. In this paper, we provide the derivation of this model and validate it experimentally. We highlight new behaviors whose explanations were ignored by physicists to date. In particular, we show that the spectra generated by Lippmann plates are in fact distorted versions of the original spectra. We also show that these distortions are influenced by the thickness of the plate and the reflection coefficient of the reflective medium used in the capture of the photographs. We verify our model with extensive experiments on our own Lippmann photographs.
△ Less
Submitted 13 July, 2022;
originally announced July 2022.
-
How Asynchronous Events Encode Video
Authors:
Karen Adam,
Adam Scholefield,
Martin Vetterli
Abstract:
As event-based sensing gains in popularity, theoretical understanding is needed to harness this technology's potential. Instead of recording video by capturing frames, event-based cameras have sensors that emit events when their inputs change, thus encoding information in the timing of events. This creates new challenges in establishing reconstruction guarantees and algorithms, but also provides a…
▽ More
As event-based sensing gains in popularity, theoretical understanding is needed to harness this technology's potential. Instead of recording video by capturing frames, event-based cameras have sensors that emit events when their inputs change, thus encoding information in the timing of events. This creates new challenges in establishing reconstruction guarantees and algorithms, but also provides advantages over frame-based video. We use time encoding machines to model event-based sensors: TEMs also encode their inputs by emitting events characterized by their timing and reconstruction from time encodings is well understood. We consider the case of time encoding bandlimited video and demonstrate a dependence between spatial sensor density and overall spatial and temporal resolution. Such a dependence does not occur in frame-based video, where temporal resolution depends solely on the frame rate of the video and spatial resolution depends solely on the pixel grid. However, this dependence arises naturally in event-based video and allows oversampling in space to provide better time resolution. As such, event-based vision encourages using more sensors that emit fewer events over time.
△ Less
Submitted 9 June, 2022;
originally announced June 2022.
-
Asynchrony Increases Efficiency: Time Encoding of Videos and Low-Rank Signals
Authors:
Karen Adam,
Adam Scholefield,
Martin Vetterli
Abstract:
In event-based sensing, many sensors independently and asynchronously emit events when there is a change in their input. Event-based sensing can present significant improvements in power efficiency when compared to traditional sampling, because (1) the output is a stream of events where the important information lies in the timing of the events, and (2) the sensor can easily be controlled to outpu…
▽ More
In event-based sensing, many sensors independently and asynchronously emit events when there is a change in their input. Event-based sensing can present significant improvements in power efficiency when compared to traditional sampling, because (1) the output is a stream of events where the important information lies in the timing of the events, and (2) the sensor can easily be controlled to output information only when interesting activity occurs at the input.
Moreover, event-based sampling can often provide better resolution than standard uniform sampling. Not only does this occur because individual event-based sensors have higher temporal resolution, it also occurs because the asynchrony of events allows for less redundant and more informative encoding. We would like to explain how such curious results come about.
To do so, we use ideal time encoding machines as a proxy for event-based sensors. We explore time encoding of signals with low rank structure, and apply the resulting theory to video. We then see how the asynchronous firing times of the time encoding machines allow for better reconstruction than in the standard sampling case, if we have a high spatial density of time encoding machines that fire less frequently.
△ Less
Submitted 29 April, 2021;
originally announced April 2021.
-
Realizability of Planar Point Embeddings from Angle Measurements
Authors:
Frederike Dümbgen,
Majed El Helou,
Adam Scholefield
Abstract:
Localization of a set of nodes is an important and a thoroughly researched problem in robotics and sensor networks. This paper is concerned with the theory of localization from inner-angle measurements. We focus on the challenging case where no anchor locations are known. Inspired by Euclidean distance matrices, we investigate when a set of inner angles corresponds to a realizable point set. In pa…
▽ More
Localization of a set of nodes is an important and a thoroughly researched problem in robotics and sensor networks. This paper is concerned with the theory of localization from inner-angle measurements. We focus on the challenging case where no anchor locations are known. Inspired by Euclidean distance matrices, we investigate when a set of inner angles corresponds to a realizable point set. In particular, we find linear and non-linear constraints that are provably necessary, and we conjecture also sufficient for characterizing realizable angle sets. We confirm this in extensive numerical simulations, and we illustrate the use of these constraints for denoising angle measurements along with the reconstruction of a valid point set.
△ Less
Submitted 9 May, 2020;
originally announced May 2020.
-
Matrix recovery from bilinear and quadratic measurements
Authors:
Michalina Pacholska,
Karen Adam,
Adam Scholefield,
Martin Vetterli
Abstract:
Matrix (or operator) recovery from linear measurements is a well-studied problem. However, there are situations where only bilinear or quadratic measurements are available. A bilinear or quadratic problem can easily be transformed into a linear one, but it raises questions when the linearized problem is solvable and what is the cost of linearization. In this work, we study a few specific cases of…
▽ More
Matrix (or operator) recovery from linear measurements is a well-studied problem. However, there are situations where only bilinear or quadratic measurements are available. A bilinear or quadratic problem can easily be transformed into a linear one, but it raises questions when the linearized problem is solvable and what is the cost of linearization. In this work, we study a few specific cases of this general problem and show when the bilinear problem is solvable. Using this result and certain properties of polynomial rings, we present a scenario when the quadratic problem can be linearized at the cost of just a linear number of additional measurements. Finally, we link our results back to two applications that inspired it: Time Encoding Machines and Continuous Localisation.
△ Less
Submitted 13 February, 2020; v1 submitted 14 January, 2020;
originally announced January 2020.
-
Encoding and Decoding Mixed Bandlimited Signals using Spiking Integrate-and-Fire Neurons
Authors:
Karen Adam,
Adam Scholefield,
Martin Vetterli
Abstract:
Conventional sampling focuses on encoding and decoding bandlimited signals by recording signal amplitudes at known time points. Alternately, sampling can be approached using biologically-inspired schemes. Among these are integrate-and-fire time encoding machines (IF-TEMs). They behave like simplified versions of spiking neurons and encode their input using spike times rather than amplitudes.
Mor…
▽ More
Conventional sampling focuses on encoding and decoding bandlimited signals by recording signal amplitudes at known time points. Alternately, sampling can be approached using biologically-inspired schemes. Among these are integrate-and-fire time encoding machines (IF-TEMs). They behave like simplified versions of spiking neurons and encode their input using spike times rather than amplitudes.
Moreover, when multiple of these neurons jointly process a set of mixed signals, they form one layer in a feedforward spiking neural network. In this paper, we investigate the encoding and decoding potential of such a layer.
We propose a setup to sample a set of bandlimited signals, by mixing them and sampling the result using different IF-TEMs. We provide conditions for perfect recovery of the set of signals from the samples in the noiseless case, and suggest an algorithm to perform the reconstruction.
△ Less
Submitted 14 February, 2020; v1 submitted 21 October, 2019;
originally announced October 2019.
-
Sampling and Reconstruction of Bandlimited Signals with Multi-Channel Time Encoding
Authors:
Karen Adam,
Adam Scholefield,
Martin Vetterli
Abstract:
Sampling is classically performed by recording the amplitude of an input signal at given time instants; however, sampling and reconstructing a signal using multiple devices in parallel becomes a more difficult problem to solve when the devices have an unknown shift in their clocks.
Alternatively, one can record the times at which a signal (or its integral) crosses given thresholds. This can mode…
▽ More
Sampling is classically performed by recording the amplitude of an input signal at given time instants; however, sampling and reconstructing a signal using multiple devices in parallel becomes a more difficult problem to solve when the devices have an unknown shift in their clocks.
Alternatively, one can record the times at which a signal (or its integral) crosses given thresholds. This can model integrate-and-fire neurons, for example, and has been studied by Lazar and Tóth under the name of ``Time Encoding Machines''. This sampling method is closer to what is found in nature.
In this paper, we show that, when using time encoding machines, reconstruction from multiple channels has a more intuitive solution, and does not require the knowledge of the shifts between machines. We show that, if single-channel time encoding can sample and perfectly reconstruct a $\mathbf{2Ω}$-bandlimited signal, then $\mathbf{M}$-channel time encoding with shifted integrators can sample and perfectly reconstruct a signal with $\mathbf{M}$ times the bandwidth.
Furthermore, we present an algorithm to perform this reconstruction and prove that it converges to the correct unique solution, in the noiseless case, without knowledge of the relative shifts between the integrators of the machines. This is quite unlike classical multi-channel sampling, where unknown shifts between sampling devices pose a problem for perfect reconstruction.
△ Less
Submitted 3 December, 2019; v1 submitted 12 July, 2019;
originally announced July 2019.
-
Embedded polarizing filters to separate diffuse and specular reflection
Authors:
Laurent Valentin Jospin,
Gilles Baechler,
Adam Scholefield
Abstract:
Polarizing filters provide a powerful way to separate diffuse and specular reflection; however, traditional methods rely on several captures and require proper alignment of the filters. Recently, camera manufacturers have proposed to embed polarizing micro-filters in front of the sensor, creating a mosaic of pixels with different polarizations. In this paper, we investigate the advantages of such…
▽ More
Polarizing filters provide a powerful way to separate diffuse and specular reflection; however, traditional methods rely on several captures and require proper alignment of the filters. Recently, camera manufacturers have proposed to embed polarizing micro-filters in front of the sensor, creating a mosaic of pixels with different polarizations. In this paper, we investigate the advantages of such camera designs. In particular, we consider different design patterns for the filter arrays and propose an algorithm to demosaic an image generated by such cameras. This essentially allows us to separate the diffuse and specular components using a single image. The performance of our algorithm is compared with a color-based method using synthetic and real data. Finally, we demonstrate how we can recover the normals of a scene using the diffuse images estimated by our method.
△ Less
Submitted 6 November, 2018;
originally announced November 2018.
-
Bound and Conquer: Improving Triangulation by Enforcing Consistency
Authors:
Adam Scholefield,
Alireza Ghasemi,
Martin Vetterli
Abstract:
We study the accuracy of triangulation in multi-camera systems with respect to the number of cameras. We show that, under certain conditions, the optimal achievable reconstruction error decays quadratically as more cameras are added to the system. Furthermore, we analyse the error decay-rate of major state-of-the-art algorithms with respect to the number of cameras. To this end, we introduce the n…
▽ More
We study the accuracy of triangulation in multi-camera systems with respect to the number of cameras. We show that, under certain conditions, the optimal achievable reconstruction error decays quadratically as more cameras are added to the system. Furthermore, we analyse the error decay-rate of major state-of-the-art algorithms with respect to the number of cameras. To this end, we introduce the notion of consistency for triangulation, and show that consistent reconstruction algorithms achieve the optimal quadratic decay, which is asymptotically faster than some other methods. Finally, we present simulations results supporting our findings. Our simulations have been implemented in MATLAB and the resulting code is available in the supplementary material.
△ Less
Submitted 27 April, 2018;
originally announced April 2018.
-
On the Accuracy of Point Localisation in a Circular Camera-Array
Authors:
Alireza Ghasemi,
Adam Scholefield,
Martin Vetterli
Abstract:
Although many advances have been made in light-field and camera-array image processing, there is still a lack of thorough analysis of the localisation accuracy of different multi-camera systems. By considering the problem from a frame-quantisation perspective, we are able to quantify the point localisation error of circular camera configurations. Specifically, we obtain closed form expressions bou…
▽ More
Although many advances have been made in light-field and camera-array image processing, there is still a lack of thorough analysis of the localisation accuracy of different multi-camera systems. By considering the problem from a frame-quantisation perspective, we are able to quantify the point localisation error of circular camera configurations. Specifically, we obtain closed form expressions bounding the localisation error in terms of the parameters describing the acquisition setup.
These theoretical results are independent of the localisation algorithm and thus provide fundamental limits on performance. Furthermore, the new frame-quantisation perspective is general enough to be extended to more complex camera configurations.
△ Less
Submitted 24 February, 2016;
originally announced February 2016.
-
SHAPE: Linear-Time Camera Pose Estimation With Quadratic Error-Decay
Authors:
Alireza Ghasemi,
Adam Scholefield,
Martin Vetterli
Abstract:
We propose a novel camera pose estimation or perspective-n-point (PnP) algorithm, based on the idea of consistency regions and half-space intersections. Our algorithm has linear time-complexity and a squared reconstruction error that decreases at least quadratically, as the number of feature point correspondences increase.
Inspired by ideas from triangulation and frame quantisation theory, we de…
▽ More
We propose a novel camera pose estimation or perspective-n-point (PnP) algorithm, based on the idea of consistency regions and half-space intersections. Our algorithm has linear time-complexity and a squared reconstruction error that decreases at least quadratically, as the number of feature point correspondences increase.
Inspired by ideas from triangulation and frame quantisation theory, we define consistent reconstruction and then present SHAPE, our proposed consistent pose estimation algorithm. We compare this algorithm with state-of-the-art pose estimation techniques in terms of accuracy and error decay rate. The experimental results verify our hypothesis on the optimal worst-case quadratic decay and demonstrate its promising performance compared to other approaches.
△ Less
Submitted 24 February, 2016;
originally announced February 2016.
