Showing 1–43 of 43 results for author: Milz, S

From Chaos to Calibration: A Geometric Mutual Information Approach to Target-Free Camera LiDAR Extrinsic Calibration

Authors: Jack Borer, Jeremy Tschirner, Florian Ölsner, Stefan Milz

Abstract: Sensor fusion is vital for the safe and robust operation of autonomous vehicles. Accurate extrinsic sensor to sensor calibration is necessary to accurately fuse multiple sensor's data in a common spatial reference frame. In this paper, we propose a target free extrinsic calibration algorithm that requires no ground truth training data, artificially constrained motion trajectories, hand engineered… ▽ More Sensor fusion is vital for the safe and robust operation of autonomous vehicles. Accurate extrinsic sensor to sensor calibration is necessary to accurately fuse multiple sensor's data in a common spatial reference frame. In this paper, we propose a target free extrinsic calibration algorithm that requires no ground truth training data, artificially constrained motion trajectories, hand engineered features or offline optimization and that is accurate, precise and extremely robust to initialization error. Most current research on online camera-LiDAR extrinsic calibration requires ground truth training data which is impossible to capture at scale. We revisit analytical mutual information based methods first proposed in 2012 and demonstrate that geometric features provide a robust information metric for camera-LiDAR extrinsic calibration. We demonstrate our proposed improvement using the KITTI and KITTI-360 fisheye data set. △ Less

Submitted 3 November, 2023; originally announced November 2023.

Characterising the Hierarchy of Multi-time Quantum Processes with Classical Memory

Authors: Philip Taranto, Marco Túlio Quintino, Mio Murao, Simon Milz

Abstract: Memory is the fundamental form of temporal complexity: when present but uncontrollable, it manifests as non-Markovian noise; conversely, if controllable, memory can be a powerful resource for information processing. Memory effects arise from/are transmitted via interactions between a system and its environment; as such, they can be either classical or quantum. From a practical standpoint, quantum… ▽ More Memory is the fundamental form of temporal complexity: when present but uncontrollable, it manifests as non-Markovian noise; conversely, if controllable, memory can be a powerful resource for information processing. Memory effects arise from/are transmitted via interactions between a system and its environment; as such, they can be either classical or quantum. From a practical standpoint, quantum processes with classical memory promise near-term applicability: they are more powerful than their memoryless counterpart, yet at the same time can be controlled over significant timeframes without being spoiled by decoherence. However, despite practical and foundational value, apart from simple two-time scenarios, the distinction between quantum and classical memory remains unexplored. Here, we analyse multi-time quantum processes with memory mechanisms that transmit only classical information forward in time. Complementing this analysis, we also study two related -- but simpler to characterise -- sets of processes that could also be considered to have classical memory from a structural perspective, and demonstrate that these lead to remarkably distinct phenomena in the multi-time setting. Subsequently, we systematically stratify the full hierarchy of memory effects in quantum mechanics, many levels of which collapse in the two-time setting, making our results genuinely multi-time phenomena. △ Less

Submitted 15 April, 2024; v1 submitted 21 July, 2023; originally announced July 2023.

Comments: 12.5+4.5 pages, 4 figures, 73 references; close to published version

Journal ref: Quantum 8, 1328 (2024)

LiDAR-BEVMTN: Real-Time LiDAR Bird's-Eye View Multi-Task Perception Network for Autonomous Driving

Authors: Sambit Mohapatra, Senthil Yogamani, Varun Ravi Kumar, Stefan Milz, Heinrich Gotzig, Patrick Mäder

Abstract: LiDAR is crucial for robust 3D scene perception in autonomous driving. LiDAR perception has the largest body of literature after camera perception. However, multi-task learning across tasks like detection, segmentation, and motion estimation using LiDAR remains relatively unexplored, especially on automotive-grade embedded platforms. We present a real-time multi-task convolutional neural network f… ▽ More LiDAR is crucial for robust 3D scene perception in autonomous driving. LiDAR perception has the largest body of literature after camera perception. However, multi-task learning across tasks like detection, segmentation, and motion estimation using LiDAR remains relatively unexplored, especially on automotive-grade embedded platforms. We present a real-time multi-task convolutional neural network for LiDAR-based object detection, semantics, and motion segmentation. The unified architecture comprises a shared encoder and task-specific decoders, enabling joint representation learning. We propose a novel Semantic Weighting and Guidance (SWAG) module to transfer semantic features for improved object detection selectively. Our heterogeneous training scheme combines diverse datasets and exploits complementary cues between tasks. The work provides the first embedded implementation unifying these key perception tasks from LiDAR point clouds achieving 3ms latency on the embedded NVIDIA Xavier platform. We achieve state-of-the-art results for two tasks, semantic and motion segmentation, and close to state-of-the-art performance for 3D object detection. By maximizing hardware efficiency and leveraging multi-task synergies, our method delivers an accurate and efficient solution tailored for real-world automated driving deployment. Qualitative results can be seen at https://youtu.be/H-hWRzv2lIY. △ Less

Submitted 17 July, 2023; originally announced July 2023.

Continuous Online Extrinsic Calibration of Fisheye Camera and LiDAR

Authors: Jack Borer, Jeremy Tschirner, Florian Ölsner, Stefan Milz

Abstract: Automated driving systems use multi-modal sensor suites to ensure the reliable, redundant and robust perception of the operating domain, for example camera and LiDAR. An accurate extrinsic calibration is required to fuse the camera and LiDAR data into a common spatial reference frame required by high-level perception functions. Over the life of the vehicle the value of the extrinsic calibration ca… ▽ More Automated driving systems use multi-modal sensor suites to ensure the reliable, redundant and robust perception of the operating domain, for example camera and LiDAR. An accurate extrinsic calibration is required to fuse the camera and LiDAR data into a common spatial reference frame required by high-level perception functions. Over the life of the vehicle the value of the extrinsic calibration can change due physical disturbances, introducing an error into the high-level perception functions. Therefore there is a need for continuous online extrinsic calibration algorithms which can automatically update the value of the camera-LiDAR calibration during the life of the vehicle using only sensor data. We propose using mutual information between the camera image's depth estimate, provided by commonly available monocular depth estimation networks, and the LiDAR pointcloud's geometric distance as a optimization metric for extrinsic calibration. Our method requires no calibration target, no ground truth training data and no expensive offline optimization. We demonstrate our algorithm's accuracy, precision, speed and self-diagnosis capability on the KITTI-360 data set. △ Less

Submitted 22 June, 2023; originally announced June 2023.

Comments: 4 pages

MSC Class: 68U99 ACM Class: I.4.9

Witnessing environment dimension through temporal correlations

Authors: Lucas B. Vieira, Simon Milz, Giuseppe Vitagliano, Costantino Budroni

Abstract: We introduce a framework to compute upper bounds for temporal correlations achievable in open quantum system dynamics, obtained by repeated measurements on the system. As these correlations arise by virtue of the environment acting as a memory resource, such bounds are witnesses for the minimal dimension of an effective environment compatible with the observed statistics. These witnesses are deriv… ▽ More We introduce a framework to compute upper bounds for temporal correlations achievable in open quantum system dynamics, obtained by repeated measurements on the system. As these correlations arise by virtue of the environment acting as a memory resource, such bounds are witnesses for the minimal dimension of an effective environment compatible with the observed statistics. These witnesses are derived from a hierarchy of semidefinite programs with guaranteed asymptotic convergence. We compute non-trivial bounds for various sequences involving a qubit system and a qubit environment, and compare the results to the best known quantum strategies producing the same outcome sequences. Our results provide a numerically tractable method to determine bounds on multi-time probability distributions in open quantum system dynamics and allow for the witnessing of effective environment dimensions through probing of the system alone. △ Less

Submitted 5 January, 2024; v1 submitted 30 May, 2023; originally announced May 2023.

Comments: 24 pages, 7 figures

Journal ref: Quantum 8, 1224 (2024)

Transformations between arbitrary (quantum) objects and the emergence of indefinite causality

Authors: Simon Milz, Marco Túlio Quintino

Abstract: Many fundamental and key objects in quantum mechanics are linear map**s between particular affine/linear spaces. This structure includes basic quantum elements such as states, measurements, channels, instruments, non-signalling channels and channels with memory, and also higher-order operations such as superchannels, quantum combs, n-time processes, testers, and process matrices which may not re… ▽ More Many fundamental and key objects in quantum mechanics are linear map**s between particular affine/linear spaces. This structure includes basic quantum elements such as states, measurements, channels, instruments, non-signalling channels and channels with memory, and also higher-order operations such as superchannels, quantum combs, n-time processes, testers, and process matrices which may not respect a definite causal order. Deducing and characterising their structural properties in terms of linear and semidefinite constraints is not only of foundational relevance, but plays an important role in enabling the numerical optimization over sets of quantum objects and allowing simpler connections between different concepts and objects. Here, we provide a general framework to deduce these properties in a direct and easy to use way. Additionally, while primarily guided by practical quantum mechanical considerations, we extend our analysis to map**s between \textit{general} linear/affine spaces and derive their properties, opening the possibility for analysing sets which are not explicitly forbidden by quantum theory, but are still not much explored. Together, these results yield versatile and readily applicable tools for all tasks that require the characterization of linear transformations, in quantum mechanics and beyond. As an application of our methods, we discuss the emergence of indefinite causality in higher-order quantum transformation. △ Less

Submitted 2 May, 2023; originally announced May 2023.

Comments: 31 pages, 8 figures

Connecting Commutativity and Classicality for Multi-Time Quantum Processes

Authors: Fattah Sakuldee, Philip Taranto, Simon Milz

Abstract: Understanding the demarcation line between classical and quantum is an important issue in modern physics. The development of such an understanding requires a clear picture of the various concurrent notions of `classicality' in quantum theory presently in use. Here, we focus on the relationship between Kolmogorov consistency of measurement statistics -- the foundational footing of classical stochas… ▽ More Understanding the demarcation line between classical and quantum is an important issue in modern physics. The development of such an understanding requires a clear picture of the various concurrent notions of `classicality' in quantum theory presently in use. Here, we focus on the relationship between Kolmogorov consistency of measurement statistics -- the foundational footing of classical stochastic processes in standard probability theory -- and the commutativity (or absence thereof) of measurement operators -- a concept at the core of quantum theory. Kolmogorov consistency implies that the statistics of sequential measurements on a (possibly quantum) system could be explained entirely by means of a classical stochastic process, thereby providing an operational notion of classicality. On the other hand, commutativity of measurement operators is a structural property that holds in classical physics and its breakdown is the origin of the uncertainty principle, a fundamentally quantum phenomenon. Here, we formalise the connection between these two a priori independent notions of classicality, demonstrate that they are distinct in general and detail their implications for memoryless multi-time quantum processes. △ Less

Submitted 26 August, 2022; v1 submitted 25 April, 2022; originally announced April 2022.

Comments: 14.5 pages, 2 figures. Close to published version

Journal ref: Phys. Rev. A 106, 022416 (2022)

Hidden Quantum Memory: Is Memory There When Somebody Looks?

Authors: Philip Taranto, Thomas J. Elliott, Simon Milz

Abstract: In classical physics, memoryless dynamics and Markovian statistics are one and the same. This is not true for quantum dynamics, first and foremost because quantum measurements are invasive. Going beyond measurement invasiveness, here we derive a novel distinction between classical and quantum processes, namely the possibility of hidden quantum memory. While Markovian statistics of classical proces… ▽ More In classical physics, memoryless dynamics and Markovian statistics are one and the same. This is not true for quantum dynamics, first and foremost because quantum measurements are invasive. Going beyond measurement invasiveness, here we derive a novel distinction between classical and quantum processes, namely the possibility of hidden quantum memory. While Markovian statistics of classical processes can always be reproduced by a memoryless dynamical model, our main result shows that this is not true in quantum mechanics: We first provide an example of quantum non-Markovianity whose manifestation depends on whether or not a previous measurement is performed -- an impossible phenomenon for memoryless dynamics; we then strengthen this result by demonstrating statistics that are Markovian independent of how they are probed, but are nonetheless still incompatible with memoryless quantum dynamics. Thus, we establish the existence of Markovian statistics gathered by probing a quantum process that nevertheless fundamentally require memory for their creation. △ Less

Submitted 24 April, 2023; v1 submitted 18 April, 2022; originally announced April 2022.

Comments: 7 + 9 pages, 5 figures

Journal ref: Quantum 7, 991 (2023)

LiMoSeg: Real-time Bird's Eye View based LiDAR Motion Segmentation

Authors: Sambit Mohapatra, Mona Hodaei, Senthil Yogamani, Stefan Milz, Heinrich Gotzig, Martin Simon, Hazem Rashed, Patrick Maeder

Abstract: Moving object detection and segmentation is an essential task in the Autonomous Driving pipeline. Detecting and isolating static and moving components of a vehicle's surroundings are particularly crucial in path planning and localization tasks. This paper proposes a novel real-time architecture for motion segmentation of Light Detection and Ranging (LiDAR) data. We use three successive scans of Li… ▽ More Moving object detection and segmentation is an essential task in the Autonomous Driving pipeline. Detecting and isolating static and moving components of a vehicle's surroundings are particularly crucial in path planning and localization tasks. This paper proposes a novel real-time architecture for motion segmentation of Light Detection and Ranging (LiDAR) data. We use three successive scans of LiDAR data in 2D Bird's Eye View (BEV) representation to perform pixel-wise classification as static or moving. Furthermore, we propose a novel data augmentation technique to reduce the significant class imbalance between static and moving objects. We achieve this by artificially synthesizing moving objects by cutting and pasting static vehicles. We demonstrate a low latency of 8 ms on a commonly used automotive embedded platform, namely Nvidia Jetson Xavier. To the best of our knowledge, this is the first work directly performing motion segmentation in LiDAR BEV space. We provide quantitative results on the challenging SemanticKITTI dataset, and qualitative results are provided in https://youtu.be/2aJ-cL8b0LI. △ Less

Submitted 22 January, 2022; v1 submitted 8 November, 2021; originally announced November 2021.

Comments: Accepted for Presentation at International Conference on Computer Vision Theory and Applications (VISAPP 2022)

Resource theory of causal connection

Authors: Simon Milz, Jessica Bavaresco, Giulio Chiribella

Abstract: The capacity of distant parties to send signals to one another is a fundamental requirement in many information-processing tasks. Such ability is determined by the causal structure connecting the parties, and more generally, by the intermediate processes carrying signals from one laboratory to another. Here we build a fully fledged resource theory of causal connection for all multi-party communica… ▽ More The capacity of distant parties to send signals to one another is a fundamental requirement in many information-processing tasks. Such ability is determined by the causal structure connecting the parties, and more generally, by the intermediate processes carrying signals from one laboratory to another. Here we build a fully fledged resource theory of causal connection for all multi-party communication scenarios, encompassing those where the parties operate in a definite causal order and also where the order is indefinite. We define and characterize the set of free processes and three different sets of free transformations thereof, resulting in three distinct resource theories of causal connection. In the causally ordered setting, we identify the most resourceful processes in the bipartite and tripartite scenarios. In the general setting, instead, our results suggest that there is no global most valuable resource. We establish the signalling robustness as a resource monotone of causal connection and provide tight bounds on it for many pertinent sets of processes. Finally, we introduce a resource theory of causal non-separability, and show that it is -- in contrast to the case of causal connection -- unique. Together our results offer a flexible and comprehensive framework to quantify and transform general quantum processes, as well as insights into their multi-layered causal connection structures. △ Less

Submitted 23 August, 2022; v1 submitted 7 October, 2021; originally announced October 2021.

Comments: 39+14 pages, 19 figures

Journal ref: Quantum 6, 788 (2022)

Extracting Quantum Dynamical Resources: Consumption of Non-Markovianity for Noise Reduction

Authors: Graeme D. Berk, Simon Milz, Felix A. Pollock, Kavan Modi

Abstract: Noise is possibly the most formidable challenge for quantum technologies. As such, a great deal of effort is dedicated to develo** methods for noise reduction. One remarkable achievement in this direction is dynamical decoupling; it details a clear set of instructions for counteracting the effects of quantum noise. Yet, the domain of its applicability remains limited to devices where exercising… ▽ More Noise is possibly the most formidable challenge for quantum technologies. As such, a great deal of effort is dedicated to develo** methods for noise reduction. One remarkable achievement in this direction is dynamical decoupling; it details a clear set of instructions for counteracting the effects of quantum noise. Yet, the domain of its applicability remains limited to devices where exercising fast control is possible. In practical terms, this is highly limiting and there is a growing need for better noise reduction tools. Here we take a significant step in this direction, by identifying the crucial ingredients required for noise suppression and the development of methods that far outperform traditional dynamical decoupling techniques. Using resource theoretic methods, we show that the key resource responsible for the efficacy of dynamical decoupling, and related protocols, is non-Markovianity (or temporal correlations). Using this insight, we then propose two methods to identify optimal pulse sequences for noise reduction. With an explicit example, we show that our methods enable a more optimal exploitation of temporal correlations, and extend the timescales at which noise suppression is viable by at least two orders of magnitude. Importantly, the corresponding tools are built on operational grounds and are easily implemented in the current generation of quantum devices. △ Less

Submitted 6 October, 2021; originally announced October 2021.

Comments: 13+7 pages, 4+2 figures

Journal ref: npj Quantum Inf 9, 104 (2023)

BEVDetNet: Bird's Eye View LiDAR Point Cloud based Real-time 3D Object Detection for Autonomous Driving

Authors: Sambit Mohapatra, Senthil Yogamani, Heinrich Gotzig, Stefan Milz, Patrick Mader

Abstract: 3D object detection based on LiDAR point clouds is a crucial module in autonomous driving particularly for long range sensing. Most of the research is focused on achieving higher accuracy and these models are not optimized for deployment on embedded systems from the perspective of latency and power efficiency. For high speed driving scenarios, latency is a crucial parameter as it provides more tim… ▽ More 3D object detection based on LiDAR point clouds is a crucial module in autonomous driving particularly for long range sensing. Most of the research is focused on achieving higher accuracy and these models are not optimized for deployment on embedded systems from the perspective of latency and power efficiency. For high speed driving scenarios, latency is a crucial parameter as it provides more time to react to dangerous situations. Typically a voxel or point-cloud based 3D convolution approach is utilized for this module. Firstly, they are inefficient on embedded platforms as they are not suitable for efficient parallelization. Secondly, they have a variable runtime due to level of sparsity of the scene which is against the determinism needed in a safety system. In this work, we aim to develop a very low latency algorithm with fixed runtime. We propose a novel semantic segmentation architecture as a single unified model for object center detection using key points, box predictions and orientation prediction using binned classification in a simpler Bird's Eye View (BEV) 2D representation. The proposed architecture can be trivially extended to include semantic segmentation classes like road without any additional computation. The proposed model has a latency of 4 ms on the embedded Nvidia Xavier platform. The model is 5X faster than other top accuracy models with a minimal accuracy degradation of 2% in Average Precision at IoU=0.5 on KITTI dataset. △ Less

Submitted 10 July, 2021; v1 submitted 21 April, 2021; originally announced April 2021.

Comments: Accepted for Oral Presentation at IEEE Intelligent Transportation Systems Conference (ITSC) 2021

SVDistNet: Self-Supervised Near-Field Distance Estimation on Surround View Fisheye Cameras

Authors: Varun Ravi Kumar, Marvin Klingner, Senthil Yogamani, Markus Bach, Stefan Milz, Tim Fingscheidt, Patrick Mäder

Abstract: A 360° perception of scene geometry is essential for automated driving, notably for parking and urban driving scenarios. Typically, it is achieved using surround-view fisheye cameras, focusing on the near-field area around the vehicle. The majority of current depth estimation approaches focus on employing just a single camera, which cannot be straightforwardly generalized to multiple cameras. The… ▽ More A 360° perception of scene geometry is essential for automated driving, notably for parking and urban driving scenarios. Typically, it is achieved using surround-view fisheye cameras, focusing on the near-field area around the vehicle. The majority of current depth estimation approaches focus on employing just a single camera, which cannot be straightforwardly generalized to multiple cameras. The depth estimation model must be tested on a variety of cameras equipped to millions of cars with varying camera geometries. Even within a single car, intrinsics vary due to manufacturing tolerances. Deep learning models are sensitive to these changes, and it is practically infeasible to train and test on each camera variant. As a result, we present novel camera-geometry adaptive multi-scale convolutions which utilize the camera parameters as a conditional input, enabling the model to generalize to previously unseen fisheye cameras. Additionally, we improve the distance estimation by pairwise and patchwise vector-based self-attention encoder networks. We evaluate our approach on the Fisheye WoodScape surround-view dataset, significantly improving over previous approaches. We also show a generalization of our approach across different camera viewing angles and perform extensive experiments to support our contributions. To enable comparison with other approaches, we evaluate the front camera data on the KITTI dataset (pinhole camera images) and achieve state-of-the-art performance among self-supervised monocular methods. An overview video with qualitative results is provided at https://youtu.be/bmX0UcU9wtA. Baseline code and dataset will be made public. △ Less

Submitted 9 April, 2021; originally announced April 2021.

Comments: To be published at IEEE Transactions on Intelligent Transportation Systems

OmniDet: Surround View Cameras based Multi-task Visual Perception Network for Autonomous Driving

Authors: Varun Ravi Kumar, Senthil Yogamani, Hazem Rashed, Ganesh Sistu, Christian Witt, Isabelle Leang, Stefan Milz, Patrick Mäder

Abstract: Surround View fisheye cameras are commonly deployed in automated driving for 360° near-field sensing around the vehicle. This work presents a multi-task visual perception network on unrectified fisheye images to enable the vehicle to sense its surrounding environment. It consists of six primary tasks necessary for an autonomous driving system: depth estimation, visual odometry, semantic segmentati… ▽ More Surround View fisheye cameras are commonly deployed in automated driving for 360° near-field sensing around the vehicle. This work presents a multi-task visual perception network on unrectified fisheye images to enable the vehicle to sense its surrounding environment. It consists of six primary tasks necessary for an autonomous driving system: depth estimation, visual odometry, semantic segmentation, motion segmentation, object detection, and lens soiling detection. We demonstrate that the jointly trained model performs better than the respective single task versions. Our multi-task model has a shared encoder providing a significant computational advantage and has synergized decoders where tasks support each other. We propose a novel camera geometry based adaptation mechanism to encode the fisheye distortion model both at training and inference. This was crucial to enable training on the WoodScape dataset, comprised of data from different parts of the world collected by 12 different cameras mounted on three different cars with different intrinsics and viewpoints. Given that bounding boxes is not a good representation for distorted fisheye images, we also extend object detection to use a polygon with non-uniformly sampled vertices. We additionally evaluate our model on standard automotive datasets, namely KITTI and Cityscapes. We obtain the state-of-the-art results on KITTI for depth estimation and pose estimation tasks and competitive performance on the other tasks. We perform extensive ablation studies on various architecture choices and task weighting methodologies. A short video at https://youtu.be/xbSjZ5OfPes provides qualitative results. △ Less

Submitted 6 June, 2023; v1 submitted 15 February, 2021; originally announced February 2021.

Comments: Best Robot Vision paper award finalist (top 4). Camera ready version accepted for RA-L and ICRA 2021 publication

Quantum stochastic processes and quantum non-Markovian phenomena

Authors: Simon Milz, Kavan Modi

Abstract: The field of classical stochastic processes forms a major branch of mathematics. They are, of course, also very well studied in biology, chemistry, ecology, geology, finance, physics, and many more fields of natural and social sciences. When it comes to quantum stochastic processes, however, the topic is plagued with pathological issues that have led to fierce debates amongst researchers. Recent d… ▽ More The field of classical stochastic processes forms a major branch of mathematics. They are, of course, also very well studied in biology, chemistry, ecology, geology, finance, physics, and many more fields of natural and social sciences. When it comes to quantum stochastic processes, however, the topic is plagued with pathological issues that have led to fierce debates amongst researchers. Recent developments have begun to untangle these issues and paved the way for generalizing the theory of classical stochastic processes to the quantum domain without ambiguities. This tutorial details the structure of quantum stochastic processes, in terms of the modern language of quantum combs, and is aimed at students in quantum physics and quantum information theory. We begin with the basics of classical stochastic processes and generalize the same ideas to the quantum domain. Along the way, we discuss the subtle structure of quantum physics that has led to troubles in forming an overarching theory for quantum stochastic processes. We close the tutorial by laying out many exciting problems that lie ahead in this branch of science. △ Less

Submitted 10 May, 2021; v1 submitted 3 December, 2020; originally announced December 2020.

Comments: 69 pages, 33 figures. Comments welcome!

Journal ref: PRX Quantum 2, 030201 (2021)

Genuine Multipartite Entanglement in Time

Authors: Simon Milz, Cornelia Spee, Zhen-Peng Xu, Felix A. Pollock, Kavan Modi, Otfried Gühne

Abstract: While spatial quantum correlations have been studied in great detail, much less is known about the genuine quantum correlations that can be exhibited by temporal processes. Employing the quantum comb formalism, processes in time can be mapped onto quantum states, with the crucial difference that temporal correlations have to satisfy causal ordering, while their spatial counterpart is not constrain… ▽ More While spatial quantum correlations have been studied in great detail, much less is known about the genuine quantum correlations that can be exhibited by temporal processes. Employing the quantum comb formalism, processes in time can be mapped onto quantum states, with the crucial difference that temporal correlations have to satisfy causal ordering, while their spatial counterpart is not constrained in the same way. Here, we exploit this equivalence and use the tools of multipartite entanglement theory to provide a comprehensive picture of the structure of correlations that (causally ordered) temporal quantum processes can display. First, focusing on the case of a process that is probed at two points in time -- which can equivalently be described by a tripartite quantum state -- we provide necessary as well as sufficient conditions for the presence of bipartite entanglement in different splittings. Next, we connect these scenarios to the previously studied concepts of quantum memory, entanglement breaking superchannels, and quantum steering, thus providing both a physical interpretation for entanglement in temporal quantum processes, and a determination of the resources required for its creation. Additionally, we construct explicit examples of W-type and GHZ-type genuinely multipartite entangled two-time processes and prove that genuine multipartite entanglement in temporal processes can be an emergent phenomenon. Finally, we show that genuinely entangled processes across multiple times exist for any number of probing times. △ Less

Submitted 17 June, 2021; v1 submitted 18 November, 2020; originally announced November 2020.

Comments: 33+14 pages, 15 figures. Close to published version

Journal ref: SciPost Phys. 10, 141 (2021)

Quantum chicken-egg dilemmas: Delayed-choice causal order and the reality of causal non-separability

Authors: Simon Milz, Dominic Jurkschat, Felix A. Pollock, Kavan Modi

Abstract: Recent frameworks describing quantum mechanics in the absence of a global causal order admit the existence of causally indefinite processes, where it is impossible to ascribe causal order for events A and B. These frameworks even allow for processes that violate the so-called causal inequalities, which are analogous to Bell's inequalities. However, the physicality of these exotic processes is, in… ▽ More Recent frameworks describing quantum mechanics in the absence of a global causal order admit the existence of causally indefinite processes, where it is impossible to ascribe causal order for events A and B. These frameworks even allow for processes that violate the so-called causal inequalities, which are analogous to Bell's inequalities. However, the physicality of these exotic processes is, in the general case, still under debate, bringing into question their foundational relevance. While it is known that causally indefinite processes can be probabilistically realised by means of a quantum circuit, along with an additional conditioning event C, concrete insights into the ontological meaning of such implementation schemes have heretofore been limited. Here, we show that causally indefinite processes can be realised with schemes where C serves only as a classical flag heralding which causally indefinite process was realised. We then show that there are processes where any pure conditioning measurement of C leads to a causally indefinite process for A and B, thus establishing causal indefiniteness as a basis-independent quantity. Finally, we demonstrate that quantum mechanics allows for phenomena where C can deterministically decide whether A comes before B or vice versa, without signalling to either. This is akin to Wheeler's famous delayed-choice experiment establishing definite causal order in quantum mechanics as instrument-\textit{dependent} property. △ Less

Submitted 10 May, 2021; v1 submitted 18 August, 2020; originally announced August 2020.

Comments: 14+6 pages, 4 figures. Close to published version

Journal ref: Phys. Rev. Research 3, 023028 (2021)

SynDistNet: Self-Supervised Monocular Fisheye Camera Distance Estimation Synergized with Semantic Segmentation for Autonomous Driving

Authors: Varun Ravi Kumar, Marvin Klingner, Senthil Yogamani, Stefan Milz, Tim Fingscheidt, Patrick Maeder

Abstract: State-of-the-art self-supervised learning approaches for monocular depth estimation usually suffer from scale ambiguity. They do not generalize well when applied on distance estimation for complex projection models such as in fisheye and omnidirectional cameras. This paper introduces a novel multi-task learning strategy to improve self-supervised monocular distance estimation on fisheye and pinhol… ▽ More State-of-the-art self-supervised learning approaches for monocular depth estimation usually suffer from scale ambiguity. They do not generalize well when applied on distance estimation for complex projection models such as in fisheye and omnidirectional cameras. This paper introduces a novel multi-task learning strategy to improve self-supervised monocular distance estimation on fisheye and pinhole camera images. Our contribution to this work is threefold: Firstly, we introduce a novel distance estimation network architecture using a self-attention based encoder coupled with robust semantic feature guidance to the decoder that can be trained in a one-stage fashion. Secondly, we integrate a generalized robust loss function, which improves performance significantly while removing the need for hyperparameter tuning with the reprojection loss. Finally, we reduce the artifacts caused by dynamic objects violating static world assumptions using a semantic masking strategy. We significantly improve upon the RMSE of previous work on fisheye by 25% reduction in RMSE. As there is little work on fisheye cameras, we evaluated the proposed method on KITTI using a pinhole model. We achieved state-of-the-art performance among self-supervised methods without requiring an external scale estimation. △ Less

Submitted 14 November, 2020; v1 submitted 10 August, 2020; originally announced August 2020.

Comments: Camera ready version + supplementary. Accepted for presentation at Winter Conference on Applications of Computer Vision 2021

UnRectDepthNet: Self-Supervised Monocular Depth Estimation using a Generic Framework for Handling Common Camera Distortion Models

Authors: Varun Ravi Kumar, Senthil Yogamani, Markus Bach, Christian Witt, Stefan Milz, Patrick Mader

Abstract: In classical computer vision, rectification is an integral part of multi-view depth estimation. It typically includes epipolar rectification and lens distortion correction. This process simplifies the depth estimation significantly, and thus it has been adopted in CNN approaches. However, rectification has several side effects, including a reduced field of view (FOV), resampling distortion, and se… ▽ More In classical computer vision, rectification is an integral part of multi-view depth estimation. It typically includes epipolar rectification and lens distortion correction. This process simplifies the depth estimation significantly, and thus it has been adopted in CNN approaches. However, rectification has several side effects, including a reduced field of view (FOV), resampling distortion, and sensitivity to calibration errors. The effects are particularly pronounced in case of significant distortion (e.g., wide-angle fisheye cameras). In this paper, we propose a generic scale-aware self-supervised pipeline for estimating depth, euclidean distance, and visual odometry from unrectified monocular videos. We demonstrate a similar level of precision on the unrectified KITTI dataset with barrel distortion comparable to the rectified KITTI dataset. The intuition being that the rectification step can be implicitly absorbed within the CNN model, which learns the distortion model without increasing complexity. Our approach does not suffer from a reduced field of view and avoids computational costs for rectification at inference time. To further illustrate the general applicability of the proposed framework, we apply it to wide-angle fisheye cameras with 190$^\circ$ horizontal field of view. The training framework UnRectDepthNet takes in the camera distortion model as an argument and adapts projection and unprojection functions accordingly. The proposed algorithm is evaluated further on the KITTI rectified dataset, and we achieve state-of-the-art results that improve upon our previous work FisheyeDistanceNet. Qualitative results on a distorted test scene video sequence indicate excellent performance https://youtu.be/K6pbx3bU4Ss. △ Less

Submitted 6 June, 2023; v1 submitted 13 July, 2020; originally announced July 2020.

Comments: Minor fixes added after IROS 2020 Camera ready submission. IROS 2020 presentation video - https://www.youtube.com/watch?v=3Br2KSWZRrY

StickyPillars: Robust and Efficient Feature Matching on Point Clouds using Graph Neural Networks

Authors: Kai Fischer, Martin Simon, Florian Oelsner, Stefan Milz, Horst-Michael Gross, Patrick Maeder

Abstract: Robust point cloud registration in real-time is an important prerequisite for many map** and localization algorithms. Traditional methods like ICP tend to fail without good initialization, insufficient overlap or in the presence of dynamic objects. Modern deep learning based registration approaches present much better results, but suffer from a heavy run-time. We overcome these drawbacks by intr… ▽ More Robust point cloud registration in real-time is an important prerequisite for many map** and localization algorithms. Traditional methods like ICP tend to fail without good initialization, insufficient overlap or in the presence of dynamic objects. Modern deep learning based registration approaches present much better results, but suffer from a heavy run-time. We overcome these drawbacks by introducing StickyPillars, a fast, accurate and extremely robust deep middle-end 3D feature matching method on point clouds. It uses graph neural networks and performs context aggregation on sparse 3D key-points with the aid of transformer based multi-head self and cross-attention. The network output is used as the cost for an optimal transport problem whose solution yields the final matching probabilities. The system does not rely on hand crafted feature descriptors or heuristic matching strategies. We present state-of-art art accuracy results on the registration problem demonstrated on the KITTI dataset while being four times faster then leading deep methods. Furthermore, we integrate our matching system into a LiDAR odometry pipeline yielding most accurate results on the KITTI odometry dataset. Finally, we demonstrate robustness on KITTI odometry. Our method remains stable in accuracy where state-of-the-art procedures fail on frame drops and higher speeds. △ Less

Submitted 19 February, 2021; v1 submitted 10 February, 2020; originally announced February 2020.

FisheyeDistanceNet: Self-Supervised Scale-Aware Distance Estimation using Monocular Fisheye Camera for Autonomous Driving

Authors: Varun Ravi Kumar, Sandesh Athni Hiremath, Stefan Milz, Christian Witt, Clement Pinnard, Senthil Yogamani, Patrick Mader

Abstract: Fisheye cameras are commonly used in applications like autonomous driving and surveillance to provide a large field of view ($>180^{\circ}$). However, they come at the cost of strong non-linear distortions which require more complex algorithms. In this paper, we explore Euclidean distance estimation on fisheye cameras for automotive scenes. Obtaining accurate and dense depth supervision is difficu… ▽ More Fisheye cameras are commonly used in applications like autonomous driving and surveillance to provide a large field of view ($>180^{\circ}$). However, they come at the cost of strong non-linear distortions which require more complex algorithms. In this paper, we explore Euclidean distance estimation on fisheye cameras for automotive scenes. Obtaining accurate and dense depth supervision is difficult in practice, but self-supervised learning approaches show promising results and could potentially overcome the problem. We present a novel self-supervised scale-aware framework for learning Euclidean distance and ego-motion from raw monocular fisheye videos without applying rectification. While it is possible to perform piece-wise linear approximation of fisheye projection surface and apply standard rectilinear models, it has its own set of issues like re-sampling distortion and discontinuities in transition regions. To encourage further research in this area, we will release our dataset as part of the WoodScape project \cite{yogamani2019woodscape}. We further evaluated the proposed algorithm on the KITTI dataset and obtained state-of-the-art results comparable to other self-supervised monocular methods. Qualitative results on an unseen fisheye video demonstrate impressive performance https://youtu.be/Sgq1WzoOmXg. △ Less

Submitted 6 October, 2020; v1 submitted 7 October, 2019; originally announced October 2019.

Comments: Minor fixes added after ICRA 2020 camera ready submission. ICRA 2020 presentation video - https://www.youtube.com/watch?v=qAsdpHP5e8c&t

Improving Map Re-localization with Deep 'Movable' Objects Segmentation on 3D LiDAR Point Clouds

Authors: Victor Vaquero, Kai Fischer, Francesc Moreno-Noguer, Alberto Sanfeliu, Stefan Milz

Abstract: Localization and Map** is an essential component to enable Autonomous Vehicles navigation, and requires an accuracy exceeding that of commercial GPS-based systems. Current odometry and map** algorithms are able to provide this accurate information. However, the lack of robustness of these algorithms against dynamic obstacles and environmental changes, even for short time periods, forces the ge… ▽ More Localization and Map** is an essential component to enable Autonomous Vehicles navigation, and requires an accuracy exceeding that of commercial GPS-based systems. Current odometry and map** algorithms are able to provide this accurate information. However, the lack of robustness of these algorithms against dynamic obstacles and environmental changes, even for short time periods, forces the generation of new maps on every session without taking advantage of previously obtained ones. In this paper we propose the use of a deep learning architecture to segment movable objects from 3D LiDAR point clouds in order to obtain longer-lasting 3D maps. This will in turn allow for better, faster and more accurate re-localization and trajectoy estimation on subsequent days. We show the effectiveness of our approach in a very dynamic and cluttered scenario, a supermarket parking lot. For that, we record several sequences on different days and compare localization errors with and without our movable objects segmentation method. Results show that we are able to accurately re-locate over a filtered map, consistently reducing trajectory errors between an average of 35.1% with respect to a non-filtered map version and of 47.9% with respect to a standalone map created on the current session. △ Less

Submitted 8 October, 2019; originally announced October 2019.

When is a non-Markovian quantum process classical?

Authors: Simon Milz, Dario Egloff, Philip Taranto, Thomas Theurer, Martin B. Plenio, Andrea Smirne, Susana F. Huelga

Abstract: More than a century after the inception of quantum theory, the question of which traits and phenomena are fundamentally quantum remains under debate. Here we give an answer to this question for temporal processes which are probed sequentially by means of projective measurements of the same observable. Defining classical processes as those that can---in principle---be simulated by means of classica… ▽ More More than a century after the inception of quantum theory, the question of which traits and phenomena are fundamentally quantum remains under debate. Here we give an answer to this question for temporal processes which are probed sequentially by means of projective measurements of the same observable. Defining classical processes as those that can---in principle---be simulated by means of classical resources only, we fully characterize the set of such processes. Based on this characterization, we show that for non-Markovian processes (i.e., processes with memory), the absence of coherence does not guarantee the classicality of observed phenomena and furthermore derive an experimentally and computationally accessible measure for non-classicality in the presence of memory. We then provide a direct connection between classicality and the vanishing of quantum discord between the evolving system and its environment. Finally, we demonstrate that---in contrast to the memoryless setting---in the non-Markovian case, there exist processes that are genuinely quantum, i.e., they display non-classical statistics independent of the measurement scheme that is employed to probe them. △ Less

Submitted 11 August, 2020; v1 submitted 12 July, 2019; originally announced July 2019.

Comments: 26+16 pages, 15 figures

Journal ref: Phys. Rev. X 10, 041049 (2020)

WoodScape: A multi-task, multi-camera fisheye dataset for autonomous driving

Authors: Senthil Yogamani, Ciaran Hughes, Jonathan Horgan, Ganesh Sistu, Padraig Varley, Derek O'Dea, Michal Uricar, Stefan Milz, Martin Simon, Karl Amende, Christian Witt, Hazem Rashed, Sumanth Chennupati, Sanjaya Nayak, Saquib Mansoor, Xavier Perroton, Patrick Perez

Abstract: Fisheye cameras are commonly employed for obtaining a large field of view in surveillance, augmented reality and in particular automotive applications. In spite of their prevalence, there are few public datasets for detailed evaluation of computer vision algorithms on fisheye images. We release the first extensive fisheye automotive dataset, WoodScape, named after Robert Wood who invented the fish… ▽ More Fisheye cameras are commonly employed for obtaining a large field of view in surveillance, augmented reality and in particular automotive applications. In spite of their prevalence, there are few public datasets for detailed evaluation of computer vision algorithms on fisheye images. We release the first extensive fisheye automotive dataset, WoodScape, named after Robert Wood who invented the fisheye camera in 1906. WoodScape comprises of four surround view cameras and nine tasks including segmentation, depth estimation, 3D bounding box detection and soiling detection. Semantic annotation of 40 classes at the instance level is provided for over 10,000 images and annotation for other tasks are provided for over 100,000 images. With WoodScape, we would like to encourage the community to adapt computer vision models for fisheye camera instead of using naive rectification. △ Less

Submitted 2 July, 2021; v1 submitted 4 May, 2019; originally announced May 2019.

Comments: Accepted for Oral Presentation at IEEE International Conference on Computer Vision (ICCV) 2019. Please refer to our website https://woodscape.valeo.com and https://github.com/valeoai/woodscape for release status and updates

Complexer-YOLO: Real-Time 3D Object Detection and Tracking on Semantic Point Clouds

Authors: Martin Simon, Karl Amende, Andrea Kraus, Jens Honer, Timo Sämann, Hauke Kaulbersch, Stefan Milz, Horst Michael Gross

Abstract: Accurate detection of 3D objects is a fundamental problem in computer vision and has an enormous impact on autonomous cars, augmented/virtual reality and many applications in robotics. In this work we present a novel fusion of neural network based state-of-the-art 3D detector and visual semantic segmentation in the context of autonomous driving. Additionally, we introduce Scale-Rotation-Translatio… ▽ More Accurate detection of 3D objects is a fundamental problem in computer vision and has an enormous impact on autonomous cars, augmented/virtual reality and many applications in robotics. In this work we present a novel fusion of neural network based state-of-the-art 3D detector and visual semantic segmentation in the context of autonomous driving. Additionally, we introduce Scale-Rotation-Translation score (SRTs), a fast and highly parameterizable evaluation metric for comparison of object detections, which speeds up our inference time up to 20\% and halves training time. On top, we apply state-of-the-art online multi target feature tracking on the object measurements to further increase accuracy and robustness utilizing temporal information. Our experiments on KITTI show that we achieve same results as state-of-the-art in all related categories, while maintaining the performance and accuracy trade-off and still run in real-time. Furthermore, our model is the first one that fuses visual semantic with 3D object detection. △ Less

Submitted 16 April, 2019; originally announced April 2019.

Exploring Deep Spiking Neural Networks for Automated Driving Applications

Authors: Sambit Mohapatra, Heinrich Gotzig, Senthil Yogamani, Stefan Milz, Raoul Zollner

Abstract: Neural networks have become the standard model for various computer vision tasks in automated driving including semantic segmentation, moving object detection, depth estimation, visual odometry, etc. The main flavors of neural networks which are used commonly are convolutional (CNN) and recurrent (RNN). In spite of rapid progress in embedded processors, power consumption and cost is still a bottle… ▽ More Neural networks have become the standard model for various computer vision tasks in automated driving including semantic segmentation, moving object detection, depth estimation, visual odometry, etc. The main flavors of neural networks which are used commonly are convolutional (CNN) and recurrent (RNN). In spite of rapid progress in embedded processors, power consumption and cost is still a bottleneck. Spiking Neural Networks (SNNs) are gradually progressing to achieve low-power event-driven hardware architecture which has a potential for high efficiency. In this paper, we explore the role of deep spiking neural networks (SNN) for automated driving applications. We provide an overview of progress on SNN and argue how it can be a good fit for automated driving applications. △ Less

Submitted 11 January, 2019; originally announced March 2019.

Comments: Accepted for Oral Presentation at VISAPP 2019

Realistic Ultrasonic Environment Simulation Using Conditional Generative Adversarial Networks

Authors: Maximilian Pöpperl, Raghavendra Gulagundi, Senthil Yogamani, Stefan Milz

Abstract: Recently, realistic data augmentation using neural networks especially generative neural networks (GAN) has achieved outstanding results. The communities main research focus is visual image processing. However, automotive cars and robots are equipped with a large suite of sensors to achieve a high redundancy. In addition to others, ultrasonic sensors are often used due to their low-costs and relia… ▽ More Recently, realistic data augmentation using neural networks especially generative neural networks (GAN) has achieved outstanding results. The communities main research focus is visual image processing. However, automotive cars and robots are equipped with a large suite of sensors to achieve a high redundancy. In addition to others, ultrasonic sensors are often used due to their low-costs and reliable near field distance measuring capabilities. Hence, Pattern recognition needs to be applied to ultrasonic signals as well. Machine Learning requires extensive data sets and those measurements are time-consuming, expensive and not flexible to hardware and environmental changes. On the other hand, there exists no method to simulate those signals deterministically. We present a novel approach for synthetic ultrasonic signal simulation using conditional GANs (cGANs). For the best of our knowledge, we present the first realistic data augmentation for automotive ultrasonics. The performance of cGANs allows us to bring the realistic environment simulation to a new level. By using setup and environmental parameters as condition, the proposed approach is flexible to external influences. Due to the low complexity and time effort for data generation, we outperform other simulation algorithms, such as finite element method. We verify the outstanding accuracy and realism of our method by applying a detailed statistical analysis and comparing the generated data to an extensive amount of measured signals. △ Less

Submitted 26 February, 2019; originally announced February 2019.

Capsule Neural Network based Height Classification using Low-Cost Automotive Ultrasonic Sensors

Authors: Maximilian Pöpperl, Raghavendra Gulagundi, Senthil Yogamani, Stefan Milz

Abstract: High performance ultrasonic sensor hardware is mainly used in medical applications. Although, the development in automotive scenarios is towards autonomous driving, the ultrasonic sensor hardware still stays low-cost and low-performance, respectively. To overcome the strict hardware limitations, we propose to use capsule neural networks. By the high classification capability of this network archit… ▽ More High performance ultrasonic sensor hardware is mainly used in medical applications. Although, the development in automotive scenarios is towards autonomous driving, the ultrasonic sensor hardware still stays low-cost and low-performance, respectively. To overcome the strict hardware limitations, we propose to use capsule neural networks. By the high classification capability of this network architecture, we can achieve outstanding results for performing a detailed height analysis of detected objects. We apply a novel resorting and resha** method to feed the neural network with ultrasonic data. This increases classification performance and computation speed. We tested the approach under different environmental conditions to verify that the proposed method is working independent of external parameters that is needed for autonomous driving. △ Less

Submitted 26 February, 2019; originally announced February 2019.

NeurAll: Towards a Unified Visual Perception Model for Automated Driving

Authors: Ganesh Sistu, Isabelle Leang, Sumanth Chennupati, Senthil Yogamani, Ciaran Hughes, Stefan Milz, Samir Rawashdeh

Abstract: Convolutional Neural Networks (CNNs) are successfully used for the important automotive visual perception tasks including object recognition, motion and depth estimation, visual SLAM, etc. However, these tasks are typically independently explored and modeled. In this paper, we propose a joint multi-task network design for learning several tasks simultaneously. Our main motivation is the computatio… ▽ More Convolutional Neural Networks (CNNs) are successfully used for the important automotive visual perception tasks including object recognition, motion and depth estimation, visual SLAM, etc. However, these tasks are typically independently explored and modeled. In this paper, we propose a joint multi-task network design for learning several tasks simultaneously. Our main motivation is the computational efficiency achieved by sharing the expensive initial convolutional layers between all tasks. Indeed, the main bottleneck in automated driving systems is the limited processing power available on deployment hardware. There is also some evidence for other benefits in improving accuracy for some tasks and easing development effort. It also offers scalability to add more tasks leveraging existing features and achieving better generalization. We survey various CNN based solutions for visual perception tasks in automated driving. Then we propose a unified CNN model for the important tasks and discuss several advanced optimization and architecture design techniques to improve the baseline model. The paper is partly review and partly positional with demonstration of several preliminary results promising for future research. We first demonstrate results of multi-stream learning and auxiliary learning which are important ingredients to scale to a large multi-task model. Finally, we implement a two-stream three-task network which performs better in many cases compared to their corresponding single-task models, while maintaining network size. △ Less

Submitted 9 March, 2024; v1 submitted 10 February, 2019; originally announced February 2019.

Comments: Accepted for Oral Presentation at IEEE Intelligent Transportation Systems Conference (ITSC) 2019

Points2Pix: 3D Point-Cloud to Image Translation using conditional Generative Adversarial Networks

Authors: Stefan Milz, Martin Simon, Kai Fischer, Maximillian Pöpperl

Abstract: We present the first approach for 3D point-cloud to image translation based on conditional Generative Adversarial Networks (cGAN). The model handles multi-modal information sources from different domains, i.e. raw point-sets and images. The generator is capable of processing three conditions, whereas the point-cloud is encoded as raw point-set and camera projection. An image background patch is us… ▽ More We present the first approach for 3D point-cloud to image translation based on conditional Generative Adversarial Networks (cGAN). The model handles multi-modal information sources from different domains, i.e. raw point-sets and images. The generator is capable of processing three conditions, whereas the point-cloud is encoded as raw point-set and camera projection. An image background patch is used as constraint to bias environmental texturing. A global approximation function within the generator is directly applied on the point-cloud (Point-Net). Hence, the representative learning model incorporates global 3D characteristics directly at the latent feature space. Conditions are used to bias the background and the viewpoint of the generated image. This opens up new ways in augmenting or texturing 3D data to aim the generation of fully individual images. We successfully evaluated our method on the Kitti and SunRGBD dataset with an outstanding object detection inception score. △ Less

Submitted 16 September, 2019; v1 submitted 26 January, 2019; originally announced January 2019.

Completely Positive Divisibility Does Not Mean Markovianity

Authors: Simon Milz, M. S. Kim, Felix A. Pollock, Kavan Modi

Abstract: In the classical domain, it is well-known that divisibility does not imply that a stochastic process is Markovian. However, for quantum processes, divisibility is often considered to be synonymous with Markovianity. We show that completely positive (CP) divisible quantum processes can still involve non-Markovian temporal correlations, that we then fully classify using the recently developed proces… ▽ More In the classical domain, it is well-known that divisibility does not imply that a stochastic process is Markovian. However, for quantum processes, divisibility is often considered to be synonymous with Markovianity. We show that completely positive (CP) divisible quantum processes can still involve non-Markovian temporal correlations, that we then fully classify using the recently developed process tensor formalism, which generalizes the theory of stochastic processes to the quantum domain. △ Less

Submitted 5 August, 2019; v1 submitted 16 January, 2019; originally announced January 2019.

Comments: 4+5 pages, 4 figures, close to published version

Journal ref: Phys. Rev. Lett. 123, 040401 (2019)

Efficient Semantic Segmentation for Visual Bird's-eye View Interpretation

Authors: Timo Sämann, Karl Amende, Stefan Milz, Christian Witt, Martin Simon, Johannes Petzold

Abstract: The ability to perform semantic segmentation in real-time capable applications with limited hardware is of great importance. One such application is the interpretation of the visual bird's-eye view, which requires the semantic segmentation of the four omnidirectional camera images. In this paper, we present an efficient semantic segmentation that sets new standards in terms of runtime and hardware… ▽ More The ability to perform semantic segmentation in real-time capable applications with limited hardware is of great importance. One such application is the interpretation of the visual bird's-eye view, which requires the semantic segmentation of the four omnidirectional camera images. In this paper, we present an efficient semantic segmentation that sets new standards in terms of runtime and hardware requirements. Our two main contributions are the decrease of the runtime by parallelizing the ArgMax layer and the reduction of hardware requirements by applying the channel pruning method to the ENet model. △ Less

Submitted 29 November, 2018; originally announced November 2018.

Journal ref: Advances in Intelligent Systems and Computing 2018

The Structure of Quantum Stochastic Processes with Finite Markov Order

Authors: Philip Taranto, Simon Milz, Felix A. Pollock, Kavan Modi

Abstract: Non-Markovian quantum processes exhibit different memory effects when measured in different ways; an unambiguous characterization of memory length requires accounting for the sequence of instruments applied to probe the system dynamics. This instrument-specific notion of quantum Markov order displays stark differences to its classical counterpart. Here, we explore the structure of quantum stochast… ▽ More Non-Markovian quantum processes exhibit different memory effects when measured in different ways; an unambiguous characterization of memory length requires accounting for the sequence of instruments applied to probe the system dynamics. This instrument-specific notion of quantum Markov order displays stark differences to its classical counterpart. Here, we explore the structure of quantum stochastic processes with finite length memory in detail. We begin by examining a generalized collision model with memory, before framing this instance within the general theory. We detail the constraints that are placed on the underlying system-environment dynamics for a process to exhibit finite Markov order with respect to natural classes of probing instruments, including deterministic (unitary) operations and sequences of generalized quantum measurements with informationally-complete preparations. Lastly, we show how processes with vanishing quantum conditional mutual information form a special case of the theory. Throughout, we provide a number of representative, pedagogical examples to display the salient features of memory effects in quantum processes. △ Less

Submitted 10 April, 2019; v1 submitted 25 October, 2018; originally announced October 2018.

Comments: 15.5+8 pages; 11 figures

Journal ref: Phys. Rev. A 99, 042108 (2019)

Quantum Markov Order

Authors: Philip Taranto, Felix A. Pollock, Simon Milz, Marco Tomamichel, Kavan Modi

Abstract: We formally extend the notion of Markov order to open quantum processes by accounting for the instruments used to probe the system of interest at different times. Our description recovers the classical Markov order property in the appropriate limit: when the stochastic process is classical and the instruments are non-invasive, \emph{i.e.}, restricted to orthogonal, projective measurements. We then… ▽ More We formally extend the notion of Markov order to open quantum processes by accounting for the instruments used to probe the system of interest at different times. Our description recovers the classical Markov order property in the appropriate limit: when the stochastic process is classical and the instruments are non-invasive, \emph{i.e.}, restricted to orthogonal, projective measurements. We then prove that there do not exist non-Markovian quantum processes that have finite Markov order with respect to all possible instruments; the same process exhibits distinct memory effects with respect to different probing instruments. This naturally leads to a relaxed definition of quantum Markov order with respect to specified sequences of instruments. The memory effects captured by different choices of instruments vary dramatically, providing a rich landscape for future exploration. △ Less

Submitted 10 April, 2019; v1 submitted 29 May, 2018; originally announced May 2018.

Comments: 4.5+2 pages, 3 figures

Journal ref: Phys. Rev. Lett. 122, 140401 (2019)

Complex-YOLO: Real-time 3D Object Detection on Point Clouds

Authors: Martin Simon, Stefan Milz, Karl Amende, Horst-Michael Gross

Abstract: Lidar based 3D object detection is inevitable for autonomous driving, because it directly links to environmental understanding and therefore builds the base for prediction and motion planning. The capacity of inferencing highly sparse 3D data in real-time is an ill-posed problem for lots of other application areas besides automated vehicles, e.g. augmented reality, personal robotics or industrial… ▽ More Lidar based 3D object detection is inevitable for autonomous driving, because it directly links to environmental understanding and therefore builds the base for prediction and motion planning. The capacity of inferencing highly sparse 3D data in real-time is an ill-posed problem for lots of other application areas besides automated vehicles, e.g. augmented reality, personal robotics or industrial automation. We introduce Complex-YOLO, a state of the art real-time 3D object detection network on point clouds only. In this work, we describe a network that expands YOLOv2, a fast 2D standard object detector for RGB images, by a specific complex regression strategy to estimate multi-class 3D boxes in Cartesian space. Thus, we propose a specific Euler-Region-Proposal Network (E-RPN) to estimate the pose of the object by adding an imaginary and a real fraction to the regression network. This ends up in a closed complex space and avoids singularities, which occur by single angle estimations. The E-RPN supports to generalize well during training. Our experiments on the KITTI benchmark suite show that we outperform current leading methods for 3D object detection specifically in terms of efficiency. We achieve state of the art results for cars, pedestrians and cyclists by being more than five times faster than the fastest competitor. Further, our model is capable of estimating all eight KITTI-classes, including Vans, Trucks or sitting pedestrians simultaneously with high accuracy. △ Less

Submitted 24 September, 2018; v1 submitted 16 March, 2018; originally announced March 2018.

Monocular Fisheye Camera Depth Estimation Using Sparse LiDAR Supervision

Authors: Varun Ravi Kumar, Stefan Milz, Martin Simon, Christian Witt, Karl Amende, Johannes Petzold, Senthil Yogamani, Timo Pech

Abstract: Near field depth estimation around a self driving car is an important function that can be achieved by four wide angle fisheye cameras having a field of view of over 180. Depth estimation based on convolutional neural networks (CNNs) produce state of the art results, but progress is hindered because depth annotation cannot be obtained manually. Synthetic datasets are commonly used but they have li… ▽ More Near field depth estimation around a self driving car is an important function that can be achieved by four wide angle fisheye cameras having a field of view of over 180. Depth estimation based on convolutional neural networks (CNNs) produce state of the art results, but progress is hindered because depth annotation cannot be obtained manually. Synthetic datasets are commonly used but they have limitations. For instance, they do not capture the extensive variability in the appearance of objects like vehicles present in real datasets. There is also a domain shift while performing inference on natural images illustrated by many attempts to handle the domain adaptation explicitly. In this work, we explore an alternate approach of training using sparse LiDAR data as ground truth for depth estimation for fisheye camera. We built our own dataset using our self driving car setup which has a 64 beam Velodyne LiDAR and four wide angle fisheye cameras. To handle the difference in view points of LiDAR and fisheye camera, an occlusion resolution mechanism was implemented. We started with Eigen's multiscale convolutional network architecture and improved by modifying activation function and optimizer. We obtained promising results on our dataset with RMSE errors comparable to the state of the art results obtained on KITTI. △ Less

Submitted 24 September, 2018; v1 submitted 16 March, 2018; originally announced March 2018.

Non-Markovian quantum control as coherent stochastic trajectories

Authors: Fattah Sakuldee, Simon Milz, Felix A. Pollock, Kavan Modi

Abstract: We develop a notion of stochastic quantum trajectories. First, we construct a basis set of trajectories, called elementary trajectories, and go on to show that any quantum dynamical process, including those that are non-Markovian, can be expressed as a linear combination of this set. We then show that the set of processes divide into two natural classes: those that can be expressed as convex mixtu… ▽ More We develop a notion of stochastic quantum trajectories. First, we construct a basis set of trajectories, called elementary trajectories, and go on to show that any quantum dynamical process, including those that are non-Markovian, can be expressed as a linear combination of this set. We then show that the set of processes divide into two natural classes: those that can be expressed as convex mixture of elementary trajectories and those that cannot be. The former are shown to be entanglement breaking processes (in each step), while the latter are dubbed coherent processes. This division of processes is analogous to separable and entangled states. In the second half of the paper, we show, with an information theoretic game, that when a process is non-Markovian, coherent trajectories allow for decoupling from the environment while preserving arbitrary quantum information encoded into the system. We give explicit expressions for the temporal correlations (quantifying non-Markovianity) and show that, in general, there are more quantum correlations than classical ones. This shows that non-Markovian quantum processes are indeed fundamentally different from their classical counterparts. Furthermore, we demonstrate how coherent trajectories (with the aid of coherent control) could turn non-Markovianity into a resource. In the final section of the paper we explore this phenomenon in a geometric picture with a convenient set of basis trajectories. △ Less

Submitted 19 September, 2018; v1 submitted 9 February, 2018; originally announced February 2018.

Comments: 18 + 6 pages, 5 figures

Journal ref: J. Phys. A: Math. Theor. 51 414014 (2018)

Kolmogorov extension theorem for (quantum) causal modelling and general probabilistic theories

Authors: Simon Milz, Fattah Sakuldee, Felix A. Pollock, Kavan Modi

Abstract: In classical physics, the Kolmogorov extension theorem lays the foundation for the theory of stochastic processes. It has been known for a long time that, in its original form, this theorem does not hold in quantum mechanics. More generally, it does not hold in any theory of stochastic processes -- classical, quantum or beyond -- that does not just describe passive observations, but allows for act… ▽ More In classical physics, the Kolmogorov extension theorem lays the foundation for the theory of stochastic processes. It has been known for a long time that, in its original form, this theorem does not hold in quantum mechanics. More generally, it does not hold in any theory of stochastic processes -- classical, quantum or beyond -- that does not just describe passive observations, but allows for active interventions. Such processes form the basis of the study of causal modelling across the sciences, including in the quantum domain. To date, these frameworks have lacked a conceptual underpinning similar to that provided by Kolmogorov's theorem for classical stochastic processes. We prove a generalized extension theorem that applies to all theories of stochastic processes, putting them on equally firm mathematical ground as their classical counterpart. Additionally, we show that quantum causal modelling and quantum stochastic processes are equivalent. This provides the correct framework for the description of experiments involving continuous control, which play a crucial role in the development of quantum technologies. Furthermore, we show that the original extension theorem follows from the generalized one in the correct limit, and elucidate how a comprehensive understanding of general stochastic processes allows one to unambiguously define the distinction between those that are classical and those that are quantum. △ Less

Submitted 9 April, 2020; v1 submitted 7 December, 2017; originally announced December 2017.

Comments: 22 pages, 4 figures, published version

Journal ref: Quantum 4, 255 (2020)

Entanglement, non-Markovianity, and causal non-separability

Authors: Simon Milz, Felix A. Pollock, Thao P. Le, Giulio Chiribella, Kavan Modi

Abstract: Quantum mechanics, in principle, allows for processes with indefinite causal order. However, most of these causal anomalies have not yet been detected experimentally. We show that every such process can be simulated experimentally by means of non-Markovian dynamics with a measurement on additional degrees of freedom. Explicitly, we provide a constructive scheme to implement arbitrary acausal proce… ▽ More Quantum mechanics, in principle, allows for processes with indefinite causal order. However, most of these causal anomalies have not yet been detected experimentally. We show that every such process can be simulated experimentally by means of non-Markovian dynamics with a measurement on additional degrees of freedom. Explicitly, we provide a constructive scheme to implement arbitrary acausal processes. Furthermore, we give necessary and sufficient conditions for open system dynamics with measurement to yield processes that respect causality locally, and find that tripartite entanglement and nonlocal unitary transformations are crucial requirements for the simulation of causally indefinite processes. These results show a direct connection between three counter-intuitive concepts: non-Markovianity, entanglement, and causal indefiniteness. △ Less

Submitted 10 November, 2017; originally announced November 2017.

Comments: 14 pages, 8 figures

Journal ref: New J. Phys. 20, 033033 (2018)

An introduction to operational quantum dynamics

Authors: Simon Milz, Felix A. Pollock, Kavan Modi

Abstract: In the summer of 2016, physicists gathered in Torun, Poland for the 48th annual Symposium on Mathematical Physics. This Symposium was special; it celebrated the 40th anniversary of the discovery of the Gorini-Kossakowski-Sudarshan-Lindblad master equation, which is widely used in quantum physics and quantum chemistry. This article forms part of a Special Volume of the journal Open Systems & Inform… ▽ More In the summer of 2016, physicists gathered in Torun, Poland for the 48th annual Symposium on Mathematical Physics. This Symposium was special; it celebrated the 40th anniversary of the discovery of the Gorini-Kossakowski-Sudarshan-Lindblad master equation, which is widely used in quantum physics and quantum chemistry. This article forms part of a Special Volume of the journal Open Systems & Information Dynamics arising from that conference; and it aims to celebrate a related discovery -- also by Sudarshan -- that of Quantum Maps (which had their 55th anniversary in the same year). Nowadays, much like the master equation, quantum maps are ubiquitous in physics and chemistry. Their importance in quantum information and related fields cannot be overstated. In this manuscript, we motivate quantum maps from a tomographic perspective, and derive their well-known representations. We then dive into the murky world beyond these maps, where recent research has yielded their generalisation to non-Markovian quantum processes. △ Less

Submitted 2 August, 2017; originally announced August 2017.

Comments: Submitted to Special OSID volume "40 years of GKLS"

Journal ref: Open Sys. Info. Dyn. 24, 1740016 (2017)

Graphs with degree complete labeling

Authors: Sebastian Milz

Abstract: In 2006 Qian [J. Qian, Degree complete graphs; Discrete Mathematics 306 (2006), 533--537] introduced the concept of degree complete graphs for labeled graphs. He also gave a characterization of these graphs in terms of two forbidden subgraphs. Furthermore, he mentioned that the property of being degree complete depends on the labeling of the graph. Related to this he stated the problem to find a c… ▽ More In 2006 Qian [J. Qian, Degree complete graphs; Discrete Mathematics 306 (2006), 533--537] introduced the concept of degree complete graphs for labeled graphs. He also gave a characterization of these graphs in terms of two forbidden subgraphs. Furthermore, he mentioned that the property of being degree complete depends on the labeling of the graph. Related to this he stated the problem to find a characterization of those (unlabeled) graphs for which every labeled version is not degree complete. We say that a (unlabeled) graph has a degree complete labeling, if there is a labeled version of the graph that is degree complete. In this paper we give three characterizations of graphs with degree complete labeling. These characterizations give us polynomial-time procedures to recognize these graphs and find a degree complete labeling, if it exists. △ Less

Submitted 14 June, 2017; originally announced June 2017.

Reconstructing open quantum system dynamics with limited control

Authors: Simon Milz, Felix A. Pollock, Kavan Modi

Abstract: The dynamics of an open quantum system can be fully described and tomographically reconstructed if the experimenter has complete control over the system of interest. Most real-world experiments do not fulfill this assumption, and the amount of control is restricted by the experimental set-up. That is, the set of performable manipulations of the system is limited. For instance, imagine a set-up whe… ▽ More The dynamics of an open quantum system can be fully described and tomographically reconstructed if the experimenter has complete control over the system of interest. Most real-world experiments do not fulfill this assumption, and the amount of control is restricted by the experimental set-up. That is, the set of performable manipulations of the system is limited. For instance, imagine a set-up where unitary operations are easy to make, but only one measurement at the end of the experiment is allowed. In this paper, we provide a general reconstruction scheme that yields operationally well-defined dynamics for any conceivable kind of experimental situation. If one additional operation can be performed, these `restricted' dynamics allow for the construction of witnesses for initial correlations and the presence of memory effects. We demonstrate the applicability of our framework for the the two important cases where the set of performable operations comprises only unitary operations or projective measurements, respectively, and show that it provides a powerful tool for the description of quantum control experiments. △ Less

Submitted 6 July, 2018; v1 submitted 7 October, 2016; originally announced October 2016.

Comments: 13 + 4 pages, 2 figures. Close to published version

Journal ref: Phys. Rev. A 98, 012108 (2018)

Volumes of conditioned bipartite state spaces

Authors: Simon Milz, Walter T. Strunz

Abstract: We analyse the metric properties of $\textit{conditioned}$ quantum state spaces $\mathcal{M}^{(n\times m)}_η$. These spaces are the convex sets of $nm \times nm$ density matrices that, when partially traced over $m$ degrees of freedom, respectively yield the given $n\times n$ density matrix $η$. For the case $n=2$, the volume of $\mathcal{M}^{(2\times m)}_η$ equipped with the Hilbert-Schmidt measu… ▽ More We analyse the metric properties of $\textit{conditioned}$ quantum state spaces $\mathcal{M}^{(n\times m)}_η$. These spaces are the convex sets of $nm \times nm$ density matrices that, when partially traced over $m$ degrees of freedom, respectively yield the given $n\times n$ density matrix $η$. For the case $n=2$, the volume of $\mathcal{M}^{(2\times m)}_η$ equipped with the Hilbert-Schmidt measure is a simple polynomial of the radius of $η$ in the Bloch-Ball. Remarkably, the probability $p_{\mathrm{sep}}^{(2\times m)}(η)$ to find a separable state in $\mathcal{M}^{(2\times m)}_η$ is independent of $η$ (except for $η$ pure). Both these results are proven analytically for the case of the family of $4\times 4$ $X$-states, and thoroughly numerically investigated for the general case. The important implications of these results for the clarification of open problems in quantum theory are pointed out and discussed. △ Less

Submitted 17 September, 2014; v1 submitted 15 August, 2014; originally announced August 2014.

Comments: 23 pages, 7 figures