-
Performance Analysis of Out-of-Distribution Detection on Trained Neural Networks
Authors:
Jens Henriksson,
Christian Berger,
Markus Borg,
Lars Tornberg,
Sankar Raman Sathyamoorthy,
Cristofer Englund
Abstract:
Several areas have been improved with Deep Learning during the past years. Implementing Deep Neural Networks (DNN) for non-safety related applications have shown remarkable achievements over the past years; however, for using DNNs in safety critical applications, we are missing approaches for verifying the robustness of such models. A common challenge for DNNs occurs when exposed to out-of-distrib…
▽ More
Several areas have been improved with Deep Learning during the past years. Implementing Deep Neural Networks (DNN) for non-safety related applications have shown remarkable achievements over the past years; however, for using DNNs in safety critical applications, we are missing approaches for verifying the robustness of such models. A common challenge for DNNs occurs when exposed to out-of-distribution samples that are outside of the scope of a DNN, but which result in high confidence outputs despite no prior knowledge of such input.
In this paper, we analyze three methods that separate between in- and out-of-distribution data, called supervisors, on four well-known DNN architectures. We find that the outlier detection performance improves with the quality of the model. We also analyse the performance of the particular supervisors during the training procedure by applying the supervisor at a predefined interval to investigate its performance as the training proceeds. We observe that understanding the relationship between training results and supervisor performance is crucial to improve the model's robustness and to indicate, what input samples require further measures to improve the robustness of a DNN. In addition, our work paves the road towards an instrument for safety argumentation for safety critical applications. This paper is an extended version of our previous work presented at 2019 SEAA (cf. [1]); here, we elaborate on the used metrics, add an additional supervisor and test them on two additional datasets.
△ Less
Submitted 26 April, 2022;
originally announced April 2022.
-
Ergo, SMIRK is Safe: A Safety Case for a Machine Learning Component in a Pedestrian Automatic Emergency Brake System
Authors:
Markus Borg,
Jens Henriksson,
Kasper Socha,
Olof Lennartsson,
Elias Sonnsjö Lönegren,
Thanh Bui,
Piotr Tomaszewski,
Sankar Raman Sathyamoorthy,
Sebastian Brink,
Mahshid Helali Moghadam
Abstract:
Integration of Machine Learning (ML) components in critical applications introduces novel challenges for software certification and verification. New safety standards and technical guidelines are under development to support the safety of ML-based systems, e.g., ISO 21448 SOTIF for the automotive domain and the Assurance of Machine Learning for use in Autonomous Systems (AMLAS) framework. SOTIF an…
▽ More
Integration of Machine Learning (ML) components in critical applications introduces novel challenges for software certification and verification. New safety standards and technical guidelines are under development to support the safety of ML-based systems, e.g., ISO 21448 SOTIF for the automotive domain and the Assurance of Machine Learning for use in Autonomous Systems (AMLAS) framework. SOTIF and AMLAS provide high-level guidance but the details must be chiseled out for each specific case. We initiated a research project with the goal to demonstrate a complete safety case for an ML component in an open automotive system. This paper reports results from an industry-academia collaboration on safety assurance of SMIRK, an ML-based pedestrian automatic emergency braking demonstrator running in an industry-grade simulator. We demonstrate an application of AMLAS on SMIRK for a minimalistic operational design domain, i.e., we share a complete safety case for its integrated ML-based component. Finally, we report lessons learned and provide both SMIRK and the safety case under an open-source licence for the research community to reuse.
△ Less
Submitted 6 December, 2022; v1 submitted 16 April, 2022;
originally announced April 2022.
-
Quantum efficiency, purity and stability of a tunable, narrowband microwave single-photon source
Authors:
Yong Lu,
Andreas Bengtsson,
Jonathan J. Burnett,
Baladitya Suri,
Sankar Raman Sathyamoorthy,
Hampus Renberg Nilsson,
Marco Scigliuzzo,
Jonas Bylander,
Göran Johansson,
Per Delsing
Abstract:
We demonstrate an on-demand source of microwave single photons with 71--99\% intrinsic quantum efficiency. The source is narrowband (300\unite{kHz}) and tuneable over a 600 MHz range around 5.2 GHz. Such a device is an important element in numerous quantum technologies and applications. The device consists of a superconducting transmon qubit coupled to the open end of a transmission line. A $π$-pu…
▽ More
We demonstrate an on-demand source of microwave single photons with 71--99\% intrinsic quantum efficiency. The source is narrowband (300\unite{kHz}) and tuneable over a 600 MHz range around 5.2 GHz. Such a device is an important element in numerous quantum technologies and applications. The device consists of a superconducting transmon qubit coupled to the open end of a transmission line. A $π$-pulse excites the qubit, which subsequently rapidly emits a single photon into the transmission line. A cancellation pulse then suppresses the reflected $π$-pulse by 33.5 dB, resulting in 0.005 photons leaking into the photon emission channel. We verify strong antibunching of the emitted photon field and determine its Wigner function. Non-radiative decay and $1/f$ flux noise both affect the quantum efficiency. We also study the device stability over time and identify uncorrelated discrete jumps of the pure dephasing rate at different qubit frequencies on a time scale of hours, which we attribute to independent two-level system defects in the device dielectrics, dispersively coupled to the qubit.
△ Less
Submitted 24 May, 2021;
originally announced May 2021.
-
Performance Analysis of Out-of-Distribution Detection on Various Trained Neural Networks
Authors:
Jens Henriksson,
Christian Berger,
Markus Borg,
Lars Tornberg,
Sankar Raman Sathyamoorthy,
Cristofer Englund
Abstract:
Several areas have been improved with Deep Learning during the past years. For non-safety related products adoption of AI and ML is not an issue, whereas in safety critical applications, robustness of such approaches is still an issue. A common challenge for Deep Neural Networks (DNN) occur when exposed to out-of-distribution samples that are previously unseen, where DNNs can yield high confidence…
▽ More
Several areas have been improved with Deep Learning during the past years. For non-safety related products adoption of AI and ML is not an issue, whereas in safety critical applications, robustness of such approaches is still an issue. A common challenge for Deep Neural Networks (DNN) occur when exposed to out-of-distribution samples that are previously unseen, where DNNs can yield high confidence predictions despite no prior knowledge of the input.
In this paper we analyse two supervisors on two well-known DNNs with varied setups of training and find that the outlier detection performance improves with the quality of the training procedure. We analyse the performance of the supervisor after each epoch during the training cycle, to investigate supervisor performance as the accuracy converges. Understanding the relationship between training results and supervisor performance is valuable to improve robustness of the model and indicates where more work has to be done to create generalized models for safety critical applications.
△ Less
Submitted 29 March, 2021;
originally announced March 2021.
-
Towards Structured Evaluation of Deep Neural Network Supervisors
Authors:
Jens Henriksson,
Christian Berger,
Markus Borg,
Lars Tornberg,
Cristofer Englund,
Sankar Raman Sathyamoorthy,
Stig Ursing
Abstract:
Deep Neural Networks (DNN) have improved the quality of several non-safety related products in the past years. However, before DNNs should be deployed to safety-critical applications, their robustness needs to be systematically analyzed. A common challenge for DNNs occurs when input is dissimilar to the training set, which might lead to high confidence predictions despite proper knowledge of the i…
▽ More
Deep Neural Networks (DNN) have improved the quality of several non-safety related products in the past years. However, before DNNs should be deployed to safety-critical applications, their robustness needs to be systematically analyzed. A common challenge for DNNs occurs when input is dissimilar to the training set, which might lead to high confidence predictions despite proper knowledge of the input. Several previous studies have proposed to complement DNNs with a supervisor that detects when inputs are outside the scope of the network. Most of these supervisors, however, are developed and tested for a selected scenario using a specific performance metric. In this work, we emphasize the need to assess and compare the performance of supervisors in a structured way. We present a framework constituted by four datasets organized in six test cases combined with seven evaluation metrics. The test cases provide varying complexity and include data from publicly available sources as well as a novel dataset consisting of images from simulated driving scenarios. The latter we plan to make publicly available. Our framework can be used to support DNN supervisor evaluation, which in turn could be used to motive development, validation, and deployment of DNNs in safety-critical applications.
△ Less
Submitted 7 March, 2019; v1 submitted 4 March, 2019;
originally announced March 2019.
-
Resonant and off-resonant microwave signal manipulation in coupled superconducting resonators
Authors:
Mathieu Pierre,
Sankar Raman Sathyamoorthy,
Ida-Maria Svensson,
Göran Johansson,
Per Delsing
Abstract:
We present an experimental demonstration as well as a theoretical model of an integrated circuit designed for the manipulation of a microwave field down to the single-photon level. The device is made of a superconducting resonator coupled to a transmission line via a second frequency-tunable resonator. The tunable resonator can be used as a tunable coupler between the fixed resonator and the trans…
▽ More
We present an experimental demonstration as well as a theoretical model of an integrated circuit designed for the manipulation of a microwave field down to the single-photon level. The device is made of a superconducting resonator coupled to a transmission line via a second frequency-tunable resonator. The tunable resonator can be used as a tunable coupler between the fixed resonator and the transmission line. Moreover, the manipulation of the microwave field between the two resonators is possible. In particular, we demonstrate the swap** of the field from one resonator to the other by pulsing the frequency detuning between the two resonators. The behavior of the system, which determines how the device can be operated, is analyzed as a function of one key parameter of the system, the dam** ratio of the coupled resonators. We show a good agreement between experiments and simulations, realized by solving a set of coupled differential equations.
△ Less
Submitted 19 March, 2019; v1 submitted 25 February, 2018;
originally announced February 2018.
-
Simple, robust and on-demand generation of single and correlated photons
Authors:
Sankar Raman Sathyamoorthy,
Andreas Bengtsson,
Steven Bens,
Michaël Simoen,
Per Delsing,
Göran Johansson
Abstract:
We propose two different setups to generate single photons on demand using an atom in front of a mirror, along with either a beam-splitter or a tunable coupling. We show that photon generation efficiency ~99% is straightforward to achieve. The proposed schemes are simple and easily tunable in frequency. The operation is relatively insensitive to dephasing and can be easily extended to generate cor…
▽ More
We propose two different setups to generate single photons on demand using an atom in front of a mirror, along with either a beam-splitter or a tunable coupling. We show that photon generation efficiency ~99% is straightforward to achieve. The proposed schemes are simple and easily tunable in frequency. The operation is relatively insensitive to dephasing and can be easily extended to generate correlated pairs of photons. They can also in principle be used to generate any photonic qubit of the form $μ|0 \rangle + ν|1\rangle$ in arbitrary wave-packets, making them very attractive for quantum communication applications.
△ Less
Submitted 5 June, 2016; v1 submitted 10 November, 2015;
originally announced November 2015.
-
Detecting itinerant single microwave photons
Authors:
Sankar Raman Sathyamoorthy,
Thomas M. Stace,
Göran Johansson
Abstract:
Single photon detectors are fundamental tools of investigation in quantum optics and play a central role in measurement theory and quantum informatics. Photodetectors based on different technologies exist at optical frequencies and much effort is currently being spent on pushing their efficiencies to meet the demands coming from the quantum computing and quantum communication proposals. In the mic…
▽ More
Single photon detectors are fundamental tools of investigation in quantum optics and play a central role in measurement theory and quantum informatics. Photodetectors based on different technologies exist at optical frequencies and much effort is currently being spent on pushing their efficiencies to meet the demands coming from the quantum computing and quantum communication proposals. In the microwave regime however, a single photon detector has remained elusive although several theoretical proposals have been put forth. In this article, we review these recent proposals, especially focusing on non-destructive detectors of propagating microwave photons. These detection schemes using superconducting artificial atoms can reach detection efficiencies of 90\% with existing technologies and are ripe for experimental investigations.
△ Less
Submitted 20 April, 2015;
originally announced April 2015.
-
Storage and on-demand release of microwaves using superconducting resonators with tunable coupling
Authors:
Mathieu Pierre,
Ida-Maria Svensson,
Sankar Raman Sathyamoorthy,
Göran Johansson,
Per Delsing
Abstract:
We present a system which allows to tune the coupling between a superconducting resonator and a transmission line. This storage resonator is addressed through a second, coupling resonator, which is frequency-tunable and controlled by a magnetic flux applied to a superconducting quantum interference device (SQUID). We experimentally demonstrate that the lifetime of the storage resonator can be tune…
▽ More
We present a system which allows to tune the coupling between a superconducting resonator and a transmission line. This storage resonator is addressed through a second, coupling resonator, which is frequency-tunable and controlled by a magnetic flux applied to a superconducting quantum interference device (SQUID). We experimentally demonstrate that the lifetime of the storage resonator can be tuned by more than three orders of magnitude. A field can be stored for 18 μs when the coupling resonator is tuned off resonance and it can be released in 14 ns when the coupling resonator is tuned on resonance. The device allows capture, storage, and on-demand release of microwaves at a tunable rate.
△ Less
Submitted 8 June, 2014;
originally announced June 2014.
-
Quantum nondemolition detection of a propagating microwave photon
Authors:
Sankar R. Sathyamoorthy,
L. Tornberg,
Anton F. Kockum,
Ben Q. Baragiola,
Joshua Combes,
C. M. Wilson,
Thomas M. Stace,
G. Johansson
Abstract:
The ability to nondestructively detect the presence of a single, traveling photon has been a long-standing goal in optics, with applications in quantum information and measurement. Realising such a detector is complicated by the fact that photon-photon interactions are typically very weak. At microwave frequencies, very strong effective photon-photon interactions in a waveguide have recently been…
▽ More
The ability to nondestructively detect the presence of a single, traveling photon has been a long-standing goal in optics, with applications in quantum information and measurement. Realising such a detector is complicated by the fact that photon-photon interactions are typically very weak. At microwave frequencies, very strong effective photon-photon interactions in a waveguide have recently been demonstrated. Here we show how this type of interaction can be used to realize a quantum nondemolition measurement of a single propagating microwave photon. The scheme we propose uses a chain of solid-state 3-level systems (transmons), cascaded through circulators which suppress photon backscattering. Our theoretical analysis shows that microwave-photon detection with fidelity around 90% can be realized with existing technologies.
△ Less
Submitted 19 December, 2013; v1 submitted 9 August, 2013;
originally announced August 2013.
-
Electronic structure, structural and optical properties of thermally evaporated CdTe thin films
Authors:
S. Lalitha,
S. Zh. Karazhanov,
P. Ravindran,
S. Senthilarasu,
R. Sathyamoorthy,
J. Janabergenov
Abstract:
Thin films of CdTe were deposited on glass substrates by thermal evaporation.
From the XRD measurements itis found that the films are of zinc-blende-type structure. Transmittance, absorption, extinction, and refractive coefficients are measured. Electronic structure, band parameters and optical spectra of CdTe were calculated from ab initio studies within the LDA and LDA+U approximations. It i…
▽ More
Thin films of CdTe were deposited on glass substrates by thermal evaporation.
From the XRD measurements itis found that the films are of zinc-blende-type structure. Transmittance, absorption, extinction, and refractive coefficients are measured. Electronic structure, band parameters and optical spectra of CdTe were calculated from ab initio studies within the LDA and LDA+U approximations. It is shown that LDA underestimates the band gap, energy levels of the Cd-4d states, s-d coupling and band dispersion. However, it calculates the spin-orbit coupling correctly. LDA+U did not increase much the band gap value, but it corrected the s-d coupling by shifting the Cd-4d levels towards the experimentally determined location and by splitting the LDA-derived single s peak into two peaks, which originates from admixture of s and d states. It is shown that the sd coupling plays an important role in absorption and reflectivity constants. The calculated optical spectra fairly agree with experimental data. Independent of wave-vector scissors operator is found to be a good first approximation to shift rigidly the band gap of CdTe underestimated by LDA.
△ Less
Submitted 5 August, 2006;
originally announced August 2006.
