-
mlf-core: a framework for deterministic machine learning
Authors:
Lukas Heumos,
Philipp Ehmele,
Luis Kuhn Cuellar,
Kevin Menden,
Edmund Miller,
Steffen Lemke,
Gisela Gabernet,
Sven Nahnsen
Abstract:
Machine learning has shown extensive growth in recent years and is now routinely applied to sensitive areas. To allow appropriate verification of predictive models before deployment, models must be deterministic. However, major machine learning libraries default to the usage of non-deterministic algorithms based on atomic operations. Solely fixing all random seeds is not sufficient for determinist…
▽ More
Machine learning has shown extensive growth in recent years and is now routinely applied to sensitive areas. To allow appropriate verification of predictive models before deployment, models must be deterministic. However, major machine learning libraries default to the usage of non-deterministic algorithms based on atomic operations. Solely fixing all random seeds is not sufficient for deterministic machine learning. To overcome this shortcoming, various machine learning libraries released deterministic counterparts to the non-deterministic algorithms. We evaluated the effect of these algorithms on determinism and runtime. Based on these results, we formulated a set of requirements for deterministic machine learning and developed a new software solution, the mlf-core ecosystem, which aids machine learning projects to meet and keep these requirements. We applied mlf-core to develop deterministic models in various biomedical fields including a single cell autoencoder with TensorFlow, a PyTorch-based U-Net model for liver-tumor segmentation in CT scans, and a liver cancer classifier based on gene expression profiles with XGBoost.
△ Less
Submitted 16 June, 2022; v1 submitted 15 April, 2021;
originally announced April 2021.
-
LGSVL Simulator: A High Fidelity Simulator for Autonomous Driving
Authors:
Guodong Rong,
Byung Hyun Shin,
Hadi Tabatabaee,
Qiang Lu,
Steve Lemke,
Mārtiņš Možeiko,
Eric Boise,
Geehoon Uhm,
Mark Gerow,
Shalin Mehta,
Eugene Agafonov,
Tae Hyung Kim,
Eric Sterner,
Keunhae Ushiroda,
Michael Reyes,
Dmitry Zelenkovsky,
Seonman Kim
Abstract:
Testing autonomous driving algorithms on real autonomous vehicles is extremely costly and many researchers and developers in the field cannot afford a real car and the corresponding sensors. Although several free and open-source autonomous driving stacks, such as Autoware and Apollo are available, choices of open-source simulators to use with them are limited. In this paper, we introduce the LGSVL…
▽ More
Testing autonomous driving algorithms on real autonomous vehicles is extremely costly and many researchers and developers in the field cannot afford a real car and the corresponding sensors. Although several free and open-source autonomous driving stacks, such as Autoware and Apollo are available, choices of open-source simulators to use with them are limited. In this paper, we introduce the LGSVL Simulator which is a high fidelity simulator for autonomous driving. The simulator engine provides end-to-end, full-stack simulation which is ready to be hooked up to Autoware and Apollo. In addition, simulator tools are provided with the core simulation engine which allow users to easily customize sensors, create new types of controllable objects, replace some modules in the core simulator, and create digital twins of particular environments.
△ Less
Submitted 21 June, 2020; v1 submitted 7 May, 2020;
originally announced May 2020.
-
Formal Scenario-Based Testing of Autonomous Vehicles: From Simulation to the Real World
Authors:
Daniel J. Fremont,
Edward Kim,
Yash Vardhan Pant,
Sanjit A. Seshia,
Atul Acharya,
Xantha Bruso,
Paul Wells,
Steve Lemke,
Qiang Lu,
Shalin Mehta
Abstract:
We present a new approach to automated scenario-based testing of the safety of autonomous vehicles, especially those using advanced artificial intelligence-based components, spanning both simulation-based evaluation as well as testing in the real world. Our approach is based on formal methods, combining formal specification of scenarios and safety properties, algorithmic test case generation using…
▽ More
We present a new approach to automated scenario-based testing of the safety of autonomous vehicles, especially those using advanced artificial intelligence-based components, spanning both simulation-based evaluation as well as testing in the real world. Our approach is based on formal methods, combining formal specification of scenarios and safety properties, algorithmic test case generation using formal simulation, test case selection for track testing, executing test cases on the track, and analyzing the resulting data. Experiments with a real autonomous vehicle at an industrial testing facility support our hypotheses that (i) formal simulation can be effective at identifying test cases to run on the track, and (ii) the gap between simulated and real worlds can be systematically evaluated and bridged.
△ Less
Submitted 12 July, 2020; v1 submitted 17 March, 2020;
originally announced March 2020.
-
Can We Count on Social Media Metrics? First Insights into the Active Scholarly Use of Social Media
Authors:
Maryam Mehrazar,
Christoph Carl Kling,
Steffen Lemke,
Athanasios Mazarakis,
Isabella Peters
Abstract:
Measuring research impact is important for ranking publications in academic search engines and for research evaluation. Social media metrics or altmetrics measure the impact of scientific work based on social media activity. Altmetrics are complementary to traditional, citation-based metrics, e.g. allowing the assessment of new publications for which citations are not yet available. Despite the in…
▽ More
Measuring research impact is important for ranking publications in academic search engines and for research evaluation. Social media metrics or altmetrics measure the impact of scientific work based on social media activity. Altmetrics are complementary to traditional, citation-based metrics, e.g. allowing the assessment of new publications for which citations are not yet available. Despite the increasing importance of altmetrics, their characteristics are not well understood: Until now it has not been researched what kind of researchers are actively using which social media services and why - important questions for scientific impact prediction. Based on a survey among 3,430 scientists, we uncover previously unknown and significant differences between social media services: We identify services which attract young and experienced researchers, respectively, and detect differences in usage motivations. Our findings have direct implications for the future design of altmetrics for scientific impact prediction.
△ Less
Submitted 8 April, 2018;
originally announced April 2018.
