Validation of Simulation-Based Testing: Bypassing Domain Shift with Label-to-Image Synthesis
Authors:
Julia Rosenzweig,
Eduardo Brito,
Hans-Ulrich Kobialka,
Maram Akila,
Nico M. Schmidt,
Peter Schlicht,
Jan David Schneider,
Fabian Hüger,
Matthias Rottmann,
Sebastian Houben,
Tim Wirtz
Abstract:
Many machine learning applications can benefit from simulated data for systematic validation - in particular if real-life data is difficult to obtain or annotate. However, since simulations are prone to domain shift w.r.t. real-life data, it is crucial to verify the transferability of the obtained results. We propose a novel framework consisting of a generative label-to-image synthesis model toget…
▽ More
Many machine learning applications can benefit from simulated data for systematic validation - in particular if real-life data is difficult to obtain or annotate. However, since simulations are prone to domain shift w.r.t. real-life data, it is crucial to verify the transferability of the obtained results. We propose a novel framework consisting of a generative label-to-image synthesis model together with different transferability measures to inspect to what extent we can transfer testing results of semantic segmentation models from synthetic data to equivalent real-life data. With slight modifications, our approach is extendable to, e.g., general multi-class classification tasks. Grounded on the transferability analysis, our approach additionally allows for extensive testing by incorporating controlled simulations. We validate our approach empirically on a semantic segmentation task on driving scenes. Transferability is tested using correlation analysis of IoU and a learned discriminator. Although the latter can distinguish between real-life and synthetic tests, in the former we observe surprisingly strong correlations of 0.7 for both cars and pedestrians.
△ Less
Submitted 10 June, 2021;
originally announced June 2021.
