Two is Better Than One: Digital Siblings to Improve Autonomous Driving Testing

This paper focuses on testing AVs that perform the lane-kee** functionality from driving samples labeled by humans. Lane-kee**, also called lane-centering or lane-following, is an automated driving assistance feature of an AV to keep the vehicle at the center of the lane. This system can be implemented at different levels, from a warning to the driver when the vehicle crosses one of the lanes up to the driverless version, which steers the vehicle automatically when it detects a departure from the center of the lane.

In this paper we consider the driverless version since it is a crucial component for the safe deployment of AVs on public roads. Indeed, according to a report by NHTSA precrashreport , off-road crashes due to failures of the lane-kee** component are first in cost ($15 billion) and second in frequency. From a technical standpoint, the lane-kee** task is implemented by behavior cloning DNNs, which learn end-to-end from supervised expert demonstrations. The training dataset consists of driving images captured with a camera sensor mounted on board of the vehicle, appropriately labeled with the driving commands of a human driver.

We consider lane-kee** DNN models, such as NVIDIA’s Dave-2 nvidia-dave2 , that predict the steering angle at which the car should steer to keep the vehicle in lane, given a single driving image. These models are generally trained with stochastic gradient descent saadonline on stationary datasets, with the goal of minimizing the error between the predicted and the ground-truth steering angles.

Such labels are typically an array of commands, i.e., steering, throttle and brake, although in the simplest case only the steering is provided, while the throttle is determined as a function of the steering and the velocity of the vehicle. Given the dataset, a DNN model, such as the Dave-2 model from Nvidia nvidia-dave2 , is trained to predict the label given an image by minimizing the Mean Squared Error (MSE) between the current prediction and the ground-truth label.

Evolutionary or metaheuristic search is a class of techniques that apply randomness and heuristics to find near-optimal solutions to optimization problems essentials . Such techniques are very general, since they only require evaluating how good a candidate solution is. The goodness of a solution is called fitness and the objective of the search algorithm is to optimize it (either maximize it or minimize it). The algorithm manipulates a solution to exploit the known parts of the search space, and creates new solutions to explore the parts that are unknown.

Search algorithms have been applied to testing problems and have been particularly effective tools for test generation evosuite ; dynamosa ; pynguin . In this paper, we use the MapElites search algorithm mapelites , implemented in the DeepHyperion tool deephyperion , to generate test cases for the DNN model under test. The algorithm explores the solution feature space at large, in order to provide a comprehensive characterization of the behaviors of the driving model.

We adopted an abstract representation of the road in each driving simulator so that only a sequence of road control points is needed when creating a new road in the driving scene. We follow the representation given by Riccio and Tonella deepjanus who defined a two-lane road using a series of control points (displayed as red stars in Figure 2). The control points are interpolated using Catmull-Rom splines 10.1145/378456.378511 , giving the road its final shape (yellow solid line).

Figure 2 shows the visualization of a test scenario generated at step ❷. Specifically, the road is defined using nine control points whereas the Catmull-Rom spline only goes through seven of them. This is because a spline segment (e.g., $P_{2}-P_{3}$) is always defined by four control points (e.g., $P_{1}$, $P_{2}$, $P_{3}$, $P_{4}$). Since two of them are on either side of the endpoints of the spline segment (e.g., $P_{1}$ and $P_{4}$), the spline cannot traverse the extreme endpoints (e.g., $P_{1}$ and $P_{9}$). Hence, $P_{2}$ defines the start point of the road (depicted as a black triangle) whereas $P_{8}$ defines the end point (depicted as a black square).

The default initial state of each test case involves positioning the vehicle in the first drivable control point (i.e., $P_{2}$ in Figure 2), at the center of the right lane following the road orientation.

We uniformed the 3D rendering of each simulator such that the driving scenarios have the same look and feel: a two-lane asphalt road, where the road is delimited by two solid white lines on each side and the two driving lanes are separated by a single solid yellow line. The road is placed on top of a green plane representing grass. Harmonization of the driving scenarios across simulators ensures that geometrical features are preserved for the collected driving images and that any color transformation applied to them during training preprocessing remains applicable nvidia-dave2 .

After interpolation, a road is deemed valid if it respects the following constraints: (1) the start and end points are different; (2) the road is contained within a squared bounding box of a predefined size (specifically 250 $\times$ 250); and, (3) there are no intersections. A test case is deemed successful when the vehicle drives within the right lane until the last road control point (e.g., $P_{8}$ in Figure 2). On the contrary, a test case failure occurs when the vehicle drives out of bound (OOB).

For the creation or fine-tuning of a self-driving model (step ❶), a labeled dataset of driving scenes is needed. We automate labeled data collection by resorting to autopilots that have global knowledge of the driving scenario such as the detailed road geometry and precise vehicle position. In particular, in each simulator, at each step of the simulation, the steering angle of the autopilot is computed by a Proportional-Integral-Differential (PID) controller farag2020complex according to the formula:

The autopilot produces a steering angle label for each image which is used to train the driving model. We aligned the frame rates of the different simulators at 20 fps such that, in each simulator, the autopilot collects a comparable number of labeled images. The speed of the vehicle, both for the autopilot and $M$, is controlled by the throttle via a linear interpolation between the minimum speed and maximum speed so that the car decreases the speed when the steering angle increases (e.g., in a curve). The following formula computes the throttle based on the speed of the vehicle and the steering:

The next step involves training the model $M$ using all simulators and the data collected in step ❶. Alternatively, if an existing trained model $M$ is available for the target DT, our approach requires fine-tuning it for all digital siblings. In both scenarios, we use hybrid training based on gradient descent 10.5555/2981562.2981583 .

Hybrid training requires combining the datasets collected for different simulators/platforms into a unified dataset, making sure that each dataset is equally represented (i.e., the unified dataset contains the same number of samples from each simulator/platform specific dataset). Then, the unified dataset is split into training and validation sets (e.g., using the standard 80/20 ratio). The training pipeline is designed in such a way that each image, of dimensions 320$\times$160, is processed according to the simulator/platform it was taken from. For example, images may be cropped differently. Depending on the vehicle size, the front part of the car may, or may not be visible in the frame captured by the camera. Another example of simulator-specific adaptation is the crop** of the above-horizon portion of the image, unnecessary for the lane-kee** task. After crop**, each image is resized to the size required for training, i.e., 320$\times$160.

The training pipeline can be further configured to use plain synthetic virtual images from the driving simulators, or pseudo-real images resembling real-world driving images. The first configuration represents the standard practice in AV testing. In the second configuration, the reality gap due to low photo-realism is reduced by an image-to-image transformation that translates the driving images of each simulator into images similar to those captured by the real-world AV during on-road driving. This practice was proposed in the literature stocco-mind and in industry wayve-sim2real to increase the transferability of the driving model tested in simulation to the real world.

More specifically, this second configuration requires training a CycleGAN model for each driving simulator cyclegan . CycleGAN entails two generators, one that learns how to translate images from simulated to real world (sim2real) and the other that learns the opposite transformation (real2sim). During training of the model, we use the sim2real generator trained for the respective simulator to translate the corresponding training set images. During testing, the sim2real generator translates images at runtime, i.e., during the execution of the simulation. We refer to the translated images as pseudo-real, since they are the output of a generative process designed to resemble real images.

Figure 3 shows an example of image translation with a CycleGAN trained for each simulator. The corresponding networks translate an image of a road curve taken in the simulated domain (left) to an image belonging to the real domain (right)—the test track of a small scale physical AV. During training and testing of the driving model in a given simulator, we use the generator of the CycleGAN trained for such simulator.

In our evaluation (Section 4), we consider both configurations of our approach, i.e., training using either simulator or pseudo-real images. We refer to the model trained on simulator images as $M_{S}$, and the model trained on pseudo-real images as $M_{R}$.

We considered three self-driving architectures, i.e., Dave-2 nvidia-dave2 , Chauffeur chauffeur and Epoch epoch . Such architectures were used in previous studies on AV testing in the literature stocco-mind ; 2020-Stocco-ICSE ; survey-lei-ma ; 2021-Jahangirova-ICST ; 2022-Stocco-ASE ; 2020-Stocco-GAUSS ; 2021-Stocco-JSEP ; deephyperion ; sbst2021 ; sbst2022 ; sbst2023 , and the respective models feature different number of parameters. The Dave-2 model has 2.8$M$ parameters, Chauffeur has 100$k$ parameters while Epoch has 26$M$ parameters (we used a reduced version of the Epoch model to reduce training and inference time stocco-mind ).

Architecturally, Dave-2 consists of five convolutional layers, followed by three fully-connected layers nvidia-dave2 . Chauffeur has six convolutional layers each followed by a dropout and a max pooling layer (except the last one) chauffeur . Epoch has three convolutional layers and one fully-connected layer, which makes up for most of the parameter count of the model epoch .

We implemented and investigated the effectiveness of DSS using the simulators BeamNG beamng and Udacity udacity-simulator . We chose them as digital siblings because: (1) they support training and testing of a DNN that performs lane-kee**, including Dave-2, Chauffeur and Epoch; (2) they are often used as simulator platforms for AV testing, as highlighted by a recent survey on autonomous driving testing survey-lei-ma ; (3) they are potentially complementary because they are developed with different technologies/game engines, and they are characterized by different physics implementations (e.g., rigid vs soft-body dynamics); (4) they are publicly available under open-source or academic-oriented licenses, hence customizable.

BeamNG beamng is a framework specialized in autonomous driving developed by BeamNG GmbH. The framework is released under an academic-oriented license, and it has been downloaded 5.5$k$ times as of January 2023. From a technical standpoint, BeamNG features a soft-body dynamics simulation based on a spring-mass model. Such a model is composed of nodes (mass points) that are connected by beams (springs), i.e., weightless elements that allow accurate vehicle deformation and other aerodynamic properties gambi-beamng .

Udacity udacity-simulator is developed with Unity 3D unity , a popular cross-platform game engine. The project has been publicly released in 2016 by the for-profit educational organization Udacity, to allow people from all over the world to access some of their technology and to contribute to an open-source self-driving car project. As of January 2023, the simulator has 3.7$k$ stars on GitHub. From a technical standpoint, Udacity is based on the Nvidia PhysX engine PhysX , featuring discrete and continuous collision detection, ray-casting, and rigid-body dynamics simulation.

We use the Donkey Car^™ open-source framework donkeycar as digital twin for our study. This platform has been used for AV testing research with physical self-driving cars in physical environments stocco-mind ; viitala ; 9412011 . The framework includes open hardware to build 1:16 scale radio-controlled cars with self-driving capabilities, a Python framework for training and testing DNN models with lane-kee** functionalities using supervised or reinforcement learning, and a simulator in which the real-world Donkey Car is faithfully modeled. This was assessed by a recent work stocco-mind reporting that, for three lane-kee** models, the steering angle distribution of the AV model driving in the real-world environment is statistically indistinguishable from the steering angle distribution of the AV model driving in the digital twin.

In the rest of the section, we refer to BeamNG as DS₁, Udacity as DS₂, the combined digital siblings as DSS, and DonkeyCar as DT.

We followed the guidelines by Bojarski et al. nvidia-dave2 to train AV autopilots. We used custom hyperparameters for each self-driving architecture, and the Adam optimizer kingma2014adam to minimize the mean squared error (MSE) between the predicted steering angles and the ground truth value. For all models, we set a learning rate of $10^{-4}$ and a batch size of 128. We used 50 epochs for Dave-2 and Chauffeur (only for the $M_{R}$ model) and 500 epochs for Epoch and the $M_{S}$ model of Chauffeur. We used an early stop** of 10 epochs for the models where the number of training epochs was 50 and an early stop** of 20 epochs otherwise.

We evaluated the performance of the trained lane-kee** models on DT, as it is the target simulator we want to approximate using the digital siblings. We collected a labeled dataset by running the autopilot on DT on 25 randomly generated roads each with 8 control points and a maximum angle of $270^{\circ}$, i.e., the same road parameters as the training set. We computed the mean squared error (MSE) between the steering angle prediction of the model on each image and the steering angle of the autopilot. Table 1 shows the MSE of all models on the first and third columns; on average, the MSE is low for both the models trained using simulated images (i.e., $M_{S}$), and the models trained using real images (i.e., $M_{R}$). We also measured the success rate of each model by driving it on the 25 randomly generated roads, and counting the number of times the model was able to arrive at the end of the road without going out of bound. Overall, each model is able to successfully complete the majority of the generated roads. Most notably, $M_{R}$ models are able to complete more than 90% of the test set roads.

We collected a labeled dataset for offline evaluation by generating 20 roads (i.e., 10 roads driven in both directions) with the same parameters as the training set. The images collected for the offline evaluation dataset amount to $\approx$26k, considering all simulators.

After training $M_{S}$ and $M_{R}$ for each self-driving architecture, we executed DeepHyperion twice to generate tests using the two digital siblings DS₁ and DS₂. We chose a population size of 20 individuals and a number of search iterations respectively equal to 150 for $M_{S}$ and 100 for $M_{R}$, as we observed from preliminary experiments that this choice of hyperparameters allows an extensive coverage of the feature maps. For both $M_{S}$ and $M_{R}$ and each digital sibling in each self-driving architecture, we repeated test generation five times to diversify the exploration of the search space and to collect multiple test cases for each cell in the feature maps. Overall, across all runs and driving models, DeepHyperion generated 10,260 tests for both siblings.

Concerning the simulations, for all simulators, we set the maximum speed for the vehicle to 30 km/h deephyperion . When testing $M_{R}$ in a given simulator, we engineered the testing pipeline to load the appropriate sim2real CycleGAN generator to translate the simulated image generated by BeamNG/Udacity into pseudo-real images in real-time during driving. For each executed test case, we collected the lateral position of the vehicle for each simulation step as well as its lateral distance. The former determines the quality of driving of the model 2021-Jahangirova-ICST , while the latter is the fitness of the test case.

For the initial ($FM_{DS_{1}}$, $FM_{DS_{2}}$) and for the union ($FM_{U_{1}}$, $FM_{U_{2}}$) feature maps, we compute the failure probability as the number of tests with a negative fitness divided by the total number of tests in the respective cell. To evaluate the quality of driving, we adopted the maximum lateral position (i.e., the distance between the center of the vehicle and the center of the lane 2020-Stocco-GAUSS ) experienced during the test case execution. Previous work showed that such metric is effective at characterizing the degradation in the quality of autonomous driving 2021-Jahangirova-ICST , since the lower the value of such metric, the higher is the quality of driving (thus, it actually measures lack of quality of driving). When considering the quality of driving, the value of each cell in a feature map represents the average of the maximum lateral positions of each test case in that cell. Furthermore, we normalized the maximum lateral position values in the interval $[0,1]$ before taking the union.

Merging the maps of the two digital siblings requires a different treatment for failure probability and quality of driving. Regarding the failure probability, the merge operator that ensures a conservative aggregation of two values is the product. Regarding the lack of quality of driving, the conservative merge operator is the minimum, since the quantities to merge are not probabilities. In fact, by taking the minimum we get a high lack of driving quality only when both simulators exhibit high values for such a metric.

We computed the prediction errors given by the difference between the predictions of the model ($M_{R}$) on images of the offline evaluation dataset (see Section 4.2), and the corresponding ground truth labels given by the autopilot. We binned the prediction errors of the model on each simulator and built the respective probability density (i.e., the number of errors in each bin is divided by the total number of prediction errors) such that different distributions could be compared.

Then, we computed the distance between each digital sibling distribution, as well as their combination, and the DT using the Wasserstein distance wgan (also known as the earth mover’s distance). Given two one-dimensional distributions $A$ and $B$, the Wasserstein distance $W(A,B)$ is defined by the following formula wasserstein :

We assess whether the difference between two distributions is statistically significant using the Wilcoxon test non-parametric-book applied to the density functions of the two error distributions to compute the $p$-value (with threshold $\alpha\leq 0.05$). We also perform power analysis (with statistical power $\beta\geq 0.8$) on the prediction errors to check whether a non-significant $p$-value is due to a low data sample size or to the difference being statistically insignificant.

For RQ₂, we computed the pairwise Pearson correlation between maps along with the corresponding $p$-value. In particular, correlations are computed between each union feature map of each digital sibling ($FM_{U_{1}}$, $FM_{U_{2}}$) and the feature map of the DT ($FM_{DT}$), and between $FM_{DSS}$ and $FM_{DT}$. For RQ₃, the setting is equivalent to that of the failure probability but considering quality of driving maps, comparing DS₁, DS₂ and DSS against the ground truth DT.

To evaluate the capabilities of the digital siblings (individually or jointly) to predict failures on DT, we computed the area under the curve Precision-Recall (AUC-PRC) at increasing thresholds, for both RQ₂ and RQ₃. This requires the discretization of failure probabilities into binary values (failure vs non-failure) for the ground truth (i.e., DT): we consider a cell in the DT feature map to be a failure cell if the associated failure probability is $>0.0$. AUC-PRC is more informative than the AUC-ROC metric (i.e., the area under of the curve of the Receiver Operating Characteristics) when dealing with imbalanced prc-justification datasets, which is the case of our study (the number of failures in the feature maps is lower than the number of non-failures with an average 10 to 20% ratio).

Table 2 reports the results for our first research question. The first column shows the simulators being compared. Columns 2–5 report the Wasserstein distance between the prediction error densities of the corresponding simulators, and the $p$-value concerning the statistical significance of the differences between the two densities, for $M_{S}$ and $M_{R}$.

For $M_{S}$ (Columns 3–4), our results show that, for Dave-2, the distance between the steering angle errors obtained for the combined digital siblings DSS and the errors obtained for DT is lower than the distance of DS₁ (0.03776 vs 0.046) and higher than the distance of DS₂ (0.02648). The distribution of the steering angle errors of DS₂ is statistically different from the errors of DT (i.e., $p$-value 0.02 $<0.05$), while the distribution of the steering angle errors of DSS is statistically indistinguishable from the errors of DT (i.e., $p$-value 0.053 $>0.05$ and power $>0.8$). This behavior is also consistent for Epoch, with the exception that the distribution of the prediction errors for DS₂ is statistically indistinguishable from that of DT. However, the distance between DSS and DT is lower than the distance of DS₁ from DT, with a statistically indistinguishable distribution of prediction errors w.r.t. DT. For Chauffeur, the combined digital siblings DSS have the only distribution of errors that is equivalent to that of DT, and its distance to it is the lowest considering the individual digital siblings.

Regarding $M_{R}$ (Columns 5–6), our results show that, for Dave-2, the distance between the steering angle errors obtained for the combined digital siblings DSS and the errors obtained for DT is 2.8 times lower than the distance of each simulator taken individually (as a percentage, the distance of DSS is respectively 70% and 56% smaller than the distance of the two individual siblings, DS₁, DS₂). The statistical test confirms that the error distributions of DSS and DT are statistically indistinguishable ($p$-value $>$ 0.05 and power $>$ 0.8), which is not the case for the error distributions of DS₁ ($p$-value $<$ 0.05). Likewise, for all the other self-driving architectures, the digital siblings DSS have the lowest distance to DT w.r.t. the individual siblings and their distribution is always statistically indistinguishable from that of DT.

Figure 5 offers a visual explanation of these scores for the Dave-2 model.¹¹1We report the plots for the other lane-kee** models in our replication package tool . The subplots compare the steering angle error distributions, respectively, of DS₁, DS₂ and DSS (shown in light red) with that of DT (shown in light blue). The $x$-axis of each subplot represents the magnitude of the prediction errors of the model $M_{R}$ w.r.t. the predictions of the autopilot, while the $y$-axis indicates their percentage for each bin.

From the plots we can see that, overall, at the model-level, $M_{R}$ makes prediction errors with small magnitudes on DS₁, DS₂ and DSS (i.e., most of the errors are between 0.0 and 0.3). On the digital sibling DS₁ (i.e., BeamNG), $M_{R}$ has a high agreement with the autopilot, as most errors have a low magnitude. It has numerous small errors ($<0.2$), while it has only a negligible portion of the distribution being above 0.2. The agreement with DT is low as $M_{R}$ under-approximates the true error distribution on DT: $M_{R}$ on DT has fewer errors with low magnitude and has a longer tail of errors greater than 0.2 (even greater than 0.3 in some cases). Differently, on the digital sibling DS₂ (i.e., Udacity), the error distribution has a longer tail than that on DT. Indeed, $M_{R}$ executed on DS₂ over-approximates the errors it would have on DT, as the errors observed on DS₂ have higher magnitude than those observed on DT.

The error distribution of the model on DSS shows why it is appropriate to combine the outcome of two simulators. At the model-level, DSS better approximates the true error distribution of the model on DT, by providing an intermediate error between DS₁ and DS₂ for both $M_{S}$ and $M_{R}$.

Table 3 shows the Pearson correlation (r), the $p$-value, and the AUC-PRC for the comparison between DS₁, DS₂, DSS and DT, respectively. The analysis is reported separately for $M_{S}$ (Columns 3–5) and $M_{R}$ (Columns 6–8).

Concerning $M_{S}$—i.e., the model driving with simulated driving scenes— the failure probabilities for Dave-2 have a high positive correlation with the true failure probability of DT ((Column 3). All such correlations are statistically significant for DSS, as well as for each individual sibling DS₁ and DS₂ ($p$-values $<$ 0.05, see Column 4). Likewise, the correlations are high and statistically significant for the other lane-kee** models (Epoch features slightly lower correlations).

However, for Dave-2 the correlation of DSS is 9% higher than the best individual correlation (i.e., DS₁) and 21% higher than the worst individual correlation (i.e., DS₂). In terms of failure prediction, DSS have the highest AUC-PRC value, 4% higher than DS₁ and 33% higher than DS₂.

This also happens with Epoch, where the correlation of DSS is slightly higher than that of the best sibling DS₁ (i.e., 0.571 vs 0.561) and 33% higher than that of the worst sibling DS₂. Regarding failure prediction on DT, DSS are 3% better than the best sibling. In the case of Chauffeur, DS₁ has the best results both in terms of correlation and failure prediction. However, DSS are better than the worst of the two siblings DS₂ both in terms of correlation and failure prediction.

Figure 6 shows the feature maps related to $M_{S}$ of Dave-2.²²2We report the plots for the other lane-kee** models in our replication package tool . The first three feature maps represent the failure probability of DS₁, DS₂ and DSS, respectively. The last feature map represents the ground truth failure probability of DT. The color of each cell ranges from green (i.e., non-failure, or failure probability = 0) to red (i.e., failure probability = 1). Let us analyze a false positive case. The test cases at coordinates (3, 0.25), whose corresponding cells are highlighted with a dashed line, represent road tracks having three curves and a maximum curvature of 0.25. In DT, this cell is green, i.e., all test cases for $M_{S}$ driving on DT succeed. On the other hand, $M_{S}$ has contrasting behaviors when the same test cases are executed on DS₁ or DS₂. These test cases did not exhibit any failure in DS₁, whereas they did trigger failures in DS₂. This disagreement is canceled out when combining the two digital siblings with the product operator and the cell is green in the DSS map. As such, digital siblings are conservative w.r.t. failures, as a failure is reported only when both digital siblings are in agreement. This can be noticed for test cases at coordinates (1, 0.23), which represent road tracks having one curve with a maximum curvature of 0.23—an instance of a true positive case (the corresponding cells in each map are highlighted with a solid line). Both DS₁ and DS₂ have a failure probability of 1 and, as a consequence, the DSS map also does. On DT, $M_{S}$ has also a high failure probability (0.5), which confirms the high effectiveness of the DSS framework at approximating the true failure probability of DT.

Concerning the failure probability for $M_{R}$—i.e., the model driving with pseudo-real driving scenes, for Dave-2 and Chauffeur, DSS are better than each individual sibling in terms of correlation with DT. For Dave-2, DS₁ better predicts the failures of DT, while for Chauffeur, the digital siblings are better than each individual sibling. Interestingly, for Epoch, DS₂ better correlates with DT but the AUC-PRC value of DSS is the higher than the individual siblings.

Table 4 shows the Pearson correlation (r), the $p$-value, and the AUC-PRC for the comparison between DS₁, DS₂, DSS and DT, respectively. The comparison considers the correlation between the quality of driving metric experienced in DS₁, DS₂, DSS and the failure probability of the model on DT, as well as the prediction of failures from the quality of driving metric. The analysis is reported separately for both $M_{S}$ (Columns 3–5) and $M_{R}$ (Columns 6–8) models.

For $M_{S}$, the correlation between DSS and DT is lower than the best individual correlation for all the lane-kee** models (0.553 of DSS vs 0.621 of DS₁ for Dave-2, 0.792 of DSS vs 0.798 of DS₁ for Chauffeur, and 0.491 of DSS vs 0.511 of DS₁ for Epoch). For Dave-2, the DSS correlation is 22% higher than the worst individual correlation (0.553 of DSS vs 0.429 of DS₂); percentages are similar for Chauffeur and Epoch. For AUC-PRC, DSS and DS₁ have the same predictive power both for Dave-2 and Chauffeur (i.e., respectively 0.659 and 0.940), while for Epoch the DSS prediction is slightly better than that of DS₁. Thus, DSS mitigate the risk of relying on the testing results of a low-quality GPSim (i.e., DS₂).

Concerning $M_{R}$, we observed a similar trend, i.e., the correlation of DS₁ with DT are higher than the correlations of DSS with DT, although DSS always have a better correlation than the worst of the two siblings, i.e., DS₂, for all lane-kee** models. The digital siblings DSS better predict the failures of DT for Dave-2 and are equivalent to DS₁ for Chauffeur. For Epoch, the best predictor of the failures of DT is DS₂, although the digital siblings are only 9% worse.

Figure 7 shows the four feature maps related to the quality of driving of the $M_{R}$ Dave-2 model on the two digital siblings and the failure probability of $M_{R}$ on DT.³³3We report the plots for the other lane-kee** models in our replication package tool . We can observe that the feature map of DS₁ and the feature map of DSS are similar. As a consequence, the two correlations are similar (0.396 of DS₁ vs 0.379 of DSS). On the other hand, the feature map of DS₂ is quite different from the failure probability map of DT, which causes the correlation to be low (0.287). We can observe that all siblings are able to capture the failure of the DT at coordinates (1, 0.23) (see the corresponding cells highlighted with a solid line). On the other hand, the test cases at coordinates (4, 0.24) triggered failures only in DS₂, and DSS correctly predict that in DT such tests will not cause a failure.

We compared all simulators under identical parameter settings. One threat to internal validity concerns our custom implementation of DeepHyperion within the simulators. We mitigated this threat by faithfully replicating the code available in the replication package of the paper deephyperion-tool . Another threat may be due to our own data collection phase and training of the lane-kee** models, which may exhibit many misbehaviors if trained inadequately. We mitigated this threat by training and fine-tuning a model which was able to drive on the majority of the training set roads consistently on all simulators.

We considered only a limited number of DNN models and simulators, which poses a threat in terms of the generalizability of our results. We tried to mitigate this threat by choosing three popular real-world DNN models, which achieved competitive scores in the Udacity challenge udacity-challenge . Their diversity in terms of both size and architectural structure determines different driving behaviors and increases the generalizability of our results. We considered two open-source GPSims, and we chose DonkeyCar as DT, as it was used as a proxy for full size self-driving cars also in previous studies stocco-mind ; 2023-Stocco-EMSE ; 2021-01-0248 ; viitala ; 9412011 . Generalizability to other GPSims or DTs would require further studies.

Our proposal focuses on testing the DNN-based lane-kee** component of an AV, by generating a large set of road scenarios. Although there are works in the literature that modify other environment objects such as weather conditions, pedestrian and other vehicles’ dynamics Abdessalem-ICSE18 ; samota ; borg , we chose to generate road scenarios to test the lane-kee** behavior of the DNN in isolation, avoiding the interference of other tasks, such as obstacle and pedestrian avoidance. Further studies are needed to assess the generalizability of our multi-simulator approach to driving tasks different from lane-kee**. On this regard, feature maps are a flexible tool to encode different characteristics of a test case (e.g., the intensity of the rain or the number of vehicles in the driving scenario), by adding new dimensions for each new desired feature.

Threats to construct validity may come from selecting inappropriate metrics to measure the agreement of the siblings with DT. To address this threat we assessed such agreement from two points of view, i.e., at the model-level (RQ₁), by measuring the distance between the two distributions under analysis and testing the statistical significance of the difference, and at the system-level, by measuring failure probability and quality of driving. Overall, our results show that the digital siblings are better at predicting the behavior of the lane-kee** model under test on DT.