Robustness Testing of Black-Box Models Against CT Degradation Through Test-Time Augmentation

Abstract

Deep learning models for medical image segmentation and object detection are becoming increasingly available as clinical products. However, details are rarely provided about the training data, thus models may unexpectedly fail when cases differ from those in the training distribution. An approach allowing potential users to independently test a model’s robustness, treating it as a ‘black-box’ and using only a few cases from their own site, is key for adoption. To address this, a method to test the robustness of these models against CT image quality variation is presented. In this work we present this framework by demonstrating that given the same training data, the model architecture and data pre-processing greatly affect the robustness of several frequently used segmentation and object detection methods to simulated CT imaging artifacts and degradation. Our framework also addresses the concern about the sustainability of deep learning models in clinical use, by considering future shifts in image quality due to scanner deterioration or imaging protocol changes which are not reflected in a limited local test dataset.

Keywords: Robustness Testing, Out of Distribution, Computed Tomography

1 Introduction

Segmentation and object detection are important medical imaging analysis tasks in clinical and research settings. Segmentation is a fundamental analysis step because it provides boundary localization and volume quantification of anatomical and pathological structures which may be key for diagnosis and treatment planning  (Giorgio and De Stefano, 2013)  (Cabral Jr. et al., 1993). Automated object detection can enhance radiological reporting by highlighting pathological features  (Choi et al., 2022), or can guide further medical image analysis to focus on important features of the image  (** et al., 2022). However, a lack of clinicians’ trust in deep learning based applications can undermine adoption, which can be remedied with scientific robustness testing of an application using data from the clinicians own center  (Ahmad, 2021). Within this field, robustness refers to a model’s ability to maintain performance when encountering data which differs from the training dataset, due to a shift in demographics, acquisition protocol, or acquisition artifacts  (Galati et al., 2022).

X-ray Computed Tomography (CT) is a frequently used medical imaging modality with many applications, including radiotherapy planning, tumor diagnosis, angiography and trauma analysis  (Liguori et al., 2015). This work presents a method to facilitate model independent robustness testing using a limited amount of CT data.

Deep neural networks based models have recently become the foremost approach for automating many segmentation and object detection tasks and can achieve expert human performance in some applications  (Kooi et al., 2017). However, models have various vulnerabilities which can make them less robust than human visual assessment  (Geirhos et al., 2018). This issue becomes particularly acute as deep learning systems are increasingly distributed as commercial products, where restricted information about the model architecture and training data render the application an effective ‘black-box’ for the end user, making their robustness unclear and undermining trust.

Robustness testing of a deep learning model should consider the types of input degradation it would encounter during real-world use, where degradation is the process by which the quality of an image is diminished or compromised. Although it is well known that models are susceptible to adversarial examples  (Szegedy et al., 2013), these cases are created by an agent deliberately attempting to fool the system which is unlikely in a medical imaging context. Also, although common degradations affecting natural image quality, e.g. noise, contrast alterations, and blurring, have been shown to affect models’ performance  (Geirhos et al., 2018), the degradations seen in medical images have a fundamentally different nature. The key concern for robustness in a medical imaging application is degradation caused by the acquisition process. This may be caused by the acquisition protocol changing, such as the resolution or CT exposure parameters. There may also be image artifacts, defined as image features not present in reality but which appear due to unintentional acquisition phenomena, such as the patient moving during the scan or the effect of metal implants.

For a model trained on data with consistent acquisition parameters and without notable artifacts, these degradations can result in a discrepancy between the characteristics of training and test data, called a distribution shift  (Quiñonero-Candela et al., 2008). Test data affected by this discrepancy is a form of out-of-distribution (OOD) data, which in medical imaging context may be frequently encountered and which can undermine the robustness of deep learning applications. Although there are many approaches to retrospectively mitigate CT artifacts, including simulations to train models to remove them  (van der Ham et al., 2022), there is little understanding of how specific parameter changes and artifacts in CT imaging can create a distribution shift which affects the performance of models and therefore their safety in a medical context.

We address this by develo** a framework to systematically assess robustness of black-box models for segmentation and object detection with CT images (see Figure 1). This would allow a user with a limited test dataset acquired from their own center, with annotations representing the optimal output for those cases according to an expert, to test a segmentation or object detection model against OOD cases not present in their dataset but which are likely to occur during future use. Our suite of tests include increased CT noise, artifacts due to metal implants, and patient motion during the scan. Our framework allows direct comparison of the robustness of multiple ‘black-box’ models designed for the same task. This aims to aid confidence in the decision to adopt deep learning based methods in clinical practice, as well as trust in the continuing robustness of these models to future changes in image quality.

Figure 1: An overview of the proposed framework for robustness testing. The potential user of a black-box model wishes to test its robustness to CT image degradation using a small locally acquired test dataset, annotated with segmentation maps or object boxes outlined by an expert. The performance of the model is evaluated after various simulation-based augmentations are applied to the input image (center). Then the evaluation results, alongside information from the test dataset noise distribution, are used to calculate robustness metrics for the model.

2 Related work

2.1 Identifying OOD images with knowledge of the model and training data

The most direct method to test a models’s robustness against OOD data is to evaluate its performance on a dataset which is known to be OOD compared to the model’s training data. In medical imaging, this approach has been used to test the accuracy of OOD detection algorithms designed to automatically identify cases which the model is not expected to be robust to  (Vasiliuk et al., 2023b)  (Vasiliuk et al., 2023a)  (Nguyen et al., 2023). OOD test sets were compiled by selecting datasets with a known distribution shift relative to the training dataset, such as different acquisition protocols or underlying patient conditions. However, this is dependent on knowledge about the training data, which is not available unless provided by the developers.

Methods do exist to explore if input data are OOD when the training dataset is not known. If access to a model’s architecture and trained weights is available, methods like Mahalanobis distance  (Anthony and Kamnitsas, 2023) and generalized ODIN  (Vasiliuk et al., 2023a) can be used to predict if an input medical image is OOD. In other situations the models weights may not be accessible, such as when it has been integrated into a product or if model obfuscation methods have been applied  (Zhou et al., 2023). In this case, the output softmax confidence score can be used to predict if an input is OOD  (Hendrycks and Gimpel, 2016)  (Liu et al., 2020). However, OOD detection methods based on confidence output have limited accuracy in natural image applications  (Liu et al., 2020), which becomes significantly worse for medical imaging applications in segmentation  (Vasiliuk et al., 2023a). Also, obtaining the softmax output may depend on having access to the final layer of the neural network architecture before a threshold is applied.

2.2 Testing robustness against OOD images from benchmark datasets

A sufficiently diverse benchmark dataset can provide a fully model-agnostic method to test a black-box model’s robustness against OOD data (Boone et al., 2023). An ideal benchmark dataset would contain a range of input images with corresponding annotations and where any potentially OOD properties of each input image is labeled. This allows evaluation of the accuracy of the final output against the annotations, summarizing how much the model’s performance degrades when it encounters each given type of OOD data. An example benchmark dataset contained cardiac Magnetic Resonance Imaging (MRI) images labeled according to the imaging center, scanner type, and disease condition, which formed part of a challenge to create generalizable segmentation models  (Campello et al., 2021). However, the report from the challenge organizers did not discuss what properties of images in certain groups led to reduced performance from the submitted models  (Campello et al., 2021). MOOD 2020 is another example benchmark dataset for brain MRI and abdominal CT (Zimmerer et al., 2022). Some images were labeled by humans as ODD due to anomalous pathology, but most were created by augmenting images from the in-distribution set. The categories of augmentations included removal of slices, blurring, global and local deformations, and randomly inserting patches of noise and sections of other images. Some have questioned whether these augmentations realistically depict anomalous medical images in the real world (Li et al., 2023a), suggesting the augmented image patches should be formed by blurring multiple extracted image features together. However, the patch based augmentations still would not reflect realistic OOD cases which result from the acquisition process.

2.3 Testing robustness against images with acquisition based augmentations

Augmentations to create potentially OOD images can instead be designed to recreate acquisition phenomena known to cause anomalous cases in the real world. For example, a patient may enter the scanner in an unusual position due to a spinal injury  (Yang et al., 2023). This could be reflected by image translation and rotation augmentations, which are already frequently included as default training augmentations (Goceri, 2023). It has been found that MRI hardware and software updates can change image properties enough to affect segmentation methods and thus distort brain tissue volume results in longitudinal studies (Medawar et al., 2021) (Potvin et al., 2019). Changing the scanner or its parameters may alter the image resolution or field of view, which can be recreated by the common training augmentations of rescaling and crop** (Goceri, 2023).

A shift in the image noise distribution is another effect of changing scanning parameters, such as when the CT radiation dose changes by altering the tube current or slice scan time (Goldman, 2007). Even though adding Gaussian distributed noise is a common training-time augmentation technique (Goceri, 2023), in the medical imaging domain actual noise has a non-Gaussian distribution associated with the acquisition modality. Speckle, Rician and Poisson distributed noise have been suggested as training-time augmentations for ultrasound (Singla et al., 2022), MRI (Boone et al., 2023), and planar X-Ray (Khalifa et al., 2022) images respectively. The noise in CT scans has a complicated spatial structure, augmentation of which requires either a physics-based simulation of the CT acquisition process (Won Kim and Kim, 2014) or the use of a generative model (Liu et al., 2022). A physics-based simulation of CT noise has been used to augment the training data for model’s to classify lung image patches for the presence of lung nodules (Omigbodun et al., 2019), which did not lead to a performance improvement for test data of various noise levels compared to Gaussian noise augmentation. However, this study only considered one specific task, and did not use the physics based model to assess the robustness of different model architectures to CT noise.

Patient motion during a scan can cause image artifacts resulting in image degradation. A simulation of MRI motion artifacts has been been used to augment the training data of a segmentation model, which improved its robustness against real world artifacts (Shaw et al., 2020).

A recent study (Boone et al., 2023) provided a benchmark dataset to test the robustness of models for MRI segmentation against OOD cases, created using a series of physics-based augmentations. This included noise, illumination field, resolution changes, spatial transformations and motion artifacts. The authors also defined a degradation metric to quantify a model’s robustness against a certain augmentation, which takes into account the mean or standard deviation of the segmentation performance metrics (e.g. Dice (Dice, 1945)) after a range of augmentation severity is applied to a test set. Since they found that the variance of the segmentation performance metrics increases with more extreme augmentations, a similar degradation metric using the variance was suggested. Both approaches allow direct comparison of the robustness of two models when tested with the same augmented dataset, by comparing the degradation value.

2.4 Testing robustness against OOD images with limited datasets

It is crucial for the end user to be able to independently test cases from their own clinical center to support their decision to adopt deep learning based applications for medical imaging analysis in a clinical context (Jacobson and Krupinski, 2021), because they must be appropriately trained for the targeted population demographics (Vayena et al., 2018), disease presentation, and acquisition system (Prior et al., 2020). Testing data has to be manually labeled by human experts, resulting in high cost and limited size of testing datasets (Koh et al., 2022), which is often exacerbated at a local level (Shaikhina and Khovanova, 2017).

If the test dataset is limited in size, the statistical significance of summative performance metrics (e.g. mean Dice) may be undermined and it may lack rarer types of OOD images which may be encountered in the future. Methods have been developed to expand small medical imaging datasets with synthetic data produced by generative adversarial networks (GANs), mostly to train models for classification tasks such as COVID-19 detection (Waheed et al., 2020). GANs have also been used to expand a training dataset for 2D retina image segmentation (Noguchi et al., 2020). Since this approach also requires the generation of accurate corresponding synthetic ground truth segmentation masks, the segmentation model being trained has to be integrated into the data synthesizing algorithm, rendering this an inappropriate way to generate test data to independently assess black-box models. Non-deep learning approaches for expanding datasets mix features in the images to create new images. A crop** and patching method has been proposed for both images and segmentation mask annotations  (Noguchi et al., 2020), however the images produced contain anatomy which is concatenated in a clearly unrealistic way. Laplacian blending has been proposed as a more realistic alternative for classification datasets  (Sanaat et al., 2022), but this method would pose problems for generating valid annotations in the transition regions for both segmentation and object detection applications.

The above methods to expand limited datasets can generate cases which are potentially OOD due to new combinations of image features. However, the patch based methods combine image features in an unrealistic way, while GAN based methods create ground truths which are not independent of the model being tested. Furthermore, these methods cannot add OOD cases resulting from acquisition artifacts/degradation not already present in the dataset. This motivates our CT simulation based framework, which is independent of the models being tested and allows them to be treated as a black-box. The augmentations generated by the simulation can also add common artifacts/degradations not initially present, or ones with greater severity, to the limited test dataset.

2.5 Contributions

To address these issues with existing approaches to independently test the robustness of black-box models with limited datasets, for the first time to our knowledge, we develop a benchmarking platform to create datasets for evaluating the robustness of models to corruptions and artifacts in CT. Analogous to previous work (Boone et al., 2023) which presented a system for robustness and out-of-distribution testing specifically to brain MRI segmentation models (ROOD-MRI), here we expand and further generalize this concept to more generic segmentation and localization road-testing tasks in CT imaging. We make the following scientific contributions to this field:

• We propose a physics-based CT simulation for modeling associated acquisition anomalies: noise variation, metal implant artifacts, and patient motion. Our simulation approach only uses parameters obtained from analysis of the CT images themselves, allowing use when details of the CT acquisition are not available.

• Using test-time data augmentations from the simulation, we demonstrate that different deep learning model architectures have weaker robustness to different CT artifacts.

• We investigate established models, designed for object detection (retinaNet) and for segmentation (U-Net, nnUnet). We also convert the tested segmentation models into object detection applications, to compare their robustness with retinaNet.

• Inspired by ROOD-MRI (Boone et al., 2023) which uses fixed weighting of the contribution of each augmentation severity level to calculate the summative noise degradation metric, here we expand on that formula by using an empirical distribution of noise in a given dataset to calibrate the weighting of noise augmentation severity levels.

The remainder of our paper is structured as follows. In Section 3 we outline the datasets we used to train models. The established model architectures that we trained to apply the robustness testing framework to are briefly described: 3D-UNet (Çiçek et al., 2016), nnUnet (Isensee et al., 2021), and retinaNet (Lin et al., 2017). Then, we describe the metrics used to evaluate their performance at segmentation or object detection using test sub-datasets. Section 4 describes how our CT simulation approach is based only on parameters extracted from the test CT images themselves, allowing general use to generate the augmentations for robustness testing. In Section 5, we show these applied to generate a summary robustness metric calibrated with the dataset. Then our results show how these methods demonstrate the strengths and weakness of different model architectures.

3 Materials and metrics

3.1 Datasets

3.1.1 LUNA 16

LUNA 16 is a lung nodule object detection challenge dataset (Jacobs and van Ginneken, 2016). We included it in this study because lung nodule annotation is crucial for lung cancer diagnosis, but manual labeling is time-consuming and error prone, so deep learning applications are of significant interest (Ren et al., 2020).

The dataset includes CT scans with a maximum slice thickness of 2.5 mm from the publicly available LIDC/IDRI database (Armato III et al., 2011). Annotations were collected by 4 experienced radiologists, and the LUNA 16 annotations consist of all nodules larger than 3 mm accepted by at least 3 out of 4 radiologists. The annotations were presented as ground truths for object detection, where all nodules were labeled as having the same class while their position and size were defined by 3D bounding box co-ordinates. We used 570 images for training models while 30 were left aside for testing.

The training images were pre-processed by resampling via bilinear interpolation to (0.703, 0.703, 1.25 mm) voxel sizes and the intensity was windowed to -1024 to 300 Hounsfield Units (HU). This pre-processing was applied to the test data after the application of CT simulation based augmentations described in Section 4.

3.1.2 Segmentation decathlon - liver task

As part of the The Medical Segmentation Decathlon challenge (Antonelli et al., 2022), the liver CT dataset consists of contrast-enhanced CT images from patients with primary cancers and metastatic liver disease. We included this dataset due to the importance of robust organ segmentation for treatment planning, such as radiotherapy (Yu et al., 2022).

The dataset contained target segmentation maps outlining the liver. It was acquired in the IRCAD Hôpitaux Universitaires, France and contained a subset of patients from the 2017 Liver Tumor Segmentation challenge (Bilic et al., 2023). We used 570 images for training models and 30 were left aside for testing.

The training images were pre-processed by resampling via bilinear interpolation to (1.5, 1.5, 1.5 mm), but the intensity was not windowed by default because the effect of training models with and without windowing was compared during robustness testing. This pre-processing was applied to the test data after the application of CT simulation based augmentations described in Section 4.

3.2 Segmentation model architectures and training

3.2.1 3D-UNet

The implementation of 3D-UNet (Çiçek et al., 2016) segmentation model provided within the MONAI library (Cardoso et al., 2022) was utilized. 3D-UNet was chosen because it has frequently been applied as a baseline architecture to compare the performance of newly developed models against during the past five years (Li et al., 2023b) (Li et al., 2023b) (Bui et al., 2019). We trained this architecture with three pre-processing approaches for the training data, to test the effect they have on robustness:

• No image augmentation, apart from random sampling of patches for training with size (96,96,48) for the liver decathlon dataset and (192,192,80) for LUNA 16.

• Image augmentation using the protocol used for nnUnet shown in Table 1, followed by the random patch selection described above.

• Image intensity windowing in the range -60 to 160 HU, followed by the random patch selection described above. This option was only used for the liver segmentation dataset because the window corresponds to the range of intensity of the liver and surrounding volume, while the intensity of lung nodules in the LUNA 16 dataset is very variable.

3.2.2 nnUNet

An integration of the nnUnet (Isensee et al., 2021) segmentation protocol within the MONAI library (Yang, 2023) was used. This trains an ensemble of UNet derived models with parameter optimization based on the data and hardware before selecting the best performing combination, as outlined in Table 1. nnUnet was chosen because it is considered a state-of-the-art segmentation protocol, after achieving the best performance across multiple medical imaging segmentation applications in the The Medical Segmentation Decathlon challenge (Antonelli et al., 2022).

Table 1: Comparison of the properties of the nnUNet and base 3D-UNet used here.
*The largest patch size possible with Batch Size 2 given the available memory.
**The best pair of the models, ensembled by averaging softmax probabilities, chosen by cross validation of the training data.

Dataset	nnUNet	3D-UNet
Training Augmentation	Rescale x0.9 or 1.2, 15%	None
	Gaussian Noise std 0.01, 15%
	Gaussian Smooth x0.5-1.15, 15%
	Intensity -0.3 or 0.3, 15%
	Flip (each axis), 50%
Learning Rate	‘poly’ schedule (initial 0.01)	0.001
Training Epochs	1000	600
Batch Size	2	1
Loss Function	Dice + Cross Entropy	Dice
Intensity Normalization	clipped: 0.5 to 99.5%	clipped: -57 to 164 HU
In-plane Image Resampling	In plane 3rd order spline	Linear interpolation
Slice Image Resampling	Nearest Neighbor	Linear interpolation
In-plane Annotation Resampling	Linear interpolation	Linear interpolation
Slice Annotation Resampling	Nearest Neighbor	Linear interpolation
Patch Size	Maximum for Batch Size 2*	96x96x96
Low-resolution Patch Size	25% of median image size	N/A
Ensemble Selection Options	3D-UNet	N/A
	2D-UNet
	Low-resolution UNet cascade
	Ensemble of two**

3.3 Object detection model architectures and training

3.3.1 retinaNet

We utilized an implementation of the retinaNet (Lin et al., 2017) architecture provided within the MONAI library. retinaNet is composed of a backbone network (ResNet in this implementation) with several downsampling layers followed by several upsampling layers. Each of the upsampling layers is sampled by two convolutional sub-networks to predict the object class and bounding box co-ordinates, giving independent box predictions considering different resolution scales which allows accurate identification of both large and small objects. This is appropriate for the LUNA 16 dataset, which contains annotated nodules varying widely in diameter from 3.0 mm to 28.3 mm. retinaNet also uses a focal loss function for classification, cross entropy weighted by uncertainty so difficult to classify examples are more heavily weighted, but there was only one class of nodule considered so this feature was not relevant.

3.3.2 Conversion of segmentation models to object detection models

Object detection models are evaluated here using the mean Average Precision (mAP), while the segmentation models are evaluated using the Dice score. Therefore, in order to allow direct comparison, we added a post-processing step to the segmentation models applied to the LUNA 16 dataset to create bounding boxes for the nodules like an object detection model, which can be evaluated using mAP. To do this, contiguous volumes of voxels labeled as ‘nodule’ within the predicted segmentation maps with a volume greater than 14.14 mm³ were selected. This is the volume of a sphere with a 3mm diameter, the minimum size of annotated nodules in the LUNA 16 dataset. For each of these contiguous volumes, the minimum enclosing box with edges parallel to the orthogonal image axes was calculated and treated as a box produced by an object detection model with that ‘nodule’ class at 100% confidence.

3.4 Evaluation metrics

3.4.1 Segmentation evaluation

The Dice coefficient (Dice, 1945) (DSC) is a de facto standard (Maier-Hein et al., 2024) used to quantify the accuracy of a segmentation method (Zijdenbos et al., 1994):

$$DSC=\frac{{2\times|A\cap B|}}{{|A|+|B|}}$$ (1)

where $A$ and $B$ represent the set of ground truth and the inferred segmentation voxels, and $|A|$ and $|B|$ are the amount of voxels in those sets. DSC ranges from 0 to 1, where 1 represents perfect overlap between the ground truth and inference. The mean DSC across a test dataset is used to evaluate the accuracy of a segmentation model. The consistency of the model may be impacted when OOD images are encountered, reflected by an increase in the standard deviation of DSC values (Boone et al., 2023).

3.4.2 Object detection evaluation

The mean Average Precision (mAP) is an evaluation metric for object detection methods used across several benchmark challenges (Padilla et al., 2020), which summarizes the precision-recall trade-off dictated by confidence levels of the predicted bounding boxes.

mAP is the mean of the Average Precision (AP) for each class, where AP is the area under a precision recall curve that has been preprocessed to remove zig-zag behavior (Padilla et al., 2021). To calculate the precision and recall at each threshold confidence value, each bounding box predicted by the model with a confidence exceeding the threshold is compared with the ground truth boxes using the metric Intersection over Union (IoU) (Padilla et al., 2021):

$$IoU=\frac{{|A\cap B|}}{{|A\cup B|}}$$ (2)

If the IoU between a predicted box and a ground truth box of the corresponding class exceeds a specified threshold, the predicted box is a true positive. If the IoU does not meet the threshold, another predicted box has already met the threshold, or the predicted class is wrong, then the predicted box is a false positive. Any ground truth boxes which does not have a corresponding true positive are counted as false negatives. The precision and recall can then be calculated for each threshold confidence value. mAP can be calculated for a single test image or across a test set. It should be noted that a ‘black-box’ object detection model may have an intrinsically fixed confidence threshold, in which case AP reduces to precision multiplied by recall.

mAP is often calculated with a range of IoU thresholds, with the mean result taken (Padilla et al., 2020). This approach is taken here, because the models being compared may perform best at different IoU thresholds. Furthermore, the IoU may be heavily affected by how tightly the image is bounded by the ground truth box, so a range of IoU thresholds helps capture this variability. This is particularly the case for annotations of lung nodules, which vary greatly in size, have boundaries which are difficult for a clinician to precisely discern (Larici et al., 2017), and which have a varying heterogeneous shape compared to an enclosing cuboid.

4 Methods for augmentation of CT data

4.1 CT simulation

A custom physics based simulation of CT was used to generate CT specific augmentations. This was based on the Astra toolbox (Van Aarle et al., 2016), which creates the system geometry then numerically simulates the stages of the acquisition process.

The pre-processing of the input images to be augmented follows a previously outlined method (Won Kim and Kim, 2014). First, the image CT values (Hounsfield Units) are converted to attenuation X-ray attenuation values $\mu$ (cm^-1):

$$\mu=\frac{{\text{{CT number}}}}{{1000}}\times\mu_{\text{{water}}}+\mu_{\text{{water}}}$$ (3)

where $\mu_{\text{{water}}}$ is the linear attenuation coefficient of water which is 0.18 cm^-1 at 120 kVp (Boedeker et al., 2007). Then, total variation based denoising (Rubin, 1992) was used to remove noise from the original acquisition from the attenuation image.

Figure 2: The geometry parameters of the simulated CT system: $\phi$ beam angle, d₁ source to image center distance, d₂ image center to array distance, d_det distance between adjacent detectors, and n_det is the number of detectors.

An outline of the geometry of the CT scanner system is required to simulate the acquisition process, but as this is often not available from image metadata, we use the following procedure to allow general CT augmentation. The geometric parameters required by the Astra toolbox are outlined in Figure 2: source to image center distance d₁, image center to array distance d₂, number of x-ray detectors n_det, and distance between adjacent detectors d_det. We assumed the beam angle $\phi$ to be 60°as is typical of modern scanners (Peyrin and Engelke, 2021), although a specific value is sometimes available in the metadata. We also assumed n_det to equal 1500, which is consistent with modern scanners (Hermena and Young, 2021) and higher values did not change the reconstructed image. The other geometric parameters can be calculated as follows.

d₁ can be obtained using $\phi$ and the diameter of the circular field of view d_fov. d_fov can measured from the input CT image by identifying the edge of the circular field of view, which is achieved by sampling a line of voxels from the corner of the central slice of the image to the image center, until an intensity value other then the uniform value outside the field of view is encountered.

$$d\textsubscript{1}=\frac{d_{fov}}{sin(\phi/2)}$$ (4)

d₂ can be assumed to be similar (approximated as equal) to d₁ because both the X-ray source and detectors are attached to the same rotating gantry, but the exact value will not affect the simulation if the air is assumed to have negligible attenuation and the values of n_det and d_det correspond to the projected size of the detector array arc given d1 and d2.

$$d_{det}=\frac{2\>\>(d_{1}+d_{2})\>\>tan(\phi/2)}{n_{det}}$$ (5)

Another parameter required by the acquisition simulation is the number of sampled angles n_$\theta$, but this was empirically tuned alongside the noise parameters in the process outlined in Figure 3.

The sinogram generated from the projection stage of the simulation was used to reconstruct a CT image using Astra’s filtered back projection function with the default Ramachandran-Lakshminarayan filter (Ramachandran and Lakshminarayanan, 1971).

Once the projection stage of the simulation has generated a sinogram for each slice, augmented noise could be added. This can be modeled as a combination of Poisson noise due to a restricted number of photons reaching the detector, and Gaussian electronic noise (Won Kim and Kim, 2014). As our augmentation approach is designed to be general, including for when specific information about the beam properties is not available, we empirically tune the incident flux of photons on each each detector point with no X-ray attenuation Q₀. This, in combination with the attenuation of the X-ray beam by the scanned object, determines the actual incident number of photons and thus Poisson noise, and Gaussian noise parameter $\sigma$. The sinogram after application of the noise augmentation $S_{ns}$ is

$$S_{ns}=S_{0}\,+\,P(\,Q\textsubscript{0}\,exp(-S_{0})\,)\,+\,G(\sigma)$$ (6)

where $S_{0}$ is the initial sinogram, $P(x)$ is a random sample from the Poisson distribution with expected value x, $G(\sigma)$ is a random sample from a Gaussian distribution with a zero mean and standard deviation $\sigma$.

Figure 3: Outline of the CT noise simulation tuning process. Step 1: The image intensity is converted from HU to attenuation units before denoising and noise extraction via the total variation method. Step 2: the simulation parameters Q₀, $\sigma$, and n_$\theta$ are selected to maximize the similarity in the radial noise power spectrum (NPS) patchwise between the noise generated by the simulation and the noise extracted from the original image, as described in Table 2. Step 3: To increase the level of noise, the incident flux Q₀ is decreased until a desired augmented image noise level is reached, as described in Section 4.2.1.

The noise model has three parameters which require empirical tuning using the input dataset (see Table 2). We did this by comparing the CT noise generated by the model with the noise extracted from the input test images by total variation denoising (see Figure 3). This utilized each test image by breaking the central slice into a 10x10 grid, calculating the radial noise power spectrum (NPS) in each cell (Dolly et al., 2016), calculating the mean of the sum of square differences between the NPS curves from the data and model, mSSE_NPS. The parameters were tuned to minimize the mSSE_NPS, which represents dissimilarity in the CT noise accounting for its spacial variation. The tuning process was a two stage coarse-to-fine grid search, using the search parameters in Table 2.

Table 2: The unitless simulation parameters tuned to recreate the CT noise extracted from the test dataset, by minimizing the mean noise power spectrum dissimilarity (mSSE_NPS). The fine grid search parameters were found by applying the multiplications or additions shown to the optimum parameter found in the coarse search stage.

Parameter	Description	Coarse Search	Fine Search
Q0	Incident photon flux	$10^{4}$,$10^{5}$,$10^{6}$,$10^{7}$	x0.5, x0.75, x1.0, x2.5, x5
$\sigma$	s.d. of Gaussian noise	0, 0.1, 1, 10	x0.5, x0.75, x1.0, x2.5, x5
n$\theta$	Number of sampled angles	720,1440,2160,2880	-360, +0, +360

4.2 Artifact generation

4.2.1 Noise

The noise in the image produced by the CT simulation can be increased by reducing the value of Q₀, modeling a lower X-ray dose, until the desired standard deviation (s.d.) of the simulated noise field is obtained. This search is done by rounding the optimized Q₀ to the nearest order of magnitude (10^$m$, where $m$ is an integer) and decreasing $m$, then fine-tuning using multiples of: [0.5, 0.75, 1.0, 1.25, 1.5], to obtain the closest s.d. value.

As shown by the simulated images in Figure 4 and the noise power spectra in Figure 5, this method of increasing the noise level does not significantly change the spatial distribution or texture of the noise field. Therefore, the strength of this augmentation can be quantified by the noise s.d., even though the augmentation is caused by altering the physical parameter Q₀.

Figure 4: The CT noise augmentation is performed by reducing the the simulation parameter Q₀, to the value resulting in the output images having an increased noise component with a given standard deviation (s.d.).

Figure 5: The radial noise power spectrum (NPS) is sampled from the noise extracted from the original image within a central patch (left) for comparison with the same patch of the simulated noise fields. The normalized NPS’s are shown to have similar profiles (right).

4.2.2 Metal implants

Metallic implants can cause streaking artifacts in CT images. In order to simulate this phenomenon, we augment the input images slices by inserting a cylindrical region with a radiodensity of 20,000 Hounsfield units, corresponding to pure steel (Bolliger et al., 2009). These implants were centered at the highest intensity point in the spine and orientated axially with a length spanning the axial range of the annotated objects or segmentation plus an extra 15mm in both directions. This arrangement was chosen because the image slices which contain part of the metal implant are affected by the characteristic streaking artifact, so in the augmented images generated by the the CT simulation, this artifact would be present in the slices containing the annotated objects or segmentation and also the region superior and inferior to them. Therefore, the effect of the metal implant artifact surrounds the annotated objects or segmentation.

We scaled the augmentation strength by increasing the radius of the implants. The output of the CT simulation contained the characteristic streaking artifacts after the input images were augmented with a metal implant. For small implants the artifacts were visually similar to examples in the LUNA 16 dataset, as shown in Figure 6. Simulated implants with a larger radius caused more extensive streaking, which is visually similar to reported observations of artifacts due to spinal implants (Barrett and Keat, 2004). However, comparably extensive streaking was not seen in the LUNA 16 dataset.

Figure 6: The simulation of a metallic implant as a form of CT data augmentation causes a streaking artifacts with a severity depending on the implant radius (white numbers). An example of the simulated artifact due to a 2.5mm radius implant (blue outline) is visually similar to an example artifact found in the LUNA 16 training dataset (orange outline).

4.2.3 Motion

We model patient motion as a rigid change in pose by tilting during the sequential acquisition of slices, by applying an axial rotation to all slices above or below a certain point and thus producing a discontinuity in the CT image. We consider two types of motion augmentation, which differ due to how their severity is scaled:

• Motion magnitude (mag): The augmentation strength is increased by increasing the amount of rotational motion in degrees, while the location of the slice discontinuity is constant at 10mm inferior to lowest point of the annotation.

• Motion proximity (prx): The augmentation strength is increased by decreasing the distance of the motion discontinuity to the to the lowest point of the annotation, while the amount of rotation is kept constant at 10°.

5 Experiments and results

5.1 Dataset analysis

Figure 7 shows the measurement of the noise distribution in the LUNA 16 dataset. This is used to find the set of weights w_s, replacing the coefficient w_s = (2/3)^s in equation (7) (Boone et al., 2023), where s is the level of augmentation severity.

$$Deg_{TM}=\frac{1}{\sum_{s=1}^{5}\omega_{s}}\sum_{s=1}^{5}\omega_{s}(mM_{clean}-mM_{T,s})\>\>,\>\>\>\text{with:}\>\>\>\>\>\omega_{s}=(2/3)^{s}$$ (7)

We obtained the weights (0.221,0.044,0.006,0,0,0) corresponding to the noise augmentation levels of (10,20,50,100,200,350,500) by dividing the frequency distribution at those points by its value at the base level of noise (see Figure 7b).

Figure 7: Measurements of the noise in the original LUNA 16 dataset allow empirical scaling of the degradation weights. The standard deviation of the noise extracted by the total variation method is measured for each case. a) We use a histogram to identify the modal level of noise as a base level (found to be 4 HU). b) We use the cases with a noise equal or higher than the base level to produce a second histogram, to which the distribution function (green) is fitted. Both histograms use a bin width of 1 HU.

5.2 Noise simulation

An example from the LUNA 16 dataset with high noise is shown in Figure 8. The CT noise is removed by total variation based denoising (Rubin, 1992) and recreated by the simulation, showing good visual similarity with the original image noise. In this case, the tuned simulation values are: Q₀ = $2.5\times 10^{6}$, $\sigma$ = $0$, and n_$\theta$ = 2160.

Figure 8: a) A slice from the LUNA 16 dataset, with a nodule labeled with a blue box. b) The slice with after denoising using the total variation method. c) the simulation is shown to recreate visually similar CT noise.

5.3 Model degradation results overview

An overview of the performance and degradation scores shows that nnUnet was overall the most robust segmentation model (see Table LABEL:tab_dce), while retinaNet was overall the most robust object detection model (see Table LABEL:tab_map). These summary results are analyzed closely in the following subsections.

Table 3: Degradation results for the segmentation models, where a lower score means better robustness ($\downarrow$). The DSC is measured between the annotation segmentation map and the model prediction, where a higher Base DSC means better model performance before augmentation ($\uparrow$). 3D-UNet (aug) is the 3D-UNet architecture with the same set of training augmentations used as nnUnet. 3D-UNet (win) is the 3D-UNet architecture with pre-processing of the input image by intensity windowing with the range -60 to 160 HU.

Dataset	Model	Base DSC $\uparrow$	Degradation metric $\downarrow$
			Noise	Metal	Motion mag	Motion prx
Liver Seg.	3D-UNet	0.808 $\pm$ 0.09	0.123	0.111	0.002	0.001
	3D-UNet (aug)	0.862 $\pm$ 0.09	0.128	0.106	0.003	0.000
	3D-UNet (win)	0.908 $\pm$ 0.10	0.217	0.155	0.000	0.001
	nnUNet	0.962 $\pm$ 0.10	0.061	0.070	0.000	0.000
LUNA 16	3D-UNet	0.751 $\pm$ 0.07	0.050	0.167	0.037	0.027
	3D-UNet (aug)	0.700 $\pm$ 0.08	0.046	0.143	0.023	0.041
	nnUNet	0.830 $\pm$ 0.07	0.031	0.166	0.038	0.051

Table 4: Degradation results for 3D Object Detection models, where a lower score means better robustness ($\downarrow$). For each image, mAP is the mean result for the IoU thresholds (0.01, 0.15… 0.9), where a higher base mAP means better performance before augmentation ($\uparrow$). 3D-UNet (aug) is the 3D-UNet architecture with the same set of training augmentations used as nnUnet. The nnUNet and 3D-UNet segmentation models are converted to detection models producing object bounding boxes, see Section 3.3.2.

Dataset	Model	Base mAP $\uparrow$	Degradation metric $\downarrow$
			Noise	Metal	Motion mag	Motion prx
LUNA 16	RetinaNet	0.709 $\pm$ 0.20	0.001	0.228	0.003	0.023
	3D-UNet	0.549 $\pm$ 0.17	0.067	0.243	0.037	0.033
	3D-UNet (aug)	0.591 $\pm$ 0.19	0.068	0.231	0.023	0.052
	nnUNet	0.660 $\pm$ 0.17	0.085	0.264	0.038	0.015

5.4 CT noise augmentation

Figure 9: The Dice (DSC) score results against the level of CT noise added by augmentation, for the segmentation models each tested with 30 cases from the liver segmentation dataset. The means with standard deviation error bars are shown in purple. nnUnet demonstrates higher robustness than the 3D-Unet models, shown by the shallower drop off in DSC with increasing noise and lower degradation metrics.

As shown in the ‘Noise’ column of Table LABEL:tab_dce and plotted in Figure 9, applying noise augmentation to the liver segmentation decathlon dataset shows that the nnUnet is more robust to increased CT noise than the 3D-UNet, as the mean DSC degradation (mDDeg) is lower. The robustness of the 3D-UNet increases when the nnUnet suite of training augmentation transforms was applied (see Table 1). Therefore, part of the increase in robustness is due to the training time augmentations, and part due to the model ensemble selection protocol and parameter tuning utilized by nnUnet.

Adding contrast windowing as part of the training and test prepossessing for 3D-UNet (HU windowing) resulted in greatly reduced robustness. This can be expected, because the model is trained to handle a smaller range of intensity values (-60 to 160 HU) and sufficient CT noise results in values being beyond this range. Although the effect of windowing on the input image shown in Figure 10 would not be visible to the user if this pre-processing step is intrinsic to a black-box model, the difference in robustness demonstrated by our framework would be.

Figure 10: Top: The output segmentation for the 3D-UNet model (without nnUnet based augmentation). Bottom: The same model trained with intensity windowing as a -60 to 160 HU pre-processing step shown below. The annotation segmentation map is blue, the predicted segmentation map is orange, and the overlap is purple. The windowing results in the effect of the noise is greater relative to the range of intensity values, resulting in the DSC reducing faster as the simulated noise level increases. Note the windowed images would not be available to a user if the windowing was integrated into the model, but we are able to reveal this as the model developer.

As shown in Table LABEL:tab_map, testing with the LUNA 16 test dataset shows that the retinaNet object detection model is very robust to increased CT noise, with no positive degradation, in contrast to any of the object detection models derived from segmentation models. When the robustness of the segmentation models were tested with the DSC metric, the nnUnet is more robust that the 3D-UNet. Adding the nnUnet suite of training augmentation transforms did not improve the robustness of 3D-UNet at this task.

A notable response of the 3D-UNet and nnUnet models to increasing CT noise is the great increase in DSC and mAP standard deviation. This is due to the metrics improving for some of the cases, which is because some false positive nodule segmentation or box generation were avoided when the nodule-like image features were obscured by noise. However, for the whole dataset this is outweighed by less true positive outputs with increased noise augmentation. This effect shows the importance of the standard deviation based degradation metrics (sMDeg and sPDeg) in complementing the metric based degradation metrics (mMDeg and mPDeg).

5.5 Metal implant augmentation

As shown in the ‘Metal’ column of Table LABEL:tab_dce, applying a simulated cylindrical implant to the liver segmentation decathlon dataset shows that the nnUnet is more robust to streak artifacts than the 3D-UNet. Like during the noise augmentation testing, the robustness of the 3D-UNet increased when the nnUnet suite of training augmentation transforms is applied. Table LABEL:tab_dce, Table LABEL:tab_map and Figure 12 also show this when the cylindrical implant augmentation was applied to the LUNA 16 test dataset and the segmentation models were tested. Figure 12 also shows that retinaNet is more robust to streak artifacts than the object detection models derived from segmentation models. Figure 11 shows the streak artifacts across a band of image slices caused by a simulated cylindrical spinal metal implant in a case from LUNA 16, as well as how this impacts the models’ output.

Figure 11: A coronal view of the streak artifacts produced by a simulated 7.5 mm radius cylindrical implant which extends 15mm above and below the annotated nodules, as well as the model outputs after this augmentation on a case from LUNA 16. A higher mean Average precision (mAP) means the output object detection boxes (orange) more accurately reflect the annotations (blue). A higher Dice score (DSC) means the output segmentation maps (orange) have a greater overlap** volume (green) with the annotation maps (blue).

Figure 12: Top: The Dice (DSC) score results against simulated implant radius for the tested segmentation models. Bottom: mean Average precision (mAP) results against simulated implant radius for the tested object detection models, including segmentation models converted to generate bounding boxes. These are shown for the 30 test cases from the LUNA 16 dataset, with a cylindrical spinal implant added which induces a simulated streak artifact. The means with standard deviation error bars are shown in purple. The models degrade at a similar rate as the implant radius is increased, but retinaNet is more robust than the converted detection models.

5.6 Motion augmentation

The ‘Motion mag’ column of Tables LABEL:tab_dce and LABEL:tab_map show that all of the models tested were robust to rotational motion augmentation when the mean metric based degradation were considered. The models’ performance do not significantly degrade as the magnitude of the rotational motion is increased beyond 10 degrees - see Figure 13. However, the standard deviation based degradation (sDDeg and sPDeg) shows a significant effect for the LUNA 16 test dataset, as the rotational discontinuity 5mm from the lowest nodule caused both increases and decreases in the segmentation DSC.

Figure 13: Top: The Dice (DSC) score results against the amount of rotational motion for the tested segmentation models. Bottom: mean Average precision (mAP) results against the amount of rotational motion for the tested object detection models, including segmentation models converted to generate bounding boxes. These are shown for the 30 test cases from the LUNA 16 dataset, with the discontinuity resulting from the single tilt motion located 5mm inferior to the lowest nodule containing slice according to the annotation. The means with standard deviation error bars are shown in purple. The models did not significantly degrade as the rotation increased beyond 10 degrees.

The reason for this is shown by the plots in Figure 14. When the discontinuity is shifted very close to the annotated object, the discontinuity sometimes improves the segmentation or detection result as the discontinuity matches the annotated object boundary in the axial direction. This effect is also seen for object detection models.

Figure 14: Top: The Dice (DSC) score results against proximity of the motion discontinuity from the lowest point containing a nodule, for the tested segmentation models. Bottom: mean Average Precision (mAP) results against proximity of the motion discontinuity from the lowest point containing a nodule, for the tested object detection models including segmentation models converted to generate bounding boxes. These are shown for each of the 30 test cases from the LUNA 16 dataset with 10 degrees of rotational motion causing a discontinuity at varying proximity to the lower edge of the lowest nodule according to the annotations. The means with standard deviation error bars are shown in purple. The models’ performance degrade and then improve as the discontinuity approaches the annotation.

An example of this effect is shown in Figure 15, where the model outputs improve when the discontinuity is adjacent to the nodule. This suggests a random motion discontinuity location should be chosen for robustness testing, rather than one based on annotation, to avoid giving the tested models implicit information which they would not have in real world use.

Figure 15: Discontinuity-nodule 10mm (top row): A coronal view of the image discontinuity produced by a 10 degree patient rotation 10mm below the annotated nodule, as well as the model outputs after this augmentation on a case from LUNA 16. Discontinuity-nodule 0mm (bottom row): The corresponding outcome with the discontinuity immediately below the annotated nodule, showing the 3D-UNet and RetinaNet performance improving due to the sharp boundary coinciding with the bottom of the annotated nodule.

6 Discussion and conclusion

In this work, we present a framework combining augmentations and evaluation metrics for evaluating the robustness of segmentation and object detection models with OOD cases produced by simulated protocol changes and artifacts in CT.

The experiments we performed to demonstrate this framework show that modern models are susceptible to augmentations modeling distribution shifts due to increased CT noise, streaking artifacts caused by high-attenuation implants, and patient movement part way through the scan. More specifically, we show different architectures have differing levels of robustness to these OOD phenomena. We find that nnUnet had higher robustness to increased CT noise and streaking artifacts compared to 3D-Unet, when segmenting the liver. A further experiment, introducing the same suite of training augmentations used in nnUnet to 3D-Unet, shows nnUnet’s superior robustness was partially due to the fixed augmentations and partially due to the use of an ensemble of Unets in its training protocol. Furthermore, we find that adding intensity windowing to the pre-processing, as is common for CT image analysis, improves the segmentation performance but reduces the robustness to increased CT noise or metal implant. This is because the windowing removes image detail in the liver and surrounding tissue, while maintaining an intensity boundary between them, making the segmentation task easier for in-distribution images. However, this also means the trained 3D-Unet is less able to handle the intensity variation caused by increased CT noise or streak artifacts, so performance for those OOD test images is worse than if windowing is not used. Experiments assessing the robustness of an object detection architecture, retinaNet, when applied to nodule detection, shows that it is highly robust to increased CT noise. When the 3D-Unet and nnUnet segmentation models are applied, or converted to object detection models, they perform worse than retinaNet without augmentation and also show worse robustness to noise. All models tested show significant weakness to the streaking artifacts caused by metal implants.

We find that the mean model performance decreases with increasing severity level of the noise and metal artifact augmentations, but the variance in the performance for individual cases increases, reflected by positive standard-deviation degradation scores. This has been seen in other research for augmentations used in robustness testing models for MRI segmentation (Boone et al., 2023). However, we see some effects which are specific to CT artifact augmentations. Abrupt boundaries between adjacent image slices introduced by the augmentations can improve the segmentation or object detection accuracy in some cases while decreasing the average performance across the dataset, thus greatly increasing the variance. This is seen when a motion discontinuity is simulated close to the boundary of the annotated organ or nodule - see Figure 15. This effect is specific to simulating the CT modality because sinograms are acquired slice by slice so the profile of artifacts may spread across an axial plane while abruptly changing from one plane to the next. This contrasts an MRI protocol which reconstructs a 3D image after sampling a 3D trajectory through k-space, resulting in artifacts which propagate in all directions (Nárai et al., 2022).

In order to empirically scale the weights for our degradation calculations we measured the distribution of CT Noise in the datasets, but a limitation is that we did not do this for the other augmentations. For CT noise, this was achieved by extracting the noise and measuring its standard deviation, which corresponds to how the severity of the noise artifact is quantified. However, for other types of augmentation, this empirical modeling is difficult given the challenges in translating observed image variances to simulation based transforms. For example, streaking artifacts are hard to retrospectively quantify in the image due to its very heterogeneous spacial profile. So instead the severity of the augmentation is defined by the metal implant size, a parameter of the simulation used to generate the artifact. Further studies quantifying the characteristics of distribution shifts of artifacts in CT may enable more specific definition of model robustness, based on degradation weights fitted to the distribution associated with a specific imaging center.

A limitation of the CT simulation emerges for some of the test CT images, which had anatomy cropped by the circular field of view. As the area beyond the field of view is assumed to be air, this would have reduced the apparent attenuation of the X-ray beams at some angles in the simulation compared to a real like CT scan, which may have altered the noise distribution. This could be mitigated in future research by registering a non-cropped atlas CT image to these cases, allowing an approximation of the attenuation beyond the field of view.

A limitation of our motion augmentation method is that it only considers a single rigid movement due to patient changing position, which occurs between the acquisition of individual slices. However, fast motion during the acquisition a sinogram and the impact of complicated artifacts which follow are not considered. In lung imaging, deformable motion due to breathing or the heartbeat during the acquisition of a slice sinogram can cause artifacts which appear as the duplication or distortion of blood vessels and nodules (van der Ham et al., 2022), which could affect a model more than a rigid discontinuity. In one study radiologists identified artifacts of this type severe enough to affect diagnostic interpretation in 30% of CT scans acquired from outpatients (Dasegowda et al., 2023). An augmentation for fast motion artifacts could be added in future work, by applying a deformable motion transform to the image before a subset of the projection angles are sampled during the simulation (van der Ham et al., 2022). However, determining a corresponding transform for the annotated segmentation maps of object box co-ordinates would be a problem for robustness testing.

Our presented framework enables direct comparison of black-box models for the same segmentation or object detection task, so a potential user with a limited annotated local dataset can test and compare the robustness of available models against potential CT artifacts or image quality changes which could occur during its use life at their site. The augmentations could also be used to generate a larger benchmarking dataset for the community to study architectural and model-design considerations that result in improved robustness. We recommend that the parameters which determine which slices are affected by simulated CT artifacts, i.e. the timing of motion, the axial position of an implant, or the implant’s extension in axial direction, are randomly varied rather than determined by the annotation, to avoid sharp discontinuities between the slices influencing the tested models with implied information about those annotations.

Acknowledgments

This research was funded by Innovate UK, grant 10033899. It was further supported by the Wellcome/EPSRC Centre for Medical Engineering [WT 203148/Z/16/Z].

Ethical Standards

This work was based on publicly available datasets, see below.

Conflicts of Interest

Julia A. Schnabel: Founder of XRnostics Ltd.
Kanwal K. Bhatia: Founder of Metalynx Ltd. trading as Aival.
Jack Highton, Quok Zong Chong, Samuel Finestone, and Arian Beqiri: employees of Metalynx Ltd. trading as Aival.

Data availability

This work was based on publicly available datasets. Luna 16: https://luna16.grand-challenge.org/ Segmentation Decathlon: http://medicaldecathlon.com/

A

Figure 16: Top: The Dice (DSC) score results against the level of CT noise added by augmentation, for the tested segmentation models. Bottom: mean Average precision (mAP) results against the level of CT noise added by augmentation, for the tested object detection models including segmentation models converted to generate bounding boxes. These are shown for each of the 30 test cases from the LUNA16 dataset with different levels of CT noise added by augmentation. The corresponding means across the test cases is shown in purple, with the error bars representing the standard deviation. The results in this plot are summarized in Tables LABEL:tab_dce and LABEL:tab_map.

Figure 17: The Dice (DSC) score results for the tested segmentation models, shown for each of the 30 test cases from the liver segmentation dataset, with simulated CT streak artifacts of varying severity caused by augmentation by insertion of a cylindrical spinal implant with varying radius. The corresponding means across the test cases is shown in purple, with the error bars representing the standard deviation. The results in this plot are summarized in Tables LABEL:tab_dce.

Figure 18: The dice (DSC) score produced by each tested model for each of the 30 test cases in the liver segmentation dataset, with simulated sudden patient motion by 10 degrees such that the resulting discontinuity at varying proximity to the liver ground truth segmentation. The corresponding means of the dice score across the test cases is shown in purple, with the error bars representing the standard deviation. The results in this plot are summarized in Tables LABEL:tab_dce.