Embarrassingly Simple Scribble Supervision for 3D Medical Segmentation

Abstract

Traditionally, segmentation algorithms require dense annotations for training, demanding significant annotation efforts, particularly within the 3D medical imaging field. Scribble-supervised learning emerges as a possible solution to this challenge, promising a reduction in annotation efforts when creating large-scale datasets.
Recently, a plethora of methods for optimized learning from scribbles have been proposed, but have so far failed to position scribble annotation as a beneficial alternative. We relate this shortcoming to two major issues: 1) the complex nature of many methods which deeply ties them to the underlying segmentation model, thus preventing a migration to more powerful state-of-the-art models as the field progresses and 2) the lack of a systematic evaluation to validate consistent performance across the broader medical domain, resulting in a lack of trust when applying these methods to new segmentation problems.
To address these issues, we propose a comprehensive scribble supervision benchmark consisting of seven datasets covering a diverse set of anatomies and pathologies imaged with varying modalities. We furthermore propose the systematic use of partial losses, i.e. losses that are only computed on annotated voxels. Contrary to most existing methods, these losses can be seamlessly integrated into state-of-the-art segmentation methods, enabling them to learn from scribble annotations while preserving their original loss formulations. Our evaluation using nnU-Net reveals that while most existing methods suffer from a lack of generalization, the proposed approach consistently delivers state-of-the-art performance. Thanks to its simplicity, our approach presents an embarrassingly simple yet effective solution to the challenges of scribble supervision. Source code as well as our extensive scribble benchmarking suite will be made publicly available upon publication.

1 Introduction

In 3D medical image segmentation, learning from densely annotated images is the predominant approach. Generating annotations is costly and time-consuming, causing this process to be the major bottleneck when creating large-scale datasets. An emerging solution to this challenge lies in approaches able to learn from sparsely annotated data, particularly in the form of scribbles (see Figure LABEL:fig:figure1:A). Given a fixed annotation budget, scribble supervised methods allow the annotation of more images, enabling the training of more robust and potentially more accurate models in a domain where datasets would otherwise be too small to capture the naturally occurring variations.

Figure 1: (A): Depiction of the annotation effort reducing benefits of scribble-supervised learning. (B): Partial losses only consider annotated voxels during the loss computation. This idea, initially introduced for the Cross-Entropy loss, can be extended to other popular segmentation losses like the Dice loss to enable state-of-the-art segmentation methods to learn from scribble annotations while preserving their original loss formulations. (C): Drawbacks of current systemic and lightweight scribble supervision methods compared to the use of partial losses.

In light of this promise, an extensive range of scribble-supervised methods have recently been proposed [8, 13, 5, 2, 23, 15, 22, 16, 7, 21, 12, 11, 6, 20, 17]. These can be categorized into systemic and lightweight, depending on the strategy used. Systemic methods are commonly designed with a specific task in mind such as cardiac segmentation and modify integral parts of the training pipeline, tightly binding them to the base method they are built on. Typical strategies of systemic methods include custom augmentations [22, 23, 15], intricate regularized loss functions [8, 13, 22, 23, 7, 12, 11, 6], pseudo-label learning [13, 16, 7, 12, 11, 6] and novel segmentation model architectures [8, 13, 5, 16, 2, 7, 21, 11, 6]. In contrast, lightweight methods typically revolve around extending the loss formulation or adding regularization terms in order to make better use of the provided scribble labels. At their core, these methods utilize the partial Cross-Entropy (pCE) to consider only annotated voxels for the loss computation. Due to their detachment from the segmentation architecture, they can easily be transferred to other segmentation methods. In principle, lightweight methods have the potential to generalize well across tasks. However, so far, evaluation in the respective publications was highly limited covering either only one dataset in the medical domain [17] or only considering natural images [20, 19].
Although scribble supervision is an active field of research, the proposed methods have so far failed to position scribble-supervised learning as a beneficial alternative to densely supervised learning. We relate this shortcoming to two hurdles that the current literature must overcome: With segmentation methods being a highly active area of research, scribble learning functionality has to be readily integrated into new state-of-the-art methods as they are released. This requirement is in contrast to the most prominent scribble learning approaches today, which require a deep systemic integration into the segmentation pipeline. Additionally, scribble supervision methods need to deliver consistent performance, across tasks and modalities, without the need to optimize a method for each new problem it is applied to. Yet, a persistent issue accompanying recent developments is the lack of a comprehensive evaluation of the proposed methods to validate claims of general functionality. To this date, scribble supervision methods are typically exclusively evaluated within a single specific task such as cardiac segmentation, omitting a broader validation across the medical domain.
We propose solutions to both of these shortcomings. In a first step, we introduce a large-scale benchmark for systematically evaluating recently proposed methods across tasks and modalities, revealing that lightweight methods have superior generalization relative to systemic methods (see Figure LABEL:fig:figure1:C). We take this as inspiration to revisit a core component of these methods: the pCE loss, which on its own already achieves competitive results. However, state-of-the-art segmentation methods rely on more than just the Cross-Entropy loss to maximize segmentation performance. We propose to generalize the idea behind pCE to arbitrary loss functions, enabling them to make use of sparsely annotated labels by ignoring all non-annotated voxels in the loss computation. By adapting state-of-the-art segmentation methods to use such partial losses, we unlock the ability to learn from sparsely annotated data without interfering with their loss composition. Using nnU-Net [9] as an example, we show how the partial loss can enable state-of-the-art segmentation methods to learn from scribble annotations, yielding improved segmentation performance relative to the competition (see Figure LABEL:fig:figure1:B). Moreover, we show that the partial loss is annotation agnostic and can be applied to arbitrary sparse annotations. We make our benchmark as well as our implementation of partial losses in nnU-Net publicly available upon acceptance of this manuscript.

2 Materials & Methods

2.1 Partial loss

We propose to generalize the simple but effective idea behind the partial Cross-Entropy loss, namely the omission of all non-annotated voxels in the loss computation, to arbitrary segmentation losses. We refer to this concept as partial loss, highlighting that the loss computation is constrained to the masked set of annotated voxels. As a lightweight extension, this idea can be seamlessly integrated into existing segmentation losses, enabling us to modify state-of-the-art segmentation methods such that they can be used to learn from scribble annotations.
Let $Y=\{Y_{i}\}_{i=1}^{N}\in\mathcal{C}\cup\xi$ be a partially annotated reference segmentation for a given set of classes $\mathcal{C}$ with non-annotated voxels being assigned to the additional ignore class $\xi$. The filtering function $A(x,Y)$ returns all values of an indexable $x$ for the corresponding subset of indices $\mathcal{S}\subseteq\{1,...,N\}$ for which $Y_{i}\in\mathcal{C}\forall i\in\mathcal{S}$. The segmentation model returns softmax outputs defining $p(\hat{Y})$. Given a loss (L), the partial loss (pL) for a single image is defined as follows:

$$\mathrm{pL}(p(\hat{Y}),Y)=L(A(p(\hat{Y}),Y),A(Y,Y))$$ (1)

We intend to couple the partial loss with the nnU-Net [9], which utilizes a combination of Cross-Entropy and Dice as a loss function to achieve state-of-the-art performance. Modifying the loss in nnU-Net yields the following equation:

$$\mathrm{pL}(p(\hat{Y}),Y)=\underbrace{\dfrac{1}{|\mathcal{C}|}\sum_{c\in\mathcal{C}}\left(1-\dfrac{2\cdot\sum_{i\in\mathcal{S}}\vmathbb{1}_{Y_{i}=c}p(\hat{Y}_{i}=Y_{i})}{\sum_{i\in\mathcal{S}}(p(\hat{Y}_{i}=c)+\vmathbb{1}_{Y_{i}=c})}\right)}_{\mathrm{pDICE}(p(\hat{Y}),Y)}\underbrace{-\sum_{i\in\mathcal{S}}\log(p(\hat{Y}_{i}=Y_{i}))}_{\mathrm{pCE}(p(\hat{Y}),Y)}$$

2.2 Benchmark Creation

The lack of a comprehensive scribble supervision benchmark has so far hindered the development of robust scribble supervision methods as well as systematic comparisons between them. Previous approaches almost exclusively relied on the ACDC [3, 21] and MSCMR [24, 25] datasets, both representing the rather narrow task of cardiac segmentation in cine MRI.
To address this, we have created a new benchmark for medical 3D scribble supervision, encompassing seven datasets of diverse segmentation tasks: LiTS [4], BraTS2020 [18], AMOS2022 [10], KiTS2023 [1], WORD [17], MSCMR [24, 25], and ACDC [3, 21]. We automate scribble generation for these datasets using the existing dense reference segmentations. For each image slice, we generate two scribbles of different types for each class: interior scribbles mimicking human annotations placed randomly within the class interior as non-uniform rational B-Splines (NURBS), and border scribbles roughly delineating a small part of the class border with random slight offsets. This dual approach follows best practices for scribble generation and ensures that both class interiors and boundaries are represented during training. We refer to Figure 2 for a visual representation of our automatically generated scribbles. Through the introduction of this benchmark, we aim to catalyze the development and comparison of new medical 3D scribble supervision methods.

Figure 2: Examples of segmentation types with (A) depicting a dense segmentation and (B) a scribble segmentation generated by our automatic scribble generation method based on the dense segmentation.

3 Results & Discussion

3.0.1 The proposed benchmark covers the diversity of medical segmentation tasks

Our benchmark encompasses seven diverse datasets reflecting a variety of anatomical and pathological structures imaged with several different modalities (CT, MRI and cine MRI). It enables an unprecedented systematic evaluation of performance and generalization capabilities of the different methods. On average, we generate 6 scribbles per slice, resulting in only 0.2% of image voxels being annotated. When respecting the class memberships of the voxels, we find that on average 8.5% of each class is annotated. The higher number relative to the naive image voxels stems from the oversampling caused by generating two scribbles per class per slice, see Section 2.2. In contrast, expert scribbles on ACDC, MSCMR, and WORD annotate 0.5% of image voxels and on average 12.3% per class. Considering scribbles disproportionately cut down on annotation time, particularly due to relaxed precision requirements at object boundaries, these numbers powerfully underline the potential of scribbles to reduce annotation costs. Detailed descriptive statistics of our benchmark are shown in Appendix 0.A and a qualitative comparison between expert and generated scribbles is depicted in Appendix 0.B.

3.0.2 Systematic benchmarking reveals generalization patterns of existing scribble supervision methods

We use the proposed benchmark to perform a systematic evaluation of existing scribble learning methods. Expert scribbles, if available (such as for the datasets ACDC, MSCMR, WORD), are included as a separate category. Existing methods included in our comparison encompass the lightweight methods DenseCRF [20] and WORD [17], which we couple with the nnU-Net as their segmentation backbone, and the systemic methods CycleMix [22], ShapePU [23] and ScribFormer [14]. Our results are summarized in Table 1. The systemic methods ShapePU, CycleMix, and ScribFormer show excellent performance on their original evaluation datasets ACDC and MSCMR. However, when tested across the entire benchmark, they fail to generalize to new tasks and therefore have the lowest performance by a substantial margin. Among the systemic methods, CycleMix is the most consistent performer with an average Dice score of 0.598. Lightweight methods show competitive generalization capabilities across all datasets and while unable to keep up with purpose-built systemic methods on cardiac datasets, their overall performance is more consistent. WORD and DenseCRF perform similarly well across all tasks and achieve average Dice scores of 0.736 and 0.758, respectively.

Table 1: Evaluation of scribble supervision methods using the Dice score. We use our systematic benchmark to compare the proposed use of partial losses to an extensive set of baselines and conduct comprehensive ablation studies. If available, we report the performance on the manually provided expert labels as well.

Method	Experts			Benchmark (Generated)
	ACDC	MSCMR	WORD	ACDC	MSCMR	WORD	LiTS	BraTS	AMOS	KiTS	Mean
ShapePU	0.843	0.837	0.226	0.765	0.530	0.436	0.210	0.060	0.322	0.233	0.365
ScribFormer	0.884	0.845	0.180	0.900	0.861	0.420	0.457	0.071	0.283	0.268	0.466
CycleMix	0.876	0.872	0.406	0.896	0.852	0.626	0.554	0.250	0.487	0.518	0.598
nnUNet+DenseCRF	0.734	0.724	0.580	0.890	0.832	0.830	0.685	0.370	0.788	0.759	0.736
nnUNet+WORD	0.519	0.652	0.729	0.728	0.698	0.788	0.728	0.725	0.836	0.806	0.758
nnUNet+pCE (2D)	0.623	0.756	0.549	0.848	0.784	0.828	0.684	0.324	0.795	0.769	0.719
nnUNet+pL (2D)	0.653	0.700	0.695	0.893	0.864	0.831	0.691	0.442	0.790	0.740	0.750
nnUNet+pCE	0.841	0.893	0.693	0.888	0.887	0.787	0.777	0.418	0.838	0.812	0.772
nnUNet+ResUNet+pL	0.837	0.880	0.763	0.895	0.889	0.856	0.750	0.518	0.842	0.834	0.798
nnUNet+pL	0.868	0.858	0.723	0.895	0.885	0.837	0.761	0.680	0.840	0.823	0.817
nnUNet (dense superv.)	0.926	0.906	0.862	0.926	0.906	0.862	0.786	0.827	0.860	0.846	0.858

3.0.3 pCE alone delivers competitive scribble supervision

The standard pCE baseline in conjunction with nnU-Net uncovers a noteworthy finding: pCE alone markedly surpasses all existing systemic and lightweight methods, delivering an average Dice score of 0.772. These results underscore the critical role of employing state-of-the-art segmentation models, establishing pCE as a robust baseline for future methodologies. WORD and DenseCRF use the pCE as a basis and propose additional regularization terms to further improve results. Interestingly, comparing the nnUNet+WORD with nnU-Net+pCE in isolation reveals a reduction in performance, casting doubt on the methodological efficacy of the WORD approach. Conversely, the DenseCRF method (which is exclusively implemented in 2D) improves over the 2D nnUNet+pCE baseline, validating its methodological innovations.

3.0.4 Partial losses achieve state-of-the-art in learning from scribbles

Modern state-of-the-art segmentation methods require more than just a Cross-Entropy loss to achieve maximum performance. We evaluate the integration of partial losses in nnU-Net, yielding the implementation of a partial Dice loss that is used in conjunction with pCE (see Section 2.1). Experiments on our comprehensive benchmark reveal that this approach not only outperforms a pCE-only baseline by a margin of 0.045, but simultaneously eclipses all systemic and lightweight methodologies. Notably, the use of partial losses showcases robust generalization capabilities across the entire benchmark, except for BraTS, a common challenge for all methods (see Section 3.0.5), and maintains consistent performance across scribble styles (see Appendix 0.B for a qualitative comparison of styles). When examining the results of partial losses in nnU-Net qualitatively, we also find that this approach achieves the most consistent class boundaries when compared against systemic and lightweight methods as shown in Figure 3. These outcomes robustly affirm the general utility of partial losses when integrated into state-of-the-art segmentation methods. We exemplary demonstrate the portability of the partial losses by applying them to a different state-of-the-art architecture, a residual U-Net (“nnUNet+ResUNet+pL”) where they maintain their superior performance against all other evaluated baselines.

Figure 3: Qualitative comparison of the top-performing systemic method, CycleMix, the top-performing lightweight method, WORD coupled with the nnU-Net, and partial losses in nnU-Net. A green image outline represents a high quality prediction while a red outline represents a critical prediction failure.

3.0.5 Partial losses can learn from arbitrary sparse annotation strategies

The versatility of partial losses extends beyond scribble annotations, accommodating any form of sparse annotation due to its lack of scribble-specific components. We perform an exploratory study examining a different sparse annotation strategy: selecting a subset of all available slices and annotating those in a traditional dense manner while leaving the remaining slices unannotated (Slices). Slices are selected until the same annotation budget (measured as % annotated, averaged over all classes) as was used for the scribbles is attained. Additionally, with the identical annotation budget, we chose a random subset of densely annotated images until the budget was satisfied, simulating a scenario where fewer images can be annotated than would be possible with sparse annotation strategies (Dense). We compare annotation efficiency using the proposed partial losses in nnU-Net against scribble annotations using the same annotation budget (Scribbles). Our study’s findings, presented in 2, illustrate that in the context of a fixed annotation budget, dense annotations underperform compared to sparse annotation strategies. A notable exception is the BraTS dataset, where Dense achieves a significantly higher Dice score. This discrepancy may be attributed to the appearance of the structures present in BraTS, characterized by classes with fuzzy borders, which makes them potentially less suited for sparse annotation methods. Among the sparse strategies, scribbles outperform slices, further highlighting the need to train methods from as diverse inputs as possible to ensure robustness.

Table 2: Comparison of annotation strategies under fixed annotation budget using the Dice score. Dense annotation forces a lower number of annotated images while sparse annotation strategies can cover all images in the respective datasets. Experiments are performed on our benchmark using partial losses in nnU-Net.

Method	ACDC	MSCMR	WORD	LiTS	BraTS	AMOS	KiTS	Mean
Dense (fewer images)	0.891	0.618	0.848	0.666	0.795	0.838	0.631	0.755
Slices	0.911	0.747	0.791	0.741	0.568	0.843	0.728	0.761
Scribbles	0.895	0.885	0.837	0.761	0.680	0.840	0.823	0.817

4 Conclusion

In this manuscript we introduced an embarrassingly simple scribble supervision method. Partial losses, a generalization of the idea behind the partial Cross-Entropy to learn only from an annotated subset of voxels, enable arbitrary state-of-the-art segmentation methods to learn from scribbles while preserving their original loss formulations. Our exemplary integration into nnU-Net yields the best results across all tested segmentation tasks, demonstrating better generalization capabilities than other lightweight methods and even matching highly specialized systemic methods on their respective tasks. We furthermore introduce a systematic benchmark for evaluating scribble supervision methods spanning a diverse range of segmentation tasks, datasets and modalities. The benchmark and our reference implementation of partial losses in nnU-Net will be made publicly available upon publication in the hope that it will be used as a starting point for catalyzing future progress in the field.

Appendix 0.A Scribble Evaluation

Table 3: Scribble evaluation results quantifying annotated scribbles with the following metrics: Scribble2Img (%): Percentage of scribble voxels in relation to image voxels averaged over all images; Scribble2Ref (%): Percentage of scribble voxels of a class in relation to the full reference segmentation voxels of a class averaged over all classes and images; Scribbles: Number of all scribbles in an image averaged over all images; ScribblesFG: Number of foreground scribbles in an image averaged over all images; ScribblesSlice: Number of all scribbles per slice averaged over all slices of all images.

Dataset		Scribble2Img (%)	Scribble2Ref (%)	Scribbles	ScribblesFG	ScribblesSlice
Exp.	ACDC	0.271	8.741	45	35	5
	MSCMR	1.061	21.925	118	66	8
	WORD	0.033	7.380	150	95	10
	Mean	0.455	12.682	104	65	8
Benchmark	ACDC	0.488	10.401	72	53	7
	MSCMR	0.241	6.140	113	82	7
	WORD	0.052	11.667	1589	1185	8
	LiTS	0.237	3.213	1318	411	3
	BraTS	0.384	9.788	605	295	4
	AMOS	0.053	12.722	1243	959	9
	KiTS	0.168	5.422	574	194	3
	Mean	0.232	8.479	788	454	6

Appendix 0.B Scribble Styles

Figure 4: Comparison of different scribble annotation styles by experts on the ACDC, MSCMR, and WORD datasets and our generated benchmark scribbles.