EFCNet: Every Feature Counts for Small Medical Object Segmentation

Abstract

This paper explores the segmentation of very small medical objects with significant clinical value. While Convolutional Neural Networks (CNNs), particularly UNet-like models, and recent Transformers have shown substantial progress in image segmentation, our empirical findings reveal their poor performance in segmenting the small medical objects and lesions concerned in this paper. This limitation may be attributed to information loss during their encoding and decoding process. In response to this challenge, we propose a novel model named EFCNet for small object segmentation in medical images. Our model incorporates two modules: the Cross-Stage Axial Attention Module (CSAA) and the Multi-Precision Supervision Module (MPS). These modules address information loss during encoding and decoding procedures, respectively. Specifically, CSAA integrates features from all stages of the encoder to adaptively learn suitable information needed in different decoding stages, thereby reducing information loss in the encoder. On the other hand, MPS introduces a novel multi-precision supervision mechanism to the decoder. This mechanism prioritizes attention to low-resolution features in the initial stages of the decoder, mitigating information loss caused by subsequent convolution and sampling processes and enhancing the model’s global perception. We evaluate our model on two benchmark medical image datasets. The results demonstrate that EFCNet significantly outperforms previous segmentation methods designed for both medical and normal images.

1 Introduction

Small medical objects, like HyperReflective Dots (HRDs) observed on Optical Coherence Tomography (OCT), are frequently encountered in disease research. Various studies [26, 15, 1, 9] have confirmed the significant relevance of these small lesions to medical diagnosis and treatment. However, the manual labeling of these small objects in medical images is a time-consuming and labor-intensive task, representing a substantial drain on medical resources. Consequently, there is a pressing need to automate the segmentation of small medical objects using computer vision algorithms. Considering the type of task, one would easily recall the numerous image segmentation models such as U-Net [27], ResUNet [39], DenseUNet [11], ResUNet++ [18], TransFuse [42] and Swin-Unet [3], as well as the recent SAM [20]. Given their generally desirable performance, it is straightforward to ask: Can these method solve the small medical objects segmentation?

Figure 1: Samples of small medical objects in two datasets (S-HRD and S-Polyp) in our work.

Figure 2: Comparison of our method (d) against conventional encoder-decoder based method in previous works (a)-(c). (a)&(b) In traditional methods [27, 39], the single-stage features of the encoder and the corresponding single-stage features of the decoder are fused by concatenation or addition, with one segmentation head at the end of the decoder. (c) Some methods [24, 41] attempt to add attention mechanisms to the encoder, which however are limited to single-stage features. And only one segmentation head is adopted at the end of the decoder. (d) Our method aggregates all features in each stages of the encoder through CSAA to guide the decoding procedure. Multi-resolution features in each stages of the decoder are segmented with multi-precision by multiple segmentation heads through MPS.

Sadly, these famous methods are generally wiped out in this specific task. Moreover, research specifically dedicated to the segmentation of small medical objects is lacking. Identified issues with previous segmentation methods reveal significant information loss, as highlighted in two points: 1) Earlier methods [27, 19] often use features from the preceding stage in the decoder and the corresponding stage in the encoder, limiting the direct use of features for each decoding stage. This contradicts previous studies [25, 8] indicating improved segmentation accuracy with both shallow and deep features and results in untapped information from various encoder stages. 2) Many prior approaches [39, 19, 14] employ only one segmentation head for supervision at the last decoder stage. In contrast, studies [43, 6, 7] highlight the strong global perception of low-resolution features in early decoding stages, valuable for small object localization. However, information in these early decoder stages is somewhat lost during convolution and upsampling processes. Additionally, small medical objects, compared to standard-sized objects, carry less information, intensifying the impact of information loss on segmentation accuracy. Some samples of small medical objects are illustrated in Fig. 1. Notably, the SAM model [20] performs inadequately in addressing this segmentation task.

To address the challenge of information loss during segmentation, we introduce a novel solution based on the encoder-decoder structure. Our model meticulously attends to all features in each stage of both the encoder and decoder, enhancing the accuracy of small medical object segmentation. Figure 2 illustrates the distinctions between our approach and prior methods. Specifically, for the encoder, we present the Cross-Stage Axial Attention Module (CSAA), leveraging the attention mechanism to integrate features from all stages. This adaptation enables the model to dynamically learn information necessary for each decoding stage. CSAA facilitates direct reference to all valuable information in the encoder during the decoding process of each stage, mitigating information loss in the encoder. Simultaneously, we introduce the Multi-Precision Supervision Module (MPS) for the decoder. This module adds segmentation heads with varying precision for supervision after each stage in the decoder. Low-precision segmentation heads focus on low-resolution features, temporarily overlooking local details to leverage their robust global perception. With MPS, the model effectively exploits information from each stage in the decoder, reducing information loss in this part of the model.

To validate the efficacy of the proposed method, we carry out comprehensive experiments on two datasets S-HRD and S-polyp, as illustrated in Fig. 1. These datasets comprise a fundus Optical Coherence Tomography (OCT) image dataset created by our team and a subset from CVC-ClinicDB [2]. Experimental results on these two datasets demonstrate that our method outperforms previous state-of-the-art models in terms of dice similarity coefficient (DSC) and intersection over union (IoU).

Our contributions are outlined as follows:
1. Innovative Segmentation Approach: We introduce a novel concept to address the challenge of small medical object segmentation. Emphasizing the significance of every feature in medical images, our model meticulously attends to all features at each stage. This approach enables the extraction of diverse information, thereby mitigating information loss associated with small medical objects.
2. Proposed Modules for Enhanced Accuracy: We devise two key modules, namely the Cross-Stage Axial Attention Module (CSAA) and the Multi-Precision Supervision Module (MPS). These modules effectively tackle information loss in the encoder and decoder, respectively, resulting in an improved segmentation accuracy for the model.
3. Benchmark Construction and Model Validation: We establish a new benchmark for evaluating small medical object segmentation. Through experiments on two datasets focusing on small medical objects, our model significantly outperforms previous state-of-the-art models. This demonstrates the robustness and superiority of our proposed approach in this challenging domain.

2 Related Works

Medical Image Segmentation. Recently, researchers have introduced several innovative methods [5, 23, 10, 12, 33] for semantic segmentation in medical images. Zhang et al. [42] proposed a unique hybrid structure that concurrently integrates CNN and Transformer, leading to a reduction in the loss of low-level details. Chen et al. [4] incorporated a Transformer module into the U-Net encoder, enhancing the model’s ability for long-range modeling. Wang et al. [36] addressed overfitting by employing a Transformer encoder and introduced the progressive locality decoder to improve local information processing in medical images. While these methods have significantly contributed to medical image segmentation, they often fall short in accounting for the impact of object size on segmentation results. Particularly, these models tend to underperform when confronted with the segmentation of small objects. Lou et al. [21] recognized the significance of considering the size of medical objects in the segmentation process, and introduced a Context Axial Reverse Attention module (CaraNet) to assist the model in detecting local information related to small medical objects. However, the use of bilinear interpolation in the decode stage of CaraNet leads to substantial information loss, significantly affecting the segmentation of small medical objects. In contrast to the aforementioned methods, our model addresses the issue of information loss in small medical objects through CSAA and MPS, effectively improving segmentation accuracy.

General Segmentation Model. Recently, Kirillov et al. [20] introduced the Segment Anything Model (SAM), a versatile segmentation model that has made significant strides in the realm of natural image segmentation. Despite its success in general applications, SAM proves unsuitable for many medical image segmentation tasks due to the intricate structures and complex boundaries present, particularly in cases involving small medical objects, without manual guidance [17, 13]. In response to these limitations, Ma et al. [22] devised MedSAM as an adaptation of SAM tailored specifically for medical image segmentation. MedSAM exhibits notable advancements in handling medical image segmentation tasks compared to SAM [20]. Nevertheless, even with these improvements, MedSAM [22] still struggles when tasked with the segmentation of small medical objects concerned in this paper.

Attention Mechanism. Numerous attention-based methods [24, 28, 16, 29, 32] have emerged in recent years, applied to diverse tasks in computer vision and natural language processing. Vaswani et al. [34] broke away from the conventional convolutional structure and introduced the Transformer, a novel architecture utilizing attention mechanisms. Woo et al. [38] enhanced convolutional neural networks by incorporating both Channel Attention and Spatial Attention. Zhang et al. [40] proposed the Pyramid Squeeze Attention Module (PSA) to enable the model to capture spatial information across different channels. Building upon the insights gained from the aforementioned attention-based methods, we introduce the Cross-Stage Axial Attention Module (CSAA). This module facilitates feature fusion and minimizes information loss within the model.

3 Method

We introduce small medical object segmentation and related notations. In Sec.3.1, we outline the overall structure of EFCNet. We detail the CSAA in Sec. 3.2 and the MPS in Sec. 3.3. Lastly, we discuss our loss function in Sec.3.4.

Problem Setup and Notations. In our segmentation task, we denote the medical picture dataset as $X=\{x_{1},...,x_{m}|x_{i}\in\mathbb{R}^{C\times H\times W},i=1,2,..,m\}$. Doctors meticulously and manually annotate lesions in each image $x_{i}$, forming the ground truth set $Y=\{y_{1},...,y_{m}|y_{i}\in\{0,1\}^{1\times H\times W},i=1,2,..,m\}$. The complete dataset is denoted as $D=\{X,Y\}$, which is splitted into a training set $D_{\text{train}}=\{X_{\text{train}},Y_{\text{train}}\}$ and a testing set $D_{\text{test}}=\{X_{\text{test}},Y_{\text{test}}\}$.

Our objective is to develop an algorithm that empowers our model to effectively segment small medical objects from $D_{\text{train}}$ and demonstrate robust performance on $D_{\text{test}}$.

3.1 Overall Architecture

Figure 3: Structure of our proposed EFCNet. (a) Overview of our method, featuring an Encoder-Decoder architecture equipped with the Cross-Stage Axial Attention Module (CSAA) and Multi-Precision Supervision Module (MPS). (b) Details of our Cross-Stage Axial Attention Module (CSAA). The CSAA combines features from each stage of the encoder, dynamically extracts information about small medical objects, and directs the decoding process of each stage in the decoder.

We propose a novel method called EFCNet to address the challenge of segmenting small medical objects. Specifically, we have designed two modules, the Cross-Stage Axial Attention Module (CSAA) and the Multi-Precision Supervision Module (MPS), to ensure that the model focuses on small medical object infomration in both the encoder and decoder.

In Fig. 3(a), an image containing small medical objects serves as input to our model. Initially, the image undergoes encoding through $k$ stages, producing feature maps encompassing diverse information about small medical objects. Subsequently, the CSAA Module processes features from all encoder stages, compelling the model to adaptively learn pertinent information for the decoding phase. The decoder then sequentially processes these feature maps under the guidance of the CSAA Module. Lastly, the feature maps from all decoder stages enter the MPS Module to yield multi-precision prediction results, receiving separate supervisions. During testing, we utilize the segmentation output from the last decoder stage as the final result of our model.

3.2 CSAA Module

Information about small medical objects is dispersed across different encoder stages, each containing varied types of data. However, much of this information is not directly usable by the decoder, and some is lost during the convolution and downsampling processes. To reduce information loss in the encoder and fully leverage insights about small medical objects, we present a novel Cross-Stage Axial Attention Module (CSAA). This module adaptively learns from features in all encoder stages and subsequently guides the decoding process. As depicted in Fig. 3(b), CSAA have four steps: resizing, W-dimensional axial attention, H-dimensional axial attention and resizing back.

Resizing. To enhance the fusion of features from all encoder stages, we resize all feature maps in each encoder stage to $(C^{*},H^{*},W^{*})$, adjusting both spatial and channel dimensions through convolution operations:

$$\begin{split}&f_{i}^{*}=\sigma(BN(conv_{i}^{e}(f_{i}^{e}))),i=1,2,..,k,\end{split}$$ (1)

where $f_{i}^{e}$ represents the feature map in stage $i$ of the encoder; $\sigma$ and $BN$ denote ReLU and Batch Normalization respectively; and $k$ is the number of stages in the encoder and decoder. We consider the k-stage resized feature maps $\{f_{i}^{*}\}_{i=1}^{k}$ as the input for the subsequent W-dimensional axial attention step.

W-Dimensional CSAA. We firstly generate Query $\{Q_{i,w}\}_{i=1}^{k}$, Key $\{K_{i,w}\}_{i=1}^{k}$, Value $\{V_{i,w}\}_{i=1}^{k}$ based on the feature maps $\{f_{i}^{*}\}_{i=1}^{k}$ in the width (W) dimension:

$$\begin{split}&Q_{i,w}=W_{i}^{Q}(f_{i,w}^{*}),i=1,2,..,k,\\ &K_{i,w}=W_{i}^{K}(f_{1,w}^{*},f_{2,w}^{*},...,f_{k,w}^{*}),i=1,2,..,k,\\ &V_{i,w}=W_{i}^{V}(f_{1,w}^{*},f_{2,w}^{*},...,f_{k,w}^{*}),i=1,2,..,k,\end{split}$$ (2)

where $\{W_{i}^{Q}\}_{i=1}^{k}$, $\{W_{i}^{K}\}_{i=1}^{k}$, $\{W_{i}^{V}\}_{i=1}^{k}$ represent the weight matrix used to generate $\{Q_{i,w}\}_{i=1}^{k}$, $\{K_{i,w}\}_{i=1}^{k}$, $\{V_{i,w}\}_{i=1}^{k}$ respectively; $k$ is the number of stages in the encoder and decoder; and $f_{i,w}^{*}$ denotes the feature $f_{i}^{*}$ in width dimension. Equation 2 shows that $K_{i,w}$ and $V_{i,w}$ merge the information from all stages of the encoder in width dimension. Next, we get the output $\{f_{i}^{w}\}_{i=1}^{k}$ of W-Dimensional Axial-Attention by

$$\begin{split}f_{i}^{w}=\text{Softmax}(\frac{Q_{i,w}K_{i,w}^{T}}{\sqrt{C^{*}H^{*}}})V_{i,w},i=1,2,..,k.\end{split}$$ (3)

H-Dimensional CSAA. Similarly above, we firstly generate Query $\{Q_{i,h}\}_{i=1}^{k}$, Key $\{K_{i,h}\}_{i=1}^{k}$, Value $\{V_{i,h}\}_{i=1}^{k}$ based on the feature maps $\{f_{i}^{w}\}_{i=1}^{k}$ in the height (H) dimension:

$$\begin{split}&Q_{i,h}=W_{i}^{Q}(f_{i,h}^{w}),i=1,2,..,k,\\ &K_{i,h}=W_{i}^{K}(f_{1,h}^{w},f_{2,h}^{w},...,f_{k,h}^{w}),i=1,2,..,k,\\ &V_{i,h}=W_{i}^{V}(f_{1,h}^{w},f_{2,h}^{w},...,f_{k,h}^{w}),i=1,2,..,k,\end{split}$$ (4)

where $\{W_{i}^{Q}\}_{i=1}^{k}$, $\{W_{i}^{K}\}_{i=1}^{k}$, $\{W_{i}^{V}\}_{i=1}^{k}$ represent the weight matrix used to generate $\{Q_{i,h}\}_{i=1}^{k}$, $\{K_{i,h}\}_{i=1}^{k}$, $\{V_{i,h}\}_{i=1}^{k}$ respectively; $k$ is the number of stages in the encoder and decoder; and $f_{i,h}^{w}$ denotes the feature $f_{i}^{w}$ in height dimension. Through Eq. 2 and Eq. 4,we merge the information from all stages of the encoder in both width dimension and height dimension. Next, we obtain the output $\{f_{i}^{h}\}_{i=1}^{k}$ of H-Dimensional Axial-Attention through the attention operation:

$$\begin{split}f_{i}^{h}=\text{Softmax}(\frac{Q_{i,h}K_{i,h}^{T}}{\sqrt{C^{*}W^{*}}})V_{i,h},i=1,2,..,k.\end{split}$$ (5)

Resizing back. To aid the guidance of the decoding process with the information acquired through axial attention, we resize the output feature $f_{i}^{h}$ of the two-step axial attention to match the dimensions of the feature map in the corresponding decoding stage $i$, adjusting both spatial and channel dimensions through convolution operations, denoted as $\{f_{i}^{attn}\in\mathbb{R}^{C_{i}\times H_{i}\times W_{i}}\}_{i=1}^{k}$,

$$\begin{split}&f_{i}^{attn}=\sigma(BN(conv_{i}^{h}(f_{i}^{h}))),i=1,..,k,\end{split}$$ (6)

where $\sigma$ and $BN$ denote ReLU and Batch Normalization respectively; and $k$ is the number of stages in the encoder and decoder. Finally, we concatenate $f_{i}^{attn}$ to the feature map of the corresponding stage in the decoder along the channel dimension.

It is noteworthy that traditional two-dimensional attention mechanism demands substantial computing resources [35]. To address this, we employ two-stage one-dimensional attention modules in CSAA, conducting attention processing sequentially on the feature maps in the width and height dimensions.

The CSAA significantly aids the model in extracting information about small medical objects from the encoder and appropriately allocates it to the corresponding stage of the decoder. Unlike prior models, our decoding process for each stage in the decoder is influenced by segmentation-related information gleaned from all stages of the encoder, facilitated by CSAA. Through CSAA, our model achieves feature fusion in the encoder, and reinforces the linkage between the encoder and decoder.

3.3 MPS Module

Low-resolution features in the decoder possess robust global perception, enhancing the model’s performance in small medical object segmentation. However, in prior models such as [27, 4, 21], the globally perceptual information is not fully harnessed; and a significant amount of useful information is lost in the subsequent convolution and upsampling processes. To tackle this issue, we introduce the Multi-Precision Supervision Module (MPS) to extract information from low-resolution features in the decoder and diminish its loss in the ensuing decoding process. Specifically, MPS consists of two steps: segmentation and upsampling.

Segmentation. To thoroughly extract information about small medical objects from each stage of the decoder, we individually feed feature maps from each decoder stage into corresponding segmentation heads. This process yields segmentation results with distinct resolutions, denoted as $\{P_{i}\in\mathbb{R}^{C_{i}\times H_{i}\times W_{i}}\}_{i=1}^{k}$,

$$\begin{split}&P_{i}=S(\sigma(BN(conv_{i}^{d}(f_{i}^{d})))),\quad i=1,..,k,\end{split}$$ (7)

where $f_{i}^{d}$ represents the feature map in stage $i$ of the decoder; $\sigma$ and $BN$ denote ReLU and Batch Normalization respectively; and $S(.)$ represents the sigmoid function.

Upsampling. We employ neighbor interpolation method to upsample the segmentation results obtained in the previous step to match the size same of the ground truth image. This process enables us to achieve multi-precision segmentation $\{M_{i}\}_{i=1}^{k}$ for small medical objects, as illustrated in Fig. 3(a).

$$M_{i}=Upsample(P_{i})\in\mathbb{R}^{C\times H\times W},i=1,2,..,k.$$ (8)

We oversee the segmentation results of different precision with the ground truth label, ensuring that each stage of the decoder encompasses sufficient information to facilitate the segmentation of small medical objects.

In MPS, we formulate a supervision strategy with varying precision for features of different resolutions. Recognizing that low-resolution features possess robust global perception but lack local details, we employ low-precision supervision for them. This decision is made to temporarily forego the emphasis on local details while capitalizing on the strengths of low-resolution features with powerful global perception. This multi-precision supervision approach preserves the advantages of the conventional single-segmentation head while preserving additional global perception from low-resolution features. Consequently, it enhances the model’s performance in small medical object segmentation.

3.4 Loss function

Considering that the positive and negative pixels are extremely unbalanced in small medical object segmentation tasks, we adopt a combination of DiceLoss [31] and Binary Cross Entropy(BCE) Loss during the training process. The loss for the segmentation maps produced by each stage of the decoder is set as follows.

$$\begin{split}&\mathcal{L}_{i}=\lambda_{1}\cdot\mathcal{L}_{\text{Dice}}(M_{i},Y)+\lambda_{2}\cdot\mathcal{L}_{\text{BCE}}(M_{i},Y),\end{split}$$ (9)

where $i=1,2,..,k$ denotes stage indexes; $M_{i}$ represents the result predicted by the model at stage $i$; and $Y$ represents the ground truth with the hyperparameters $\lambda_{1}$ and $\lambda_{2}$ to balance DiceLoss and BCELoss. Taking into account all segmentation results output by each stage of the decoder, the total loss of the model is

$$\mathcal{L}_{\text{total}}=\sum_{i=1}^{k}\alpha_{i}\cdot\mathcal{L}_{i},$$ (10)

where hyperparameter $\{\alpha_{i}\}_{i=1}^{k}$ leverage the losses of segmentation results with different precision.

4 Experiment

S-HRD Dataset. We have gathered a dataset comprising 313 optical coherence tomography (OCT) images from patients with macular edema, with the objective of segmenting small HyperReflective Dots (small HRDs) within them. We refer to this dataset as S-HRD, where ’S’ indicates ’Small’. In S-HRD, the area of each lesion is less than 1 percent of the entire image size. All the ground truths have been manually labeled by experienced eye doctors with over ten years of expertise. We have ensured that appropriate consent has been obtained for the utilization and presentation of images in our research. For further insights into the data collection process, privacy considerations and relevant medical knowledge, please refer to the Supplementary.

S-Polyp Dataset. We build a small-polyp segmentation dataset by excluding images with sizable medical lesions in CVC-ClinicDB [2]. From this selection, we retain 229 images where all lesions are small medical objects. We label this dataset as S-Polyp. In S-Polyp, the area of each lesion is less than 5 percent of the entire image size. Examples of both S-HRD and S-Polyp are illustrated in Fig. 1.

To mitigate limitations and address the specificity of the two datasets, we employ a five-fold cross-validation approach to assess the performance of our model.

Definition of Small Medical Objects. Given the absence of a consistent definition for small medical objects in previous works, we establish our own criteria. In our framework, an object is considered a small medical object if the ratio of its pixel count $n$ to the total pixel count $N$ in the entire image is less than 5$\%$. For objects with a ratio below 1$\%$, we classify them as extremely small medical objects. In the S-Polyp dataset, all objects fall under the category of small medical objects, while in S-HRD, all objects are classified as extremely small medical objects.

Evaluation Metrics. We use two common metrics to compare our model with previous state-of-the-art models. Dice Similariy Coefficient (DSC) is defined as:

$$DSC=\frac{2\times|P\cap G|}{|P|+|G|},$$ (11)

where $P$ represents the area of the predicted label and $G$ represents the area of the ground truth. Intersection over Union (IoU) is defined as:

$$IoU=\frac{S_{i}}{S_{u}},$$ (12)

where $S_{i}$ represents the area where the predicted label and ground truth overlap; and $S_{u}$ for the total area of the two.

Implementation Details. We conduct our experiments using one NVIDIA RTX A6000 GPU equipped with 48GB of memory. The SGD optimizer is employed with an initial learning rate of 0.01. Our training spans 200 epochs, employing a batch size of 4. All input images are resized uniformly to $352\times 352$. The model configuration includes 4 stages in both the encoder and decoder. We set $\lambda_{1}$ and $\lambda_{2}$ to 0.7 and 0.3 respectively to balance DiceLoss and BCELoss. And $\alpha_{1}$, $\alpha_{2}$, $\alpha_{3}$, $\alpha_{4}$ are set to 1.0, 0.9, 0.8,0.7 respectively to balance losses of muti-precision segmentation results.

Competitors. Given the limited number of works dedicated specifically to small medical object segmentation, we reference recent models from the broader field of medical image segmentation, including state-of-the-art models: (1) CNN based methods: U-Net [27], Attention-UNet [24], MSU-Net [30], CaraNet [21]; (2) Transformer based methods: TransFuse [42], TransUNet [4], SSFormer [36], Swin-UNet [3]; (3) Segment Anything Model (SAM) [20] and the related works: SAM without any prompt, SAM with point, SAM with box and MedSAM [22]. (4) Additionally, we assess the performance of an enlarged version of U-Net (U-Net-Large), where both the encoder and decoder are scaled up from 4 layers to 12 layers. This exploration aims to understand the impact of model size on segmentation accuracy.

4.1 Quantitative Results

Table 1: Comparison of our EFCNet with competitors on S-HRD and S-Polyp in DSC ($\%$) and IoU ($\%$).

Metrics	Methods	S-HRD						S-Polyp
		Fold0	Fold1	Fold2	Fold3	Fold4	Mean	Fold0	Fold1	Fold2	Fold3	Fold4	Mean
DSC ($\%$)	U-Net [27]	42.09	36.51	37.74	35.90	41.29	38.71	75.90	75.47	81.95	72.06	76.73	76.42
	U-Net-Large [27]	43.02	36.72	38.24	37.82	41.60	39.48	78.93	78.32	82.76	73.70	77.84	78.31
	Attn-UNet [24]	44.33	39.97	39.07	38.46	39.90	40.35	79.89	81.17	81.19	77.01	74.68	78.79
	MSU-Net [30]	44.06	40.94	38.66	38.54	39.44	40.33	87.60	86.64	86.03	82.99	75.56	83.76
	CaraNet [21]	17.03	13.79	12.46	13.40	13.07	13.95	85.12	78.66	88.14	75.02	82.56	81.90
	TransUNet [4]	32.84	30.34	30.71	31.54	32.81	31.65	84.83	82.13	88.24	76.53	82.85	82.92
	TransFuse [42]	14.95	10.11	11.59	11.96	11.13	11.95	80.73	72.43	82.33	72.53	81.04	77.81
	SSFormer [36]	34.66	27.74	27.06	28.49	25.97	28.78	85.92	84.44	88.00	78.80	83.37	84.11
	Swin-UNet [3]	07.90	04.27	04.29	08.08	07.32	06.37	48.16	53.91	50.00	49.57	61.68	52.66
	SAM [20]	03.81	03.66	01.47	01.40	02.89	02.64	44.65	40.57	49.94	56.43	46.31	47.58
	SAM (box) [20]	03.99	02.78	02.60	02.67	02.55	02.92	77.21	80.40	82.65	77.73	78.60	79.32
	SAM (point) [20]	10.49	07.48	05.16	04.66	07.79	07.12	72.53	73.27	75.30	67.80	72.08	72.20
	MedSAM [22]	04.07	03.29	03.16	02.86	03.11	03.30	76.46	79.26	80.97	75.26	78.70	78.13
	EFCNet(Ours)	49.10	44.91	43.72	43.46	44.95	45.23	89.11	90.74	89.20	85.39	83.58	87.60
IoU ($\%$)	U-Net [27]	28.83	24.09	24.20	24.12	26.90	25.63	65.98	68.77	73.42	63.67	58.63	66.09
	U-Net-Large [27]	29.99	25.09	25.21	25.13	27.88	26.66	68.93	69.99	76.75	62.55	65.98	68.84
	Attn-UNet [24]	30.75	27.47	26.27	25.95	26.68	27.42	72.69	72.59	73.30	68.58	68.38	71.11
	MSU-Net [30]	30.20	27.98	25.60	25.48	25.76	27.00	79.51	78.79	77.54	72.72	68.21	75.35
	CaraNet [21]	10.36	08.04	07.21	07.77	07.50	08.18	77.24	72.04	81.40	67.29	75.56	74.71
	TransUNet [4]	22.00	19.87	19.55	21.02	21.21	20.73	75.40	75.29	81.02	68.83	75.20	75.15
	TransFuse [42]	09.05	05.75	06.58	07.15	06.25	06.96	70.77	64.74	74.47	65.13	73.01	69.62
	SSFormer [36]	22.65	17.67	16.57	17.95	16.02	18.17	78.16	77.30	80.93	72.74	75.52	76.93
	Swin-UNet [3]	05.03	02.38	02.32	05.08	04.16	03.79	37.36	41.84	39.23	38.00	50.10	41.31
	SAM [20]	02.14	02.11	00.77	00.73	01.62	01.47	39.17	35.44	44.97	51.01	41.50	42.42
	SAM (box) [20]	02.09	01.45	01.34	01.40	01.31	01.52	67.04	70.62	72.97	67.71	69.34	69.54
	SAM (point) [20]	06.86	04.49	03.10	02.65	04.85	04.39	64.17	65.89	67.54	60.15	65.60	64.67
	MedSAM [22]	02.13	01.71	01.66	01.49	01.64	01.73	66.47	69.21	71.61	65.59	69.13	68.40
	EFCNet(Ours)	35.06	31.45	29.84	29.35	30.25	31.19	82.54	83.71	82.08	76.98	75.59	80.18

As shown in Tab. 1, our model consistently outperforms previous state-of-the-art models across all folds for both S-HRD and S-Polyp, measured by DSC and IoU.

On S-HRD, our model demonstrates a noteworthy improvement of 4.88$\%$ in DSC and 3.77$\%$ in IoU compared to earlier methods. Similarly, on S-Polyp, our model exhibits a performance boost of 3.49$\%$ in DSC and 3.25$\%$ in IoU.

Table 1 shows that methods tend to perform poorly on S-HRD. That is because the objects in S-HRD are smaller in size compared with S-Polyp, which means that there is less information available in images. Nevertheless, our model still performs best among all methods.

Furthermore, the results on S-HRD and S-Polyp indicate that the smaller the medical objects in the datasets, the more significant the improvement of our model compared to previous SOTA methods. This underscores the superiority of our model in small medical object segmentation.

Additionally, simply increasing the size of U-Net (U-Net-Large) yields only marginal improvements in segmentation performance compared to the standard-sized U-Net. In contrast, our EFCNet demonstrates substantial improvement. This indicates that the superior performance of EFCNet in segmentation is primarily attributed to our model design rather than the larger model size. While the addition of CSAA and MPS increases the model’s cost, we believe the improvement justifies the associated costs in the realm of small medical object segmentation. We provide detailed model costs comparison in the Supplementary.

4.2 Visualization

Figure 4: Visualization of EFCNet (ours) and previous SOTA methods on S-HRD and S-Polyp. The previous SOTA method on S-HRD is Attn-UNet [24], and the previous SOTA method on S-Polyp is SSFormer [36]. The green circle areas show extremely small medical objects captured by our method that are not captured by previous SOTA methods. The yellow circle areas show that the segmentation of the boundaries of small medical objects in our method is significantly better than the previous SOTA method. The red circle areas show the wrong segmentation of small medical objects in the previous SOTA method while our method is correct.

Some visual results of different methods on S-HRD and S-Polyp are shown in Fig. 4. We compare our EFCNet with the previous SOTA methods. According to our experimental results in Tab. 1, the previous SOTA method on S-HRD is Attn-UNet [24], and the previous SOTA method on S-Polyp is SSFormer [36].

As illustrated in Fig. 4, the strengths of our method are predominantly evident in three aspects: (1) Our method excels at capturing extremely small medical objects, as highlighted in the green circle areas. (2) Our method demonstrates higher accuracy in terms of segmenting the boundaries of small medical objects, as evidenced by the yellow circle areas. (3) Our method is significantly less prone to erroneously segmenting the background into small medical objects, as indicated by the red circle areas.

On one hand, CSAA facilitates the application of valuable local information from low-level features in the encoder to the segmentation process, ensuring the model’s capability to capture fine details of small medical objects. On the other hand, MPS enables the model to leverage global perception inherent in low-resolution features in the initial stages of the decoder, enhancing its ability to locate small medical objects.

4.3 Ablation Studies

Table 2: Ablation study on CSAA module and MPS module on S-HRD and S-Polyp in DSC ($\%$) and IoU ($\%$).

Methods	S-HRD		S-Polyp
	DSC($\%$)	IoU($\%$)	DSC($\%$)	IoU($\%$)
U-Net	36.58	24.14	69.98	63.27
U-Net+CSAA	39.19	27.09	78.84	70.66
U-Net+MPS	38.98	26.85	80.20	74.14
U-Net+CSAA+MPS (Ours)	41.82	29.69	83.26	76.29

We perform several sets of ablation experiments on S-HRD and S-Polyp, confirming the positive effect of CSAA and MPS on segmentation ability for small medical objects respectively.

Effectiveness of CSAA and MPS. We incorporate CSAA and MPS into our U-Net backbone individually, and the experimental results are presented in Tab. 2. It is evident that each module contributes to the improvement of our model’s performance. Furthermore, with the addition of both CSAA and MPS, the segmentation ability of our model experiences further enhancement.

Table 3: Ablation study on the number of stages that CSAA aggregates on S-HRD and S-Polyp in DSC ($\%$) and IoU ($\%$).

Methods	S-HRD		S-Polyp
	DSC ($\%$)	IoU ($\%$)	DSC ($\%$)	IoU ($\%$)
Concat-One	38.98	26.85	80.20	74.14
AA-One	40.37	28.13	82.36	75.94
AA-All (Ours)	41.82	29.69	83.26	76.29

Number of Stages in CSAA. We change the number of stages aggregated in CSAA module: AA-All aggregates features in all stages of the encoder, which is exactly the CSAA applied in our final model. AA-One only performs axial attention on features in one stage of the encoder. Concat-One only concatenates features in each stage of the encoder to the corresponding decoder without any other processing. The performance of these three methods on S-HRD and S-Polyp is shown in Tab. 3. It can be seen that among the three models, Concat-One performs the worst. Compared with Concat-One, AA-One can improve the segmentation ability of the model. CSAA aggregates features of all stages of the encoder and performs best among these three methods above.

Table 4: Ablation study on the number of MPS connected to the decoder on S-HRD and S-Polyp in DSC ($\%$) and IoU ($\%$).

Methods	S-HRD		S-Polyp
	DSC ($\%$)	IoU ($\%$)	DSC ($\%$)	IoU ($\%$)
MPS-1	39.19	27.09	78.84	70.66
MPS-2	40.07	27.17	82.00	73.87
MPS-3	40.18	28.32	82.51	75.13
MPS-4 (Ours)	41.82	29.69	83.26	76.29

Number of Supervisions in MPS. We vary the number of supervisions in the MPS: MPS-4 connects segmentation heads to all stages of the decoder, representing the MPS configuration in our final model. MPS-3, MPS-2, and MPS-1 connect three, two and one segmentation heads to the decoder respectively. The performance of these four models on S-HRD and S-Polyp is shown in Tab. 4. It is evident that among these four models, increased supervision correlates with improved model performance.

5 Conclusion

We introduce a novel model called EFCNet to address the challenging task of small object segmentation in medical images. EFCNet pays sufficient attention to all features of each stage in the model, effectively reducing the information loss of small medical objects and improving the segmentation accuracy. Specifically, we propose Cross-Stage Axial Attention Module (CSAA) and Multi-Precision Supervision Module (MPS), which alleviate the loss of information in the encoder and decoder respectively, leading to a substantial enhancement in model performance. Moreover, we establish a new benchmark for small medical object segmentation research. Our experiments on two datasets demonstrate that CSAA and MPS contribute to improved segmentation accuracy, with our model significantly outperforming previous state-of-the-art models.

Supplementary Material

6 Rationale

Having the supplementary compiled together with the main paper means that:

• The supplementary can back-reference sections of the main paper, for example, we can refer to Sec. 1;

• The main paper can forward reference sub-sections within the supplementary explicitly (e.g. referring to a particular experiment);

• When submitted to arXiv, the supplementary will already included at the end of the paper.

To split the supplementary pages from the main paper, you can use Preview (on macOS), Adobe Acrobat (on all OSs), as well as command line tools.

7 Overview

In the supplementary material, we firstly provide additional analysis of model cost in Sec. 8. Then, we provide additional details about the architecture of our EFCNet in Sec. 9 and experiment settings in Sec. 10. In the end, we introduce ethical considerations in Sec. 11.

8 Analysis of Model Cost

We provide model cost comparison of our EFCNet and other UNet-based methods in Tab. 5 and performance improvement compared to U-Net [27] in Tab. 6. We can draw a conclusion that in the field of small medical object segmentation, simply increasing the model size like U-Net-Large cannot bring significant improvement in segmentation performance based on the standard-sized U-Net. In comparison, our EFCNet achieves far better segmentation performance than other UNet-based methods with model cost less than that of U-Net-Large. Indeed, our model remains relatively large to effectively address the intricate challenge of the segmentation task.

9 Additional Details about Model Architecture

We provide additional details about the backbone network, the CSAA Module and the MPS Module in our EFCNet as shown in Tab. 7, Tab. 8 and Tab. 9 respectively.

10 Additional Details about Experiment Settings

Experimental environment. The environment of our experiment is as follows. GPU: NVIDIA RTX A6000; CUDA Version: 11.7; Python Version: 3.10.4; Torch Version: 1.13.1.

Split of Datasets. We perform five-fold cross-validation in our experiments. In each split, datasets are separated into training set, validation set, testing set by a ratio of 7:1:2.

Table 5: Model cost comparison of our EFCNet with other UNet-based methods.

Methods	FLOPs (G)	Params (M)
U-Net [27]	91.94	37.66
Attn-UNet [24]	125.94	34.88
MSU-Net [30]	143.05	47.09
U-Net-Large [27]	430.57	100.37
U-Net+CSAA	375.52	87.37
U-Net+MPS	101.48	38.81
U-Net+CSAA+MPS (EFCNet)	385.06	88.52

Table 6: Performance improvement among our EFCNet and other UNet-based methods compared to U-Net [27].

Methods	S-HRD		S-Polyp
	$\Delta$DSC($\%$)	$\Delta$IoU($\%$)	$\Delta$DSC($\%$)	$\Delta$IoU($\%$)
U-Net [27]	+0.00	+0.00	+0.00	+0.00
Attn-UNet [24]	+1.64	+1.79	+2.37	+5.02
MSU-Net [30]	+1.62	+1.37	+7.34	+9.26
U-Net-Large [27]	+0.77	+1.03	+1.89	+2.75
U-Net+CSAA+MPS (EFCNet)	+6.52	+5.56	+11.18	+14.09

Table 7: Details of the backbone network of our EFCNet.

Stage	Layer	Output shape
Input	-	(3, 352, 352)
Encoder stage1	(Conv, BN, ReLU) $\times$ 2, Downsample	(64, 176, 176)
Encoder stage2	(Conv, BN, ReLU) $\times$ 2, Downsample	(128, 88, 88)
Encoder stage3	(Conv, BN, ReLU) $\times$ 2, Downsample	(256, 44, 44)
Encoder stage4	(Conv, BN, ReLU) $\times$ 2, Downsample	(512, 22, 22)
Decoder stage4	(Conv, BN, ReLU) $\times$ 2, Upsample, Concat	(1024, 44, 44)
Decoder stage3	(Conv, BN, ReLU) $\times$ 2, UPsample, Concat	(512, 88, 88)
Decoder stage2	(Conv, BN, ReLU) $\times$ 2, UPsample, Concat	(256, 176, 176)
Decoder stage1	(Conv, BN, ReLU) $\times$ 2, UPsample, Concat	(128, 352, 352)

Table 8: Details of the CSAA Module in our EFCNet.

Stage	Layer	Output shape
CSAA stage1	Resize: ((Conv, BN, ReLU) $\times$ 2)	($C^{*}$, $H^{*}$, $W^{*}$)
	W-CSAA: one-dimensional attention module	($C^{*}$, $H^{*}$, $W^{*}$)
	H-CSAA: one-dimensional attention module	($C^{*}$, $H^{*}$, $W^{*}$)
	Resize back: ((Conv, BN, ReLU) $\times$ 2)	(64, 352, 352)
CSAA stage2	Resize: ((Conv, BN, ReLU) $\times$ 2)	($C^{*}$, $H^{*}$, $W^{*}$)
	W-CSAA: one-dimensional attention module	($C^{*}$, $H^{*}$, $W^{*}$)
	H-CSAA: one-dimensional attention module	($C^{*}$, $H^{*}$, $W^{*}$)
	Resize back: ((Conv, BN, ReLU) $\times$ 2)	(128, 176, 176)
CSAA stage3	Resize: ((Conv, BN, ReLU) $\times$ 2)	($C^{*}$, $H^{*}$, $W^{*}$)
	W-CSAA: one-dimensional attention module	($C^{*}$, $H^{*}$, $W^{*}$)
	H-CSAA: one-dimensional attention module	($C^{*}$, $H^{*}$, $W^{*}$)
	Resize back: ((Conv, BN, ReLU) $\times$ 2)	(256, 88, 88)
CSAA stage4	Resize: ((Conv, BN, ReLU) $\times$ 2)	($C^{*}$, $H^{*}$, $W^{*}$)
	W-CSAA: one-dimensional attention module	($C^{*}$, $H^{*}$, $W^{*}$)
	H-CSAA: one-dimensional attention module	($C^{*}$, $H^{*}$, $W^{*}$)
	Resize back: ((Conv, BN, ReLU) $\times$ 2)	(512, 44, 44)

Table 9: Details of the MPS Module in our EFCNet.

Stage	Layer	Output shape
MPS stage1	Segmentation: (Conv, BN, ReLU) $\times$ 2, Conv, Sigmoid	(1, 352, 352)
MPS stage2	Segmentation: (Conv, BN, ReLU) $\times$ 2, Conv, Sigmoid	(1, 176, 176)
	Upsample: Nearest interpolation	(1, 352, 352)
MPS stage3	Segmentation: (Conv, BN, ReLU) $\times$ 2, Conv, Sigmoid	(1, 88, 88)
	Upsample: Nearest interpolation	(1, 352, 352)
MPS stage4	Segmentation: (Conv, BN, ReLU) $\times$ 2, Conv, Sigmoid	(1, 44, 44)
	Upsample: Nearest interpolation	(1, 352, 352)

11 Ethical Considerations

The collection and utilization of human data in our research project adheres to the highest ethical standards. Our study has received the full approval of the Institutional Review Board and the Ethics Committee of a hospital. This approval process is conducted in strict accordance with the principles outlined in the Declaration of Helsinki [37], which provides ethical guidelines for medical research involving human subjects. All recruited patients have signed the informed consent to publish this paper. Here, we provide the information on data collection and annotation, appropriate consent and privacy considerations.

Data collection and annotation. In this investigation, retinal OCT scans are collected from eyes of 313 patients who seek treatment for macular edema associated with diabetic retinopathy or retinal vein occlusion at a hospital within the past six months. The investigation has been approved by the Institutional Review Board and the Ethics Committee of a hospital, in accordance with the principles of the Declaration of Helsinki [37]. Each OCT scan is centered on the fovea, either vertically or horizontally. Macular edema is defined as a central retinal thickness (CRT) greater than 300 mm. Small hyperreflective dots (small HRDs) are defined as discrete tiny dots with diameter between 20 micron and 40 micron, characterized by reflectivity similar to that of the nerve fiber layer and the absence of back shadowing [15]. The ground truths of small HRDs have been manually labeled by experienced eye doctors with over ten years of expertise.

Appropriate consent and privacy considerations. We confirm that appropriate consent has been obtained for the use and display of images in our research. To uphold the privacy and confidentiality of the individuals involved, we take rigorous measures to ensure that all identifying information, including names, genders, and birth dates of patients, has been thoroughly removed from the images prior to any processing or analysis. These steps are taken to safeguard patient privacy and adhere to ethical standards. We understand the critical importance of addressing privacy concerns when dealing with medical images, and we are committed to upholding the highest ethical standards in our research practices.