Demystifying the Effect of Receptive Field Size in U-Net Models for Medical Image Segmentation

Abstract

Purpose: Medical image segmentation is a critical task in healthcare applications, and U-Nets have demonstrated promising results in this domain. This work delves into the understudied aspect of receptive field (RF) size and its impact on the U-Net and Attention U-Net architectures used for medical imaging segmentation.
Approach: This work explores several critical elements including the relationship between RF size, characteristics of the region of interest, and model performance, as well as the balance between RF size and computational costs for U-Net and Attention U-Net methods for different datasets. This work also proposes a mathematical notation for representing the theoretical receptive field (TRF) of a given layer in a network and proposes two new metrics namely - effective receptive field (ERF) rate and the Object rate to quantify the fraction of significantly contributing pixels within the ERF against the TRF area and assessing the relative size of the segmentation object compared to the TRF size respectively.
Results: The results demonstrate that there exists an optimal TRF size that successfully strikes a balance between capturing a wider global context and maintaining computational efficiency, thereby optimizing model performance. Interestingly, a distinct correlation is observed between the data complexity and the required TRF size; segmentation based solely on contrast achieved peak performance even with smaller TRF sizes, whereas more complex segmentation tasks necessitated larger TRFs. Attention U-Net models consistently outperformed their U-Net counterparts, highlighting the value of attention mechanisms regardless of TRF size.
Conclusions: These novel insights present an invaluable resource for develo** more efficient U-Net-based architectures for medical imaging and pave the way for future exploration of other segmentation architectures. A tool is also developed that calculates the TRF for a U-Net (and Attention U-Net) model, and also suggest an appropriate TRF size for a given model and dataset.

*Navchetan Awasthi, [email protected]

1 Introduction

Medical imaging, a cornerstone of modern healthcare, provides non-invasive means for diagnosing and monitoring a wide range of diseases. However, the interpretation of medical images often requires expert knowledge and can be time-consuming, leading to a growing interest in automated analysis methods [1].

Figure 1: Variable attention U-Net in which the depth ($n$), kernel size of the convolution layers ($k$), and number of channels ($c$) can be tuned to alter the size of the TRF. It can be converted to a regular U-Net by simply removing the attention gates and gating signals.

Semantic segmentation, a key task in computer vision, plays a crucial role in this context. It involves the categorization of pixels in an image into predefined classes, enabling the delineation of anatomical structures and pathological regions in medical images [2]. The U-Net architecture, a convolutional neural network (CNN) designed specifically for biomedical image segmentation, has emerged as a popular choice for semantic segmentation tasks in medical imaging [3]. As illustrated in Figure 1, it employs an encoder-decoder structure. The encoder progressively reduces the spatial dimensionality while increasing the feature representation, capturing the global context of the image. The decoder, on the other hand, gradually recovers the spatial information, enabling precise localization [4]. The U-Net is renowned for its accuracy in semantic segmentation tasks [3]. An extended version, Attention U-Net, integrates an attention mechanism to enhance feature capturing to improve overall performance [5].

Within these networks, the concept of the receptive field (RF) is crucial. It refers to the region in the input space that affects a feature in a CNN [6]. There are two kinds of receptive fields: the theoretical receptive field (TRF) and the effective receptive field (ERF). The TRF is defined as the maximum region of the input image that influences a specific pixel of the output, considering only the receptive field from the preceding layers that are relevant to the current layer [7]. This is in contrast to the ERF, which is the actual region of the input image that contributes to the activation of a particular neuron in the network, taking into account the impact of operations such as pooling [6]. An example of the TRF and ERF is illustrated in Figure 2.

Figure 2: Example of Theoretical Receptive Field (TRF) and Effective Receptive Field (ERF) in an image. The yellow square denotes the TRF, the maximum input area influencing the output pixel located at the centre of the square. The gray pixels, representing the ERF, show the actual input area affecting a neuron’s activation, with intensity indicating the impact level.

Previous studies have started investigating the role of RF size on U-Net performance for image segmentation tasks, but not all aspects have been explored. In one such study, [8] focused on ultrasound image segmentation, demonstrating that the RF size has a more critical role than the network’s depth or the number of parameters. They suggested that a computationally efficient shallow network could replace a deep one without performance loss by manipulating the RF size. However, their study was limited to the U-Net architecture and a single dataset, comparing only a deep and a shallow network, leaving room for a more exhaustive investigation.

In another study, [9] delved into the influence of RF size and network complexity on a CNN’s performance for transmission electron microscopy (TEM) image analysis. They found that the RF size’s influence varied with TEM image resolution and contrast characteristics. For low-resolution TEM images, where contrast is crucial, RF size had a minimal influence. But for high-resolution TEM images, where identification is less dependent on contrast changes, the RF size was vital, especially for low contrast images. However, they only considered TEM images and regular U-Nets, leaving the applicability of these findings to other medical imaging tasks unexplored.

Our study builds upon these insights by examining the influence of RF size on U-Net and Attention U-Net architectures across multiple medical image segmentation datasets with certain characteristics in the region of interest (RoI). We compare ten different U-Net architectures shown in Table-1 with varied RF sizes and equal total parameters, thus isolating RF size’s impact on performance while taking into consideration specific dataset characteristics. Moreover, we repeat the experiments on eight different synthetic datasets and six real-world medical datasets. We also extend our investigation to the Attention U-Net architecture, thereby expanding the study beyond regular U-Net architecture. Our aim is to offer critical insights for U-Net-based architectures’ design, considering RF size as a key parameter.

Specifically, this paper makes the following contributions to the field of medical image segmentation with U-Nets:

1. We provide a comprehensive analysis of the role of the RF size in the performance of U-Net and Attention U-Net architectures, demonstrating its significance in capturing the necessary context for accurate segmentation.

2. We propose a mathematical notation to represent the TRF at a given layer within a network, utilizing a 4-dimensional tensor and provide the mathematical framework for calculating TRF for the different deep learning layers.

3. We propose two new metrics called the ERF rate and the Object rate to quantify the fraction of significantly contributing pixels within the ERF against the TRF area and assessing the relative size of the segmentation object compared to the TRF size respectively.

4. We explore the trade-off between RF size and computational cost for a variety of medical imaging datasets and synthetic datasets.

5. We compare the performance of U-Net and Attention U-Net architectures for the same RF size, highlighting the effectiveness of the attention mechanism in improving the model’s overall performance.

6. We present a nuanced analysis of the performance trends across datasets with different characteristics in the RoI, particularly its size and contrast to the surrounding area.

7. We provide a tool that calculates the TRF for a U-Net (and Attention U-Net) model, and also suggest an appropriate TRF size for a given model and dataset.

2 Methodology

This study explores the role of the receptive field in the performance of U-Net and Attention U-Net models in semantic segmentation tasks. Through a series of experiments with varying TRF sizes, we evaluated these models on diverse range of datasets. In this section, we further provide comprehensive descriptions of the model architectures, as well as the TRF and ERF computation.

2.1 U-Net Design and TRF Tuning

The configuration of the hyper-parameters of a U-Net model significantly impacts the size of its TRF. As established by [10], the TRF size is primarily determined by the number of pooling layers and the convolutional kernel sizes. To elaborate, Figure 1 illustrates a variable attention U-Net diagram, where the TRF size can be adjusted in two different ways. Firstly, when the vertical depth ($d$) of the network is increased, one encoder and one decoder block is added before and after the bottleneck respectively. This increases the number of pooling layers and therefore increases the TRF size. Changing the network depth on its own does not result in a significant impact on the model’s performance [8]. Secondly, the TRF size can be varied by changing the kernel sizes of the convolutional layers within the network. The mathematical details of the effects of various layers on the TRF size are provided in Section 2.2.

It should be noted that adjusting these hyper-parameters also impacts the total number of parameters in the model. To ensure a fair comparison between the performance of various configurations, the total number of parameters must remain approximately equal. According to [11], this can be achieved by modifying the number of output channels in each convolution layer within the network blocks. Table 1 provides an overview of all configurations utilized in this study. It is important to mention that the parameter count is based on the standard U-Net architecture. The Attention U-Net introduces additional parameters due to the inclusion of an attention block at each layer. However, this increase varies approximately on the order of 100,000, which is relatively insignificant and can be considered negligible in this context.

Table 1: All U-Net configurations with different TRF sizes. The TRF size is influenced by the convolutional kernel size ($k$) and the vertical depth of the network ($d$).

TRF size	$k,d$	Out channels per layer	# Parameters
54	3, 2	[230, 456, 765, 1245]	31,013,720
100	3, 3	[145, 256, 512, 1024]	31,012,268
146	3, 4	[133, 244, 355, 791]	31,032,960
204	4, 3	[64, 128, 256, 512, 1024]	31,042,369
230	3, 6	[63, 170, 256, 512]	31,031,345
298	4, 4	[25, 44, 110, 451, 756]	31,043,816
360	3, 8	[47, 83, 180, 360]	31,062,482
412	5, 3	[63, 64, 115, 255, 512, 1024]	31,043,945
486	4, 6	[28, 58, 146, 270, 510]	31,027,119
570	4, 7	[24, 55, 101, 223, 481]	31,041,124

2.2 Computing the TRF

Formally, the TRF refers to the maximum region of the input image $\mathbf{X}\in[0,1]^{h\times w}$ that potentially influences a specific pixel in the output layer. To represent the TRF at layer $d$ in a U-Net architecture of depth $D$, we introduce a 4-dimensional tensor $\mathbf{T}^{(d)}\in\mathds{R}^{h\times w\times 2\times 2}$. Here, the first two dimensions correspond to the $y$ and $x$ axes of the input image, respectively, while the third and fourth dimensions represent the top-left (t-l) and bottom-right (b-r) coordinates of the TRF at layer $d$. For a given pixel located at position $(i,j)$ in the output layer $D$, the TRF can be expressed as a $2\times 2$ matrix in which the first row corresponds to the top-left corner of the TRF, and the second row corresponds to the bottom-right corner of the TRF:

$$\mathbf{T}_{i,j}^{(D)}=\begin{bmatrix}t_{i,j}^{(D)}&l_{i,j}^{(D)}\\ b_{i,j}^{(D)}&r_{i,j}^{(D)}\end{bmatrix}$$ (1)

Empirically, it has been observed that all pixels have an equal TRF size in the output layer, except those located around the border because of the padded zeroes. Based on this observation, a single (maximum) TRF value can be assigned to the entire U-Net. In the remainder of this paper, we define the TRF size of a U-Net as the size of the TRF of the center pixel $(u,v)=(h/2,w/2)$ in the output layer:

$$\text{TRF}=\sqrt{\left(\mathbf{T}_{u,v,1,0}^{(D)}-\mathbf{T}_{u,v,0,0}^{(D)}\right)\cdot\left(\mathbf{T}_{u,v,1,1}^{(D)}-\mathbf{T}_{u,v,0,1}^{(D)}\right)}$$ (2)

To compute the values of the TRF matrix in Equation 1, we traverse the network from the first to the final layer, tracking the TRF of each pixel at every layer based on the previous layer’s pixels until reaching the output layer [7]. Therefore, the TRF of the pixel at position $(i,j)$ in a layer at depth $d$ can be expressed as:

$$\mathbf{T}_{i,j}^{(d)}=\begin{bmatrix}t_{i,j}^{(d)}&l_{i,j}^{(d)}\\ b_{i,j}^{(d)}&r_{i,j}^{(d)}\end{bmatrix}$$ (3)

In the input layer $0$, the TRF of each pixel corresponds to the pixel itself:

$$\mathbf{T}_{i,j}^{(0)}=\begin{bmatrix}i&j\\ i&j\end{bmatrix}$$ (4)

The computation of the TRF in subsequent layers depends on the U-Net’s configuration. Here are all possible layers that a U-Net may include.

2.2.1 Convolution

In a 2D convolution layer, a filter or kernel is applied to a 2D image, performing a dot product at each position [12]. The kernel size ($k$) impacts the detail level captured, while the stride ($s$) affects the kernel shift amount. Padding ($p$), set to ‘same’ in this study, ensures the output feature map matches the input image dimensions, permitting edge-based convolution operations.

If the padding is set to same, the number of arrays that must be padded on every side of the $\mathbf{T}^{(d-1)}$ tensor to simulate a convolution while maintaining the previous layer’s dimensions can be calculated. For a $h\times w$ layer, the padding values along the $y$ and $x$ axes are computed as follows:

$$p_{y} =\left\lfloor\frac{(h-1)\cdot s+k-h}{2}\right\rfloor$$ (5)

$$p_{x} =\left\lfloor\frac{(w-1)\cdot s+k-w}{2}\right\rfloor$$ (6)

Therefore, along the first and second axes of the 4-dimensional tensor $\mathbf{T}^{(d-1)}$, the tensor is padded with $p_{y}$ and $p_{x}$, 2-dimensional tensors that contain the same values as the edges along the first and second axes of $\mathbf{T}^{(d-1)}$. Let $\mathbf{P}^{(d-1)}$ denote this padded tensor. For each position $(i,j)$, the top-left and bottom-right pixels from the previous layer’s TRF can be fetched from $\mathbf{P}^{(d-1)}$ at the indices $(i\cdot s,j\cdot s)$ and $(i\cdot s+k-1,j\cdot s+k-1)$, respectively. Thus, the TRF at position $(i,j)$ for a convolutional layer at depth $d$ can be denoted as:

$$\mathbf{T}_{i,j}^{(d)}=\begin{bmatrix}\mathbf{P}_{i\cdot s,j\cdot s,0,0}^{(d-1)}&\mathbf{P}_{i\cdot s,j\cdot s,0,1}^{(d-1)}\\ \\ \mathbf{P}_{i\cdot s+k-1,j\cdot s+k-1,1,0}^{(d-1)}&\mathbf{P}_{i\cdot s+k-1,j\cdot s+k-1,1,1}^{(d-1)}\end{bmatrix}$$ (7)

2.2.2 Max pooling

2D max pooling is a feature map reduction method where a rectangular kernel selects maximum values within regions, creating a smaller feature map [12]. The kernel size ($k$) defines the sliding window size over the input, and the stride ($s$) – in our study equal to $k$ in order to simplify the computation – controls the window’s movement.

For a given position $(i,j)$, the topmost and leftmost pixels from the previous layer’s TRF can be accessed from the $\mathbf{T}^{(d-1)}$ tensor at the index of $(i\cdot k,j\cdot k)$, while the bottom-most and rightmost pixels can be accessed at the index of $(i\cdot k+k-1,j\cdot k+k-1)$. As such, the TRF at position $(i,j)$ for a max pooling layer at depth $d$ can be expressed as follows:

$$\mathbf{T}_{i,j}^{(d)}=\begin{bmatrix}\mathbf{T}_{i\cdot k,j\cdot k,0,0}^{(d-1)}&\mathbf{T}_{i\cdot k,j\cdot k,0,1}^{(d-1)}\\ \\ \mathbf{T}_{i\cdot k+k-1,j\cdot k+k-1,1,0}^{(d-1)}&\mathbf{T}_{i\cdot k+k-1,j\cdot k+k-1,1,1}^{(d-1)}\end{bmatrix}$$ (8)

2.2.3 Upsampling

Upsampling is a technique used to increase the spatial resolution of feature maps. In particular, it is implemented through transposed convolution or deconvolution, which is the reverse operation of convolution. During the transposed convolution operation, a kernel of size $k$ is applied to the input feature map to generate an output feature map with a higher spatial resolution. The stride $s$ determines the amount of shift in the output feature map for each kernel application [12]. When the stride is set to $k$, the size of the output feature map is equal to the size of the input feature map multiplied by the stride.

However, when the stride $s$ is different from the kernel size $k$, there may be overlaps in the values of the output feature map. Therefore, an iterative method is required to identify the corners of the TRF for each pixel in the output feature map. Specifically, Algorithm 1 is applied to each pixel $(i,j)$ in the input map, computing the range in which the pixel is copied to the output feature map by multiplying the top and left indices with the stride and the bottom and right indices with the stride and then adding the kernel size. The algorithm then iterates over the pixels $(m,n)$ in the output feature map within this range. If there is no overlap, the indices from the previous layer at $(i,j)$ are simply copied. Otherwise, for the top and left of the TRF, the algorithm takes the minimum of the current index and a potentially overlap** index, while for the bottom and right TRF, it takes the maximum.

Algorithm 1 TRF at layer $d$ and pixel $(i,j)$ after upsampling

for $m\leftarrow i\cdot s$ to $i\cdot s+k$ do

     for $n\leftarrow j\cdot s$ to $j\cdot s+k$ do

         if $\mathbf{T}_{m,n}^{(d)}$ is None then $\triangleright$ If no overlap (yet)

              $\mathbf{T}_{m,n}^{(d)}\leftarrow\mathbf{T}_{i,j}^{(d-1)}$ $\triangleright$ Copy from previous layer

              continue $\triangleright$ Go to next pixel

         end if

$\triangleright$ If there is overlap:

         if $\mathbf{T}_{i,j,0,0}^{(d-1)}\leq\mathbf{T}_{m,n,0,0}^{(d)}$ then $\triangleright$ Get smallest value

              $\mathbf{T}_{m,n,0,0}^{(d)}\leftarrow\mathbf{T}_{i,j,0,0}^{(d-1)}$ $\triangleright$ Update top

         end if

         if $\mathbf{T}_{i,j,0,1}^{(d-1)}\leq\mathbf{T}_{m,n,0,1}^{(d)}$ then $\triangleright$ Get smallest value

              $\mathbf{T}_{m,n,0,1}^{(d)}\leftarrow\mathbf{T}_{i,j,0,1}^{(d-1)}$ $\triangleright$ Update left

         end if

         if $\mathbf{T}_{i,j,1,0}^{(d-1)}\geq\mathbf{T}_{m,n,1,0}^{(d)}$ then $\triangleright$ Get largest value

              $\mathbf{T}_{m,n,1,0}^{(d)}\leftarrow\mathbf{T}_{i,j,1,0}^{(d-1)}$ $\triangleright$ Update bottom

         end if

         if $\mathbf{T}_{i,j,1,1}^{(d-1)}\geq\mathbf{T}_{m,n,1,1}^{(d)}$ then $\triangleright$ Get largest value

              $\mathbf{T}_{m,n,1,1}^{(d)}\leftarrow\mathbf{T}_{i,j,1,1}^{(d-1)}$ $\triangleright$ Update right

         end if

     end for

end for

2.2.4 Concatenations

Within the U-Net architecture, skip connections from layer $d-c$ are integrated into the decoder blocks by concatenating them with the output of the upsampling layer $d-1$ [13]. To achieve this, the TRF of the tensors being concatenated, denoted as $\mathbf{T}_{i,j}^{(d-c)}$ and $\mathbf{T}_{i,j}^{(d-1)}$, must first be determined. The TRF of each pixel after concatenation, denoted as $\mathbf{T}_{i,j}^{(d)}$, is obtained by selecting the lowest indices for the left and top of both TRFs, and the highest indices for the right and bottom of both TRFs. This approach ensures that the largest possible TRF is obtained.

$$\mathbf{T}_{i,j,0,0}^{(d)} =\min\left(\left\{\mathbf{T}_{i,j,0,0}^{(d-1)},\mathbf{T}_{i,j,0,0}^{(d-c)}\right\}\right)$$

$$\mathbf{T}_{i,j,0,1}^{(d)} =\min\left(\left\{\mathbf{T}_{i,j,0,1}^{(d-1)},\mathbf{T}_{i,j,0,1}^{(d-c)}\right\}\right)$$

$$\mathbf{T}_{i,j,1,0}^{(d)} =\max\left(\left\{\mathbf{T}_{i,j,1,0}^{(d-1)},\mathbf{T}_{i,j,1,0}^{(d-c)}\right\}\right)$$

$$\mathbf{T}_{i,j,1,1}^{(d)} =\max\left(\left\{\mathbf{T}_{i,j,1,1}^{(d-1)},\mathbf{T}_{i,j,1,1}^{(d-c)}\right\}\right)$$

2.2.5 Activation functions

While nonlinear activation functions like ReLU and sigmoid do affect the ERF by potentially reducing its size when certain parameters are set to zero [14], they have no effect on the TRF, as these functions act element-wise on the previous layer. Therefore, in a layer $d$ with an activation function, it can be concluded that $\mathbf{T}_{i,j}^{(d)}=\mathbf{T}_{i,j}^{(d-1)}$.

2.2.6 Attention gates

Attention gates, a key component of the Attention U-Net architecture (illustrated in Figure 1), receive input features from a layer denoted as $x^{\prime}$, and a gating signal from a layer $g$ [5]. The inputs are then subjected to $1\times 1$ convolutions, followed by element-wise addition. At this point, the TRF is equivalent to the maximum range of the TRF of either input, as the TRF is not modified by the $1\times 1$ convolution. Next, a ReLU and sigmoid function are applied, which leave the TRF unchanged, as described in the previous section. Finally, element-wise multiplication is performed on the output, which results in the TRF being equivalent to the maximum range of the TRF of either input. As a result, the TRF size of a U-Net and Attention U-Net with the same depth and convolution kernel sizes are equivalent.

Therefore, similar to concatenations, the TRF of an attention gate $a$ is the maximal range of the TRF from its input features of layer $x^{\prime}$ and the gating signal of layer $g$.

$$\mathbf{T}_{i,j,0,0}^{(a)} =\min\left(\left\{\mathbf{T}_{i,j,0,0}^{(x^{\prime})},\mathbf{T}_{i,j,0,0}^{(g)}\right\}\right)$$

$$\mathbf{T}_{i,j,0,1}^{(a)} =\min\left(\left\{\mathbf{T}_{i,j,0,1}^{(x^{\prime})},\mathbf{T}_{i,j,0,1}^{(g)}\right\}\right)$$

$$\mathbf{T}_{i,j,1,0}^{(a)} =\max\left(\left\{\mathbf{T}_{i,j,1,0}^{(x^{\prime})},\mathbf{T}_{i,j,1,0}^{(g)}\right\}\right)$$

$$\mathbf{T}_{i,j,1,1}^{(a)} =\max\left(\left\{\mathbf{T}_{i,j,1,1}^{(x^{\prime})},\mathbf{T}_{i,j,1,1}^{(g)}\right\}\right)$$

2.3 Computing the ERF

For each pixel $x_{i,j}$ in the input image $\mathbf{X}\in[0,1]^{h\times w}$, its impact on the center pixel of the output image $y_{h/2,w/2}$ is measured by computing the partial derivative of the center output pixel with respect to each input pixel $\partial y_{h/2,w/2}/\partial x_{i,j}$. This method quantifies how much $y_{h/2,w/2}$ changes if $x_{i,j}$ is changed by a small amount [6]. For a TRF, the corresponding ERF ($\mathbf{E}\in\mathds{R}^{m\times n}$) can be expressed as a matrix as shown in equation 9.

$$\mathbf{E}=\begin{bmatrix}\dfrac{\partial y_{h/2,w/2}}{\partial x_{t,l}}&\ldots&\dfrac{\partial y_{h/2,w/2}}{\partial x_{t,r}}\\ \vdots&\ddots&\vdots\\ \dfrac{\partial y_{h/2,w/2}}{\partial x_{b,l}}&\ldots&\dfrac{\partial y_{h/2,w/2}}{\partial x_{b,r}}\end{bmatrix}$$ (9)

The actual computation of the ERF can be done easily with most deep learning frameworks by back-propagating the value of one certain output pixel to the entire input, and taking the $m\times n$ slice of the input at the position of the TRF.

3 Experiment

3.1 Training protocol

All the models were trained on a high-performance computing node featuring two Intel Xeon Platinum 8360Y CPUs and an NVIDIA A100 GPU with 40 GB of HBM2 memory. We used the PyTorch framework [15] and employed Binary Cross-Entropy with Logits Loss as our loss function, with the Adam optimizer facilitating training due to its efficiency and minimal memory requirements [16].

The initial learning rate was set at $10^{-4}$, and a learning rate scheduling strategy was implemented to optimize learning. This strategy reduces the learning rate by 0.1 when the validation loss plateaus for four epochs, enabling more substantial updates in early training phases and smaller updates as the model nears convergence. Training lasted up to 200 epochs, with early stop** [17] implemented to prevent overfitting. If the validation loss remained static over 20 consecutive epochs, training was ceased, and the parameters that achieved the lowest validation loss were saved.

A	A large	A contour	A large contour	B	B large	B contour
B large contour	Fetal head	Fetal head 2	Kidneys	Lungs	Thyroid	Nerve

Figure 3: Typical images and segmentation masks for the synthetic datasets (A and B) and medical datasets (Fetal head, Fetal head 2, Kidneys, Lungs, Thyroid, Nerve).

3.2 Datasets

Our study utilized a wide array of datasets, both synthetic and real-world medical images. The synthetic datasets were specifically designed to evaluate certain hypotheses under controlled conditions. Following this, we applied our hypotheses to medical imaging datasets, which encompassed a variety of imaging techniques and anatomical structures, adding a layer of complexity and realism to our evaluations. Illustrative examples of images and corresponding masks from each dataset can be found in Figure 3.

3.2.1 Synthetic Datasets

The synthetic shape datasets are designed to provide a controlled environment for investigating the impact of the TRF on the performance of the models. The datasets consist of synthetic images with predefined shapes and configurations, allowing for a systematic exploration of the models’ behaviour under different conditions.

There are a total of 8 datasets with generated images. These are of two types, referred to as Type A and Type B. Both types include three non-overlap** shapes – a circle, a triangle, and a square – that are randomly placed and rotated, with a random gray value assigned to each shape. For Type A images, the masks are identical to the shapes in the images. For Type B images, the masks are the same, but the mask of the square is omitted, adding an additional level of complexity to the segmentation task.

For each type, four datasets are created. Two of them contain small shapes placed on an invisible $3\times 3$ grid, and two of them contain large shapes placed on an invisible $2\times 2$ grid. For both the small and large datasets one of them contains images with filled shapes and filled masks, and the other one contains images with only the contours of the shapes with filled masks. Each dataset contains a total of 1000 images of which 700 are used for training, 150 for validation, and 150 for testing.

These synthetic shape datasets offer valuable insights into the role of the TRF on the performance of the models. By comparing the performance of the models on images with small shapes versus large shapes, we can assess how the TRF size affects the model’s ability to capture features of different scales. Specifically, it allows us to determine to what degree it matters if the TRF is smaller than the shape, or if the shape fits into the TRF.

The comparison between images with filled shapes and those with contour shapes allows us to determine what happens if the TRF does not capture the entire shape, but only a part of it, such as the part which is completely black in the image but is filled in the mask because it is within the contours. This is particularly relevant for real-world applications, where the images often contain complex structures that the model needs to accurately segment.

Furthermore, the use of Type B images, where the mask of the square is omitted, enables us to examine how the models handle irrelevant features in the images. This is particularly relevant for real-world applications, where the images often contain irrelevant or distracting features that the model needs to ignore to perform the task effectively.

3.2.2 Medical Datasets

The experiments were carried out using below listed 6 medical datasets. The datasets are classified into two categories: high-contrast, where the RoI can be visually distinguished solely based on its contrast with the background, and low-contrast, which requires additional details like the RoI’s contour or shape to distinguish it from the background.

1. Fetal Head – This low-contrast dataset consists of 2D ultrasound images of fetal heads [dataset] [18]. It includes 350 training images, 74 validation images, and 76 test images. The images were obtained using a standard clinical ultrasound system, and the fetal head circumference was manually annotated by expert sonographers.

2. Fetal Head 2 – This low-contrast dataset is another set of 2D ultrasound images of fetal heads, with a larger number of images [dataset] [19, 20]. It includes 14560 training images, 3240 validation images, and 2875 test images. The images in this dataset were collected from multiple hospitals and were annotated by experienced radiologists.

3. Kidneys – This low-contrast dataset consists of 3D MRI images of kidneys [dataset] [21, 22]. It includes 454 training images, 91 validation images, and 104 test images. The images were acquired using a 3T MRI scanner and the kidney regions were manually segmented by radiologists.

4. Lungs – This high-contrast dataset consists of 2D X-Ray images of lungs [dataset] [23, 24]. It includes 396 training images, 84 validation images, and 86 test images. The images were collected from a variety of patients with different lung conditions, providing a diverse dataset for training and testing.

5. Thyroid – This low-contrast dataset consists of 3D ultrasound images of the thyroid [dataset] [25]. It includes 3160 training images, 439 validation images, and 510 test images. The images were acquired using a high-frequency linear array transducer and the thyroid regions were manually segmented by experienced clinicians.

6. Nerve – This low-contrast dataset consists of 2D ultrasound images of nerves [dataset] [26]. It includes 1610 training images, 364 validation images, and 349 test images. The images were collected from a variety of patients and the nerve structures were manually annotated by expert radiologists.

3.3 Data Pre-processing

All images in the datasets were pre-processed to ensure consistency and optimal performance of the models. The pre-processing steps included resizing all images to a uniform size of 576 $\times$ 576 pixels. For the 3D datasets, all 2D slices were extracted and used as separate images.

The datasets were split into training, validation, and test sets, with approximately 70% of the images used for training, 15% for validation, and 15% for testing. However, to prevent overfitting, slices from one 3D volume or 2D images from the same patient were included in only one of the train, validation, or test sets. This means that the split is not always exactly in these ratios.

Finally, on some of the smaller datasets random data augmentation was applied in order to improve the absolute results. On each sample, four random combinations of a horizontal flip, vertical flip and rotation with 90, 180 or 270 degrees were applied, where each part of the combination is applied with a probability of 0.5.

3.4 Evaluation Measures

In the realm of image segmentation, five principal metrics are typically utilized to assess performance [27, 28]. The Dice Similarity Coefficient (DSC) serves as a statistical metric, measuring the similarity between two sets by calculating the ratio of twice the intersection area to the total size of both sets. Sensitivity, or recall, appraises the model’s ability to accurately identify positive instances, hence providing insight into the model’s efficacy in segmenting intended areas. Specificity evaluates the model’s proficiency in correctly recognizing negative instances, or in other words, its capability to exclude regions not meant to be segmented. Accuracy gauges the model’s overall correctness in assigning classifications. Lastly, the Jaccard Index (JI) is an intersection-over-union measure that quantifies the similarity between the predicted and actual segmentations, providing a rigorous assessment of model performance in segmenting images.

Moreover, to understand fully the impact of TRF and ERF on model performance, two additional metrics are proposed in this work: ERF rate and object rate. We also factor in the training time (epochs) as a metric, quantifying the epochs needed to attain the lowest validation loss. This allows us to comparatively analyze the training cost across various models.

3.4.1 ERF rate

We propose a new metric called ERF rate to measure the ERF distribution. It quantifies the fraction of significantly contributing pixels within the ERF against the TRF area, utilizing the absolute value of the ERF pixels. The ERF rate (equation 10) accounts for all the meaningful pixels above a certain threshold ($\varepsilon$) in the ERF, giving more weight to pixels with higher values and normalizing the result with the TRF area. The metric is computed for each test image, reporting the mean ERF rate as the overall score.

$$r=\frac{\sum_{y\in\mathbf{E}}[|y|>\varepsilon]\cdot(1+|y|)}{m\cdot n}$$ (10)

We use Kernel Density Estimation (KDE) to find the threshold ($\varepsilon$) for key contributing pixels, estimating the Probability Density Function (PDF) of a continuous variable based on observed samples [29].

The density function $f(x)$ of ERF values can be calculated using the formula in equation 11, where $\hat{f}(x)$ is the estimated PDF, $K(x)$ is a kernel function with bandwidth $h$, and $m\cdot n$ is the number of observations in $\mathbf{E}$. It is centered at each observation $y$.

$$\hat{f}(x)=\frac{1}{m\cdot n}\cdot\sum_{y\in\mathbf{E}}\frac{1}{h}K\left(\frac{x-|y|}{h}\right)$$ (11)

To identify the ideal parameters for KDE, we examined the ERF absolute value histogram for a large dataset sample. It reveals two different types of distributions: (i) ERFs with both contributing and non-contributing pixels have a bimodal distribution with a left peak representing non-contributing pixels and a right peak representing contributing pixels, and (ii) ERFs with mostly non-contributing pixels have a highly positively skewed distribution. The first parameter, the bandwidth ($h$), controls the kernel width and PDF smoothing level. Silverman’s rule-of-thumb [30] was used to automatically determine $h$ ($h=1.06\cdot\hat{\sigma}mn$, where $\hat{\sigma}$ is the standard deviation of sample of size $m\cdot n$), because it performs well on both bimodal and skewed distributions [31].

Finally, the threshold ($\varepsilon$) was selected based on the trough in bimodal distributions or the stop** point of decrease in skewed distributions (Figure 4). To reduce the number of troughs, thus making it easier to find the optimal threshold, a Gaussian kernel function was used to smooth the estimated PDF [29].

Figure 4: Examples of determining the threshold ($\varepsilon$) for the ERF rate with KDE for bimodally distributed ERF pixel values (top row) and positively skewed distributed ERF pixel values (bottom row).

3.4.2 Object rate

In order to assess the relative size of the object to be segmented in comparison to the TRF size, a new metric denoted as object rate is proposed. This metric is computed by dividing the total area of a rectangle encompassing the edges of the object by the total area of the TRF size, or $TRF^{2}$ as defined in equation 2. Thus, for an object with its highest point at $t$, lowest at $b$, leftmost at $l$, and rightmost at $r$, the object rate can be calculated as follows:

$$\text{OR}=\frac{(b-t)\cdot(r-l)}{\text{TRF}^{2}}$$ (12)

4 Results & Discussion

Detailed results of the performance of the U-Net model for the different metrics on all medical datasets can be found in Table 2. Results of the Attention U-Net on the medical datasets and the U-Net on the synthetic datasets of type A and B can be found in A, B, and C respectively. In the following section we present different plots to interpret and discuss these results.

Table 2: All the results for the different evaluation measures on the medical datasets (fetal head, fetal head 2, kidneys, lungs, nerve, and thyroid) for the U-Net.

TRF size	trf54	trf100	trf146	trf204	trf230	trf298	trf360	trf412	trf486	trf570
Fetal head
Training time (epochs)	38	31	21	26	34	21	24	26	28	30
ERF rate before training	0.0135	0.0097	0.0047	0.0011	0.0046	0.0014	0.0044	0.0005	0.0012	0.0009
ERF rate	0.8898	0.9380	0.9153	0.8614	0.8196	0.5970	0.5785	0.4618	0.4175	0.2309
Dice score	0.7752	0.8866	0.9224	0.9527	0.9506	0.9526	0.9623	0.9614	0.9650	0.9665
Object rate	67.3558	16.8389	7.4840	4.0325	2.6942	1.7922	1.0997	0.9870	0.6452	0.4481
Accuracy	0.8687	0.9300	0.9518	0.9690	0.9693	0.9704	0.9749	0.9735	0.9761	0.9773
Sensitivity	0.8458	0.8914	0.9395	0.9508	0.9707	0.9709	0.9675	0.9570	0.9627	0.9680
Specificity	0.8831	0.9530	0.9612	0.9828	0.9739	0.9745	0.9844	0.9878	0.9887	0.9873
Jaccard index	0.6577	0.8152	0.8723	0.9212	0.9210	0.9248	0.9390	0.9371	0.9435	0.9466
Fetal head 2
Training time (epochs)	1	1	1	6	1	8	7	7	25	14
ERF rate before training	0.1531	0.0330	0.0300	0.0097	0.0245	0.0073	0.0163	0.0019	0.0051	0.0054
ERF rate	1.0005	0.0944	0.2275	0.0095	0.0168	0.0051	0.2761	0.0010	0.3159	0.1735
Dice score	0.6009	0.6261	0.7582	0.8745	0.7950	0.8588	0.9028	0.9071	0.9116	0.9214
Object rate	35.5599	8.8900	3.9511	2.1289	1.4224	0.9462	0.5806	0.5211	0.3406	0.2365
Accuracy	0.9889	0.8888	0.9207	0.9557	0.9315	0.9525	0.9654	0.9675	0.9686	0.9725
Sensitivity	0.7058	0.7818	0.8512	0.9133	0.9284	0.9502	0.9567	0.9524	0.9479	0.9616
Specificity	0.9913	0.9004	0.9303	0.9627	0.9316	0.9514	0.9645	0.9679	0.9708	0.9723
Jaccard index	0.5154	0.4764	0.6265	0.7876	0.6841	0.7805	0.8365	0.8445	0.8513	0.8656
Kidneys
Training time (epochs)	21	34	31	32	38	47	54	47	48	54
ERF rate before training	0.1715	0.0496	0.0303	0.0126	0.0287	0.0058	0.0217	0.0024	0.0059	0.0065
ERF rate	0.0123	0.0341	0.0227	0.0057	0.0162	0.0035	0.0088	0.0012	0.0038	0.0038
Dice score	0.7560	0.8367	0.8477	0.8524	0.8617	0.8364	0.8865	0.8657	0.8439	0.8802
Object rate	27.4954	6.8738	3.0550	1.6461	1.0998	0.7316	0.4489	0.4029	0.2634	0.1829
Accuracy	0.9832	0.9904	0.9911	0.9911	0.9917	0.9889	0.9923	0.9902	0.9900	0.9918
Sensitivity	0.7808	0.8856	0.8762	0.8783	0.8645	0.8814	0.8993	0.8914	0.8647	0.8884
Specificity	0.9892	0.9935	0.9945	0.9948	0.9963	0.9916	0.9961	0.9935	0.9942	0.9954
Jaccard index	0.6320	0.7509	0.7679	0.7717	0.7853	0.7490	0.8119	0.7836	0.7607	0.8055
Lungs
Training time (epochs)	15	20	29	26	36	26	51	28	33	40
ERF rate before training	0.0417	0.0419	0.0216	0.0127	0.0122	0.0069	0.0134	0.0009	0.0036	0.0061
ERF rate	0.0614	0.1298	0.0315	0.0040	0.0137	0.0031	0.0245	0.0012	0.0011	0.0039
Dice score	0.9601	0.9673	0.9687	0.9686	0.9683	0.9666	0.9689	0.9683	0.9662	0.9673
Object rate	84.4219	21.1055	9.3802	5.0542	3.3769	2.2463	1.3784	1.2371	0.8087	0.5616
Accuracy	0.9784	0.9823	0.9830	0.9829	0.9829	0.9820	0.9830	0.9827	0.9818	0.9824
Sensitivity	0.9650	0.9677	0.9697	0.9681	0.9695	0.9654	0.9694	0.9654	0.9746	0.9776
Specificity	0.9825	0.9870	0.9872	0.9878	0.9869	0.9873	0.9875	0.9886	0.9834	0.9834
Jaccard index	0.9240	0.9371	0.9398	0.9396	0.9391	0.9361	0.9402	0.9389	0.9353	0.9373
Nerve
Training time (epochs)	7	13	15	8	14	17	10	8	10	12
ERF rate before training	0.1425	0.0444	0.0381	0.0107	0.0287	0.0096	0.0213	0.0019	0.0104	0.0078
ERF rate	0.9312	0.7345	0.6953	0.0057	0.3348	0.0224	0.1363	0.0008	0.1244	0.0399
Dice score	0.4685	0.7329	0.7531	0.7745	0.7792	0.7863	0.7965	0.7951	0.7960	0.7947
Object rate	7.3183	1.8296	0.8131	0.4381	0.2927	0.1947	0.1195	0.1072	0.0701	0.0487
Accuracy	0.9758	0.9848	0.9859	0.9868	0.9873	0.9872	0.9876	0.9878	0.9881	0.9880
Sensitivity	0.6442	0.7519	0.7637	0.7803	0.7978	0.7990	0.7901	0.8068	0.8301	0.8289
Specificity	0.9808	0.9914	0.9923	0.9930	0.9928	0.9930	0.9940	0.9934	0.9927	0.9927
Jaccard index	0.3281	0.6030	0.6321	0.6572	0.6614	0.6701	0.6827	0.6800	0.6786	0.6785
Thyroid
Training time (epochs)	1	1	3	2	2	5	3	2	7	4
ERF rate before training	0.1652	0.0434	0.0280	0.0094	0.0227	0.0093	0.0180	0.0021	0.0089	0.0054
ERF rate	0.1041	0.1439	0.1823	0.0124	0.0482	0.0240	0.0269	0.0038	0.0152	0.0054
Dice score	0.5155	0.5829	0.6456	0.7043	0.7124	0.6907	0.6680	0.6667	0.7457	0.7284
Object rate	14.8609	3.7152	1.6512	0.8897	0.5944	0.3954	0.2426	0.2178	0.1424	0.0989
Accuracy	0.9718	0.9807	0.9840	0.9860	0.9822	0.9864	0.9854	0.9859	0.9871	0.9837
Sensitivity	0.6309	0.7427	0.7374	0.7516	0.7021	0.7563	0.7449	0.7705	0.7602	0.7268
Specificity	0.9788	0.9827	0.9879	0.9912	0.9928	0.9905	0.9884	0.9879	0.9935	0.9933
Jaccard index	0.3881	0.4779	0.5481	0.6107	0.6168	0.6036	0.5785	0.5746	0.6526	0.6246

Figure 5: Performance of the shapes datasets (A and B) for the regular U-Net

Figure 6: Various plots for the analyses of the medical datasets for U-Net (a) DSC Vs TRF size, (b) ERF rate Vs TRF size, (c) Training time (epochs) Vs TRF size, (d) Dice Score Vs TRF, and for Attention U-Net (e) DSC Vs TRF size, (f) ERF rate Vs TRF size, (g) Training time (epochs) Vs TRF size, (h) Dice Score Vs TRF.

TRF = 54	TRF = 230	TRF = 486

(a) Examples on the fetal head dataset.

TRF = 54	TRF = 230	TRF = 486

(b) Examples on the kidneys dataset.

TRF = 54	TRF = 230	TRF = 486

(c) Examples on the nerve dataset.

TRF = 54	TRF = 230	TRF = 486

(d) Examples on the thyroid dataset.

Figure 7: Examples of the TRF (yellow square), ERF (pixels within the TRF) in the top row in each subfigure and the predicted segmentation for various TRF sizes in the bottom row, on the samples from the datasets in Figure 3.

TRF = 54	TRF = 230	TRF = 486

Figure 8: Examples of the TRF (yellow square), ERF (pixels within the TRF) in the top row and the predicted segmentation for various TRF sizes in the bottom row, on the sample from the lungs dataset from Figure 3.

4.1 The Role of Contrast

Figure 5 illustrates the relationship between the TRF size and the DSC for all synthetic shape datasets, encompassing both Type A and Type B for the U-Net model. For all datasets that can be segmented solely based on contrast (A, A large), the model attains perfect performance even at the smallest TRF size (Figure 5 (a)). For datasets, that present an added layer of complexity by either representing only contours of RoI in input images (A contour, A large contour) or by excluding the square from the mask (Type B), require a larger TRF to reach peak performance (Figure 5 (a), Figure 5 (b)). These datasets with an added complexity in segmentation show a model performance trend where DSC starts at a lower point for a small TRF and requires a larger TRF to reach peak performance unlike the consistent perfect performance in the contrast-based datasets.

The same pattern is present in the medical datasets: all datasets which have a low-contrast RoI show a trend of increasing DSC as the TRF size grows, whereas the high-contrast lung dataset attains peak performance starting at the lowest TRF (Figs. 6(a) and 6(e)). The segmentation output for the datasets of fetal head, kidneys, nerve and thyroid for different TRF’s are shown in Figs. 7 (a), 7 (b), 7 (c), and 7 (d) respectively and the combined results for the U-net model are shown in Table-2. Fig. 8 shows the results for the lung dataset for different TRF’s. Since, in the lung dataset, the RoI can be identified visually using the contrast, the DSC attains close to peak value even for a very small TRF. It is clear that the predicted segmentation improves significantly with increasing TRF for all datasets except the lung dataset.

This pattern is further highlighted in Table 2, where all low-contrast datasets consistently show a trend of increasing DSC with TRF, and all high-contrast datasets do not show the same trend.

TRF = 54	100	146	204	230	298	360	412	486	570

Figure 9: Examples of the TRF (yellow square), ERF (pixels within the TRF) in the top row and the predicted segmentation for various TRF sizes in the bottom row, on the sample from the fetal head 2 dataset from Figure 3.

4.2 Optimal TRF Size

In Figs. 6(b) and 6(f), a trend is visible which shows that the ERF rate shrinks with the enlargement of the TRF size for the U-Net and the Attention U-Net respectively. This suggests that as the TRF size increases, a smaller proportion of pixels actually contribute to the predicted segmentation. Moreover, as the TRF size increases, the training time (epochs) also tends to increase as displayed in Figs. 6(c) and 6(g) for the U-Net and the Attention U-Net respectively. This finding implies that an excessively large TRF size may lead to unnecessary computations, potentially explaining the observed increase in training time (epochs) with the expansion of the TRF size.

In this context, the object rate, plotted against the DSC in Figs. 6(d) (U-Net) and 6(h) (Attention U-Net), also seems to play a role. When the object rate, i.e. the size of the RoI relative to the TRF, increases, the DSC degrades. This is corroborated by the two rightmost columns in Table 3, where for most low-contrast datasets where the TRF size plays a major role, the optimal TRF size, i.e. the TRF size at which the DSC saturates, is usually only slightly smaller than the size of the RoI.

Despite the overall trend of increasing DSC with expanding TRF size, we observe slight drops. This can be interpreted in light of the concept of variability in neural networks, as discussed by [32]. Variability, as they define it, represents the richness of landscape patterns in the data space with respect to well-scaled random weights. As the TRF size increases, the model starts to incorporate more global context into its predictions. While this can be beneficial for capturing larger-scale structures in the image, it may also introduce more noise into the model’s predictions, especially if the larger TRF includes irrelevant or distracting features. This could result in slight decreases in the DSC.

4.3 Attention Mechanism and TRF Size

In Figs. 6(a) and 6(e) the TRF size is plotted against the DSC for the U-Net and Attention U-Net respectively for all the medical datasets. In both instances, the trend of an increasing DSC as the TRF grows is present. Fig. 9 shows the segmentation for the fetal head 2 dataset for all the TRF’s and the corresponding TRF and ERF. As the TRF size increases, the segmentation accuracy increases and the overall trend can be seen in Table-2. The same trend is also visible for the Attention U-Net model, the results of which are shown in A. However, all absolute scores are higher in the case of the Attention U-Net. Table-3 column 4 shows the summary of results for all the datasets if they follow the pattern of DSC using Attention U-Net model. All the medical imaging datasets except the lung dataset follow the pattern for both U-Net as well as Attention U-Net model, with a higher absolute scores for attention counterpart. Hence, it can be said that the attention mechanism will consistently improve the performance, regardless of TRF size. Even with attention mechanism, TRF plays an important role and a larger TRF might further improve the performance of Attention U-net models.

4.4 Designing Efficient Architectures

In this work, we performed the experiments for different TRF’s having same number of total parameters for different datasets as can be seen in Table-1. Detailed results of the performance of the U-Net model for the different metrics on all medical datasets can be found in Table 2. Results of the Attention U-Net on the medical datasets and the U-Net on the synthetic datasets of type A and B can be found in A, B, and C respectively. These results show that even for the same number of parameters there is a very high effect on the performance of the network if the TRF is changed. Inclusion of TRF size as a parameter for models can lead to a more fair comparison among their performance. It will also help in designing efficient architectures, ones with optimal trade-off between performance and number of parameters.

Table 3: Summary of the insights from the results. Values with no* mean that the RoI can be identified visually by contrast, but that not all regions which have this contrast are also included in the mask.

Dataset	Dataset Type	RoI can be identified visually only by contrast	Pattern of increasing DSC with TRF	Pattern retained with Attention U-Net, but higher absolute score	Average dimension of RoI	DSC saturates between TRF sizes
Nerve	Clinical	o	Yes	Yes	159	298-360
B contour	Synthetic	No	Yes	N/A	168	100-146
A contour	Synthetic	No	Yes	N/A	169	100-146
Thyroid	Clinical	No	Yes	Yes	187	146-204
B large contour	Synthetic	No	Yes	N/A	237	146-204
A large contour	Synthetic	No	Yes	N/A	242	146-204
Fetal head 2	Clinical	No	Yes	Yes	255	146-204
Fetal head	Clinical	No	Yes	Yes	260	146-204
Kidneys	Clinical	No*	Yes	Yes	101	298-360
B	Synthetic	No*	Yes	N/A	168	54-100
B large	Synthetic	No*	Yes	N/A	238	100-146
Lungs	Clinical	Yes	No	N/A	329	0-54
A	Synthetic	es	No	N/A	168	0-54
A large	Synthetic	Yes	No	N/A	244	0-54

5 Conclusion

This work highlights the essential role of the TRF size in semantic segmentation tasks with U-Net and Attention U-Net architectures across datasets of various modalities. We discovered that an optimal TRF size, the one which balances capturing of global context and computational efficiency, can significantly enhance model performance. This implies that an excessively large TRF size may lead to unnecessary computational costs without corresponding improvements in performance. Additionally, our results also emphasize the added value of the attention mechanism in boosting segmentation accuracy, irrespective of the TRF size.

Our findings suggest that the datasets where RoI can be visually identified by contrast comparison alone, typically attain peak performance with even small TRF. Conversely, this is not the case when additional complexities are present, such as contrast not being only criteria for identifying RoI or contours demarcating RoI. This implies that the model’s performance also depends on factors like the complexity of the task and the size of the RoI relative to the TRF size.

Furthermore, our study indicates that the DSC tends to plateau at a certain TRF size depending on the dataset. This suggests that there exists an optimal TRF size for each dataset, beyond which further expansion of the TRF size does not significantly improve the DSC. These findings can have practical implications for the design of segmentation models, suggesting that increasing TRF size may not always be necessary or beneficial.

These insights provide a valuable reference for designing and optimizing U-Net-based architectures for various tasks and datasets in medical imaging. While our study focused on the U-Net and Attention U-Net architectures, there are many other architectures used for semantic segmentation tasks, such as SegNet [33], PSPNet [34], and DeepLab [35]. Future research could investigate the impact of the TRF size on the performance of these architectures.