High-resolution segmentations of the hypothalamus and its subregions for training of segmentation models

Abstract

Segmentation of brain structures on magnetic resonance imaging (MRI) is a highly relevant neuroimaging topic, as it is a prerequisite for different analyses such as volumetry or shape analysis. Automated segmentation facilitates the study of brain structures in larger cohorts when compared with manual segmentation, which is time-consuming. However, the development of most automated methods relies on large and manually annotated datasets, which limits the generalizability of these methods. Recently, new techniques using synthetic images have emerged, reducing the need for manual annotation. Here we provide HELM, Hypothalamic ex vivo Label Maps, a dataset composed of label maps built from publicly available ultra-high resolution ex vivo MRI from 10 whole hemispheres, which can be used to develop segmentation methods using synthetic data. The label maps are obtained with a combination of manual labels for the hypothalamic regions and automated segmentations for the rest of the brain, and mirrored to simulate entire brains. We also provide the pre-processed ex vivo scans, as this dataset can support future projects to include other structures after these are manually segmented.

Background & Summary

The hypothalamus is a small cone-shaped region located in the medial part of the brain, comprising small subnuclei containing the cell bodies of various neuron subtypes. It plays a significant role in maintaining body homeostasis and regulating sleep, body temperature, appetite, and emotions  [1, 2, 3]. Many authors have already linked hypothalamic volume variation with numerous conditions and diseases such as Alzheimer’s disease [4], Huntington’s disease [5, 6], Behavioral Variant Frontotemporal Dementia (bvFTD)  [7, 8], Amyotrophic Lateral Sclerosis (ALS) [9, 10], among others [11, 12, 13, 14]. However, to be able to measure volume differences in the hypothalamus, all these neuroimaging studies require the delineation of hypothalamic boundaries. Despite being considered the gold standard, manual annotation is a costly and time-consuming task, requiring neuroanatomical expertise and dedicated resources. When applied to the hypothalamus, the challenges are further compounded by its small size and the limited contrast it exhibits with neighboring tissues. This difficulty poses an obstacle to the development of large-scale studies on this brain structure.

Large-scale brain imaging studies using magnetic resonance imaging (MRI) have large potential to enhance our understanding of the human brain in both health and disease conditions. However, these studies are often limited by the need for manual annotation. Automated segmentation methods have been developed to circumvent this problem and allow for a greater quantity of data to be utilized. Existing methods include classical atlas-based approaches [15, 16] and more modern deep learning networks [17, 18]. While deep learning showcases outstanding performance in segmentation problems, these methods only work on a specific type of MRI modality and resolution. For instance, we can find a few automated segmentation models for the hypothalamus focusing on T1w [19, 20, 21], however only one is able to segment both T1w and T2w images [22]. So far, models trained solely with T1-weighted MR images do not perform effectively on other MRI modalities (T2w, PD, FLAIR, and others) due to the so-called “domain gap”.

While domain adaptation techniques [23] can sometimes mitigate this problem, it does not fully close the domain gap. More recently, synthetic image-based methods have been proposed to obtain networks that generalize well across different dataset [24, 25, 26, 27, 28]. Recently, Billot et al proposed Synthseg[27], a new segmentation method trained solely with synthetic images derived from label maps. By randomizing the appearance and resolution of the synthetic scans continuously during training, SynthSeg can readily segment images of any contrast and resolution during testing, without retraining

Therefore, in this article, we introduce a dataset of 3D label maps that may be used to train deep-learning-based networks using synthetic data. The presented dataset is derived from 10 post mortem ultra-high MRI acquisitions of brains provided by the Distributed Archives for Neurophysiology Data Integration (DANDI Archive) [29]. Using this sample of post mortem MRI as a starting point, we trained a deep learning network capable of segmenting the hypothalamus and its subregions across various MRI modalities and resolutions.

The label maps used for the network training are openly available for further research. Also, we provide the pre-processed ex vivo scans, as this dataset has the potential to be extended to different brain structures by manually segmenting them

Methods

Ultra-high resolution ex vivo MRI

HELM is derived from 10 post mortem ultra-high MRI acquisitions of brains from the public DANDI Archive [29]. The post mortem images are openly available and comply with all relevant ethical regulations [30]. The MRI of the ex vivo brain hemispheres was obtained using multi-echo fast low-angle shot (ME FLASH or MEF) on a 7 T Siemens MR scanner with Repetition Time (TR) of 34 ms, time to echo (TE) of 5.65, 11.95, 18.25, and 24.55 ms, and field of view (FOV) of 192 mm by 81.3 mm. Before the MRI, the specimens were fixed in 10% formalin for a minimum of 90 days and packed in a 2% buffered paraformaldehyde solution. The images present an equal distribution of 5 male and 5 female control specimens of individuals who died of natural causes with no clinical diagnoses or neuropathology. As the images come from post mortem brains, the acquisition is free from motion artifacts and has a high resolution, ranging from 120 to 150$\mu$m isotropic, which improves the visualization of hypothalamic boundaries. Further details on the specimens and MRI acquisition can be found in the original publication [30]. The age at the time of death ranges from 54 to 79 years, with an average of 66.4 $\pm$ 8.46 years and they include only a single hemisphere of the brain (four right, six left). In practice, we left-right flip the right hemispheres and work with 10 left hemispheres. This is common practice when working with ex vivo datasets with high resolution but limited sample sizes, e.g.,  [31, 32]. We note that combining left and (flipped) right hemispheres into a single model has negligible impact on the subsequent training of machine learning models, due to the strong lateral symmetry of the hypothalamus [33] and to the aggressive geometric augmentation that is often used in training.

Using these MRI images as a starting point, we performed data pre-processing and generated automated whole-brain segmentation using unsupervised clustering. Unlike the hypothalamic labels, these automated segmentations of the rest of the brain are used only for image synthesis purposes and not as segmentation targets. Therefore, they can be noisy and not correspond directly to brain structures – so unsupervised clustering suffices.

The focus is on capturing context around the hypothalamus to produce synthetic intensities. Subsequently, we mirrored the hemispheres to create label maps that would serve as synthetic images for the input of the deep neural network without having to segment scans into hemispheres during testing.

Pre-processing of ex vivo scans

The first step of the method was the pre-processing of the images. The primary objective at this stage is to standardize the dataset and remove any background elements that could interfere with the subsequent steps. (Fig. 1).

• Orientation: Given that the primary objective of the dataset is to facilitate the development of automated segmentation methods, it was essential to ensure that all images were uniformly oriented. We decided to use the positive RAS (right-anterior-superior) standard. While our dataset includes images of both hemispheres, we left-right flipped all right hemispheres and with the idea of training neural networks to process left hemispheres; these networks can be used to analyze right hemispheres by simply flip** the input MRI scan.

• Background segmentation: The ex vivo brains were packaged in a bag for scanning, which is discernible in the images and undesirable in our model. To address this issue, we employed a Bayesian automated image segmentation approach adaptive to contrast [34] to create a mask, which was then utilized to eliminate non-cerebral elements that were not related to brain structures.

• Voxel resampling: The original images have a voxel resolution ranging from 0.1 to 0.15$mm^{3}$. While this resolution assists in distinguishing structures during manual segmentation, it significantly prolongs image processing, particularly when the ultimate goal is to employ them in deep learning networks. Therefore, we adjusted the voxel resolution to a constant resolution equal to $0.3\times 0.3\times 0.3mm$ isotropic, which provides a compromise between a high level of details and being storage- and processing-friendly

• Bias Field Correction: Finally, the last step in image pre-processing is the bias field correction [34]. This step is essential, as in the generation of the whole brain segmentation, we utilize an unsupervised clustering method that can be directly affected by the bias field.

Figure 1: Examples of original ex vivo images (up) and images after the pre-processing steps (down). First, we re-orient the images to positive RAS standard and remove non-cerebral elements from the background. Then, we resample the images voxels to $0.3\times 0.3\times 0.3mm$ isotropic and perform bias field correction.

Segmentation

In the second stage, we generated the label maps for the hypothalamus, its subregions, and the whole brain:

• Hypothalamus manual segmentation: All images were traced by one non-specialist trained by MFJ. For manual segmentation, we relied on protocols focused on in vivo images as described in the literature. In particular, we followed the criteria described in Rodrigues et al [19] (whole hypothalamus) and Bocchetta et al [7] (for the subregions). At 0.3$mm$ isotropic resolution, approximately 40-50 coronal slices include the hypothalamus.

  • Whole structure: The segmentation of the whole hypothalamus occurs on coronal view. To ensure the correct delineation of the landmarks, we also simultaneously inspected the sagittal and axial views. In in vivo images, the hypothalamus lies around the third ventricle. However, In the case of the ex vivo images, is not always possible to distinguish the third ventricle, since we only have one hemisphere of the brain. Therefore, on the coronal view, we use the recess dorsal to the hypothalamus to define its most superior boundary (Fig 2).

    

    Figure 2: Recess of the hypothalamus used for the delineation of superior boundary. Tub_sup = Tuberal Superior subregion;Tub_inf = Tuberal inferior subregion

    Ventrally, the hypothalamus is defined by the optic tract and the hypothalamic sulcus in the most rostral slices. The most anterior coronal slice is defined as the one where the anterior commissure is visible, while the most caudal coronal slice is where the mammillary bodies (MB) are no longer visible. The mammillary bodies were included in the segmentation, while the fornix and optical tract were excluded from the segmentation.

  • Subregions: We subdivided the hypothalamus into 5 subregions: Anterior inferior, Anterior Superior, Tuberal inferior, Tuberal Superior, and Posterior. The manual segmentation of the whole hypothalamus (its outer boundaries with the surrounding structures) relied completely on image contrast and anatomical landmarks. However, the division of the internal subregions relied on a combination of image intensities and geometric criteria, specifically for the subdivision of their rostro–caudal and dorso–ventral borders. Specifically, these geometric criteria included horizontal lines relying on anatomical landmarks, which have been defined and validated in Bocchetta et al [7] (see Table 1). As we only have one hemisphere for each brain, we could not segment both hypothalami for each subject. The most rostral slice of the anterior subregion coincides with the most rostral slice of the hypothalamus when looking at the coronal plane. On in vivo images, the anterior subregion is included from the most rostral coronal slice of the hypothalamus to the most rostral part of the infundibulum. However, for ex vivo images we used the anterior commissure visible from the sagittal view as a landmark to delineate the most caudal part of the anterior regions (Fig. 3(a)). The tuberal subregions begin posteriorly to the coronal slice where the anterior regions are visible (as defined by the anterior commissure sagitally) and extend to the most rostral slice where the MB are visible, which are included in the posterior subregion (Fig. 3(b)). To delineate the superior and inferior portions of both the anterior and tuberal subregions, we drew a horizontal line on the coronal slice connecting the most medial to the most lateral point of the hypothalamus (Fig. 3(c,d)).

    Table 1: Landmarks used for hypothalamus division into subregions

    	Anterior	Tuberal
    Most rostral	Most rostral coronal slice where	Posterior to last coronal slice where the anterior
    landmarks	the hypothalamus is visible	subregions are visible (defined by the
    		anterior commissure on the sagittal view (Fig. 3(a,b)))
    Most caudal	Defined by the anterior	First coronal slice just rostral to the slice where
    landmarks	commissure on the sagittal view	the mammillary bodies are clearly visible
    Superior/Inferior	Defined on all coronal slices by a horizontal line connecting the most
    landmarks	medial to the most lateral point of the hypothalamus (Fig. 3(c,d))
    Lateral/medial	Defined as the same boundaries of the
    landmarks	whole hypothalamus

    

    Figure 3: Segmentation protocol: (a) Subregions delineation: Anterior superior (pink), anterior inferior (red), tuberal superior (yellow), tuberal inferior (blue) and posterior (green) (b)Sagittal landmarks for subregion delineation (AC = Anterior Comissure and MB=mamillary body) (c) Coronal View of superior and inferior tuberal subregions (d) Coronal landmark for superior/inferior division

• Whole brain segmentation: Unlike the hypothalamic subregions, semantic meaning is not necessary for the labels of the rest of the brain – since they are only used to generate synthetic image contrast, and not as segmentation targets. That means we do not need to correctly delineate the morphological borders of the other structures. We just need to establish context around the hypothalamus to generate the synthetic images. We decided to use k-means, a non-supervised clustering method, to model the non-hypothalamic tissue based on grayscale levels of the pre-processed images. Seeking to increase data variability, we ran this step for values of k ranging from 4 to 9. As a result, for each of the 10 images, we have 6 different maps, totaling 60 label maps (Fig. 4)

  

  Figure 4: Examples of three different label maps derived from the same image. From left to right: $k=4$, $k=6$, $k=8$

• Segmentation merge: Finally, it is necessary to merge both segmentations. In this stage, we needed to ensure that there would be no discrepancy during the overlay process. To achieve this, we employed mathematical morphology (specifically, a closing operator with a spherical structuring element, radius = 1.2mm) to refine the delineation of the fornix and eliminate false positive voxels in the third ventricle area (Fig. 5).

Figure 5: (a) Whole brain segmentation: example with k=4 (b) Manual segmentation simply overlap** the whole brain segmentation. We can see that there are a few inconsistent voxels, that should be labeled either as hypothalamus or background that have different labels. (c) To fix these inconsistencies, a mathematical morphology-based algorithm is applied.

Hemisphere mirroring

The ex vivo images were acquired from single brain hemispheres. Given that the hypothalamus is situated in the most medial part of the brain, a model covering a single hemisphere lacks contextual information regarding its surroundings. To address this concern, we mirrored the images to generate a complete brain. In the mirroring process, two major concerns needed to be addressed: the original and mirrored hemispheres should not overlap, and they should be as close as possible to each other (Fig. 6).

To tackle this issue, we encode the two constraints into softened versions and combine them into a weighted cost function that is optimized with respect to a rigid transformation T:

$$\hat{T}=argmin_{T}\sum_{i\in\Omega}\delta[x(v_{i};T)>0]x(v_{i};T)-\alpha\sum_{i\in\Omega}x(v_{i};T)$$

where $\Omega$ is the brain mask, $x(v_{i};T)$ is the $x$ real-world coordinate after rigid transformation with $T$, $\delta$ is Kronecker’s delta, and $\alpha$ a trade-off value that changes according to the image. The rationale behind the cost function is the following: assuming a virtual mirror located on the plane $x=0$, we want positive values of $x$ to be penalized (Fig. 6b) so that the hemisphere does not cross the mirror (i.e., the real and mirrored hemispheres do not overlap, as in Fig. 6d). This soft constraint corresponds to the first term of the equation. At the same time, the hemisphere should not be too distant from the mirror at $x=0$ (Fig. 6c), to prevent gaps between the real and mirrored hemispheres(Fig. 6e). This is encoded into the second term of the equation – which also includes relative weight $\alpha$ that represents a trade-off between the two terms. After optimization, we mirror the transformed hemisphere and merge the real and mirrored hemispheres into one image, ending up with 10 subregions (five from each hemisphere, see Fig. 6f).

Figure 6: Label map creation: Following the segmentation step, a half-brain label map is generated (a). However, given the hypothalamus’s central location within the brain, mirroring is essential to provide contextual information. For the mirroring process, translation and rotation are applied to the RAS coordinates. This involves moving the brain close to the x=0 axis from the negative side, without surpassing into the positive side. Essentially, a final cost function is computed, penalizing positive values of x (b) and high negative values. Finally, we prevent the overlap between brain hemispheres (d) and also prevent them from ending up at unnaturally distant positions (e). After the optimization and mirroring, we end up with the final label map (f).

Data Records

HELM is available at https://www.nitrc.org/projects/hsynex_data/(release 0.1) [35] under CC-BY license. The data is in NIfTI format and the image naming system corresponds to the original dataset. The following files are provided:

• preprocessed_exvivo_images.zip: For reproducibility purposes, and to facilitate the development of segmentation methods for other surrounding brain structures, we provide in this file the 10 pre-processed ex vivo scans (including reorientation, background segmentation, and bias field correction); see further details in the Code availability Section

• hypothalamus_manual_seg.zip: Manual segmentation of the hypothalamus and its subregions of the ex vivo images following the protocol explained in the Segmentation section.

• whole_brain_segmentation.zip: For each subject, there are six whole brain segmentation for context. They are available to be merged with the manual segmentation

• label_maps_1.zip and label_maps_2.zip: Our final label maps using the manual segmentation of the hypothalamus

Technical Validation

Finally, we assessed the data quality and usability of the dataset in training neural networks.

• Manual segmentation quality assessment: To test the intra-rater reliability of the manual segmentations, one LMR (trained by MFJ) manually segmented 5 out of the 10 images a second time, using the same protocol. The images were randomly selected from the total sample and were re-segmented blindly, with the same software and computer settings, after four months from the first manual segmentation. For evaluation, we used the Dice Coefficient and Average Hausdorff Distance [36], which are summarized in Table 2. We note that the low image contrast between adjacent hypothalamic subregions and their small size lead to generally low intra-rater Dice. For example, Billot et al [20] reported Dice scores between 0.70 and 0.87 for different subregions of the hypothalamus. Estrada et al [22] reported an intra-rater Dice coefficient of 0.82 for the whole hypothalamus. In both studies, the authors highlighted the challenges associated with manual delineation due to the small size and low contrast of the hypothalamus.

  Table 2: Intra-rater metrics for 5 subjects.

  	Dice Coefficient	Average Hausdorff Distance
  Whole	0.82$\pm$0.03	0.37$\pm$0.14
  Anterior	0.65$\pm$0.2	0.84$\pm$0.75
  Tuberal	0.78$\pm$0.05	0.43$\pm$0.17
  Posterior	0.79$\pm$0.04	0.32$\pm$0.08

• Usability: The dataset’s usability assessment involved implementing a deep learning-based network method for hypothalamus segmentation. We generated synthetic images as Gaussian mixture models (with random means and variances) conditioned on the geometrically augmented labels and blurred them with random kernels to simulate images of different orientations and slice thicknesses [27]. We used these images and the corresponding hypothalamic labels to train two cascaded 3D U-Nets [37]: one segmenting the whole hypothalamus, and a second subdividing into subregions. Model validation was conducted using a validation set comprising five different MR sequences (T1, T2, PD, FA, and qT1). The random parameter sampling ensures generalization to any contrast and resolution in the input data [27]. For evaluation, we used in vivo MRI from two publicly available datasets: IXI [38] and ADNI [39]. Further information regarding the methodology, such as the datasets utilized for validation and testing, as well as quantitative evaluation metrics, are available in Rodrigues et al. [40]

Figure 7: Hypothalamus and subregions segmentation on in vivo images using the method trained on the synthetic dataset. The method was capable of segmenting the hypothalamus in T1w, T2w, PD, and FLAIR sequences, the last one presenting a spacing of 5$mm$

Usage Notes

The label maps described here could be employed for training different networks dedicated to the hypothalamus or different brain structures. To achieve this, three steps need to be followed:

• Manual segmentation: Manually segment the target structure. Given that there are only 10 images, the manual segmentation will not demand as much effort as the ones typically used in supervised learning.

• Merge with whole brain segmentation: Merge the manual segmentation with the provided whole brain segmentation.

• Mirroring: Run the provided mirroring codes to generate the final label maps.

We also encourage the usage of this dataset in tasks other than segmentation. For instance, we can find in the literature the use of synthetic images applied to registration [41] and conversion of different MRIs into high-resolution T1 scans [28].

Code availability

The 10 pre-processed textitex vivo images, automated brain segmentations, manual hypothalamus segmentations, as well as the final label maps used for training the hypothalamus segmentation model are available on https://www.nitrc.org/projects/hsynex_data/. The codes for creating the label maps are available on https://github.com/liviamarodrigues/hsynex