BOrg: A Brain Organoid-Based Mitosis Dataset for Automatic Analysis of Brain Diseases

Abstract

Recent advances have enabled the study of human brain development using brain organoids derived from stem cells. Quantifying cellular processes like mitosis in these organoids offers insights into neurodevelopmental disorders, but the manual analysis is time-consuming, and existing datasets lack specific details for brain organoid studies. We introduce BOrg, a dataset designed to study mitotic events in the embryonic development of the brain using confocal microscopy images of brain organoids. BOrg utilizes an efficient annotation pipeline with sparse point annotations and techniques that minimize expert effort, overcoming limitations of standard deep learning approaches on sparse data. We adapt and benchmark state-of-the-art object detection and cell counting models on BOrg for detecting and analyzing mitotic cells across prophase, metaphase, anaphase, and telophase stages. Our results demonstrate these adapted models significantly improve mitosis analysis efficiency and accuracy for brain organoid research compared to existing methods. BOrg facilitates the development of automated tools to quantify statistics like mitosis rates, aiding mechanistic studies of neurodevelopmental processes and disorders. Data and code are available at BOrg’s GitHub page.

1 Introduction

Understanding early human brain development is crucial for deciphering the mechanisms underlying various neurological conditions like genetic disorders and neurodegenerative diseases. However, directly studying the develo** human brain presents significant technical and ethical challenges. Consequently, researchers have heavily relied on animal models, such as rodents, which lack key features of the human brain, including cortical folding believed to be essential for higher cognitive functions [14]. This fundamental limitation hinders our ability to fully comprehend human brain development and its associated diseases.

Brain organoids - 3D cell cultures mimicking early brain development - offer a promising alternative to studying the human brain directly [15, 11]. This approach avoids ethical concerns and enables researchers to investigate brain development and neurodegenerative diseases in a controlled laboratory setting. However, studying these organoids requires extensive domain expertise and is time-consuming. Manual monitoring of the mitotic process is time-consuming and error-prone, representing a major bottleneck in research.

Figure 1: A sample from the BOrg dataset. Left: a large field-of-view (FoV) patch highlights multiple rosette-like zones in the brain organoids, which mimic the cortical folding observed in the human brain. Right: a detailed close-up of a singular rosette-like germinal zone with point and class annotations as well as a close-up of individual cells in various stages of mitosis.

To address this challenge, we propose a deep learning approach to automate and analyze mitosis in brain organoids, ultimately aiding in understanding brain diseases. Mitosis, the process of cell division, plays a critical role in early brain development. Its proper regulation is crucial; any disruptions can lead to tumor formation or developmental disorders. We reformulate the quantification problem into cell identification and counting across different stages of mitosis. The detection and counting information can be used to calculate various statistics and gain deeper insight into brain development and disease.

Our approach involves collecting confocal microscopy images of custom-designed on-chip brain organoids. These brain organoids comprise neural progenitor cells arranged in rosette-like germinal zones that closely mimic early brain development [14]. We then developed an efficient data annotation pipeline that utilizes multiple techniques, such as sparse annotations and collaboration between experts and non-experts, to maximize the efficiency of domain expertise. This effort resulted in the creation of the Brain Organoids (BOrg) dataset, consisting of cell images annotated with 737 instances across four mitotic phases: prophase, metaphase, anaphase, and telophase, as depicted in Fig. 1. Finally, we adapt state-of-the-art deep learning models for cell detection and counting to detect cells and measure mitosis dynamics. By quantifying these cellular processes, we aim to assist researchers in studying early brain development and deciphering the mechanisms of brain diseases.

2 Related Work

Cell Datasets: Several public datasets exist for various cell analysis tasks using microscopic images, such as nuclei detection, cell segmentation, detection, and counting. These datasets cover diverse cell types and conditions as presented in [4] for the cell segmentation challenge. Additionally, histopathology datasets (e.g., TUPAC16 [27], [8], MoNuSAC [26]) and related algorithms have been developed for cell segmentation [29], cell detection [5, 1, 28, 23], and cell counting [25].

Cell Detection: Detecting cells in the biomedical domain is challenging due to their small size and complex appearances. Existing approaches have employed both general purpose and specialized deep learning models such as R-CNN [7], Yolo [21, 13, 12] and RetinaNet [17] and DeGPR [25]. Mitosis detection focuses specifically on identifying nuclei undergoing cell division. Several works have explored multi-stage approaches for mitosis detection [16, 28]. Similarly, several other works have also been proposed for mitosis detection [3, 19, 20, 24].

Cell Counting: General cell counting methods fall under two categories - detection-based and regression-based. Detection-based methods first localize cells using a detector, followed by a simple counting step. DeGPR [25] introduced a detection-based counting method that detected cells using an improved Yolov5 [25] by incorporating supervised contrastive posterior regularization. Similar general object detection models have also been explored [2, 6]. Regression-based methods formulate cell counting as a regression problem, directly predicting the number of cells [10, 18, 9]. However, this approach has limitations when counting different cell classes.

3 BOrg Dataset

The BOrg dataset is a collection of confocal microscopy images of brain organoids. These organoids are specifically designed to mimic the development of the human embryonic brain. The images are systematically captured at regular intervals throughout the development of these organoids with a confocal microscope. Subsequently, each image is efficiently annotated with point annotations corresponding to the mitosis phase, leveraging expert and non-expert skills with an efficient annotation pipeline. Finally, we trained models and demonstrated their effectiveness in detecting and counting cells in different phases. The following section explains our data collection and annotation process in detail in the following section.

Figure 2: Example of cells in four different phases of mitosis. The nuclei are visible with red fluorescence, and the rest of the cell is green fluorescent.

3.1 Data Collection

The dataset consists of microscopy images captured at different phases of mitosis in on-chip brain organoids. These brain organoids are composed of neural progenitor cells arranged in rosette-like germinal zones that closely mimic the neuroepithelium stage of early brain development. These neural progenitor cells are proliferating cells that undergo repeated cycles of mitosis to maintain the progenitor cell population and give rise to newly-born neuronal cells.

These cells have been genetically manipulated such that the nuclei are tagged with red fluorescence, and the rest of the cell is green fluorescent. This helps us to observe the cells during mitosis in real-time by live imaging with confocal microscopy. Fig. 1 shows the multiple rosettes in a large FoV and the mitosis events occurring inside each rosette. The cell cycle of these proliferating progenitor cells can be divided into two broad phases, like most other dividing cells – the Interphase and Mitosis phases.

The elongated cells populating the outer edge of the rosettes are in interphase. When the cell decides to enter the mitosis phase, the nuclei become more rounded than elongated and move towards the center of the rosette, where it undergoes all phases of mitosis in succession, namely, Prophase, Metaphase, Anaphase, and Telophase, giving rise to two daughter cells that move back outwards in the rosette. This characteristic cell cycle coordinated movement of the nuclei in neural progenitor cells is called Interkinetic Nuclear Migration (INM) and contributes to normal human brain development. Hence, studying mitosis at this early developmental stage is crucial as it is the foundation of the birthing process of neurons that will later mature and be functional in the brain.

Phase Name	Train Annotations	Validation Annotations	Total Annotations
Prophase	282	82	364
Metaphase	146	62	208
Anaphase	69	24	93
Telophase	59	13	72

Table 1: Annotation statistics and their split.

The on-chip brain organoids are generated according to the published protocol from human embryonic stem cell line WIBR3 [14]. Some genetic manipulations are done to tag the nuclei with red fluorescence and the remaining cells with green fluorescence for real-time tracing of dividing cells. The organoids were allowed to grow till 14 days of development, following which the tissue dynamics were recorded in a confocal microscope. We recorded cell activities for 16 hours at 5-minute intervals using a 40X objective lens in an Oxford Dragonfly spinning disc confocal microscope. The 16-hour movie is split into 16 movies of 1 hour, each containing approximately 20 to 25 frames. Each video shows multiple cells in different phases of mitotic division. These images are then processed and annotated in the files with the Spots feature of Imaris v.6.2. The actual annotations contain the centroid of the cells that undergo mitosis along with their phase of mitotic division.

3.2 Efficient Data Annotation Pipeline

Traditionally, analyzing these cellular activities requires extensive expertise in annotating microscopic images. Our work addresses this by introducing an efficient annotation pipeline. This pipeline simplifies the annotation task by focusing on relevant information for the final analysis. It leverages a combination of expert and non-expert under a sparse annotation labeling approach, minimizing the need for manual inputs from domain specialists. Finally, a pre-processing step ensures the data is compatible with training machine learning models.

In the first stage, we aim to transform the ultimate objective into a task that is efficient in annotating and enables leveraging progress in deep learning by utilizing off-the-shelf models. Given our goal to quantify various statistics, we label each cell undergoing mitosis, intentionally excluding non-dividing cells from this process. Moreover, since our focus is not on precisely detecting cell boundaries, we adopt a strategy of marking dividing cells with dot annotations rather than employing segmentation masks or detection boxes. This approach of sparsely labeling only the cells in the division process substantially conserves expert time.

We implemented a two-stage labeling strategy to expedite the annotation process and alleviate the burden on domain experts. Initially, a domain expert annotates only the start of division for each cell phase. This task necessitates their specialized knowledge due to the significant resemblance between dividing and non-dividing cells and the difficulty for non-experts to distinguish between various cell phases. In the subsequent stage, following the initial expert annotation, a non-expert annotator fills in the remaining frames, thus completing the annotation process. This strategy significantly reduces the required effort; for example, if a cell completes its division over 30 frames, a domain expert needs to annotate only 4 of these.

Upon completing the annotation phase, the final step involves pre-processing the data to ready it for model training. We decided to use 2D projection of 3D data as it enables efficient training and utilizing considerable progress in 2D models. We use mean projection to convert 3D images into 2D images. Finally, by utilizing this framework, we annotated 262 frames taken for 16 hours of microscopy imaging captured every 5 minutes. This way, we collected 737 annotations for four different phases of mitosis. Detailed statistics of our data are shown in Tab. 1, and a few examples of cells in different phases of mitosis are shown in Fig. 2.

3.3 Benchmarking by Adapting Object Detection and Cell Counting Algorithms

We selected two main categories of models and adapted them for our dataset. These two categories include state-of-the-art (SOTA) off-the-shelf object detection models and algorithms specialized in cell counting. We introduce several modifications to adapt these models for our dataset. These adaptations include converting point annotation to bounding boxes, calculating implicit features, joint pre-training of backbones, etc.

Model	Precision	Recall	mAP	MAE Pro	MAE Meta	MAE Ana	MAE Telo
FasterRCNN [22]	0.364	0.385	0.384	1.564	1.512	0.615	0.333
RetinaNet 5 [17]	0.321	0.215	0.310	2.1032	1.59	0.615	0.333
YOLOv8 [12]	0.518	0.453	0.461	1.308	0.718	0.436	0.359
YOLOv5 [13]	0.563	0.373	0.426	1.589	1.051	0.436	0.410
YOLOv3 [21]	0.677	0.245	0.399	1.103	1.051	0.564	0.308
DeGPR++ [25]	0.535	0.445	0.449	1.487	0.795	0.385	0.308

Table 2: Comparison of detection and counting performance of various state-of-the-art models adapted for the BOrg dataset. Precision, Recall, mAP (Mean Average Precision), and class-wise MAE (Mean Average Error) are utilized for comparison. DeGPR++ is our adaptation of DeGPR (cell counting algorithm) [25].

Adapting Off-the-Shelf Detection Models for BOrg. For our dataset, object detection models can be utilized to compute useful insights. Hence, we first benchmark the BOrg dataset with off-the-shelf object detection algorithms. As discussed earlier, we transformed dot annotation into detection boxes by utilizing class-wise average cell diameter. Object detection models require detection bounding boxes rather than the singular point annotations present in our dataset. We computed an average diameter for each cell per class to address this discrepancy and used this value to generate synthetic object bounding boxes. This approach is justified by our objective, which requires the identification rather than the precise detection of different cell types.

Given an image I as input, the objective is to predict bounding boxes, $B=\{B_{1},B_{2},..,B_{n}\}$ and corresponding classes $c=\{c_{1},c_{2},...,c_{n}\}$. The detector is trained on the ground truth data, where point annotations are converted into box annotations by using the point as the center and drawing a box square box around with an average class diameter.

To see the effectiveness of different families of object detection methods, we choose three different types: single-stage object detectors [17], two-stage object detectors [22] and YOLO [21, 13, 12] family which allows efficient processing on input images.

Adapting DeGPR for Cell Counting. In addition to object detection models, we adapt DeGPR [25], a state-of-the-art cell detection and counting algorithm, to the BOrg dataset. DeGPR formulates cell counting as a detection problem and employs posterior regularization during the training of YOLOv5 [13]. The key idea is to leverage both explicit and implicit discriminative features to train the detector with an additional posterior regularization loss. Handcrafted attributes such as cell size and intensity are extracted from the bounding box regions for explicit features. A ResNet18 encoder computes implicit features by embedding the cropped bounding box patches. The encoder is trained with a supervised contrastive loss to generate class-discriminative embeddings. These explicit and implicit features are concatenated and fitted with a Gaussian Mixture Model (GMM). KL divergence is applied between the GMM fits of the true and predicted bounding boxes to enforce consistency. To adapt DeGPR to the BOrg dataset, we first transform the point annotations to bounding boxes using average cell diameters. We then compute explicit handcrafted features from the training set and incorporate them into the model. Finally, we pre-train the embedding encoder on the BOrg dataset to learn discriminative representations tailored to our task. These modifications enable DeGPR to operate effectively on the sparse annotations and unique challenges posed by the BOrg dataset.

Figure 3: (a) Class-wise mean average precision (mAP) of different detection models. (b) Comparison of the effect of different projections on final mAP. (c) Effect of three different pre-training strategies for DeGPR performance on BOrg dataset.

4 Experiments

Evaluation metrics. We utilize detection and counting evaluation metrics since the purpose of our dataset is to calculate various subcellular statistics for later analysis. Specifically, we use precision, recall, and mAP@50 (mean average precision), which measures the accuracy of detection models by considering precision and recall simultaneously, and MAE (mean average error), which quantifies the discrepancy between predicted and ground truth counts of cells at different phases of mitosis.

Dataset. For annotation, we utilize Imaris , a microscopy analysis software that enables 3D annotation of cells. In addition to benchmarking our models on our proposed dataset, we also leverage MoNuSAC [26] as a pre-training dataset and demonstrate successful reproduction of DeGPR. MoNuSAC is a histopathology cell dataset comprising four classes (Epithelial, Lymphocyte, Neutrophil, and Macrophage) across 295 images, with 209 for training and 85 for testing. To maintain consistency, we resize each image to 640x640 pixels, irrespective of its original dimensions. Our novel dataset, BOrg, consists of confocal microscopy frames with an initial size of 654x588 pixels, which we resize to 640x640 for compatibility with our models. The dataset encompasses 262 images, randomly divided into an 80-20 train-validation split (see Tab. 1).

Implementation Details. For both MoNuSAC and BOrg datasets, we trained detection models for 300 epochs using the SGD optimizer with a training batch size of 32. We employed a learning rate of 0.03 for the MoNuSAC dataset and 0.0005 for the BOrg dataset. In order to find the implicit features for performing posterior regression in DeGPR, the encoder is trained for 300 epochs at a learning rate of 0.001. The model generates 512-dimensional embeddings, which we then reduce using PCA by preserving 90% of variance. Additionally, to validate the correct implementation of DeGPR, we initially replicated it on the MoNuSAC dataset, achieving a mean average precision (mAP) close to the reported values (0.481 compared to 0.489). The experiments are conducted on a Quadro RTX 6000 GPU.

Figure 4: Qualitative comparison of detection and cell counting models adapted for BOrg dataset.

Results. Table 2 presents the detection and counting performance of various state-of-the-art models adapted for the BOrg dataset. Among the different model families, we observe that the efficient YOLO models significantly outperform single-stage (RetinaNet5 [17]) and two-stage (Faster RCNN [7]) object detectors in both detection and counting tasks. The YOLO models strike a favorable balance between accuracy and efficiency, making it suitable for practical analysis tools on consumer hardware. Furthermore, despite slightly lower detection scores, DeGPR [25] performs well in counting tasks due to its specialized training for cell classification. Figure 3(a) illustrates the class-wise mean Average Precision (mAP) of the top three models, highlighting the impact of class distribution on overall performance.

To improve the models’ effectiveness, we investigate the role of different preprocessing techniques and training strategies through ablation studies. Figure 3(b) compares the effect of various 3D-to-2D projection methods on the final mAP. We select the color mean projection based on these results for subsequent analysis. Additionally, Figure 3(c) evaluates three different pre-training strategies for DeGPR, with the combined pre-training on both BOrg and MoNuSAC datasets yielding the best performance.

Overall, our results demonstrate the effectiveness of adapting off-the-shelf object detection and cell counting methods to the BOrg dataset, showcasing their potential to aid researchers in understanding embryonic brain development and associated disorders through automated analysis of mitotic events.

5 Conclusion

We introduce BOrg, a novel brain organoid-based dataset designed to support the analysis of early brain development. This dataset comprises images collected via confocal microscopy of on-chip brain organoids that recapitulate critical aspects of the embryonic development of the human brain in its early stage. Subsequently, the data is annotated using our efficient annotation mechanism, which effectively employs a blend of expert and non-expert input. Finally, we adapt several object detection and cell counting models on this dataset to show its potential in assisting analysis in deciphering brain mechanisms.