-
On the Value of PHH3 for Mitotic Figure Detection on H&E-stained Images
Authors:
Jonathan Ganz,
Christian Marzahl,
Jonas Ammeling,
Barbara Richter,
Chloé Puget,
Daniela Denk,
Elena A. Demeter,
Flaviu A. Tabaran,
Gabriel Wasinger,
Karoline Lipnik,
Marco Tecilla,
Matthew J. Valentine,
Michael J. Dark,
Niklas Abele,
Pompei Bolfa,
Ramona Erber,
Robert Klopfleisch,
Sophie Merz,
Taryn A. Donovan,
Samir Jabari,
Christof A. Bertram,
Katharina Breininger,
Marc Aubreville
Abstract:
The count of mitotic figures (MFs) observed in hematoxylin and eosin (H&E)-stained slides is an important prognostic marker as it is a measure for tumor cell proliferation. However, the identification of MFs has a known low inter-rater agreement. Deep learning algorithms can standardize this task, but they require large amounts of annotated data for training and validation. Furthermore, label nois…
▽ More
The count of mitotic figures (MFs) observed in hematoxylin and eosin (H&E)-stained slides is an important prognostic marker as it is a measure for tumor cell proliferation. However, the identification of MFs has a known low inter-rater agreement. Deep learning algorithms can standardize this task, but they require large amounts of annotated data for training and validation. Furthermore, label noise introduced during the annotation process may impede the algorithm's performance. Unlike H&E, the mitosis-specific antibody phospho-histone H3 (PHH3) specifically highlights MFs. Counting MFs on slides stained against PHH3 leads to higher agreement among raters and has therefore recently been used as a ground truth for the annotation of MFs in H&E. However, as PHH3 facilitates the recognition of cells indistinguishable from H&E stain alone, the use of this ground truth could potentially introduce noise into the H&E-related dataset, impacting model performance. This study analyzes the impact of PHH3-assisted MF annotation on inter-rater reliability and object level agreement through an extensive multi-rater experiment. We found that the annotators' object-level agreement increased when using PHH3-assisted labeling. Subsequently, MF detectors were evaluated on the resulting datasets to investigate the influence of PHH3-assisted labeling on the models' performance. Additionally, a novel dual-stain MF detector was developed to investigate the interpretation-shift of PHH3-assisted labels used in H&E, which clearly outperformed single-stain detectors. However, the PHH3-assisted labels did not have a positive effect on solely H&E-based models. The high performance of our dual-input detector reveals an information mismatch between the H&E and PHH3-stained images as the cause of this effect.
△ Less
Submitted 28 June, 2024;
originally announced June 2024.
-
Deep Learning model predicts the c-Kit-11 mutational status of canine cutaneous mast cell tumors by HE stained histological slides
Authors:
Chloé Puget,
Jonathan Ganz,
Julian Ostermaier,
Thomas Konrad,
Eda Parlak,
Christof Albert Bertram,
Matti Kiupel,
Katharina Breininger,
Marc Aubreville,
Robert Klopfleisch
Abstract:
Numerous prognostic factors are currently assessed histopathologically in biopsies of canine mast cell tumors to evaluate clinical behavior. In addition, PCR analysis of the c-Kit exon 11 mutational status is often performed to evaluate the potential success of a tyrosine kinase inhibitor therapy. This project aimed at training deep learning models (DLMs) to identify the c-Kit-11 mutational status…
▽ More
Numerous prognostic factors are currently assessed histopathologically in biopsies of canine mast cell tumors to evaluate clinical behavior. In addition, PCR analysis of the c-Kit exon 11 mutational status is often performed to evaluate the potential success of a tyrosine kinase inhibitor therapy. This project aimed at training deep learning models (DLMs) to identify the c-Kit-11 mutational status of MCTs solely based on morphology without additional molecular analysis. HE slides of 195 mutated and 173 non-mutated tumors were stained consecutively in two different laboratories and scanned with three different slide scanners. This resulted in six different datasets (stain-scanner variations) of whole slide images. DLMs were trained with single and mixed datasets and their performances was assessed under scanner and staining domain shifts. The DLMs correctly classified HE slides according to their c-Kit 11 mutation status in, on average, 87% of cases for the best-suited stain-scanner variant. A relevant performance drop could be observed when the stain-scanner combination of the training and test dataset differed. Multi-variant datasets improved the average accuracy but did not reach the maximum accuracy of algorithms trained and tested on the same stain-scanner variant. In summary, DLM-assisted morphological examination of MCTs can predict c-Kit-exon 11 mutational status of MCTs with high accuracy. However, the recognition performance is impeded by a change of scanner or staining protocol. Larger data sets with higher numbers of scans originating from different laboratories and scanners may lead to more robust DLMs to identify c-Kit mutations in HE slides.
△ Less
Submitted 2 January, 2024;
originally announced January 2024.
-
Domain generalization across tumor types, laboratories, and species -- insights from the 2022 edition of the Mitosis Domain Generalization Challenge
Authors:
Marc Aubreville,
Nikolas Stathonikos,
Taryn A. Donovan,
Robert Klopfleisch,
Jonathan Ganz,
Jonas Ammeling,
Frauke Wilm,
Mitko Veta,
Samir Jabari,
Markus Eckstein,
Jonas Annuscheit,
Christian Krumnow,
Engin Bozaba,
Sercan Cayir,
Hongyan Gu,
Xiang 'Anthony' Chen,
Mostafa Jahanifar,
Adam Shephard,
Satoshi Kondo,
Satoshi Kasai,
Sujatha Kotte,
VG Saipradeep,
Maxime W. Lafarge,
Viktor H. Koelzer,
Ziyue Wang
, et al. (5 additional authors not shown)
Abstract:
Recognition of mitotic figures in histologic tumor specimens is highly relevant to patient outcome assessment. This task is challenging for algorithms and human experts alike, with deterioration of algorithmic performance under shifts in image representations. Considerable covariate shifts occur when assessment is performed on different tumor types, images are acquired using different digitization…
▽ More
Recognition of mitotic figures in histologic tumor specimens is highly relevant to patient outcome assessment. This task is challenging for algorithms and human experts alike, with deterioration of algorithmic performance under shifts in image representations. Considerable covariate shifts occur when assessment is performed on different tumor types, images are acquired using different digitization devices, or specimens are produced in different laboratories. This observation motivated the inception of the 2022 challenge on MItosis Domain Generalization (MIDOG 2022). The challenge provided annotated histologic tumor images from six different domains and evaluated the algorithmic approaches for mitotic figure detection provided by nine challenge participants on ten independent domains. Ground truth for mitotic figure detection was established in two ways: a three-expert consensus and an independent, immunohistochemistry-assisted set of labels. This work represents an overview of the challenge tasks, the algorithmic strategies employed by the participants, and potential factors contributing to their success. With an $F_1$ score of 0.764 for the top-performing team, we summarize that domain generalization across various tumor domains is possible with today's deep learning-based recognition pipelines. However, we also found that domain characteristics not present in the training set (feline as new species, spindle cell shape as new morphology and a new scanner) led to small but significant decreases in performance. When assessed against the immunohistochemistry-assisted reference standard, all methods resulted in reduced recall scores, but with only minor changes in the order of participants in the ranking.
△ Less
Submitted 31 January, 2024; v1 submitted 27 September, 2023;
originally announced September 2023.
-
Nuclear Pleomorphism in Canine Cutaneous Mast Cell Tumors: Comparison of Reproducibility and Prognostic Relevance between Estimates, Manual Morphometry and Algorithmic Morphometry
Authors:
Andreas Haghofer,
Eda Parlak,
Alexander Bartel,
Taryn A. Donovan,
Charles-Antoine Assenmacher,
Pompei Bolfa,
Michael J. Dark,
Andrea Fuchs-Baumgartinger,
Andrea Klang,
Kathrin Jäger,
Robert Klopfleisch,
Sophie Merz,
Barbara Richter,
F. Yvonne Schulman,
Hannah Janout,
Jonathan Ganz,
Josef Scharinger,
Marc Aubreville,
Stephan M. Winkler,
Matti Kiupel,
Christof A. Bertram
Abstract:
Variation in nuclear size and shape is an important criterion of malignancy for many tumor types; however, categorical estimates by pathologists have poor reproducibility. Measurements of nuclear characteristics (morphometry) can improve reproducibility, but manual methods are time consuming. The aim of this study was to explore the limitations of estimates and develop alternative morphometric sol…
▽ More
Variation in nuclear size and shape is an important criterion of malignancy for many tumor types; however, categorical estimates by pathologists have poor reproducibility. Measurements of nuclear characteristics (morphometry) can improve reproducibility, but manual methods are time consuming. The aim of this study was to explore the limitations of estimates and develop alternative morphometric solutions for canine cutaneous mast cell tumors (ccMCT). We assessed the following nuclear evaluation methods for measurement accuracy, reproducibility, and prognostic utility: 1) anisokaryosis (karyomegaly) estimates by 11 pathologists; 2) gold standard manual morphometry of at least 100 nuclei; 3) practicable manual morphometry with stratified sampling of 12 nuclei by 9 pathologists; and 4) automated morphometry using a deep learning-based segmentation algorithm. The study dataset comprised 96 ccMCT with available outcome information. The study dataset comprised 96 ccMCT with available outcome information. Inter-rater reproducibility of karyomegaly estimates was low ($κ$ = 0.226), while it was good (ICC = 0.654) for practicable morphometry of the standard deviation (SD) of nuclear size. As compared to gold standard manual morphometry (AUC = 0.839, 95% CI: 0.701 - 0.977), the prognostic value (tumor-specific survival) of SDs of nuclear area for practicable manual morphometry (12 nuclei) and automated morphometry were high with an area under the ROC curve (AUC) of 0.868 (95% CI: 0.737 - 0.991) and 0.943 (95% CI: 0.889 - 0.996), respectively. This study supports the use of manual morphometry with stratified sampling of 12 nuclei and algorithmic morphometry to overcome the poor reproducibility of estimates.
△ Less
Submitted 23 May, 2024; v1 submitted 26 September, 2023;
originally announced September 2023.
-
Multi-Scanner Canine Cutaneous Squamous Cell Carcinoma Histopathology Dataset
Authors:
Frauke Wilm,
Marco Fragoso,
Christof A. Bertram,
Nikolas Stathonikos,
Mathias Öttl,
**gna Qiu,
Robert Klopfleisch,
Andreas Maier,
Katharina Breininger,
Marc Aubreville
Abstract:
In histopathology, scanner-induced domain shifts are known to impede the performance of trained neural networks when tested on unseen data. Multi-domain pre-training or dedicated domain-generalization techniques can help to develop domain-agnostic algorithms. For this, multi-scanner datasets with a high variety of slide scanning systems are highly desirable. We present a publicly available multi-s…
▽ More
In histopathology, scanner-induced domain shifts are known to impede the performance of trained neural networks when tested on unseen data. Multi-domain pre-training or dedicated domain-generalization techniques can help to develop domain-agnostic algorithms. For this, multi-scanner datasets with a high variety of slide scanning systems are highly desirable. We present a publicly available multi-scanner dataset of canine cutaneous squamous cell carcinoma histopathology images, composed of 44 samples digitized with five slide scanners. This dataset provides local correspondences between images and thereby isolates the scanner-induced domain shift from other inherent, e.g. morphology-induced domain shifts. To highlight scanner differences, we present a detailed evaluation of color distributions, sharpness, and contrast of the individual scanner subsets. Additionally, to quantify the inherent scanner-induced domain shift, we train a tumor segmentation network on each scanner subset and evaluate the performance both in- and cross-domain. We achieve a class-averaged in-domain intersection over union coefficient of up to 0.86 and observe a cross-domain performance decrease of up to 0.38, which confirms the inherent domain shift of the presented dataset and its negative impact on the performance of deep neural networks.
△ Less
Submitted 27 February, 2023; v1 submitted 11 January, 2023;
originally announced January 2023.
-
Deep Learning-Based Automatic Assessment of AgNOR-scores in Histopathology Images
Authors:
Jonathan Ganz,
Karoline Lipnik,
Jonas Ammeling,
Barbara Richter,
Chloé Puget,
Eda Parlak,
Laura Diehl,
Robert Klopfleisch,
Taryn A. Donovan,
Matti Kiupel,
Christof A. Bertram,
Katharina Breininger,
Marc Aubreville
Abstract:
Nucleolar organizer regions (NORs) are parts of the DNA that are involved in RNA transcription. Due to the silver affinity of associated proteins, argyrophilic NORs (AgNORs) can be visualized using silver-based staining. The average number of AgNORs per nucleus has been shown to be a prognostic factor for predicting the outcome of many tumors. Since manual detection of AgNORs is laborious, automat…
▽ More
Nucleolar organizer regions (NORs) are parts of the DNA that are involved in RNA transcription. Due to the silver affinity of associated proteins, argyrophilic NORs (AgNORs) can be visualized using silver-based staining. The average number of AgNORs per nucleus has been shown to be a prognostic factor for predicting the outcome of many tumors. Since manual detection of AgNORs is laborious, automation is of high interest. We present a deep learning-based pipeline for automatically determining the AgNOR-score from histopathological sections. An additional annotation experiment was conducted with six pathologists to provide an independent performance evaluation of our approach. Across all raters and images, we found a mean squared error of 0.054 between the AgNOR- scores of the experts and those of the model, indicating that our approach offers performance comparable to humans.
△ Less
Submitted 15 December, 2022;
originally announced December 2022.
-
Mind the Gap: Scanner-induced domain shifts pose challenges for representation learning in histopathology
Authors:
Frauke Wilm,
Marco Fragoso,
Christof A. Bertram,
Nikolas Stathonikos,
Mathias Öttl,
**gna Qiu,
Robert Klopfleisch,
Andreas Maier,
Marc Aubreville,
Katharina Breininger
Abstract:
Computer-aided systems in histopathology are often challenged by various sources of domain shift that impact the performance of these algorithms considerably. We investigated the potential of using self-supervised pre-training to overcome scanner-induced domain shifts for the downstream task of tumor segmentation. For this, we present the Barlow Triplets to learn scanner-invariant representations…
▽ More
Computer-aided systems in histopathology are often challenged by various sources of domain shift that impact the performance of these algorithms considerably. We investigated the potential of using self-supervised pre-training to overcome scanner-induced domain shifts for the downstream task of tumor segmentation. For this, we present the Barlow Triplets to learn scanner-invariant representations from a multi-scanner dataset with local image correspondences. We show that self-supervised pre-training successfully aligned different scanner representations, which, interestingly only results in a limited benefit for our downstream task. We thereby provide insights into the influence of scanner characteristics for downstream applications and contribute to a better understanding of why established self-supervised methods have not yet shown the same success on histopathology data as they have for natural images.
△ Less
Submitted 29 November, 2022;
originally announced November 2022.
-
Pan-tumor CAnine cuTaneous Cancer Histology (CATCH) dataset
Authors:
Frauke Wilm,
Marco Fragoso,
Christian Marzahl,
**gna Qiu,
Chloé Puget,
Laura Diehl,
Christof A. Bertram,
Robert Klopfleisch,
Andreas Maier,
Katharina Breininger,
Marc Aubreville
Abstract:
Due to morphological similarities, the differentiation of histologic sections of cutaneous tumors into individual subtypes can be challenging. Recently, deep learning-based approaches have proven their potential for supporting pathologists in this regard. However, many of these supervised algorithms require a large amount of annotated data for robust development. We present a publicly available da…
▽ More
Due to morphological similarities, the differentiation of histologic sections of cutaneous tumors into individual subtypes can be challenging. Recently, deep learning-based approaches have proven their potential for supporting pathologists in this regard. However, many of these supervised algorithms require a large amount of annotated data for robust development. We present a publicly available dataset of 350 whole slide images of seven different canine cutaneous tumors complemented by 12,424 polygon annotations for 13 histologic classes, including seven cutaneous tumor subtypes. In inter-rater experiments, we show a high consistency of the provided labels, especially for tumor annotations. We further validate the dataset by training a deep neural network for the task of tissue segmentation and tumor subtype classification. We achieve a class-averaged Jaccard coefficient of 0.7047, and 0.9044 for tumor in particular. For classification, we achieve a slide-level accuracy of 0.9857. Since canine cutaneous tumors possess various histologic homologies to human tumors the added value of this dataset is not limited to veterinary pathology but extends to more general fields of application.
△ Less
Submitted 26 August, 2022; v1 submitted 27 January, 2022;
originally announced January 2022.
-
First steps on Gamification of Lung Fluid Cells Annotations in the Flower Domain
Authors:
Sonja Kunzmann,
Christian Marzahl,
Felix Denzinger,
Christof A. Bertram,
Robert Klopfleisch,
Katharina Breininger,
Vincent Christlein,
Andreas Maier
Abstract:
Annotating data, especially in the medical domain, requires expert knowledge and a lot of effort. This limits the amount and/or usefulness of available medical data sets for experimentation. Therefore, develo** strategies to increase the number of annotations while lowering the needed domain knowledge is of interest. A possible strategy is the use of gamification, i.e. transforming the annotatio…
▽ More
Annotating data, especially in the medical domain, requires expert knowledge and a lot of effort. This limits the amount and/or usefulness of available medical data sets for experimentation. Therefore, develo** strategies to increase the number of annotations while lowering the needed domain knowledge is of interest. A possible strategy is the use of gamification, i.e. transforming the annotation task into a game. We propose an approach to gamify the task of annotating lung fluid cells from pathological whole slide images (WSIs). As the domain is unknown to non-expert annotators, we transform images of cells to the domain of flower images using a CycleGAN architecture. In this more assessable domain, non-expert annotators can be (t)asked to annotate different kinds of flowers in a playful setting. In order to provide a proof of concept, this work shows that the domain transfer is possible by evaluating an image classification network trained on real cell images and tested on the cell images generated by the CycleGAN network (reconstructed cell images) as well as real cell images. The classification network reaches an average accuracy of 94.73 % on the original lung fluid cells and 95.25 % on the transformed lung fluid cells, respectively. Our study lays the foundation for future research on gamification using CycleGANs.
△ Less
Submitted 17 January, 2022; v1 submitted 5 November, 2021;
originally announced November 2021.
-
Inter-Species Cell Detection: Datasets on pulmonary hemosiderophages in equine, human and feline specimens
Authors:
Christian Marzahl,
Jenny Hill,
Jason Stayt,
Dorothee Bienzle,
Lutz Welker,
Frauke Wilm,
Jörn Voigt,
Marc Aubreville,
Andreas Maier,
Robert Klopfleisch,
Katharina Breininger,
Christof A. Bertram
Abstract:
Pulmonary hemorrhage (P-Hem) occurs among multiple species and can have various causes. Cytology of bronchoalveolarlavage fluid (BALF) using a 5-tier scoring system of alveolar macrophages based on their hemosiderin content is considered the most sensitive diagnostic method. We introduce a novel, fully annotated multi-species P-Hem dataset which consists of 74 cytology whole slide images (WSIs) wi…
▽ More
Pulmonary hemorrhage (P-Hem) occurs among multiple species and can have various causes. Cytology of bronchoalveolarlavage fluid (BALF) using a 5-tier scoring system of alveolar macrophages based on their hemosiderin content is considered the most sensitive diagnostic method. We introduce a novel, fully annotated multi-species P-Hem dataset which consists of 74 cytology whole slide images (WSIs) with equine, feline and human samples. To create this high-quality and high-quantity dataset, we developed an annotation pipeline combining human expertise with deep learning and data visualisation techniques. We applied a deep learning-based object detection approach trained on 17 expertly annotated equine WSIs, to the remaining 39 equine, 12 human and 7 feline WSIs. The resulting annotations were semi-automatically screened for errors on multiple types of specialised annotation maps and finally reviewed by a trained pathologists. Our dataset contains a total of 297,383 hemosiderophages classified into five grades. It is one of the largest publicly availableWSIs datasets with respect to the number of annotations, the scanned area and the number of species covered.
△ Less
Submitted 19 August, 2021;
originally announced August 2021.
-
Quantifying the Scanner-Induced Domain Gap in Mitosis Detection
Authors:
Marc Aubreville,
Christof Bertram,
Mitko Veta,
Robert Klopfleisch,
Nikolas Stathonikos,
Katharina Breininger,
Natalie ter Hoeve,
Francesco Ciompi,
Andreas Maier
Abstract:
Automated detection of mitotic figures in histopathology images has seen vast improvements, thanks to modern deep learning-based pipelines. Application of these methods, however, is in practice limited by strong variability of images between labs. This results in a domain shift of the images, which causes a performance drop of the models. Hypothesizing that the scanner device plays a decisive role…
▽ More
Automated detection of mitotic figures in histopathology images has seen vast improvements, thanks to modern deep learning-based pipelines. Application of these methods, however, is in practice limited by strong variability of images between labs. This results in a domain shift of the images, which causes a performance drop of the models. Hypothesizing that the scanner device plays a decisive role in this effect, we evaluated the susceptibility of a standard mitosis detection approach to the domain shift introduced by using a different whole slide scanner. Our work is based on the MICCAI-MIDOG challenge 2021 data set, which includes 200 tumor cases of human breast cancer and four scanners.
Our work indicates that the domain shift induced not by biochemical variability but purely by the choice of acquisition device is underestimated so far. Models trained on images of the same scanner yielded an average F1 score of 0.683, while models trained on a single other scanner only yielded an average F1 score of 0.325. Training on another multi-domain mitosis dataset led to mean F1 scores of 0.52. We found this not to be reflected by domain-shifts measured as proxy A distance-derived metric.
△ Less
Submitted 30 March, 2021;
originally announced March 2021.
-
Learning to be EXACT, Cell Detection for Asthma on Partially Annotated Whole Slide Images
Authors:
Christian Marzahl,
Christof A. Bertram,
Frauke Wilm,
Jörn Voigt,
Ann K. Barton,
Robert Klopfleisch,
Katharina Breininger,
Andreas Maier,
Marc Aubreville
Abstract:
Asthma is a chronic inflammatory disorder of the lower respiratory tract and naturally occurs in humans and animals including horses. The annotation of an asthma microscopy whole slide image (WSI) is an extremely labour-intensive task due to the hundreds of thousands of cells per WSI. To overcome the limitation of annotating WSI incompletely, we developed a training pipeline which can train a deep…
▽ More
Asthma is a chronic inflammatory disorder of the lower respiratory tract and naturally occurs in humans and animals including horses. The annotation of an asthma microscopy whole slide image (WSI) is an extremely labour-intensive task due to the hundreds of thousands of cells per WSI. To overcome the limitation of annotating WSI incompletely, we developed a training pipeline which can train a deep learning-based object detection model with partially annotated WSIs and compensate class imbalances on the fly. With this approach we can freely sample from annotated WSIs areas and are not restricted to fully annotated extracted sub-images of the WSI as with classical approaches. We evaluated our pipeline in a cross-validation setup with a fixed training set using a dataset of six equine WSIs of which four are partially annotated and used for training, and two fully annotated WSI are used for validation and testing. Our WSI-based training approach outperformed classical sub-image-based training methods by up to 15\% $mAP$ and yielded human-like performance when compared to the annotations of ten trained pathologists.
△ Less
Submitted 13 January, 2021;
originally announced January 2021.
-
Dataset on Bi- and Multi-Nucleated Tumor Cells in Canine Cutaneous Mast Cell Tumors
Authors:
Christof A. Bertram,
Taryn A. Donovan,
Marco Tecilla,
Florian Bartenschlager,
Marco Fragoso,
Frauke Wilm,
Christian Marzahl,
Katharina Breininger,
Andreas Maier,
Robert Klopfleisch,
Marc Aubreville
Abstract:
Tumor cells with two nuclei (binucleated cells, BiNC) or more nuclei (multinucleated cells, MuNC) indicate an increased amount of cellular genetic material which is thought to facilitate oncogenesis, tumor progression and treatment resistance. In canine cutaneous mast cell tumors (ccMCT), binucleation and multinucleation are parameters used in cytologic and histologic grading schemes (respectively…
▽ More
Tumor cells with two nuclei (binucleated cells, BiNC) or more nuclei (multinucleated cells, MuNC) indicate an increased amount of cellular genetic material which is thought to facilitate oncogenesis, tumor progression and treatment resistance. In canine cutaneous mast cell tumors (ccMCT), binucleation and multinucleation are parameters used in cytologic and histologic grading schemes (respectively) which correlate with poor patient outcome. For this study, we created the first open source data-set with 19,983 annotations of BiNC and 1,416 annotations of MuNC in 32 histological whole slide images of ccMCT. Labels were created by a pathologist and an algorithmic-aided labeling approach with expert review of each generated candidate. A state-of-the-art deep learning-based model yielded an $F_1$ score of 0.675 for BiNC and 0.623 for MuNC on 11 test whole slide images. In regions of interest ($2.37 mm^2$) extracted from these test images, 6 pathologists had an object detection performance between 0.270 - 0.526 for BiNC and 0.316 - 0.622 for MuNC, while our model archived an $F_1$ score of 0.667 for BiNC and 0.685 for MuNC. This open dataset can facilitate development of automated image analysis for this task and may thereby help to promote standardization of this facet of histologic tumor prognostication.
△ Less
Submitted 5 January, 2021;
originally announced January 2021.
-
How Many Annotators Do We Need? -- A Study on the Influence of Inter-Observer Variability on the Reliability of Automatic Mitotic Figure Assessment
Authors:
Frauke Wilm,
Christof A. Bertram,
Christian Marzahl,
Alexander Bartel,
Taryn A. Donovan,
Charles-Antoine Assenmacher,
Kathrin Becker,
Mark Bennett,
Sarah Corner,
Brieuc Cossic,
Daniela Denk,
Martina Dettwiler,
Beatriz Garcia Gonzalez,
Corinne Gurtner,
Annika Lehmbecker,
Sophie Merz,
Stephanie Plog,
Anja Schmidt,
Rebecca C. Smedley,
Marco Tecilla,
Tuddow Thaiwong,
Katharina Breininger,
Matti Kiupel,
Andreas Maier,
Robert Klopfleisch
, et al. (1 additional authors not shown)
Abstract:
Density of mitotic figures in histologic sections is a prognostically relevant characteristic for many tumours. Due to high inter-pathologist variability, deep learning-based algorithms are a promising solution to improve tumour prognostication. Pathologists are the gold standard for database development, however, labelling errors may hamper development of accurate algorithms. In the present work…
▽ More
Density of mitotic figures in histologic sections is a prognostically relevant characteristic for many tumours. Due to high inter-pathologist variability, deep learning-based algorithms are a promising solution to improve tumour prognostication. Pathologists are the gold standard for database development, however, labelling errors may hamper development of accurate algorithms. In the present work we evaluated the benefit of multi-expert consensus (n = 3, 5, 7, 9, 11) on algorithmic performance. While training with individual databases resulted in highly variable F$_1$ scores, performance was notably increased and more consistent when using the consensus of three annotators. Adding more annotators only resulted in minor improvements. We conclude that databases by few pathologists and high label accuracy may be the best compromise between high algorithmic performance and time investment.
△ Less
Submitted 8 January, 2021; v1 submitted 4 December, 2020;
originally announced December 2020.
-
A completely annotated whole slide image dataset of canine breast cancer to aid human breast cancer research
Authors:
Marc Aubreville,
Christof A. Bertram,
Taryn A. Donovan,
Christian Marzahl,
Andreas Maier,
Robert Klopfleisch
Abstract:
Canine mammary carcinoma (CMC) has been used as a model to investigate the pathogenesis of human breast cancer and the same grading scheme is commonly used to assess tumor malignancy in both. One key component of this grading scheme is the density of mitotic figures (MF). Current publicly available datasets on human breast cancer only provide annotations for small subsets of whole slide images (WS…
▽ More
Canine mammary carcinoma (CMC) has been used as a model to investigate the pathogenesis of human breast cancer and the same grading scheme is commonly used to assess tumor malignancy in both. One key component of this grading scheme is the density of mitotic figures (MF). Current publicly available datasets on human breast cancer only provide annotations for small subsets of whole slide images (WSIs). We present a novel dataset of 21 WSIs of CMC completely annotated for MF. For this, a pathologist screened all WSIs for potential MF and structures with a similar appearance. A second expert blindly assigned labels, and for non-matching labels, a third expert assigned the final labels. Additionally, we used machine learning to identify previously undetected MF. Finally, we performed representation learning and two-dimensional projection to further increase the consistency of the annotations. Our dataset consists of 13,907 MF and 36,379 hard negatives. We achieved a mean F1-score of 0.791 on the test set and of up to 0.696 on a human breast cancer dataset.
△ Less
Submitted 27 November, 2020; v1 submitted 24 August, 2020;
originally announced August 2020.
-
Are pathologist-defined labels reproducible? Comparison of the TUPAC16 mitotic figure dataset with an alternative set of labels
Authors:
Christof A. Bertram,
Mitko Veta,
Christian Marzahl,
Nikolas Stathonikos,
Andreas Maier,
Robert Klopfleisch,
Marc Aubreville
Abstract:
Pathologist-defined labels are the gold standard for histopathological data sets, regardless of well-known limitations in consistency for some tasks. To date, some datasets on mitotic figures are available and were used for development of promising deep learning-based algorithms. In order to assess robustness of those algorithms and reproducibility of their methods it is necessary to test on sever…
▽ More
Pathologist-defined labels are the gold standard for histopathological data sets, regardless of well-known limitations in consistency for some tasks. To date, some datasets on mitotic figures are available and were used for development of promising deep learning-based algorithms. In order to assess robustness of those algorithms and reproducibility of their methods it is necessary to test on several independent datasets. The influence of different labeling methods of these available datasets is currently unknown. To tackle this, we present an alternative set of labels for the images of the auxiliary mitosis dataset of the TUPAC16 challenge. Additional to manual mitotic figure screening, we used a novel, algorithm-aided labeling process, that allowed to minimize the risk of missing rare mitotic figures in the images. All potential mitotic figures were independently assessed by two pathologists. The novel, publicly available set of labels contains 1,999 mitotic figures (+28.80%) and additionally includes 10,483 labels of cells with high similarities to mitotic figures (hard examples). We found significant difference comparing F_1 scores between the original label set (0.549) and the new alternative label set (0.735) using a standard deep learning object detection architecture. The models trained on the alternative set showed higher overall confidence values, suggesting a higher overall label consistency. Findings of the present study show that pathologists-defined labels may vary significantly resulting in notable difference in the model performance. Comparison of deep learning-based algorithms between independent datasets with different labeling methods should be done with caution.
△ Less
Submitted 10 July, 2020;
originally announced July 2020.
-
EXACT: A collaboration toolset for algorithm-aided annotation of images with annotation version control
Authors:
Christian Marzahl,
Marc Aubreville,
Christof A. Bertram,
Jennifer Maier,
Christian Bergler,
Christine Kröger,
Jörn Voigt,
Katharina Breininger,
Robert Klopfleisch,
Andreas Maier
Abstract:
In many research areas, scientific progress is accelerated by multidisciplinary access to image data and their interdisciplinary annotation. However, kee** track of these annotations to ensure a high-quality multi-purpose data set is a challenging and labour intensive task. We developed the open-source online platform EXACT (EXpert Algorithm Collaboration Tool) that enables the collaborative int…
▽ More
In many research areas, scientific progress is accelerated by multidisciplinary access to image data and their interdisciplinary annotation. However, kee** track of these annotations to ensure a high-quality multi-purpose data set is a challenging and labour intensive task. We developed the open-source online platform EXACT (EXpert Algorithm Collaboration Tool) that enables the collaborative interdisciplinary analysis of images from different domains online and offline. EXACT supports multi-gigapixel medical whole slide images as well as image series with thousands of images. The software utilises a flexible plugin system that can be adapted to diverse applications such as counting mitotic figures with a screening mode, finding false annotations on a novel validation view, or using the latest deep learning image analysis technologies. This is combined with a version control system which makes it possible to keep track of changes in the data sets and, for example, to link the results of deep learning experiments to specific data set versions. EXACT is freely available and has already been successfully applied to a broad range of annotation tasks, including highly diverse applications like deep learning supported cytology scoring, interdisciplinary multi-centre whole slide image tumour annotation, and highly specialised whale sound spectroscopy clustering.
△ Less
Submitted 19 July, 2021; v1 submitted 30 April, 2020;
originally announced April 2020.
-
Are fast labeling methods reliable? A case study of computer-aided expert annotations on microscopy slides
Authors:
Christian Marzahl,
Christof A. Bertram,
Marc Aubreville,
Anne Petrick,
Kristina Weiler,
Agnes C. Gläsel,
Marco Fragoso,
Sophie Merz,
Florian Bartenschlager,
Judith Hoppe,
Alina Langenhagen,
Anne Jasensky,
Jörn Voigt,
Robert Klopfleisch,
Andreas Maier
Abstract:
Deep-learning-based pipelines have shown the potential to revolutionalize microscopy image diagnostics by providing visual augmentations to a trained pathology expert. However, to match human performance, the methods rely on the availability of vast amounts of high-quality labeled data, which poses a significant challenge. To circumvent this, augmented labeling methods, also known as expert-algori…
▽ More
Deep-learning-based pipelines have shown the potential to revolutionalize microscopy image diagnostics by providing visual augmentations to a trained pathology expert. However, to match human performance, the methods rely on the availability of vast amounts of high-quality labeled data, which poses a significant challenge. To circumvent this, augmented labeling methods, also known as expert-algorithm-collaboration, have recently become popular. However, potential biases introduced by this operation mode and their effects for training neuronal networks are not entirely understood. This work aims to shed light on some of the effects by providing a case study for three pathologically relevant diagnostic settings. Ten trained pathology experts performed a labeling tasks first without and later with computer-generated augmentation. To investigate different biasing effects, we intentionally introduced errors to the augmentation. Furthermore, we developed a novel loss function which incorporates the experts' annotation consensus in the training of a deep learning classifier. In total, the pathology experts annotated 26,015 cells on 1,200 images in this novel annotation study. Backed by this extensive data set, we found that the consensus of multiple experts and the deep learning classifier accuracy, was significantly increased in the computer-aided setting, versus the unaided annotation. However, a significant percentage of the deliberately introduced false labels was not identified by the experts. Additionally, we showed that our loss function profited from multiple experts and outperformed conventional loss functions. At the same time, systematic errors did not lead to a deterioration of the trained classifier accuracy. Furthermore, a classifier trained with annotations from a single expert with computer-aided support can outperform the combined annotations from up to nine experts.
△ Less
Submitted 13 April, 2020;
originally announced April 2020.
-
Fooling the Crowd with Deep Learning-based Methods
Authors:
Christian Marzahl,
Marc Aubreville,
Christof A. Bertram,
Stefan Gerlach,
Jennifer Maier,
Jörn Voigt,
Jenny Hill,
Robert Klopfleisch,
Andreas Maier
Abstract:
Modern, state-of-the-art deep learning approaches yield human like performance in numerous object detection and classification tasks. The foundation for their success is the availability of training datasets of substantially high quantity, which are expensive to create, especially in the field of medical imaging. Recently, crowdsourcing has been applied to create large datasets for a broad range o…
▽ More
Modern, state-of-the-art deep learning approaches yield human like performance in numerous object detection and classification tasks. The foundation for their success is the availability of training datasets of substantially high quantity, which are expensive to create, especially in the field of medical imaging. Recently, crowdsourcing has been applied to create large datasets for a broad range of disciplines. This study aims to explore the challenges and opportunities of crowd-algorithm collaboration for the object detection task of grading cytology whole slide images. We compared the classical crowdsourcing performance of twenty participants with their results from crowd-algorithm collaboration. All participants performed both modes in random order on the same twenty images. Additionally, we introduced artificial systematic flaws into the precomputed annotations to estimate a bias towards accepting precomputed annotations. We gathered 9524 annotations on 800 images from twenty participants organised into four groups in concordance to their level of expertise with cytology. The crowd-algorithm mode improved on average the participants' classification accuracy by 7%, the mean average precision by 8% and the inter-observer Fleiss' kappa score by 20%, and reduced the time spent by 31%. However, two thirds of the artificially modified false labels were not recognised as such by the contributors. This study shows that crowd-algorithm collaboration is a promising new approach to generate large datasets when it is ensured that a carefully designed setup eliminates potential biases.
△ Less
Submitted 30 November, 2019;
originally announced December 2019.
-
Learning New Tricks from Old Dogs -- Inter-Species, Inter-Tissue Domain Adaptation for Mitotic Figure Assessment
Authors:
Marc Aubreville,
Christof A. Bertram,
Samir Jabari,
Christian Marzahl,
Robert Klopfleisch,
Andreas Maier
Abstract:
For histopathological tumor assessment, the count of mitotic figures per area is an important part of prognostication. Algorithmic approaches - such as for mitotic figure identification - have significantly improved in recent times, potentially allowing for computer-augmented or fully automatic screening systems in the future. This trend is further supported by whole slide scanning microscopes bec…
▽ More
For histopathological tumor assessment, the count of mitotic figures per area is an important part of prognostication. Algorithmic approaches - such as for mitotic figure identification - have significantly improved in recent times, potentially allowing for computer-augmented or fully automatic screening systems in the future. This trend is further supported by whole slide scanning microscopes becoming available in many pathology labs and could soon become a standard imaging tool.
For an application in broader fields of such algorithms, the availability of mitotic figure data sets of sufficient size for the respective tissue type and species is an important precondition, that is, however, rarely met. While algorithmic performance climbed steadily for e.g. human mammary carcinoma, thanks to several challenges held in the field, for most tumor types, data sets are not available.
In this work, we assess domain transfer of mitotic figure recognition using domain adversarial training on four data sets, two from dogs and two from humans. We were able to show that domain adversarial training considerably improves accuracy when applying mitotic figure classification learned from the canine on the human data sets (up to +12.8% in accuracy) and is thus a helpful method to transfer knowledge from existing data sets to new tissue types and species.
△ Less
Submitted 25 November, 2019;
originally announced November 2019.
-
Deep Learning-Based Quantification of Pulmonary Hemosiderophages in Cytology Slides
Authors:
Christian Marzahl,
Marc Aubreville,
Christof A. Bertram,
Jason Stayt,
Anne-Katherine Jasensky,
Florian Bartenschlager,
Marco Fragoso-Garcia,
Ann K. Barton,
Svenja Elsemann,
Samir Jabari,
Jens Krauth,
Prathmesh Madhu,
Jörn Voigt,
Jenny Hill,
Robert Klopfleisch,
Andreas Maier
Abstract:
Purpose: Exercise-induced pulmonary hemorrhage (EIPH) is a common syndrome in sport horses with negative impact on performance. Cytology of bronchoalveolar lavage fluid by use of a scoring system is considered the most sensitive diagnostic method. Macrophages are classified depending on the degree of cytoplasmic hemosiderin content. The current gold standard is manual grading, which is however mon…
▽ More
Purpose: Exercise-induced pulmonary hemorrhage (EIPH) is a common syndrome in sport horses with negative impact on performance. Cytology of bronchoalveolar lavage fluid by use of a scoring system is considered the most sensitive diagnostic method. Macrophages are classified depending on the degree of cytoplasmic hemosiderin content. The current gold standard is manual grading, which is however monotonous and time-consuming. Methods: We evaluated state-of-the-art deep learning-based methods for single cell macrophage classification and compared them against the performance of nine cytology experts and evaluated inter- and intra-observer variability. Additionally, we evaluated object detection methods on a novel data set of 17 completely annotated cytology whole slide images (WSI) containing 78,047 hemosiderophages. Resultsf: Our deep learning-based approach reached a concordance of 0.85, partially exceeding human expert concordance (0.68 to 0.86, $μ$=0.73, $σ$ =0.04). Intra-observer variability was high (0.68 to 0.88) and inter-observer concordance was moderate (Fleiss kappa = 0.67). Our object detection approach has a mean average precision of 0.66 over the five classes from the whole slide gigapixel image and a computation time of below two minutes. Conclusion: To mitigate the high inter- and intra-rater variability, we propose our automated object detection pipeline, enabling accurate, reproducible and quick EIPH scoring in WSI.
△ Less
Submitted 12 August, 2019;
originally announced August 2019.
-
Deep learning algorithms out-perform veterinary pathologists in detecting the mitotically most active tumor region
Authors:
Marc Aubreville,
Christof A. Bertram,
Christian Marzahl,
Corinne Gurtner,
Martina Dettwiler,
Anja Schmidt,
Florian Bartenschlager,
Sophie Merz,
Marco Fragoso,
Olivia Kershaw,
Robert Klopfleisch,
Andreas Maier
Abstract:
Manual count of mitotic figures, which is determined in the tumor region with the highest mitotic activity, is a key parameter of most tumor grading schemes. It can be, however, strongly dependent on the area selection due to uneven mitotic figure distribution in the tumor section.We aimed to assess the question, how significantly the area selection could impact the mitotic count, which has a know…
▽ More
Manual count of mitotic figures, which is determined in the tumor region with the highest mitotic activity, is a key parameter of most tumor grading schemes. It can be, however, strongly dependent on the area selection due to uneven mitotic figure distribution in the tumor section.We aimed to assess the question, how significantly the area selection could impact the mitotic count, which has a known high inter-rater disagreement. On a data set of 32 whole slide images of H&E-stained canine cutaneous mast cell tumor, fully annotated for mitotic figures, we asked eight veterinary pathologists (five board-certified, three in training) to select a field of interest for the mitotic count. To assess the potential difference on the mitotic count, we compared the mitotic count of the selected regions to the overall distribution on the slide.Additionally, we evaluated three deep learning-based methods for the assessment of highest mitotic density: In one approach, the model would directly try to predict the mitotic count for the presented image patches as a regression task. The second method aims at deriving a segmentation mask for mitotic figures, which is then used to obtain a mitotic density. Finally, we evaluated a two-stage object-detection pipeline based on state-of-the-art architectures to identify individual mitotic figures. We found that the predictions by all models were, on average, better than those of the experts. The two-stage object detector performed best and outperformed most of the human pathologists on the majority of tumor cases. The correlation between the predicted and the ground truth mitotic count was also best for this approach (0.963 to 0.979). Further, we found considerable differences in position selection between pathologists, which could partially explain the high variance that has been reported for the manual mitotic count.
△ Less
Submitted 21 October, 2020; v1 submitted 12 February, 2019;
originally announced February 2019.
-
Field Of Interest Proposal for Augmented Mitotic Cell Count: Comparison of two Convolutional Networks
Authors:
Marc Aubreville,
Christof A. Bertram,
Robert Klopfleisch,
Andreas Maier
Abstract:
Most tumor grading systems for human as for veterinary histopathology are based upon the absolute count of mitotic figures in a certain reference area of a histology slide. Since time for prognostication is limited in a diagnostic setting, the pathologist will often almost arbitrarily choose a certain field of interest assumed to have the highest mitotic activity. However, as mitotic figures are c…
▽ More
Most tumor grading systems for human as for veterinary histopathology are based upon the absolute count of mitotic figures in a certain reference area of a histology slide. Since time for prognostication is limited in a diagnostic setting, the pathologist will often almost arbitrarily choose a certain field of interest assumed to have the highest mitotic activity. However, as mitotic figures are commonly very sparse on the slide and often have a patchy distribution, this poses a sampling problem which is known to be able to influence the tumor prognostication. On the other hand, automatic detection of mitotic figures can't yet be considered reliable enough for clinical application. In order to aid the work of the human expert and at the same time reduce variance in tumor grading, it is beneficial to assess the whole slide image (WSI) for the highest mitotic activity and use this as a reference region for human counting. For this task, we compare two methods for region of interest proposal, both based on convolutional neural networks (CNN). For both approaches, the CNN performs a segmentation of the WSI to assess mitotic activity. The first method performs a segmentation at the original image resolution, while the second approach performs a segmentation operation at a significantly reduced resolution, cutting down on processing complexity. We evaluate the approach using a dataset of 32 completely annotated whole slide images of canine mast cell tumors, where 22 were used for training of the network and 10 for test. Our results indicate that, while the overall correlation to the ground truth mitotic activity is considerably higher (0.94 vs. 0.83) for the approach based upon the fine resolution network, the field of interest choices are only marginally better. Both approaches propose fields of interest that contain a mitotic count in the upper quartile of respective slides.
△ Less
Submitted 22 October, 2018;
originally announced October 2018.
-
Augmented Mitotic Cell Count using Field Of Interest Proposal
Authors:
Marc Aubreville,
Christof A. Bertram,
Robert Klopfleisch,
Andreas Maier
Abstract:
Histopathological prognostication of neoplasia including most tumor grading systems are based upon a number of criteria. Probably the most important is the number of mitotic figures which are most commonly determined as the mitotic count (MC), i.e. number of mitotic figures within 10 consecutive high power fields. Often the area with the highest mitotic activity is to be selected for the MC. Howev…
▽ More
Histopathological prognostication of neoplasia including most tumor grading systems are based upon a number of criteria. Probably the most important is the number of mitotic figures which are most commonly determined as the mitotic count (MC), i.e. number of mitotic figures within 10 consecutive high power fields. Often the area with the highest mitotic activity is to be selected for the MC. However, since mitotic activity is not known in advance, an arbitrary choice of this region is considered one important cause for high variability in the prognostication and grading.
In this work, we present an algorithmic approach that first calculates a mitotic cell map based upon a deep convolutional network. This map is in a second step used to construct a mitotic activity estimate. Lastly, we select the image segment representing the size of ten high power fields with the overall highest mitotic activity as a region proposal for an expert MC determination. We evaluate the approach using a dataset of 32 completely annotated whole slide images, where 22 were used for training of the network and 10 for test. We find a correlation of r=0.936 in mitotic count estimate.
△ Less
Submitted 1 October, 2018;
originally announced October 2018.
-
SlideRunner - A Tool for Massive Cell Annotations in Whole Slide Images
Authors:
Marc Aubreville,
Christof Bertram,
Robert Klopfleisch,
Andreas Maier
Abstract:
Large-scale image data such as digital whole-slide histology images pose a challenging task at annotation software solutions. Today, a number of good solutions with varying scopes exist. For cell annotation, however, we find that many do not match the prerequisites for fast annotations. Especially in the field of mitosis detection, it is assumed that detection accuracy could significantly benefit…
▽ More
Large-scale image data such as digital whole-slide histology images pose a challenging task at annotation software solutions. Today, a number of good solutions with varying scopes exist. For cell annotation, however, we find that many do not match the prerequisites for fast annotations. Especially in the field of mitosis detection, it is assumed that detection accuracy could significantly benefit from larger annotation databases that are currently however very troublesome to produce. Further, multiple independent (blind) expert labels are a big asset for such databases, yet there is currently no tool for this kind of annotation available.
To ease this tedious process of expert annotation and grading, we introduce SlideRunner, an open source annotation and visualization tool for digital histopathology, developed in close cooperation with two pathologists. SlideRunner is capable of setting annotations like object centers (for e.g. cells) as well as object boundaries (e.g. for tumor outlines). It provides single-click annotations as well as a blind mode for multi-annotations, where the expert is directly shown the microscopy image containing the cells that he has not yet rated.
△ Less
Submitted 7 February, 2018;
originally announced February 2018.
-
A Guided Spatial Transformer Network for Histology Cell Differentiation
Authors:
Marc Aubreville,
Maximilian Krappmann,
Christof Bertram,
Robert Klopfleisch,
Andreas Maier
Abstract:
Identification and counting of cells and mitotic figures is a standard task in diagnostic histopathology. Due to the large overall cell count on histological slides and the potential sparse prevalence of some relevant cell types or mitotic figures, retrieving annotation data for sufficient statistics is a tedious task and prone to a significant error in assessment. Automatic classification and seg…
▽ More
Identification and counting of cells and mitotic figures is a standard task in diagnostic histopathology. Due to the large overall cell count on histological slides and the potential sparse prevalence of some relevant cell types or mitotic figures, retrieving annotation data for sufficient statistics is a tedious task and prone to a significant error in assessment. Automatic classification and segmentation is a classic task in digital pathology, yet it is not solved to a sufficient degree.
We present a novel approach for cell and mitotic figure classification, based on a deep convolutional network with an incorporated Spatial Transformer Network. The network was trained on a novel data set with ten thousand mitotic figures, about ten times more than previous data sets. The algorithm is able to derive the cell class (mitotic tumor cells, non-mitotic tumor cells and granulocytes) and their position within an image. The mean accuracy of the algorithm in a five-fold cross-validation is 91.45%.
In our view, the approach is a promising step into the direction of a more objective and accurate, semi-automatized mitosis counting supporting the pathologist.
△ Less
Submitted 26 July, 2017;
originally announced July 2017.
