Showing 1–16 of 16 results for author: Hipp, J

PLUTO: Pathology-Universal Transformer

Authors: Dinkar Juyal, Harshith Padigela, Chintan Shah, Daniel Shenker, Natalia Harguindeguy, Yi Liu, Blake Martin, Yibo Zhang, Michael Nercessian, Miles Markey, Isaac Finberg, Kelsey Luu, Daniel Borders, Syed Ashar Javed, Emma Krause, Raymond Biju, Aashish Sood, Allen Ma, Jackson Nyman, John Shamshoian, Guillaume Chhor, Darpan Sanghavi, Marc Thibault, Limin Yu, Fedaa Najdawi , et al. (8 additional authors not shown)

Abstract: Pathology is the study of microscopic inspection of tissue, and a pathology diagnosis is often the medical gold standard to diagnose disease. Pathology images provide a unique challenge for computer-vision-based analysis: a single pathology Whole Slide Image (WSI) is gigapixel-sized and often contains hundreds of thousands to millions of objects of interest across multiple resolutions. In this wor… ▽ More Pathology is the study of microscopic inspection of tissue, and a pathology diagnosis is often the medical gold standard to diagnose disease. Pathology images provide a unique challenge for computer-vision-based analysis: a single pathology Whole Slide Image (WSI) is gigapixel-sized and often contains hundreds of thousands to millions of objects of interest across multiple resolutions. In this work, we propose PathoLogy Universal TransfOrmer (PLUTO): a light-weight pathology FM that is pre-trained on a diverse dataset of 195 million image tiles collected from multiple sites and extracts meaningful representations across multiple WSI scales that enable a large variety of downstream pathology tasks. In particular, we design task-specific adaptation heads that utilize PLUTO's output embeddings for tasks which span pathology scales ranging from subcellular to slide-scale, including instance segmentation, tile classification, and slide-level prediction. We compare PLUTO's performance to other state-of-the-art methods on a diverse set of external and internal benchmarks covering multiple biologically relevant tasks, tissue types, resolutions, stains, and scanners. We find that PLUTO matches or outperforms existing task-specific baselines and pathology-specific foundation models, some of which use orders-of-magnitude larger datasets and model sizes when compared to PLUTO. Our findings present a path towards a universal embedding to power pathology image analysis, and motivate further exploration around pathology foundation models in terms of data diversity, architectural improvements, sample efficiency, and practical deployability in real-world applications. △ Less

Submitted 13 May, 2024; originally announced May 2024.

Statistical Modelling of Driving Scenarios in Road Traffic using Fleet Data of Production Vehicles

Authors: Christian Reichenbächer, Jochen Hipp, Oliver Bringmann

Abstract: Ensuring the safety of road vehicles at an acceptable level requires the absence of any unreasonable risk arising from all potential hazards linked to the intended au-tomated driving function and its implementation. The assurance that there are no unreasonable risks stemming from hazardous behaviours associated to functional insufficiencies is denoted as safety of intended functionality (SOTIF), a… ▽ More Ensuring the safety of road vehicles at an acceptable level requires the absence of any unreasonable risk arising from all potential hazards linked to the intended au-tomated driving function and its implementation. The assurance that there are no unreasonable risks stemming from hazardous behaviours associated to functional insufficiencies is denoted as safety of intended functionality (SOTIF), a concept outlined in the ISO 21448 standard. In this context, the acquisition of real driving data is considered essential for the verification and validation. For this purpose, we are currently develo** a method with which data collect-ed representatively from production vehicles can be modelled into a knowledge-based system in the future. A system that represents the probabilities of occur-rence of concrete driving scenarios over the statistical population of road traffic and makes them usable. The method includes the qualitative and quantitative ab-straction of the drives recorded by the sensors in the vehicles, the possibility of subsequent wireless transmission of the abstracted data from the vehicles and the derivation of the distributions and correlations of scenario parameters. This paper provides a summary of the research project and outlines its central idea. To this end, among other things, the needs for statistical information and da-ta from road traffic are elaborated from ISO 21448, the current state of research is addressed, and methodical aspects are discussed. △ Less

Submitted 9 April, 2024; originally announced April 2024.

Comments: 12 pages, 4 figures, the article has been accepted for publication and presentation during the 9th International ATZ Conference on Automated Driving 2024

Learning to Predict RNA Sequence Expressions from Whole Slide Images with Applications for Search and Classification

Authors: Amir Safarpoor, Jason D. Hipp, H. R. Tizhoosh

Abstract: Deep learning methods are widely applied in digital pathology to address clinical challenges such as prognosis and diagnosis. As one of the most recent applications, deep models have also been used to extract molecular features from whole slide images. Although molecular tests carry rich information, they are often expensive, time-consuming, and require additional tissue to sample. In this paper,… ▽ More Deep learning methods are widely applied in digital pathology to address clinical challenges such as prognosis and diagnosis. As one of the most recent applications, deep models have also been used to extract molecular features from whole slide images. Although molecular tests carry rich information, they are often expensive, time-consuming, and require additional tissue to sample. In this paper, we propose tRNAsfomer, an attention-based topology that can learn both to predict the bulk RNA-seq from an image and represent the whole slide image of a glass slide simultaneously. The tRNAsfomer uses multiple instance learning to solve a weakly supervised problem while the pixel-level annotation is not available for an image. We conducted several experiments and achieved better performance and faster convergence in comparison to the state-of-the-art algorithms. The proposed tRNAsfomer can assist as a computational pathology tool to facilitate a new generation of search and classification methods by combining the tissue morphology and the molecular fingerprint of the biopsy samples. △ Less

Submitted 26 March, 2022; originally announced March 2022.

Identifying Scenarios in Field Data to Enable Validation of Highly Automated Driving Systems

Authors: Christian Reichenbächer, Maximilian Rasch, Zafer Kayatas, Florian Wirthmüller, Jochen Hipp, Thao Dang, Oliver Bringmann

Abstract: Scenario-based approaches for the validation of highly automated driving functions are based on the search for safety-critical characteristics of driving scenarios using software-in-the-loop simulations. This search requires information about the shape and probability of scenarios in real-world traffic. The scope of this work is to develop a method that identifies redefined logical driving scenari… ▽ More Scenario-based approaches for the validation of highly automated driving functions are based on the search for safety-critical characteristics of driving scenarios using software-in-the-loop simulations. This search requires information about the shape and probability of scenarios in real-world traffic. The scope of this work is to develop a method that identifies redefined logical driving scenarios in field data, so that this information can be derived subsequently. More precisely, a suitable approach is developed, implemented and validated using a traffic scenario as an example. The presented methodology is based on qualitative modelling of scenarios, which can be detected in abstracted field data. The abstraction is achieved by using universal elements of an ontology represented by a domain model. Already published approaches for such an abstraction are discussed and concretised with regard to the given application. By examining a first set of test data, it is shown that the developed method is a suitable approach for the identification of further driving scenarios. △ Less

Submitted 4 May, 2022; v1 submitted 7 March, 2022; originally announced March 2022.

Comments: 9 pages, 5 figures

Journal ref: In Proceedings of the 8th International Conference on Vehicle Technology and Intelligent Transport Systems - VEHITS, 134-142, 2022

The Atlas of Lane Changes: Investigating Location-dependent Lane Change Behaviors Using Measurement Data from a Customer Fleet

Authors: Florian Wirthmüller, Jochen Hipp, Christian Reichenbächer, Manfred Reichert

Abstract: The prediction of surrounding traffic participants behavior is a crucial and challenging task for driver assistance and autonomous driving systems. Today's approaches mainly focus on modeling dynamic aspects of the traffic situation and try to predict traffic participants behavior based on this. In this article we take a first step towards extending this common practice by calculating location-spe… ▽ More The prediction of surrounding traffic participants behavior is a crucial and challenging task for driver assistance and autonomous driving systems. Today's approaches mainly focus on modeling dynamic aspects of the traffic situation and try to predict traffic participants behavior based on this. In this article we take a first step towards extending this common practice by calculating location-specific a-priori lane change probabilities. The idea behind this is straight forward: The driving behavior of humans may vary in exactly the same traffic situation depending on the respective location. E.g. drivers may ask themselves: Should I pass the truck in front of me immediately or should I wait until reaching the less curvy part of my route lying only a few kilometers ahead? Although, such information is far away from allowing behavior prediction on its own, it is obvious that today's approaches will greatly benefit when incorporating such location-specific a-priori probabilities into their predictions. For example, our investigations show that highway interchanges tend to enhance driver's motivation to perform lane changes, whereas curves seem to have lane change-dampening effects. Nevertheless, the investigation of all considered local conditions shows that superposition of various effects can lead to unexpected probabilities at some locations. We thus suggest dynamically constructing and maintaining a lane change probability map based on customer fleet data in order to support onboard prediction systems with additional information. For deriving reliable lane change probabilities a broad customer fleet is the key to success. △ Less

Submitted 9 July, 2021; v1 submitted 23 June, 2021; originally announced July 2021.

Comments: the article has been accepted for publication during the 24th IEEE Intelligent Transportation Systems Conference (ITSC), 8 pages, 11 figures

Predicting the Time Until a Vehicle Changes the Lane Using LSTM-based Recurrent Neural Networks

Authors: Florian Wirthmüller, Marvin Klimke, Julian Schlechtriemen, Jochen Hipp, Manfred Reichert

Abstract: To plan safe and comfortable trajectories for automated vehicles on highways, accurate predictions of traffic situations are needed. So far, a lot of research effort has been spent on detecting lane change maneuvers rather than on estimating the point in time a lane change actually happens. In practice, however, this temporal information might be even more useful. This paper deals with the develop… ▽ More To plan safe and comfortable trajectories for automated vehicles on highways, accurate predictions of traffic situations are needed. So far, a lot of research effort has been spent on detecting lane change maneuvers rather than on estimating the point in time a lane change actually happens. In practice, however, this temporal information might be even more useful. This paper deals with the development of a system that accurately predicts the time to the next lane change of surrounding vehicles on highways using long short-term memory-based recurrent neural networks. An extensive evaluation based on a large real-world data set shows that our approach is able to make reliable predictions, even in the most challenging situations, with a root mean squared error around 0.7 seconds. Already 3.5 seconds prior to lane changes the predictions become highly accurate, showing a median error of less than 0.25 seconds. In summary, this article forms a fundamental step towards downstreamed highly accurate position predictions. △ Less

Submitted 3 February, 2021; v1 submitted 2 February, 2021; originally announced February 2021.

Comments: the article has been accepted for publication in IEEE Robotics and Automation Letters (RA-L); the article has been submitted to RA-L with IEEE ICRA conference option; if the article will be presented during the conference will be decided independently; 8 pages, 5 figures, 6 tables

A Fleet Learning Architecture for Enhanced Behavior Predictions during Challenging External Conditions

Authors: Florian Wirthmüller, Marvin Klimke, Julian Schlechtriemen, Jochen Hipp, Manfred Reichert

Abstract: Already today, driver assistance systems help to make daily traffic more comfortable and safer. However, there are still situations that are quite rare but are hard to handle at the same time. In order to cope with these situations and to bridge the gap towards fully automated driving, it becomes necessary to not only collect enormous amounts of data but rather the right ones. This data can be use… ▽ More Already today, driver assistance systems help to make daily traffic more comfortable and safer. However, there are still situations that are quite rare but are hard to handle at the same time. In order to cope with these situations and to bridge the gap towards fully automated driving, it becomes necessary to not only collect enormous amounts of data but rather the right ones. This data can be used to develop and validate the systems through machine learning and simulation pipelines. Along this line this paper presents a fleet learning-based architecture that enables continuous improvements of systems predicting the movement of surrounding traffic participants. Moreover, the presented architecture is applied to a testing vehicle in order to prove the fundamental feasibility of the system. Finally, it is shown that the system collects meaningful data which are helpful to improve the underlying prediction systems. △ Less

Submitted 24 September, 2020; v1 submitted 23 September, 2020; originally announced September 2020.

Comments: the article has been accepted for publication during the 2020 IEEE Symposium Series on Computational Intelligence (SSCI) within the IEEE Symposium on Computational Intelligence in Vehicles and Transportation Systems (CIVTS), 7 pages, 6 figures

Towards Incorporating Contextual Knowledge into the Prediction of Driving Behavior

Authors: Florian Wirthmüller, Julian Schlechtriemen, Jochen Hipp, Manfred Reichert

Abstract: Predicting the behavior of surrounding traffic participants is crucial for advanced driver assistance systems and autonomous driving. Most researchers however do not consider contextual knowledge when predicting vehicle motion. Extending former studies, we investigate how predictions are affected by external conditions. To do so, we categorize different kinds of contextual information and provide… ▽ More Predicting the behavior of surrounding traffic participants is crucial for advanced driver assistance systems and autonomous driving. Most researchers however do not consider contextual knowledge when predicting vehicle motion. Extending former studies, we investigate how predictions are affected by external conditions. To do so, we categorize different kinds of contextual information and provide a carefully chosen definition as well as examples for external conditions. More precisely, we investigate how a state-of-the-art approach for lateral motion prediction is influenced by one selected external condition, namely the traffic density. Our investigations demonstrate that this kind of information is highly relevant in order to improve the performance of prediction algorithms. Therefore, this study constitutes the first step towards the integration of such information into automated vehicles. Moreover, our motion prediction approach is evaluated based on the public highD data set showing a maneuver prediction performance with areas under the ROC curve above 97% and a median lateral prediction error of only 0.18m on a prediction horizon of 5s. △ Less

Submitted 4 July, 2020; v1 submitted 15 June, 2020; originally announced June 2020.

Comments: the article has been accepted for publication during the 23rd IEEE Intelligent Transportation Systems Conference (ITSC), 7 pages, 6 figures, 1 table

Spatial Heterogeneity Can Lead to Substantial Local Variations in COVID-19 Timing and Severity

Authors: Loring J. Thomas, Peng Huang, Fan Yin, Xiaoshuang Iris Luo, Zack W. Almquist, John R. Hipp, Carter T. Butts

Abstract: Standard epidemiological models for COVID-19 employ variants of compartment (SIR) models at local scales, implicitly assuming spatially uniform local mixing. Here, we examine the effect of employing more geographically detailed diffusion models based on known spatial features of interpersonal networks, most particularly the presence of a long-tailed but monotone decline in the probability of inter… ▽ More Standard epidemiological models for COVID-19 employ variants of compartment (SIR) models at local scales, implicitly assuming spatially uniform local mixing. Here, we examine the effect of employing more geographically detailed diffusion models based on known spatial features of interpersonal networks, most particularly the presence of a long-tailed but monotone decline in the probability of interaction with distance, on disease diffusion. Based on simulations of unrestricted COVID-19 diffusion in 19 U.S cities, we conclude that heterogeneity in population distribution can have large impacts on local pandemic timing and severity, even when aggregate behavior at larger scales mirrors a classic SIR-like pattern. Impacts observed include severe local outbreaks with long lag time relative to the aggregate infection curve, and the presence of numerous areas whose disease trajectories correlate poorly with those of neighboring areas. A simple catchment model for hospital demand illustrates potential implications for health care utilization, with substantial disparities in the timing and extremity of impacts even without distancing interventions. Likewise, analysis of social exposure to others who are morbid or deceased shows considerable variation in how the epidemic can appear to individuals on the ground, potentially affecting risk assessment and compliance with mitigation measures. These results demonstrate the potential for spatial network structure to generate highly non-uniform diffusion behavior even at the scale of cities, and suggest the importance of incorporating such structure when designing models to inform healthcare planning, predict community outcomes, or identify potential disparities. △ Less

Submitted 20 May, 2020; originally announced May 2020.

Journal ref: Proceedings of the National Academy of Sciences Sep 2020, 117 (39) 24180-24187; DOI: 10.1073/pnas.2011656117

Teaching Vehicles to Anticipate: A Systematic Study on Probabilistic Behavior Prediction Using Large Data Sets

Authors: Florian Wirthmüller, Julian Schlechtriemen, Jochen Hipp, Manfred Reichert

Abstract: By observing their environment as well as other traffic participants, humans are enabled to drive road vehicles safely. Vehicle passengers, however, perceive a notable difference between non-experienced and experienced drivers. In particular, they may get the impression that the latter ones anticipate what will happen in the next few moments and consider these foresights in their driving behavior.… ▽ More By observing their environment as well as other traffic participants, humans are enabled to drive road vehicles safely. Vehicle passengers, however, perceive a notable difference between non-experienced and experienced drivers. In particular, they may get the impression that the latter ones anticipate what will happen in the next few moments and consider these foresights in their driving behavior. To make the driving style of automated vehicles comparable to the one of human drivers with respect to comfort and perceived safety, the aforementioned anticipation skills need to become a built-in feature of self-driving vehicles. This article provides a systematic comparison of methods and strategies to generate this intention for self-driving cars using machine learning techniques. To implement and test these algorithms we use a large data set collected over more than 30000 km of highway driving and containing approximately 40000 real-world driving situations. We further show that it is possible to classify driving maneuvers upcoming within the next 5 s with an Area Under the ROC Curve (AUC) above 0.92 for all defined maneuver classes. This enables us to predict the lateral position with a prediction horizon of 5 s with a median lateral error of less than 0.21 m. △ Less

Submitted 10 June, 2020; v1 submitted 17 October, 2019; originally announced October 2019.

Comments: the paper has been accepted for publication in IEEE Transcations on Intelligent Transportation Systems (T-ITS) 16 pages 13 figures 12 tables

Human-Centered Tools for Co** with Imperfect Algorithms during Medical Decision-Making

Authors: Carrie J. Cai, Emily Reif, Narayan Hegde, Jason Hipp, Been Kim, Daniel Smilkov, Martin Wattenberg, Fernanda Viegas, Greg S. Corrado, Martin C. Stumpe, Michael Terry

Abstract: Machine learning (ML) is increasingly being used in image retrieval systems for medical decision making. One application of ML is to retrieve visually similar medical images from past patients (e.g. tissue from biopsies) to reference when making a medical decision with a new patient. However, no algorithm can perfectly capture an expert's ideal notion of similarity for every case: an image that is… ▽ More Machine learning (ML) is increasingly being used in image retrieval systems for medical decision making. One application of ML is to retrieve visually similar medical images from past patients (e.g. tissue from biopsies) to reference when making a medical decision with a new patient. However, no algorithm can perfectly capture an expert's ideal notion of similarity for every case: an image that is algorithmically determined to be similar may not be medically relevant to a doctor's specific diagnostic needs. In this paper, we identified the needs of pathologists when searching for similar images retrieved using a deep learning algorithm, and developed tools that empower users to cope with the search algorithm on-the-fly, communicating what types of similarity are most important at different moments in time. In two evaluations with pathologists, we found that these refinement tools increased the diagnostic utility of images found and increased user trust in the algorithm. The tools were preferred over a traditional interface, without a loss in diagnostic accuracy. We also observed that users adopted new strategies when using refinement tools, re-purposing them to test and understand the underlying algorithm and to disambiguate ML errors from their own errors. Taken together, these findings inform future human-ML collaborative systems for expert decision-making. △ Less

Submitted 8 February, 2019; originally announced February 2019.

Similar Image Search for Histopathology: SMILY

Authors: Narayan Hegde, Jason D. Hipp, Yun Liu, Michael E. Buck, Emily Reif, Daniel Smilkov, Michael Terry, Carrie J. Cai, Mahul B. Amin, Craig H. Mermel, Phil Q. Nelson, Lily H. Peng, Greg S. Corrado, Martin C. Stumpe

Abstract: The increasing availability of large institutional and public histopathology image datasets is enabling the searching of these datasets for diagnosis, research, and education. Though these datasets typically have associated metadata such as diagnosis or clinical notes, even carefully curated datasets rarely contain annotations of the location of regions of interest on each image. Because pathology… ▽ More The increasing availability of large institutional and public histopathology image datasets is enabling the searching of these datasets for diagnosis, research, and education. Though these datasets typically have associated metadata such as diagnosis or clinical notes, even carefully curated datasets rarely contain annotations of the location of regions of interest on each image. Because pathology images are extremely large (up to 100,000 pixels in each dimension), further laborious visual search of each image may be needed to find the feature of interest. In this paper, we introduce a deep learning based reverse image search tool for histopathology images: Similar Medical Images Like Yours (SMILY). We assessed SMILY's ability to retrieve search results in two ways: using pathologist-provided annotations, and via prospective studies where pathologists evaluated the quality of SMILY search results. As a negative control in the second evaluation, pathologists were blinded to whether search results were retrieved by SMILY or randomly. In both types of assessments, SMILY was able to retrieve search results with similar histologic features, organ site, and prostate cancer Gleason grade compared with the original query. SMILY may be a useful general-purpose tool in the pathologist's arsenal, to improve the efficiency of searching large archives of histopathology images, without the need to develop and implement specific tools for each application. △ Less

Submitted 5 February, 2019; v1 submitted 30 January, 2019; originally announced January 2019.

Comments: 23 Pages with 6 figures and 3 tables. The file also has 6 pages of supplemental material. Improved figure resolution, edited metadata

Journal ref: Nature Partner Journal Digital Medicine (2019)

Whole-Slide Image Focus Quality: Automatic Assessment and Impact on AI Cancer Detection

Authors: Timo Kohlberger, Yun Liu, Melissa Moran, Po-Hsuan, Chen, Trissia Brown, Craig H. Mermel, Jason D. Hipp, Martin C. Stumpe

Abstract: Digital pathology enables remote access or consults and powerful image analysis algorithms. However, the slide digitization process can create artifacts such as out-of-focus (OOF). OOF is often only detected upon careful review, potentially causing rescanning and workflow delays. Although scan-time operator screening for whole-slide OOF is feasible, manual screening for OOF affecting only parts of… ▽ More Digital pathology enables remote access or consults and powerful image analysis algorithms. However, the slide digitization process can create artifacts such as out-of-focus (OOF). OOF is often only detected upon careful review, potentially causing rescanning and workflow delays. Although scan-time operator screening for whole-slide OOF is feasible, manual screening for OOF affecting only parts of a slide is impractical. We developed a convolutional neural network (ConvFocus) to exhaustively localize and quantify the severity of OOF regions on digitized slides. ConvFocus was developed using our refined semi-synthetic OOF data generation process, and evaluated using real whole-slide images spanning 3 different tissue types and 3 different stain types that were digitized by two different scanners. ConvFocus's predictions were compared with pathologist-annotated focus quality grades across 514 distinct regions representing 37,700 35x35 $μ$m image patches, and 21 digitized "z-stack" whole-slide images that contain known OOF patterns. When compared to pathologist-graded focus quality, ConvFocus achieved Spearman rank coefficients of 0.81 and 0.94 on two scanners, and reproduced the expected OOF patterns from z-stack scanning. We also evaluated the impact of OOF on the accuracy of a state-of-the-art metastatic breast cancer detector and saw a consistent decrease in performance with increasing OOF. Comprehensive whole-slide OOF categorization could enable rescans prior to pathologist review, potentially reducing the impact of digitization focus issues on the clinical workflow. We show that the algorithm trained on our semi-synthetic OOF data generalizes well to real OOF regions across tissue types, stains, and scanners. Finally, quantitative OOF maps can flag regions that might otherwise be misclassified by image analysis algorithms, preventing OOF-induced errors. △ Less

Submitted 5 February, 2019; v1 submitted 14 January, 2019; originally announced January 2019.

Journal ref: Pathology Informatics (2019)

Microscope 2.0: An Augmented Reality Microscope with Real-time Artificial Intelligence Integration

Authors: Po-Hsuan Cameron Chen, Krishna Gadepalli, Robert MacDonald, Yun Liu, Kunal Nagpal, Timo Kohlberger, Jeffrey Dean, Greg S. Corrado, Jason D. Hipp, Martin C. Stumpe

Abstract: The brightfield microscope is instrumental in the visual examination of both biological and physical samples at sub-millimeter scales. One key clinical application has been in cancer histopathology, where the microscopic assessment of the tissue samples is used for the diagnosis and staging of cancer and thus guides clinical therapy. However, the interpretation of these samples is inherently subje… ▽ More The brightfield microscope is instrumental in the visual examination of both biological and physical samples at sub-millimeter scales. One key clinical application has been in cancer histopathology, where the microscopic assessment of the tissue samples is used for the diagnosis and staging of cancer and thus guides clinical therapy. However, the interpretation of these samples is inherently subjective, resulting in significant diagnostic variability. Moreover, in many regions of the world, access to pathologists is severely limited due to lack of trained personnel. In this regard, Artificial Intelligence (AI) based tools promise to improve the access and quality of healthcare. However, despite significant advances in AI research, integration of these tools into real-world cancer diagnosis workflows remains challenging because of the costs of image digitization and difficulties in deploying AI solutions. Here we propose a cost-effective solution to the integration of AI: the Augmented Reality Microscope (ARM). The ARM overlays AI-based information onto the current view of the sample through the optical pathway in real-time, enabling seamless integration of AI into the regular microscopy workflow. We demonstrate the utility of ARM in the detection of lymph node metastases in breast cancer and the identification of prostate cancer with a latency that supports real-time workflows. We anticipate that ARM will remove barriers towards the use of AI in microscopic analysis and thus improve the accuracy and efficiency of cancer diagnosis. This approach is applicable to other microscopy tasks and AI algorithms in the life sciences and beyond. △ Less

Submitted 4 December, 2018; v1 submitted 21 November, 2018; originally announced December 2018.

Journal ref: Nature Medicine (2019)

Development and Validation of a Deep Learning Algorithm for Improving Gleason Scoring of Prostate Cancer

Authors: Kunal Nagpal, Davis Foote, Yun Liu, Po-Hsuan, Chen, Ellery Wulczyn, Fraser Tan, Niels Olson, Jenny L. Smith, Arash Mohtashamian, James H. Wren, Greg S. Corrado, Robert MacDonald, Lily H. Peng, Mahul B. Amin, Andrew J. Evans, Ankur R. Sangoi, Craig H. Mermel, Jason D. Hipp, Martin C. Stumpe

Abstract: For prostate cancer patients, the Gleason score is one of the most important prognostic factors, potentially determining treatment independent of the stage. However, Gleason scoring is based on subjective microscopic examination of tumor morphology and suffers from poor reproducibility. Here we present a deep learning system (DLS) for Gleason scoring whole-slide images of prostatectomies. Our syst… ▽ More For prostate cancer patients, the Gleason score is one of the most important prognostic factors, potentially determining treatment independent of the stage. However, Gleason scoring is based on subjective microscopic examination of tumor morphology and suffers from poor reproducibility. Here we present a deep learning system (DLS) for Gleason scoring whole-slide images of prostatectomies. Our system was developed using 112 million pathologist-annotated image patches from 1,226 slides, and evaluated on an independent validation dataset of 331 slides, where the reference standard was established by genitourinary specialist pathologists. On the validation dataset, the mean accuracy among 29 general pathologists was 0.61. The DLS achieved a significantly higher diagnostic accuracy of 0.70 (p=0.002) and trended towards better patient risk stratification in correlations to clinical follow-up data. Our approach could improve the accuracy of Gleason scoring and subsequent therapy decisions, particularly where specialist expertise is unavailable. The DLS also goes beyond the current Gleason system to more finely characterize and quantitate tumor morphology, providing opportunities for refinement of the Gleason system itself. △ Less

Submitted 15 November, 2018; originally announced November 2018.

Journal ref: Nature Partner Journal Digital Medicine (2019)

Detecting Cancer Metastases on Gigapixel Pathology Images

Authors: Yun Liu, Krishna Gadepalli, Mohammad Norouzi, George E. Dahl, Timo Kohlberger, Aleksey Boyko, Subhashini Venugopalan, Aleksei Timofeev, Philip Q. Nelson, Greg S. Corrado, Jason D. Hipp, Lily Peng, Martin C. Stumpe

Abstract: Each year, the treatment decisions for more than 230,000 breast cancer patients in the U.S. hinge on whether the cancer has metastasized away from the breast. Metastasis detection is currently performed by pathologists reviewing large expanses of biological tissues. This process is labor intensive and error-prone. We present a framework to automatically detect and localize tumors as small as 100 x… ▽ More Each year, the treatment decisions for more than 230,000 breast cancer patients in the U.S. hinge on whether the cancer has metastasized away from the breast. Metastasis detection is currently performed by pathologists reviewing large expanses of biological tissues. This process is labor intensive and error-prone. We present a framework to automatically detect and localize tumors as small as 100 x 100 pixels in gigapixel microscopy images sized 100,000 x 100,000 pixels. Our method leverages a convolutional neural network (CNN) architecture and obtains state-of-the-art results on the Camelyon16 dataset in the challenging lesion-level tumor detection task. At 8 false positives per image, we detect 92.4% of the tumors, relative to 82.7% by the previous best automated approach. For comparison, a human pathologist attempting exhaustive search achieved 73.2% sensitivity. We achieve image-level AUC scores above 97% on both the Camelyon16 test set and an independent set of 110 slides. In addition, we discover that two slides in the Camelyon16 training set were erroneously labeled normal. Our approach could considerably reduce false negative rates in metastasis detection. △ Less

Submitted 7 March, 2017; v1 submitted 3 March, 2017; originally announced March 2017.

Comments: Fig 1: normal and tumor patches were accidentally reversed - now fixed. Minor grammatical corrections in appendix, section "Image Color Normalization"

Journal ref: MICCAI Tutorial (2017)