-
Differences between human and machine perception in medical diagnosis
Authors:
Taro Makino,
Stanislaw Jastrzebski,
Witold Oleszkiewicz,
Celin Chacko,
Robin Ehrenpreis,
Naziya Samreen,
Chloe Chhor,
Eric Kim,
Jiyon Lee,
Kristine Pysarenko,
Beatriu Reig,
Hildegard Toth,
Divya Awal,
Linda Du,
Alice Kim,
James Park,
Daniel K. Sodickson,
Laura Heacock,
Linda Moy,
Kyunghyun Cho,
Krzysztof J. Geras
Abstract:
Deep neural networks (DNNs) show promise in image-based medical diagnosis, but cannot be fully trusted since their performance can be severely degraded by dataset shifts to which human perception remains invariant. If we can better understand the differences between human and machine perception, we can potentially characterize and mitigate this effect. We therefore propose a framework for comparin…
▽ More
Deep neural networks (DNNs) show promise in image-based medical diagnosis, but cannot be fully trusted since their performance can be severely degraded by dataset shifts to which human perception remains invariant. If we can better understand the differences between human and machine perception, we can potentially characterize and mitigate this effect. We therefore propose a framework for comparing human and machine perception in medical diagnosis. The two are compared with respect to their sensitivity to the removal of clinically meaningful information, and to the regions of an image deemed most suspicious. Drawing inspiration from the natural image domain, we frame both comparisons in terms of perturbation robustness. The novelty of our framework is that separate analyses are performed for subgroups with clinically meaningful differences. We argue that this is necessary in order to avert Simpson's paradox and draw correct conclusions. We demonstrate our framework with a case study in breast cancer screening, and reveal significant differences between radiologists and DNNs. We compare the two with respect to their robustness to Gaussian low-pass filtering, performing a subgroup analysis on microcalcifications and soft tissue lesions. For microcalcifications, DNNs use a separate set of high frequency components than radiologists, some of which lie outside the image regions considered most suspicious by radiologists. These features run the risk of being spurious, but if not, could represent potential new biomarkers. For soft tissue lesions, the divergence between radiologists and DNNs is even starker, with DNNs relying heavily on spurious high frequency components ignored by radiologists. Importantly, this deviation in soft tissue lesions was only observable through subgroup analysis, which highlights the importance of incorporating medical domain knowledge into our comparison framework.
△ Less
Submitted 27 November, 2020;
originally announced November 2020.
-
Deep Neural Networks Improve Radiologists' Performance in Breast Cancer Screening
Authors:
Nan Wu,
Jason Phang,
Jungkyu Park,
Yiqiu Shen,
Zhe Huang,
Masha Zorin,
Stanisław Jastrzębski,
Thibault Févry,
Joe Katsnelson,
Eric Kim,
Stacey Wolfson,
Ujas Parikh,
Sushma Gaddam,
Leng Leng Young Lin,
Kara Ho,
Joshua D. Weinstein,
Beatriu Reig,
Yiming Gao,
Hildegard Toth,
Kristine Pysarenko,
Alana Lewin,
Jiyon Lee,
Krystal Airola,
Eralda Mema,
Stephanie Chung
, et al. (7 additional authors not shown)
Abstract:
We present a deep convolutional neural network for breast cancer screening exam classification, trained and evaluated on over 200,000 exams (over 1,000,000 images). Our network achieves an AUC of 0.895 in predicting whether there is a cancer in the breast, when tested on the screening population. We attribute the high accuracy of our model to a two-stage training procedure, which allows us to use…
▽ More
We present a deep convolutional neural network for breast cancer screening exam classification, trained and evaluated on over 200,000 exams (over 1,000,000 images). Our network achieves an AUC of 0.895 in predicting whether there is a cancer in the breast, when tested on the screening population. We attribute the high accuracy of our model to a two-stage training procedure, which allows us to use a very high-capacity patch-level network to learn from pixel-level labels alongside a network learning from macroscopic breast-level labels. To validate our model, we conducted a reader study with 14 readers, each reading 720 screening mammogram exams, and find our model to be as accurate as experienced radiologists when presented with the same data. Finally, we show that a hybrid model, averaging probability of malignancy predicted by a radiologist with a prediction of our neural network, is more accurate than either of the two separately. To better understand our results, we conduct a thorough analysis of our network's performance on different subpopulations of the screening population, model design, training procedure, errors, and properties of its internal representations.
△ Less
Submitted 19 March, 2019;
originally announced March 2019.
-
Behavioral subty** through typed assertions
Authors:
Herbert Toth
Abstract:
This paper presents a critical discussion of popular approaches to ensure the Liskov substitution principle in class hierarchies (e.g. Design by Contract(TM), specification inheritance). It will be shown that they have some deficiencies which are due to the way how effective constraints are calculated for subclass methods. A new mechanism, called client conformance, is introduced that takes the cl…
▽ More
This paper presents a critical discussion of popular approaches to ensure the Liskov substitution principle in class hierarchies (e.g. Design by Contract(TM), specification inheritance). It will be shown that they have some deficiencies which are due to the way how effective constraints are calculated for subclass methods. A new mechanism, called client conformance, is introduced that takes the client's view on the program state into account more properly: The client's static type determines the context in which reasoning about program state is to be done. This is the context to which the runtime assertion checking (RAC) of server methods must be adapted appropriately. In a stepwise argumentation we show the improvements for RAC that can be reached following this approach in a natural way, preserving the percolation pattern mechanism: Clients will neither be confronted with unsafe or surprising executions, nor with surprising failures of server methods.
△ Less
Submitted 31 July, 2013; v1 submitted 30 December, 2010;
originally announced January 2011.
