Evaluating AI systems under uncertain ground truth: a case study in dermatology
Authors:
David Stutz,
Ali Taylan Cemgil,
Abhijit Guha Roy,
Tatiana Matejovicova,
Melih Barsbey,
Patricia Strachan,
Mike Schaekermann,
Jan Freyberg,
Rajeev Rikhye,
Beverly Freeman,
Javier Perez Matos,
Umesh Telang,
Dale R. Webster,
Yuan Liu,
Greg S. Corrado,
Yossi Matias,
Pushmeet Kohli,
Yun Liu,
Arnaud Doucet,
Alan Karthikesalingam
Abstract:
For safety, AI systems in health undergo thorough evaluations before deployment, validating their predictions against a ground truth that is assumed certain. However, this is actually not the case and the ground truth may be uncertain. Unfortunately, this is largely ignored in standard evaluation of AI models but can have severe consequences such as overestimating the future performance. To avoid…
▽ More
For safety, AI systems in health undergo thorough evaluations before deployment, validating their predictions against a ground truth that is assumed certain. However, this is actually not the case and the ground truth may be uncertain. Unfortunately, this is largely ignored in standard evaluation of AI models but can have severe consequences such as overestimating the future performance. To avoid this, we measure the effects of ground truth uncertainty, which we assume decomposes into two main components: annotation uncertainty which stems from the lack of reliable annotations, and inherent uncertainty due to limited observational information. This ground truth uncertainty is ignored when estimating the ground truth by deterministically aggregating annotations, e.g., by majority voting or averaging. In contrast, we propose a framework where aggregation is done using a statistical model. Specifically, we frame aggregation of annotations as posterior inference of so-called plausibilities, representing distributions over classes in a classification setting, subject to a hyper-parameter encoding annotator reliability. Based on this model, we propose a metric for measuring annotation uncertainty and provide uncertainty-adjusted metrics for performance evaluation. We present a case study applying our framework to skin condition classification from images where annotations are provided in the form of differential diagnoses. The deterministic adjudication process called inverse rank normalization (IRN) from previous work ignores ground truth uncertainty in evaluation. Instead, we present two alternative statistical models: a probabilistic version of IRN and a Plackett-Luce-based model. We find that a large portion of the dataset exhibits significant ground truth uncertainty and standard IRN-based evaluation severely over-estimates performance without providing uncertainty estimates.
△ Less
Submitted 5 July, 2023;
originally announced July 2023.
Characterization of harmonic and subharmonic functions via mean-value properties
Authors:
Pedro Freitas,
Joao Palhoto Matos
Abstract:
We give a characterization of harmonic and subharmonic functions in terms of their mean values in balls and on spheres. This includes the converse of an inequality of Beardon's for subharmonic functions. We also obtain integral inequalities of Harnack type between these two means in general domains.
We give a characterization of harmonic and subharmonic functions in terms of their mean values in balls and on spheres. This includes the converse of an inequality of Beardon's for subharmonic functions. We also obtain integral inequalities of Harnack type between these two means in general domains.
△ Less
Submitted 11 June, 2001;
originally announced June 2001.
