-
A study of why we need to reassess full reference image quality assessment with medical images
Authors:
Anna Breger,
Ander Biguri,
Malena Sabaté Landman,
Ian Selby,
Nicole Amberg,
Elisabeth Brunner,
Janek Gröhl,
Sepideh Hatamikia,
Clemens Karner,
Lipeng Ning,
Sören Dittmer,
Michael Roberts,
AIX-COVNET Collaboration,
Carola-Bibiane Schönlieb
Abstract:
Image quality assessment (IQA) is not just indispensable in clinical practice to ensure high standards, but also in the development stage of novel algorithms that operate on medical images with reference data. This paper provides a structured and comprehensive collection of examples where the two most common full reference (FR) image quality measures prove to be unsuitable for the assessment of no…
▽ More
Image quality assessment (IQA) is not just indispensable in clinical practice to ensure high standards, but also in the development stage of novel algorithms that operate on medical images with reference data. This paper provides a structured and comprehensive collection of examples where the two most common full reference (FR) image quality measures prove to be unsuitable for the assessment of novel algorithms using different kinds of medical images, including real-world MRI, CT, OCT, X-Ray, digital pathology and photoacoustic imaging data. In particular, the FR-IQA measures PSNR and SSIM are known and tested for working successfully in many natural imaging tasks, but discrepancies in medical scenarios have been noted in the literature. Inconsistencies arising in medical images are not surprising, as they have very different properties than natural images which have not been targeted nor tested in the development of the mentioned measures, and therefore might imply wrong judgement of novel methods for medical images. Therefore, improvement is urgently needed in particular in this era of AI to increase explainability, reproducibility and generalizability in machine learning for medical imaging and beyond. On top of the pitfalls we will provide ideas for future research as well as suggesting guidelines for the usage of FR-IQA measures applied to medical images.
△ Less
Submitted 29 May, 2024;
originally announced May 2024.
-
Navigating the challenges in creating complex data systems: a development philosophy
Authors:
Sören Dittmer,
Michael Roberts,
Julian Gilbey,
Ander Biguri,
AIX-COVNET Collaboration,
Jacobus Preller,
James H. F. Rudd,
John A. D. Aston,
Carola-Bibiane Schönlieb
Abstract:
In this perspective, we argue that despite the democratization of powerful tools for data science and machine learning over the last decade, develo** the code for a trustworthy and effective data science system (DSS) is getting harder. Perverse incentives and a lack of widespread software engineering (SE) skills are among many root causes we identify that naturally give rise to the current syste…
▽ More
In this perspective, we argue that despite the democratization of powerful tools for data science and machine learning over the last decade, develo** the code for a trustworthy and effective data science system (DSS) is getting harder. Perverse incentives and a lack of widespread software engineering (SE) skills are among many root causes we identify that naturally give rise to the current systemic crisis in reproducibility of DSSs. We analyze why SE and building large complex systems is, in general, hard. Based on these insights, we identify how SE addresses those difficulties and how we can apply and generalize SE methods to construct DSSs that are fit for purpose. We advocate two key development philosophies, namely that one should incrementally grow -- not biphasically plan and build -- DSSs, and one should always employ two types of feedback loops during development: one which tests the code's correctness and another that evaluates the code's efficacy.
△ Less
Submitted 21 October, 2022;
originally announced October 2022.
-
Classification of datasets with imputed missing values: does imputation quality matter?
Authors:
Tolou Shadbahr,
Michael Roberts,
Jan Stanczuk,
Julian Gilbey,
Philip Teare,
Sören Dittmer,
Matthew Thorpe,
Ramon Vinas Torne,
Evis Sala,
Pietro Lio,
Mishal Patel,
AIX-COVNET Collaboration,
James H. F. Rudd,
Tuomas Mirtti,
Antti Rannikko,
John A. D. Aston,
**g Tang,
Carola-Bibiane Schönlieb
Abstract:
Classifying samples in incomplete datasets is a common aim for machine learning practitioners, but is non-trivial. Missing data is found in most real-world datasets and these missing values are typically imputed using established methods, followed by classification of the now complete, imputed, samples. The focus of the machine learning researcher is then to optimise the downstream classification…
▽ More
Classifying samples in incomplete datasets is a common aim for machine learning practitioners, but is non-trivial. Missing data is found in most real-world datasets and these missing values are typically imputed using established methods, followed by classification of the now complete, imputed, samples. The focus of the machine learning researcher is then to optimise the downstream classification performance. In this study, we highlight that it is imperative to consider the quality of the imputation. We demonstrate how the commonly used measures for assessing quality are flawed and propose a new class of discrepancy scores which focus on how well the method recreates the overall distribution of the data. To conclude, we highlight the compromised interpretability of classifier models trained using poorly imputed data.
△ Less
Submitted 16 June, 2022;
originally announced June 2022.
-
The Average Size of Giant Components Between the Double-Jump
Authors:
Vlady Ravelomanana,
the Projet PAI Amadeus Collaboration
Abstract:
We study the sizes of connected components according to their excesses during a random graph process built with $n$ vertices. The considered model is the continuous one defined in Janson 2000. An ${\ell}$-component is a connected component with ${\ell}$ edges more than vertices. $\ell$ is also called the \textit{excess} of such component. As our main result, we show that when $\ell$ and…
▽ More
We study the sizes of connected components according to their excesses during a random graph process built with $n$ vertices. The considered model is the continuous one defined in Janson 2000. An ${\ell}$-component is a connected component with ${\ell}$ edges more than vertices. $\ell$ is also called the \textit{excess} of such component. As our main result, we show that when $\ell$ and ${n \over \ell}$ are both large, the expected number of vertices that ever belong to an $\ell$-component is about ${12}^{1/3} {\ell}^{1/3} n^{2/3}$. We also obtain limit theorems for the number of creations of $\ell$-components.
△ Less
Submitted 12 July, 2006;
originally announced July 2006.
