-
Fighting the disagreement in Explainable Machine Learning with consensus
Authors:
Antonio Jesus Banegas-Luna,
Carlos Martınez-Cortes,
Horacio Perez-Sanchez
Abstract:
Machine learning (ML) models are often valued by the accuracy of their predictions. However, in some areas of science, the inner workings of models are as relevant as their accuracy. To understand how ML models work internally, the use of interpretability algorithms is the preferred option. Unfortunately, despite the diversity of algorithms available, they often disagree in explaining a model, lea…
▽ More
Machine learning (ML) models are often valued by the accuracy of their predictions. However, in some areas of science, the inner workings of models are as relevant as their accuracy. To understand how ML models work internally, the use of interpretability algorithms is the preferred option. Unfortunately, despite the diversity of algorithms available, they often disagree in explaining a model, leading to contradictory explanations. To cope with this issue, consensus functions can be applied once the models have been explained. Nevertheless, the problem is not completely solved because the final result will depend on the selected consensus function and other factors. In this paper, six consensus functions have been evaluated for the explanation of five ML models. The models were previously trained on four synthetic datasets whose internal rules were known in advance. The models were then explained with model-agnostic local and global interpretability algorithms. Finally, consensus was calculated with six different functions, including one developed by the authors. The results demonstrated that the proposed function is fairer than the others and provides more consistent and accurate explanations.
△ Less
Submitted 3 July, 2023;
originally announced July 2023.
-
ALMERIA: Boosting pairwise molecular contrasts with scalable methods
Authors:
Rafael Mena-Yedra,
Juana L. Redondo,
Horacio Pérez-Sánchez,
Pilar M. Ortigosa
Abstract:
Searching for potential active compounds in large databases is a necessary step to reduce time and costs in modern drug discovery pipelines. Such virtual screening methods seek to provide predictions that allow the search space to be narrowed down. Although cheminformatics has made great progress in exploiting the potential of available big data, caution is needed to avoid introducing bias and pro…
▽ More
Searching for potential active compounds in large databases is a necessary step to reduce time and costs in modern drug discovery pipelines. Such virtual screening methods seek to provide predictions that allow the search space to be narrowed down. Although cheminformatics has made great progress in exploiting the potential of available big data, caution is needed to avoid introducing bias and provide useful predictions with new compounds. In this work, we propose the decision-support tool ALMERIA (Advanced Ligand Multiconformational Exploration with Robust Interpretable Artificial Intelligence) for estimating compound similarities and activity prediction based on pairwise molecular contrasts while considering their conformation variability. The methodology covers the entire pipeline from data preparation to model selection and hyperparameter optimization. It has been implemented using scalable software and methods to exploit large volumes of data -- in the order of several terabytes -- , offering a very quick response even for a large batch of queries. The implementation and experiments have been performed in a distributed computer cluster using a benchmark, the public access DUD-E database. In addition to cross-validation, detailed data split criteria have been used to evaluate the models on different data partitions to assess their true generalization ability with new compounds. Experiments show state-of-the-art performance for molecular activity prediction (ROC AUC: $0.99$, $0.96$, $0.87$), proving that the chosen data representation and modeling have good properties to generalize. Molecular conformations -- prediction performance and sensitivity analysis -- have also been evaluated. Finally, an interpretability analysis has been performed using the SHAP method.
△ Less
Submitted 28 April, 2023;
originally announced May 2023.
-
SIBILA: A novel interpretable ensemble of general-purpose machine learning models applied to medical contexts
Authors:
Antonio Jesús Banegas-Luna,
Horacio Pérez-Sánchez
Abstract:
Personalized medicine remains a major challenge for scientists. The rapid growth of Machine learning and Deep learning has made them a feasible al- ternative for predicting the most appropriate therapy for individual patients. However, the need to develop a custom model for every dataset, the lack of interpretation of their results and high computational requirements make many reluctant to use the…
▽ More
Personalized medicine remains a major challenge for scientists. The rapid growth of Machine learning and Deep learning has made them a feasible al- ternative for predicting the most appropriate therapy for individual patients. However, the need to develop a custom model for every dataset, the lack of interpretation of their results and high computational requirements make many reluctant to use these methods. Aiming to save time and bring light to the way models work internally, SIBILA has been developed. SIBILA is an ensemble of machine learning and deep learning models that applies a range of interpretability algorithms to identify the most relevant input features. Since the interpretability algo- rithms may not be in line with each other, a consensus stage has been imple- mented to estimate the global attribution of each variable to the predictions. SIBILA is containerized to be run on any high-performance computing plat- form. Although conceived as a command-line tool, it is also available to all users free of charge as a web server at https://bio-hpc.ucam.edu/sibila. Thus, even users with few technological skills can take advantage of it. SIBILA has been applied to two medical case studies to show its ability to predict in classification problems. Even though it is a general-purpose tool, it has been developed with the aim of becoming a powerful decision-making tool for clinicians, but can actually be used in many other domains. Thus, other two non-medical examples are supplied as supplementary material to prove that SIBILA still works well with noise and in regression problems.
△ Less
Submitted 27 April, 2023; v1 submitted 12 May, 2022;
originally announced May 2022.
-
When will the mist clear? On the Interpretability of Machine Learning for Medical Applications: a survey
Authors:
Antonio-Jesús Banegas-Luna,
Jorge Peña-García,
Adrian Iftene,
Fiorella Guadagni,
Patrizia Ferroni,
Noemi Scarpato,
Fabio Massimo Zanzotto,
Andrés Bueno-Crespo,
Horacio Pérez-Sánchez
Abstract:
Artificial Intelligence is providing astonishing results, with medicine being one of its favourite playgrounds. In a few decades, computers may be capable of formulating diagnoses and choosing the correct treatment, while robots may perform surgical operations, and conversational agents could interact with patients as virtual coaches. Machine Learning and, in particular, Deep Neural Networks are b…
▽ More
Artificial Intelligence is providing astonishing results, with medicine being one of its favourite playgrounds. In a few decades, computers may be capable of formulating diagnoses and choosing the correct treatment, while robots may perform surgical operations, and conversational agents could interact with patients as virtual coaches. Machine Learning and, in particular, Deep Neural Networks are behind this revolution. In this scenario, important decisions will be controlled by standalone machines that have learned predictive models from provided data. Among the most challenging targets of interest in medicine are cancer diagnosis and therapies but, to start this revolution, software tools need to be adapted to cover the new requirements. In this sense, learning tools are becoming a commodity in Python and Matlab libraries, just to name two, but to exploit all their possibilities, it is essential to fully understand how models are interpreted and which models are more interpretable than others. In this survey, we analyse current machine learning models, frameworks, databases and other related tools as applied to medicine - specifically, to cancer research - and we discuss their interpretability, performance and the necessary input data. From the evidence available, ANN, LR and SVM have been observed to be the preferred models. Besides, CNNs, supported by the rapid development of GPUs and tensor-oriented programming libraries, are gaining in importance. However, the interpretability of results by doctors is rarely considered which is a factor that needs to be improved. We therefore consider this study to be a timely contribution to the issue.
△ Less
Submitted 1 October, 2020;
originally announced October 2020.
