Challenges with the application and adoption of artificial intelligence for drug discovery
Authors:
Ghita Ghislat,
Saiveth Hernandez-Hernandez,
Chayanit Piwajanusorn,
Pedro J. Ballester
Abstract:
Artificial intelligence (AI) is exhibiting tremendous potential to reduce the massive costs and long timescales of drug discovery. There are however important challenges limiting the impact and scope of AI models. Typically, these models are evaluated on benchmarks that are unlikely to anticipate their prospective performance, which inadvertently misguides their development. Indeed, while all the…
▽ More
Artificial intelligence (AI) is exhibiting tremendous potential to reduce the massive costs and long timescales of drug discovery. There are however important challenges limiting the impact and scope of AI models. Typically, these models are evaluated on benchmarks that are unlikely to anticipate their prospective performance, which inadvertently misguides their development. Indeed, while all the developed models excel in a selected benchmark, only a small proportion of them are ultimately reported to have prospective value (e.g. by discovering potent and innovative drug leads for a therapeutic target). Here we discuss a range of data issues (bias, inconsistency, skewness, irrelevance, small size, high dimensionality), how they challenge AI models and which issue-specific mitigations have been effective. Next, we point out the challenges faced by uncertainty quantification techniques aimed at enhancing these AI models. We also discuss how conceptual errors, unrealistic benchmarks and performance misestimation can confound the evaluation of models and thus their development. Lastly, we explain how human bias, whether from AI experts or drug discovery experts, constitutes another challenge that can be alleviated with prospective studies.
△ Less
Submitted 6 July, 2024;
originally announced July 2024.
Scaffold Splits Overestimate Virtual Screening Performance
Authors:
Qianrong Guo,
Saiveth Hernandez-Hernandez,
Pedro J Ballester
Abstract:
Virtual Screening (VS) of vast compound libraries guided by Artificial Intelligence (AI) models is a highly productive approach to early drug discovery. Data splitting is crucial for better benchmarking of such AI models. Traditional random data splits produce similar molecules between training and test sets, conflicting with the reality of VS libraries which mostly contain structurally distinct c…
▽ More
Virtual Screening (VS) of vast compound libraries guided by Artificial Intelligence (AI) models is a highly productive approach to early drug discovery. Data splitting is crucial for better benchmarking of such AI models. Traditional random data splits produce similar molecules between training and test sets, conflicting with the reality of VS libraries which mostly contain structurally distinct compounds. Scaffold split, grou** molecules by shared core structure, is widely considered to reflect this real-world scenario. However, here we show that the scaffold split also overestimates VS performance. The reason is that molecules with different chemical scaffolds are often similar, which hence introduces unrealistically high similarities between training molecules and test molecules following a scaffold split. Our study examined three representative AI models on 60 NCI-60 datasets, each with approximately 30,000 to 50,000 molecules tested on a different cancer cell line. Each dataset was split with three methods: scaffold, Butina clustering and the more accurate Uniform Manifold Approximation and Projection (UMAP) clustering. Regardless of the model, model performance is much worse with UMAP splits from the results of the 2100 models trained and evaluated for each algorithm and split. These robust results demonstrate the need for more realistic data splits to tune, compare, and select models for VS. For the same reason, avoiding the scaffold split is also recommended for other molecular property prediction problems. The code to reproduce these results is available at https://github.com/ScaffoldSplitsOverestimateVS
△ Less
Submitted 30 June, 2024; v1 submitted 2 June, 2024;
originally announced June 2024.
