Showing 1–13 of 13 results for author: Leenings, R

GateNet: A novel Neural Network Architecture for Automated Flow Cytometry Gating

Authors: Lukas Fisch, Michael O. Heming, Andreas Schulte-Mecklenbeck, Catharina C. Gross, Stefan Zumdick, Carlotta Barkhau, Daniel Emden, Jan Ernsting, Ramona Leenings, Kelvin Sarink, Nils R. Winter, Udo Dannlowski, Heinz Wiendl, Gerd Meyer zu Hörste, Tim Hahn

Abstract: Flow cytometry is widely used to identify cell populations in patient-derived fluids such as peripheral blood (PB) or cerebrospinal fluid (CSF). While ubiquitous in research and clinical practice, flow cytometry requires gating, i.e. cell type identification which requires labor-intensive and error-prone manual adjustments. To facilitate this process, we designed GateNet, the first neural network… ▽ More Flow cytometry is widely used to identify cell populations in patient-derived fluids such as peripheral blood (PB) or cerebrospinal fluid (CSF). While ubiquitous in research and clinical practice, flow cytometry requires gating, i.e. cell type identification which requires labor-intensive and error-prone manual adjustments. To facilitate this process, we designed GateNet, the first neural network architecture enabling full end-to-end automated gating without the need to correct for batch effects. We train GateNet with over 8,000,000 events based on N=127 PB and CSF samples which were manually labeled independently by four experts. We show that for novel, unseen samples, GateNet achieves human-level performance (F1 score ranging from 0.910 to 0.997). In addition we apply GateNet to a publicly available dataset confirming generalization with an F1 score of 0.936. As our implementation utilizes graphics processing units (GPU), gating only needs 15 microseconds per event. Importantly, we also show that GateNet only requires ~10 samples to reach human-level performance, rendering it widely applicable in all domains of flow cytometry. △ Less

Submitted 12 December, 2023; originally announced December 2023.

DenseNet and Support Vector Machine classifications of major depressive disorder using vertex-wise cortical features

Authors: Vladimir Belov, Tracy Erwin-Grabner, Ling-Li Zeng, Christopher R. K. Ching, Andre Aleman, Alyssa R. Amod, Zeynep Basgoze, Francesco Benedetti, Bianca Besteher, Katharina Brosch, Robin Bülow, Romain Colle, Colm G. Connolly, Emmanuelle Corruble, Baptiste Couvy-Duchesne, Kathryn Cullen, Udo Dannlowski, Christopher G. Davey, Annemiek Dols, Jan Ernsting, Jennifer W. Evans, Lukas Fisch, Paola Fuentes-Claramonte, Ali Saffet Gonul, Ian H. Gotlib , et al. (63 additional authors not shown)

Abstract: Major depressive disorder (MDD) is a complex psychiatric disorder that affects the lives of hundreds of millions of individuals around the globe. Even today, researchers debate if morphological alterations in the brain are linked to MDD, likely due to the heterogeneity of this disorder. The application of deep learning tools to neuroimaging data, capable of capturing complex non-linear patterns, h… ▽ More Major depressive disorder (MDD) is a complex psychiatric disorder that affects the lives of hundreds of millions of individuals around the globe. Even today, researchers debate if morphological alterations in the brain are linked to MDD, likely due to the heterogeneity of this disorder. The application of deep learning tools to neuroimaging data, capable of capturing complex non-linear patterns, has the potential to provide diagnostic and predictive biomarkers for MDD. However, previous attempts to demarcate MDD patients and healthy controls (HC) based on segmented cortical features via linear machine learning approaches have reported low accuracies. In this study, we used globally representative data from the ENIGMA-MDD working group containing an extensive sample of people with MDD (N=2,772) and HC (N=4,240), which allows a comprehensive analysis with generalizable results. Based on the hypothesis that integration of vertex-wise cortical features can improve classification performance, we evaluated the classification of a DenseNet and a Support Vector Machine (SVM), with the expectation that the former would outperform the latter. As we analyzed a multi-site sample, we additionally applied the ComBat harmonization tool to remove potential nuisance effects of site. We found that both classifiers exhibited close to chance performance (balanced accuracy DenseNet: 51%; SVM: 53%), when estimated on unseen sites. Slightly higher classification performance (balanced accuracy DenseNet: 58%; SVM: 55%) was found when the cross-validation folds contained subjects from all sites, indicating site effect. In conclusion, the integration of vertex-wise morphometric features and the use of the non-linear classifier did not lead to the differentiability between MDD and HC. Our results support the notion that MDD classification on this combination of features and classifiers is unfeasible. △ Less

Submitted 18 November, 2023; originally announced November 2023.

Deepbet: Fast brain extraction of T1-weighted MRI using Convolutional Neural Networks

Authors: Lukas Fisch, Stefan Zumdick, Carlotta Barkhau, Daniel Emden, Jan Ernsting, Ramona Leenings, Kelvin Sarink, Nils R. Winter, Benjamin Risse, Udo Dannlowski, Tim Hahn

Abstract: Brain extraction in magnetic resonance imaging (MRI) data is an important segmentation step in many neuroimaging preprocessing pipelines. Image segmentation is one of the research fields in which deep learning had the biggest impact in recent years enabling high precision segmentation with minimal compute. Consequently, traditional brain extraction methods are now being replaced by deep learning-b… ▽ More Brain extraction in magnetic resonance imaging (MRI) data is an important segmentation step in many neuroimaging preprocessing pipelines. Image segmentation is one of the research fields in which deep learning had the biggest impact in recent years enabling high precision segmentation with minimal compute. Consequently, traditional brain extraction methods are now being replaced by deep learning-based methods. Here, we used a unique dataset comprising 568 T1-weighted (T1w) MR images from 191 different studies in combination with cutting edge deep learning methods to build a fast, high-precision brain extraction tool called deepbet. deepbet uses LinkNet, a modern UNet architecture, in a two stage prediction process. This increases its segmentation performance, setting a novel state-of-the-art performance during cross-validation with a median Dice score (DSC) of 99.0% on unseen datasets, outperforming current state of the art models (DSC = 97.8% and DSC = 97.9%). While current methods are more sensitive to outliers, resulting in Dice scores as low as 76.5%, deepbet manages to achieve a Dice score of > 96.9% for all samples. Finally, our model accelerates brain extraction by a factor of ~10 compared to current methods, enabling the processing of one image in ~2 seconds on low level hardware. △ Less

Submitted 14 August, 2023; originally announced August 2023.

From Group-Differences to Single-Subject Probability: Conformal Prediction-based Uncertainty Estimation for Brain-Age Modeling

Authors: Jan Ernsting, Nils R. Winter, Ramona Leenings, Kelvin Sarink, Carlotta B. C. Barkhau, Lukas Fisch, Daniel Emden, Vincent Holstein, Jonathan Repple, Dominik Grotegerd, Susanne Meinert, NAKO Investigators, Klaus Berger, Benjamin Risse, Udo Dannlowski, Tim Hahn

Abstract: The brain-age gap is one of the most investigated risk markers for brain changes across disorders. While the field is progressing towards large-scale models, recently incorporating uncertainty estimates, no model to date provides the single-subject risk assessment capability essential for clinical application. In order to enable the clinical use of brain-age as a biomarker, we here combine uncerta… ▽ More The brain-age gap is one of the most investigated risk markers for brain changes across disorders. While the field is progressing towards large-scale models, recently incorporating uncertainty estimates, no model to date provides the single-subject risk assessment capability essential for clinical application. In order to enable the clinical use of brain-age as a biomarker, we here combine uncertainty-aware deep Neural Networks with conformal prediction theory. This approach provides statistical guarantees with respect to single-subject uncertainty estimates and allows for the calculation of an individual's probability for accelerated brain-aging. Building on this, we show empirically in a sample of N=16,794 participants that 1. a lower or comparable error as state-of-the-art, large-scale brain-age models, 2. the statistical guarantees regarding single-subject uncertainty estimation indeed hold for every participant, and 3. that the higher individual probabilities of accelerated brain-aging derived from our model are associated with Alzheimer's Disease, Bipolar Disorder and Major Depressive Disorder. △ Less

Submitted 10 February, 2023; originally announced February 2023.

Comments: arXiv admin note: text overlap with arXiv:2107.07977

More Alike than Different: Quantifying Deviations of Brain Structure and Function in Major Depressive Disorder across Neuroimaging Modalities

Authors: Nils R. Winter, Ramona Leenings, Jan Ernsting, Kelvin Sarink, Lukas Fisch, Daniel Emden, Julian Blanke, Janik Goltermann, Nils Opel, Carlotta Barkhau, Susanne Meinert, Katharina Dohm, Jonathan Repple, Marco Mauritz, Marius Gruber, Elisabeth J. Leehr, Dominik Grotegerd, Ronny Redlich, Andreas Jansen, Igor Nenadic, Markus Nöthen, Andreas Forstner, Marcella Rietschel, Joachim Groß, Jochen Bauer , et al. (6 additional authors not shown)

Abstract: Introduction: Identifying neurobiological differences between patients suffering from Major Depressive Disorder (MDD) and healthy individuals has been a mainstay of clinical neuroscience for decades. However, recent meta- and mega-analyses have raised concerns regarding the replicability and clinical relevance of brain alterations in depression. Methods: Here, we systematically investigate healthy… ▽ More Introduction: Identifying neurobiological differences between patients suffering from Major Depressive Disorder (MDD) and healthy individuals has been a mainstay of clinical neuroscience for decades. However, recent meta- and mega-analyses have raised concerns regarding the replicability and clinical relevance of brain alterations in depression. Methods: Here, we systematically investigate healthy controls and MDD patients across a comprehensive range of modalities including structural magnetic resonance imaging (MRI), diffusion tensor imaging, functional task-based and resting-state MRI under near-ideal conditions. To this end, we quantify the upper bounds of univariate effect sizes, predictive utility, and distributional dissimilarity in a fully harmonized cohort of N=1,809 participants. We compare the results to an MDD polygenic risk score (PRS) and environmental variables. Results: The upper bound of the effect sizes range from partial eta squared = .004 to .017, distributions overlap between 89% and 95%, with classification accuracies ranging between 54% and 55% across neuroimaging modalities. This pattern remains virtually unchanged when considering only acutely or chronically depressed patients. Differences are comparable to those found for PRS, but substantially smaller than for environmental variables. Discussion: We provide a large-scale, multimodal analysis of univariate biological differences between MDD patients and controls and show that even under near-ideal conditions and for maximum biological differences, deviations are extremely small and similarity dominates. We sketch an agenda for a new focus of future research in biological psychiatry facilitating quantitative, theory-driven research, an emphasis on computational psychiatry and multivariate machine learning approaches, as well as the utilization of ecologically valid phenoty**. △ Less

Submitted 20 December, 2021; originally announced December 2021.

Comments: 12 pages, 3 figures

Towards a Network Control Theory of Electroconvulsive Therapy Response

Authors: Tim Hahn, Hamidreza Jamalabadi, Erfan Nozari, Nils R. Winter, Jan Ernsting, Marius Gruber, Marco J. Mauritz, Pascal Grumbach, Lukas Fisch, Ramona Leenings, Kelvin Sarink, Julian Blanke, Leon Kleine Vennekate, Daniel Emden, Nils Opel, Dominik Grotegerd, Verena Enneking, Susanne Meinert, Tiana Borgers, Melissa Klug, Elisabeth J. Leehr, Katharina Dohm, Walter Heindel, Joachim Gross, Udo Dannlowski , et al. (2 additional authors not shown)

Abstract: Electroconvulsive Therapy (ECT) is arguably the most effective intervention for treatment-resistant depression. While large interindividual variability exists, a theory capable of predicting individual response to ECT remains elusive. To address this, we posit a quantitative, mechanistic framework of ECT response based on Network Control Theory (NCT). Then, we empirically test our approach and emp… ▽ More Electroconvulsive Therapy (ECT) is arguably the most effective intervention for treatment-resistant depression. While large interindividual variability exists, a theory capable of predicting individual response to ECT remains elusive. To address this, we posit a quantitative, mechanistic framework of ECT response based on Network Control Theory (NCT). Then, we empirically test our approach and employ it to predict ECT treatment response. To this end, we derive a formal association between Postictal Suppression Index (PSI) - an ECT seizure quality index - and whole-brain modal and average controllability, NCT metrics based on white matter brain network architecture, respectively. Exploiting the known association of ECT response and PSI, we then hypothesized an association between our controllability metrics and ECT response mediated by PSI. We formally tested this conjecture in N=50 depressive patients undergoing ECT. We show that whole-brain controllability metrics based on pre-ECT structural connectome data predict ECT response in accordance with our hypotheses. In addition, we show the expected mediation effects via PSI. Importantly, our theoretically motivated metrics are at least on par with extensive machine learning models based on pre-ECT connectome data. In summary, we derived and tested a control-theoretic framework capable of predicting ECT response based on individual brain network architecture. It makes testable, quantitative predictions regarding individual therapeutic response, which are corroborated by strong empirical evidence. Our work might constitute a starting point for a comprehensive, quantitative theory of personalized ECT interventions rooted in control theory. △ Less

Submitted 14 December, 2021; originally announced December 2021.

A Network Control Theory Approach to Longitudinal Symptom Dynamics in Major Depressive Disorder

Authors: Tim Hahn, Hamidreza Jamalabadi, Daniel Emden, Janik Goltermann, Jan Ernsting, Nils R. Winter, Lukas Fisch, Ramona Leenings, Kelvin Sarink, Vincent Holstein, Marius Gruber, Dominik Grotegerd, Susanne Meinert, Katharina Dohm, Elisabeth J. Leehr, Maike Richter, Lisa Sindermann, Verena Enneking, Hannah Lemke, Stephanie Witt, Marcella Rietschel, Katharina Brosch, Julia-Katharina Pfarr, Tina Meller, Kai Gustav Ringwald , et al. (9 additional authors not shown)

Abstract: Background: The evolution of symptoms over time is at the heart of understanding and treating mental disorders. However, a principled, quantitative framework explaining symptom dynamics remains elusive. Here, we propose a Network Control Theory of Psychopathology allowing us to formally derive a theoretical control energy which we hypothesize quantifies resistance to future symptom improvement in… ▽ More Background: The evolution of symptoms over time is at the heart of understanding and treating mental disorders. However, a principled, quantitative framework explaining symptom dynamics remains elusive. Here, we propose a Network Control Theory of Psychopathology allowing us to formally derive a theoretical control energy which we hypothesize quantifies resistance to future symptom improvement in Major Depressive Disorder (MDD). We test this hypothesis and investigate the relation to genetic and environmental risk as well as resilience. Methods: We modelled longitudinal symptom-network dynamics derived from N=2,059 Beck Depression Inventory measurements acquired over a median of 134 days in a sample of N=109 patients suffering from MDD. We quantified the theoretical energy required for each patient and time-point to reach a symptom-free state given individual symptom-network topology (E 0 ) and 1) tested if E 0 predicts future symptom improvement and 2) whether this relationship is moderated by Polygenic Risk Scores (PRS) of mental disorders, childhood maltreatment experience, and self-reported resilience. Outcomes: We show that E 0 indeed predicts symptom reduction at the next measurement and reveal that this coupling between E 0 and future symptom change increases with higher genetic risk and childhood maltreatment while it decreases with resilience. Interpretation: Our study provides a mechanistic framework capable of predicting future symptom improvement based on individual symptom-network topology and clarifies the role of genetic and environmental risk as well as resilience. Our control-theoretic framework makes testable, quantitative predictions for individual therapeutic response and provides a starting-point for the theory-driven design of personalized interventions. Funding: German Research Foundation and Interdisciplinary Centre for Clinical Research, Münster △ Less

Submitted 21 July, 2021; originally announced July 2021.

Comments: 21 pages, 2 figures

Genetic, Individual, and Familial Risk Correlates of Brain Network Controllability in Major Depressive Disorder

Authors: Tim Hahn, Nils R. Winter, Jan Ernsting, Marius Gruber, Marco J. Mauritz, Lukas Fisch, Ramona Leenings, Kelvin Sarink, Julian Blanke, Vincent Holstein, Daniel Emden, Marie Beisemann, Nils Opel, Dominik Grotegerd, Susanne Meinert, Walter Heindel, Stephanie Witt, Marcella Rietschel, Markus M. Nöthen, Andreas J. Forstner, Tilo Kircher, Igor Nenadic, Andreas Jansen, Bertram Müller-Myhsok, Till F. M. Andlauer , et al. (5 additional authors not shown)

Abstract: Background: A therapeutic intervention in psychiatry can be viewed as an attempt to influence the brain's large-scale, dynamic network state transitions underlying cognition and behavior. Building on connectome-based graph analysis and control theory, Network Control Theory is emerging as a powerful tool to quantify network controllability - i.e., the influence of one brain region over others rega… ▽ More Background: A therapeutic intervention in psychiatry can be viewed as an attempt to influence the brain's large-scale, dynamic network state transitions underlying cognition and behavior. Building on connectome-based graph analysis and control theory, Network Control Theory is emerging as a powerful tool to quantify network controllability - i.e., the influence of one brain region over others regarding dynamic network state transitions. If and how network controllability is related to mental health remains elusive. Methods: From Diffusion Tensor Imaging data, we inferred structural connectivity and inferred calculated network controllability parameters to investigate their association with genetic and familial risk in patients diagnosed with major depressive disorder (MDD, n=692) and healthy controls (n=820). Results: First, we establish that controllability measures differ between healthy controls and MDD patients while not varying with current symptom severity or remission status. Second, we show that controllability in MDD patients is associated with polygenic scores for MDD and psychiatric cross-disorder risk. Finally, we provide evidence that controllability varies with familial risk of MDD and bipolar disorder as well as with body mass index. Conclusions: We show that network controllability is related to genetic, individual, and familial risk in MDD patients. We discuss how these insights into individual variation of network controllability may inform mechanistic models of treatment response prediction and personalized intervention-design in mental health. △ Less

Submitted 21 July, 2021; originally announced July 2021.

Comments: 24 pages, 1 figure

An Uncertainty-Aware, Shareable and Transparent Neural Network Architecture for Brain-Age Modeling

Authors: Tim Hahn, Jan Ernsting, Nils R. Winter, Vincent Holstein, Ramona Leenings, Marie Beisemann, Lukas Fisch, Kelvin Sarink, Daniel Emden, Nils Opel, Ronny Redlich, Jonathan Repple, Dominik Grotegerd, Susanne Meinert, Jochen G. Hirsch, Thoralf Niendorf, Beate Endemann, Fabian Bamberg, Thomas Kröncke, Robin Bülow, Henry Völzke, Oyunbileg von Stackelberg, Ramona Felizitas Sowade, Lale Umutlu, Börge Schmidt , et al. (9 additional authors not shown)

Abstract: The deviation between chronological age and age predicted from neuroimaging data has been identified as a sensitive risk-marker of cross-disorder brain changes, growing into a cornerstone of biological age-research. However, Machine Learning models underlying the field do not consider uncertainty, thereby confounding results with training data density and variability. Also, existing models are com… ▽ More The deviation between chronological age and age predicted from neuroimaging data has been identified as a sensitive risk-marker of cross-disorder brain changes, growing into a cornerstone of biological age-research. However, Machine Learning models underlying the field do not consider uncertainty, thereby confounding results with training data density and variability. Also, existing models are commonly based on homogeneous training sets, often not independently validated, and cannot be shared due to data protection issues. Here, we introduce an uncertainty-aware, shareable, and transparent Monte-Carlo Dropout Composite-Quantile-Regression (MCCQR) Neural Network trained on N=10,691 datasets from the German National Cohort. The MCCQR model provides robust, distribution-free uncertainty quantification in high-dimensional neuroimaging data, achieving lower error rates compared to existing models across ten recruitment centers and in three independent validation samples (N=4,004). In two examples, we demonstrate that it prevents spurious associations and increases power to detect accelerated brain-aging. We make the pre-trained model publicly available. △ Less

Submitted 16 July, 2021; originally announced July 2021.

Predicting brain-age from raw T 1 -weighted Magnetic Resonance Imaging data using 3D Convolutional Neural Networks

Authors: Lukas Fisch, Jan Ernsting, Nils R. Winter, Vincent Holstein, Ramona Leenings, Marie Beisemann, Kelvin Sarink, Daniel Emden, Nils Opel, Ronny Redlich, Jonathan Repple, Dominik Grotegerd, Susanne Meinert, Niklas Wulms, Heike Minnerup, Jochen G. Hirsch, Thoralf Niendorf, Beate Endemann, Fabian Bamberg, Thomas Kröncke, Annette Peters, Robin Bülow, Henry Völzke, Oyunbileg von Stackelberg, Ramona Felizitas Sowade , et al. (11 additional authors not shown)

Abstract: Age prediction based on Magnetic Resonance Imaging (MRI) data of the brain is a biomarker to quantify the progress of brain diseases and aging. Current approaches rely on preparing the data with multiple preprocessing steps, such as registering voxels to a standardized brain atlas, which yields a significant computational overhead, hampers widespread usage and results in the predicted brain-age to… ▽ More Age prediction based on Magnetic Resonance Imaging (MRI) data of the brain is a biomarker to quantify the progress of brain diseases and aging. Current approaches rely on preparing the data with multiple preprocessing steps, such as registering voxels to a standardized brain atlas, which yields a significant computational overhead, hampers widespread usage and results in the predicted brain-age to be sensitive to preprocessing parameters. Here we describe a 3D Convolutional Neural Network (CNN) based on the ResNet architecture being trained on raw, non-registered T$_ 1$-weighted MRI data of N=10,691 samples from the German National Cohort and additionally applied and validated in N=2,173 samples from three independent studies using transfer learning. For comparison, state-of-the-art models using preprocessed neuroimaging data are trained and validated on the same samples. The 3D CNN using raw neuroimaging data predicts age with a mean average deviation of 2.84 years, outperforming the state-of-the-art brain-age models using preprocessed data. Since our approach is invariant to preprocessing software and parameter choices, it enables faster, more robust and more accurate brain-age modeling. △ Less

Submitted 22 March, 2021; originally announced March 2021.

The PHOTON Wizard -- Towards Educational Machine Learning Code Generators

Authors: Ramona Leenings, Nils Ralf Winter, Kelvin Sarink, Jan Ernsting, Xiaoyi Jiang, Udo Dannlowski, Tim Hahn

Abstract: Despite the tremendous efforts to democratize machine learning, especially in applied-science, the application is still often hampered by the lack of coding skills. As we consider programmatic understanding key to building effective and efficient machine learning solutions, we argue for a novel educational approach that builds upon the accessibility and acceptance of graphical user interfaces to c… ▽ More Despite the tremendous efforts to democratize machine learning, especially in applied-science, the application is still often hampered by the lack of coding skills. As we consider programmatic understanding key to building effective and efficient machine learning solutions, we argue for a novel educational approach that builds upon the accessibility and acceptance of graphical user interfaces to convey programming skills to an applied-science target group. We outline a proof-of-concept, open-source web application, the PHOTON Wizard, which dynamically translates GUI interactions into valid source code for the Python machine learning framework PHOTON. Thereby, users possessing theoretical machine learning knowledge gain key insights into the model development workflow as well as an intuitive understanding of custom implementations. Specifically, the PHOTON Wizard integrates the concept of Educational Machine Learning Code Generators to teach users how to write code for designing, training, optimizing and evaluating custom machine learning pipelines. △ Less

Submitted 13 February, 2020; originally announced February 2020.

PHOTONAI -- A Python API for Rapid Machine Learning Model Development

Authors: Ramona Leenings, Nils Ralf Winter, Lucas Plagwitz, Vincent Holstein, Jan Ernsting, Jakob Steenweg, Julian Gebker, Kelvin Sarink, Daniel Emden, Dominik Grotegerd, Nils Opel, Benjamin Risse, Xiaoyi Jiang, Udo Dannlowski, Tim Hahn

Abstract: PHOTONAI is a high-level Python API designed to simplify and accelerate machine learning model development. It functions as a unifying framework allowing the user to easily access and combine algorithms from different toolboxes into custom algorithm sequences. It is especially designed to support the iterative model development process and automates the repetitive training, hyperparameter optimiza… ▽ More PHOTONAI is a high-level Python API designed to simplify and accelerate machine learning model development. It functions as a unifying framework allowing the user to easily access and combine algorithms from different toolboxes into custom algorithm sequences. It is especially designed to support the iterative model development process and automates the repetitive training, hyperparameter optimization and evaluation tasks. Importantly, the workflow ensures unbiased performance estimates while still allowing the user to fully customize the machine learning analysis. PHOTONAI extends existing solutions with a novel pipeline implementation supporting more complex data streams, feature combinations, and algorithm selection. Metrics and results can be conveniently visualized using the PHOTONAI Explorer and predictive models are shareable in a standardized format for further external validation or application. A growing add-on ecosystem allows researchers to offer data modality specific algorithms to the community and enhance machine learning in the areas of the life sciences. Its practical utility is demonstrated on an exemplary medical machine learning problem, achieving a state-of-the-art solution in few lines of code. Source code is publicly available on Github, while examples and documentation can be found at www.photon-ai.com. △ Less

Submitted 7 July, 2021; v1 submitted 13 February, 2020; originally announced February 2020.

Systematic Misestimation of Machine Learning Performance in Neuroimaging Studies of Depression

Authors: Claas Flint, Micah Cearns, Nils Opel, Ronny Redlich, David M. A. Mehler, Daniel Emden, Nils R. Winter, Ramona Leenings, Simon B. Eickhoff, Tilo Kircher, Axel Krug, Igor Nenadic, Volker Arolt, Scott Clark, Bernhard T. Baune, Xiaoyi Jiang, Udo Dannlowski, Tim Hahn

Abstract: We currently observe a disconcerting phenomenon in machine learning studies in psychiatry: While we would expect larger samples to yield better results due to the availability of more data, larger machine learning studies consistently show much weaker performance than the numerous small-scale studies. Here, we systematically investigated this effect focusing on one of the most heavily studied ques… ▽ More We currently observe a disconcerting phenomenon in machine learning studies in psychiatry: While we would expect larger samples to yield better results due to the availability of more data, larger machine learning studies consistently show much weaker performance than the numerous small-scale studies. Here, we systematically investigated this effect focusing on one of the most heavily studied questions in the field, namely the classification of patients suffering from major depressive disorder (MDD) and healthy control (HC) based on neuroimaging data. Drawing upon structural magnetic resonance imaging (MRI) data from a balanced sample of $N = 1,868$ MDD patients and HC from our recent international Predictive Analytics Competition (PAC), we first trained and tested a classification model on the full dataset which yielded an accuracy of $61\,\%$. Next, we mimicked the process by which researchers would draw samples of various sizes ($N = 4$ to $N = 150$) from the population and showed a strong risk of misestimation. Specifically, for small sample sizes ($N = 20$), we observe accuracies of up to $95\,\%$. For medium sample sizes ($N = 100$) accuracies up to $75\,\%$ were found. Importantly, further investigation showed that sufficiently large test sets effectively protect against performance misestimation whereas larger datasets per se do not. While these results question the validity of a substantial part of the current literature, we outline the relatively low-cost remedy of larger test sets, which is readily available in most cases. △ Less

Submitted 3 May, 2021; v1 submitted 13 December, 2019; originally announced December 2019.

Journal ref: Neuropsychopharmacology 46 (2021) 1510-1517