Showing 1–9 of 9 results for author: Saidi, P

Unraveling overoptimism and publication bias in ML-driven science

Authors: Pouria Saidi, Gautam Dasarathy, Visar Berisha

Abstract: Machine Learning (ML) is increasingly used across many disciplines with impressive reported results across many domain areas. However, recent studies suggest that the published performance of ML models are often overoptimistic. Validity concerns are underscored by findings of an inverse relationship between sample size and reported accuracy in published ML models, contrasting with the theory of le… ▽ More Machine Learning (ML) is increasingly used across many disciplines with impressive reported results across many domain areas. However, recent studies suggest that the published performance of ML models are often overoptimistic. Validity concerns are underscored by findings of an inverse relationship between sample size and reported accuracy in published ML models, contrasting with the theory of learning curves where accuracy should improve or remain stable with increasing sample size. This paper investigates factors contributing to overoptimistic accuracy reports in ML-driven science, focusing on data leakage and publication bias. We introduce a novel stochastic model for observed accuracy, integrating parametric learning curves and the aforementioned biases. We then construct an estimator that corrects for these biases in observed data. Theoretical and empirical results show that our framework can estimate the underlying learning curve, providing realistic performance assessments from published results. Applying the model to meta-analyses in ML-driven science, including neuroimaging-based and speech-based classifications of neurological conditions, we find prevalent overoptimism and estimate the inherent limits of ML-based prediction in each domain. △ Less

Submitted 10 June, 2024; v1 submitted 23 May, 2024; originally announced May 2024.

Comments: 31 pages, 7 figures, 6 tables

Anisotropic diffusion of radiation-induced self-interstitial clusters in HCP zirconium: a molecular dynamics and rate-theory assessment

Authors: Amir Ghorbani, Yu Luo, Peyman Saidi, Laurent Karim Beland

Abstract: Under irradiation, Zr and Zr alloys undergo growth in the absence of applied stress. This phenomenon is thought to be associated with the anisotropy of diffusion of either or both radiation-induced point defects and defect clusters. In this work, molecular dynamic simulations are used to study the anisotropy of diffusion of self-interstitial atom clusters. Both near-equilibrium clusters generated… ▽ More Under irradiation, Zr and Zr alloys undergo growth in the absence of applied stress. This phenomenon is thought to be associated with the anisotropy of diffusion of either or both radiation-induced point defects and defect clusters. In this work, molecular dynamic simulations are used to study the anisotropy of diffusion of self-interstitial atom clusters. Both near-equilibrium clusters generated by aggregation of self-interstitial atoms and cascade-induced clusters were considered. The cascade-induced clusters display more anisotropy than their counterparts produced by aggregation. In addition to 1-dimensional diffusing clusters, 2-dimensional diffusing clusters were observed. Using our molecular dynamic simulations, the input parameters for the "self-interstitial atom cluster bias" rate-theory model were estimated. The radiation-induced growth strains predicted using this model are largely consistent with experiments, but are highly sensitive to the choice of interatomic interaction potential. △ Less

Submitted 11 September, 2023; originally announced September 2023.

Comments: Manuscript and Supplementary Materials included

Active Sequential Two-Sample Testing

Authors: Weizhi Li, Prad Kadambi, Pouria Saidi, Karthikeyan Natesan Ramamurthy, Gautam Dasarathy, Visar Berisha

Abstract: A two-sample hypothesis test is a statistical procedure used to determine whether the distributions generating two samples are identical. We consider the two-sample testing problem in a new scenario where the sample measurements (or sample features) are inexpensive to access, but their group memberships (or labels) are costly. To address the problem, we devise the first \emph{active sequential two… ▽ More A two-sample hypothesis test is a statistical procedure used to determine whether the distributions generating two samples are identical. We consider the two-sample testing problem in a new scenario where the sample measurements (or sample features) are inexpensive to access, but their group memberships (or labels) are costly. To address the problem, we devise the first \emph{active sequential two-sample testing framework} that not only sequentially but also \emph{actively queries}. Our test statistic is a likelihood ratio where one likelihood is found by maximization over all class priors, and the other is provided by a probabilistic classification model. The classification model is adaptively updated and used to predict where the (unlabelled) features have a high dependency on labels; labeling the ``high-dependency'' features leads to the increased power of the proposed testing framework. In theory, we provide the proof that our framework produces an \emph{anytime-valid} $p$-value. In addition, we characterize the proposed framework's gain in testing power by analyzing the mutual information between the feature and label variables in asymptotic and finite-sample scenarios. In practice, we introduce an instantiation of our framework and evaluate it using several experiments; the experiments on the synthetic, MNIST, and application-specific datasets demonstrate that the testing power of the instantiated active sequential test significantly increases while the Type I error is under control. △ Less

Submitted 27 June, 2024; v1 submitted 29 January, 2023; originally announced January 2023.

Support Recovery of Periodic Mixtures with Nested Periodic Dictionaries

Authors: Pouria Saidi, George K. Atia

Abstract: Periodic signals composed of periodic mixtures admit sparse representations in nested periodic dictionaries (NPDs). Therefore, their underlying hidden periods can be estimated by recovering the exact support of said representations. In this paper, support recovery guarantees of such signals are derived both in noise-free and noisy settings. While exact recovery conditions have been studied in the… ▽ More Periodic signals composed of periodic mixtures admit sparse representations in nested periodic dictionaries (NPDs). Therefore, their underlying hidden periods can be estimated by recovering the exact support of said representations. In this paper, support recovery guarantees of such signals are derived both in noise-free and noisy settings. While exact recovery conditions have been studied in the theory of compressive sensing, existing conditions fall short of yielding meaningful achievability regions in the context of periodic signals with sparse representations in NPDs, in part since existing bounds do not capture structures intrinsic to these dictionaries. We leverage known properties of NPDs to derive several conditions for exact sparse recovery of periodic mixtures in the noise-free setting. These conditions rest on newly introduced notions of nested periodic coherence and restricted coherence, which can be efficiently computed. In the presence of noise, we obtain improved conditions for recovering the exact support set of the sparse representation of the periodic mixture via orthogonal matching pursuit based on the introduced notions of coherence. The theoretical findings are corroborated using numerical experiments for different families of NPDs. Our results show significant improvement over generic recovery bounds as the conditions hold over a larger range of sparsity levels. △ Less

Submitted 3 June, 2024; v1 submitted 25 October, 2021; originally announced October 2021.

Comments: 32 pages, 10 figures

Deep Learning and Crystal Plasticity: A Preconditioning Approach for Accurate Orientation Evolution Prediction

Authors: Peyman Saidi, Hadi Pirgazi, Mehdi Sanjari, Saeed Tamimi, Mohsen Mohammadi, Laurent K. Beland, Mark R. Daymond, Isaac Tamblyn

Abstract: Efficient and precise prediction of plasticity by data-driven models relies on appropriate data preparation and a well-designed model. Here we introduce an unsupervised machine learning-based data preparation method to maximize the trainability of crystal orientation evolution data during deformation. For Taylor model crystal plasticity data, the preconditioning procedure improves the test score o… ▽ More Efficient and precise prediction of plasticity by data-driven models relies on appropriate data preparation and a well-designed model. Here we introduce an unsupervised machine learning-based data preparation method to maximize the trainability of crystal orientation evolution data during deformation. For Taylor model crystal plasticity data, the preconditioning procedure improves the test score of an artificial neural network from 0.831 to 0.999, while decreasing the training iterations by an order of magnitude. The efficacy of the approach was further improved with a recurrent neural network. Electron backscattered (EBSD) lab measurements of crystal rotation during rolling were compared with the results of the surrogate model, and despite error introduced by Taylor model simplifying assumptions, very reasonable agreement between the surrogate model and experiment was observed. Our method is foundational for further data-driven studies, enabling the efficient and precise prediction of texture evolution from experimental and simulated crystal plasticity results. △ Less

Submitted 23 June, 2021; originally announced June 2021.

Comments: 15 pages, 6 figures

Interstitialcy-based reordering kinetics of Ni$_3$Al precipitates in irradiated Ni-based super alloys

Authors: Keyvan Ferasat, Thomas D. Swinburne, Peyman Saidi, Mark R. Daymond, Zhongwen Yao, Laurent Karim Béland

Abstract: Neutron irradiation tends to promote disorder in ordered alloys through the action of the thermal spikes that it generates, while simultaneously introducing point defects and defect clusters. As they migrate, these point defects will promote reordering of the alloys, acting against irradiation-induced disordering. In this study, classical molecular dynamics and a highly parallel accelerated sampli… ▽ More Neutron irradiation tends to promote disorder in ordered alloys through the action of the thermal spikes that it generates, while simultaneously introducing point defects and defect clusters. As they migrate, these point defects will promote reordering of the alloys, acting against irradiation-induced disordering. In this study, classical molecular dynamics and a highly parallel accelerated sampling method are used to study the reordering kinetics of Ni$_3$Al under the diffusion of self-interstitial atoms (SIA). By monitoring the order parameter and potential energy from atomistic simulations, we show that the SIA acts as a reordering agent in Ni$_3$Al. A mean-field rate theory model of the interstitialcy-based reordering kinetics is introduced, which reproduces simulation data and predicts reordering at temperatures as low as 500 K. △ Less

Submitted 30 August, 2021; v1 submitted 23 April, 2021; originally announced April 2021.

Journal ref: Materialia. 19 (2021) 101180

Effect of He on the Order-Disorder Transition in Ni3Al under Irradiation

Authors: Peyman Saidi, Pooyan Changizian, Eric Nicholson, He Ken Zhang, Yu Luo, Zhongwen Yao, Chandra Veer Singh, Mark R. Daymond, Laurent Karim Beland

Abstract: The order-disorder transition in Ni-Al alloys under irradiation represents an interplay between various re-ordering processes and disordering due to thermal spikes generated by incident high energy particles. Typically, ordering in enabled by diffusion of thermally-generated vacancies, and can only take place at temperatures where they are mobile and in sufficiently high concentration. Here, in-si… ▽ More The order-disorder transition in Ni-Al alloys under irradiation represents an interplay between various re-ordering processes and disordering due to thermal spikes generated by incident high energy particles. Typically, ordering in enabled by diffusion of thermally-generated vacancies, and can only take place at temperatures where they are mobile and in sufficiently high concentration. Here, in-situ transmission electron micrographs reveal that the presence of He, usually considered to be a deleterious immiscible atom in this material, promotes reordering in Ni3Al at temperatures where vacancies are not effective ordering agents. A rate-theory model is presented, that quantitatively explains this behavior, based on parameters extracted from atomistic simulations. These calculations show that the V2He complex is an effective agent through its high stability and mobility. It is surmised that immiscible atoms may stabilize reordering agents in other materials undergoing driven processes, and preserve ordered phases at temperature where the driven processes would otherwise lead to disorder. △ Less

Submitted 26 February, 2020; originally announced February 2020.

Primary damage production in the presence of extended defects and growth of vacancy-type dislocation loops in hcp zirconium

Authors: Cong Dai, Fei Long, Peyman Saidi, Laurent Karim Beland, Zhongwen Yao, Mark R. Daymond

Abstract: Production rates in long-term predictive radiation damage accumulation models are generally considered independent of the material's microstructure for reactor components. In this study, the effect of pre-existing microstructural elements on primary damage production in alpha-Zr -- and vice-versa -- is assessed by molecular dynamics (MD) simulations. a-type dislocation loops, c-component dislocati… ▽ More Production rates in long-term predictive radiation damage accumulation models are generally considered independent of the material's microstructure for reactor components. In this study, the effect of pre-existing microstructural elements on primary damage production in alpha-Zr -- and vice-versa -- is assessed by molecular dynamics (MD) simulations. a-type dislocation loops, c-component dislocation loops and a tilt grain boundary (GB) were considered. Primary damage production is reduced in the presence of all these microstructural elements, and clustering behavior is dependent on the microstructure. Collision cascades do not cause a-type loop growth or shrinkage, but they cause c-component loop shrinkage. Cascades in the presence of the GBs produce more vacancies than interstitials. This result, as well as other theoretical, MD and experimental evidence, confirm that vacancy loops will grow in the vacancy supersaturated environment near GBs. Distinct temperature-dependent growth regimes are identified. Also, MD reveals cascade-induced events where a-type vacancy loops are absorbed by GBs. Fe segregation at the loops inhibits this cascade-induced absorption. △ Less

Submitted 27 March, 2019; originally announced March 2019.

Detection of Brain Stimuli Using Ramanujan Periodicity Transforms

Authors: Pouria Saidi, Azadeh Vosoughi, George Atia

Abstract: The ability to efficiently match the frequency of the brain's response to repetitive visual stimuli in real time is the basis for reliable SSVEP-based Brain-Computer-Interfacing (BCI). The detection of different stimuli is posed as a composite hypothesis test, where SSVEPs are assumed to admit a sparse representation in a Ramanujan Periodicity Transform (RPT) dictionary. For the binary case, we de… ▽ More The ability to efficiently match the frequency of the brain's response to repetitive visual stimuli in real time is the basis for reliable SSVEP-based Brain-Computer-Interfacing (BCI). The detection of different stimuli is posed as a composite hypothesis test, where SSVEPs are assumed to admit a sparse representation in a Ramanujan Periodicity Transform (RPT) dictionary. For the binary case, we develop and analyze the performance of an RPT detector based on a derived generalized likelihood ratio test. Our approach is extended to multi-hypothesis multi-electrode settings, where we capture the spatial correlation between the electrodes using pre-stimulus data. We also introduce a new metric for evaluating SSVEP detection schemes based on their achievable efficiency and discrimination rate tradeoff for given system resources. We obtain exact distributions of the test statistic in terms of confluent hypergeometric functions. Results based on extensive simulations with both synthesized and real data indicate that the RPT detector substantially outperforms spectral-based methods. Its performance also surpasses the state-of-the-art Canonical Correlation Analysis (CCA) methods with respect to accuracy and sample complexity in short data lengths regimes crucial for real-time applications. The proposed approach is asymptotically optimal as it closes the gap to a perfect measurement bound as the data length increases. In contrast to existing supervised methods which are highly data-dependent, the RPT detector only uses pre-stimulus data to estimate the per-subject spatial correlation, thereby dispensing with considerable overhead associated with data collection for a large number of subjects and stimuli. Our work advances the theory and practice of emerging real-time BCI and affords a new framework for comparing SSVEP detection schemes across a wider spectrum of operating regimes. △ Less

Submitted 8 December, 2018; v1 submitted 27 January, 2018; originally announced January 2018.