Showing 1–50 of 166 results for author: Wu, X

A Partition-insensitive Parallel Framework for Distributed Model Fitting

Authors: Xiaofei Wu, Rongmei Liang, Fabio Roli, Marcello Pelillo, **g Yuan

Abstract: Distributed model fitting refers to the process of fitting a mathematical or statistical model to the data using distributed computing resources, such that computing tasks are divided among multiple interconnected computers or nodes, often organized in a cluster or network. Most of the existing methods for distributed model fitting are to formulate it in a consensus optimization problem, and then… ▽ More Distributed model fitting refers to the process of fitting a mathematical or statistical model to the data using distributed computing resources, such that computing tasks are divided among multiple interconnected computers or nodes, often organized in a cluster or network. Most of the existing methods for distributed model fitting are to formulate it in a consensus optimization problem, and then build up algorithms based on the alternating direction method of multipliers (ADMM). This paper introduces a novel parallel framework for achieving a distributed model fitting. In contrast to previous consensus frameworks, the introduced parallel framework offers two notable advantages. Firstly, it exhibits insensitivity to sample partitioning, meaning that the solution of the algorithm remains unaffected by variations in the number of slave nodes or/and the amount of data each node carries. Secondly, fewer variables are required to be updated at each iteration, so that the proposed parallel framework performs in a more succinct and efficient way, and adapts to high-dimensional data. In addition, we prove that the algorithms under the new parallel framework have a worst-case linear convergence rate in theory. Numerical experiments confirm the generality, robustness, and accuracy of our proposed parallel framework. △ Less

Submitted 2 June, 2024; originally announced June 2024.

On the Role of Attention Masks and LayerNorm in Transformers

Authors: Xinyi Wu, Amir Ajorlou, Yifei Wang, Stefanie Jegelka, Ali Jadbabaie

Abstract: Self-attention is the key mechanism of transformers, which are the essential building blocks of modern foundation models. Recent studies have shown that pure self-attention suffers from an increasing degree of rank collapse as depth increases, limiting model expressivity and further utilization of model depth. The existing literature on rank collapse, however, has mostly overlooked other critical… ▽ More Self-attention is the key mechanism of transformers, which are the essential building blocks of modern foundation models. Recent studies have shown that pure self-attention suffers from an increasing degree of rank collapse as depth increases, limiting model expressivity and further utilization of model depth. The existing literature on rank collapse, however, has mostly overlooked other critical components in transformers that may alleviate the rank collapse issue. In this paper, we provide a general analysis of rank collapse under self-attention, taking into account the effects of attention masks and layer normalization (LayerNorm). In particular, we find that although pure masked attention still suffers from exponential collapse to a rank one subspace, local masked attention can provably slow down the collapse rate. In the case of self-attention with LayerNorm, we first show that for certain classes of value matrices, collapse to a rank one subspace still happens exponentially. However, through construction of nontrivial counterexamples, we then establish that with proper choice of value matrices, a general class of sequences may not converge to a rank one subspace, and the self-attention dynamics with LayerNorm can simultaneously possess a rich set of equilibria with any possible rank between one and full. Our result refutes the previous hypothesis that LayerNorm plays no role in the rank collapse of self-attention and suggests that self-attention with LayerNorm constitutes a much more expressive, versatile nonlinear dynamical system than what was originally thought. △ Less

Submitted 29 May, 2024; originally announced May 2024.

Towards Efficient Disaster Response via Cost-effective Unbiased Class Rate Estimation through Neyman Allocation Stratified Sampling Active Learning

Authors: Yanbing Bai, Xinyi Wu, Lai Xu, Jihan Pei, Erick Mas, Shunichi Koshimura

Abstract: With the rapid development of earth observation technology, we have entered an era of massively available satellite remote-sensing data. However, a large amount of satellite remote sensing data lacks a label or the label cost is too high to hinder the potential of AI technology mining satellite data. Especially in such an emergency response scenario that uses satellite data to evaluate the degree… ▽ More With the rapid development of earth observation technology, we have entered an era of massively available satellite remote-sensing data. However, a large amount of satellite remote sensing data lacks a label or the label cost is too high to hinder the potential of AI technology mining satellite data. Especially in such an emergency response scenario that uses satellite data to evaluate the degree of disaster damage. Disaster damage assessment encountered bottlenecks due to excessive focus on the damage of a certain building in a specific geographical space or a certain area on a larger scale. In fact, in the early days of disaster emergency response, government departments were more concerned about the overall damage rate of the disaster area instead of single-building damage, because this helps the government decide the level of emergency response. We present an innovative algorithm that constructs Neyman stratified random sampling trees for binary classification and extends this approach to multiclass problems. Through extensive experimentation on various datasets and model structures, our findings demonstrate that our method surpasses both passive and conventional active learning techniques in terms of class rate estimation and model enhancement with only 30\%-60\% of the annotation cost of simple sampling. It effectively addresses the 'sampling bias' challenge in traditional active learning strategies and mitigates the 'cold start' dilemma. The efficacy of our approach is further substantiated through application to disaster evaluation tasks using Xview2 Satellite imagery, showcasing its practical utility in real-world contexts. △ Less

Submitted 27 May, 2024; originally announced May 2024.

Enhancing Uncertain Demand Prediction in Hospitals Using Simple and Advanced Machine Learning

Authors: Annie Hu, Samuel Stockman, Xun Wu, Richard Wood, Bangdong Zhi, Oliver Y. Chén

Abstract: Early and timely prediction of patient care demand not only affects effective resource allocation but also influences clinical decision-making as well as patient experience. Accurately predicting patient care demand, however, is a ubiquitous challenge for hospitals across the world due, in part, to the demand's time-varying temporal variability, and, in part, to the difficulty in modelling trends… ▽ More Early and timely prediction of patient care demand not only affects effective resource allocation but also influences clinical decision-making as well as patient experience. Accurately predicting patient care demand, however, is a ubiquitous challenge for hospitals across the world due, in part, to the demand's time-varying temporal variability, and, in part, to the difficulty in modelling trends in advance. To address this issue, here, we develop two methods, a relatively simple time-vary linear model, and a more advanced neural network model. The former forecasts patient arrivals hourly over a week based on factors such as day of the week and previous 7-day arrival patterns. The latter leverages a long short-term memory (LSTM) model, capturing non-linear relationships between past data and a three-day forecasting window. We evaluate the predictive capabilities of the two proposed approaches compared to two naïve approaches - a reduced-rank vector autoregressive (VAR) model and the TBATS model. Using patient care demand data from Rambam Medical Center in Israel, our results show that both proposed models effectively capture hourly variations of patient demand. Additionally, the linear model is more explainable thanks to its simple architecture, whereas, by accurately modelling weekly seasonal trends, the LSTM model delivers lower prediction errors. Taken together, our explorations suggest the utility of machine learning in predicting time-varying patient care demand; additionally, it is possible to predict patient care demand with good accuracy (around 4 patients) three days or a week in advance using machine learning. △ Less

Submitted 29 April, 2024; originally announced April 2024.

Causal Diffusion Autoencoders: Toward Counterfactual Generation via Diffusion Probabilistic Models

Authors: Aneesh Komanduri, Chen Zhao, Feng Chen, Xintao Wu

Abstract: Diffusion probabilistic models (DPMs) have become the state-of-the-art in high-quality image generation. However, DPMs have an arbitrary noisy latent space with no interpretable or controllable semantics. Although there has been significant research effort to improve image sample quality, there is little work on representation-controlled generation using diffusion models. Specifically, causal mode… ▽ More Diffusion probabilistic models (DPMs) have become the state-of-the-art in high-quality image generation. However, DPMs have an arbitrary noisy latent space with no interpretable or controllable semantics. Although there has been significant research effort to improve image sample quality, there is little work on representation-controlled generation using diffusion models. Specifically, causal modeling and controllable counterfactual generation using DPMs is an underexplored area. In this work, we propose CausalDiffAE, a diffusion-based causal representation learning framework to enable counterfactual generation according to a specified causal model. Our key idea is to use an encoder to extract high-level semantically meaningful causal variables from high-dimensional data and model stochastic variation using reverse diffusion. We propose a causal encoding mechanism that maps high-dimensional data to causally related latent factors and parameterize the causal mechanisms among latent factors using neural networks. To enforce the disentanglement of causal variables, we formulate a variational objective and leverage auxiliary label information in a prior to regularize the latent space. We propose a DDIM-based counterfactual generation procedure subject to do-interventions. Finally, to address the limited label supervision scenario, we also study the application of CausalDiffAE when a part of the training data is unlabeled, which also enables granular control over the strength of interventions in generating counterfactuals during inference. We empirically show that CausalDiffAE learns a disentangled latent space and is capable of generating high-quality counterfactual images. △ Less

Submitted 8 May, 2024; v1 submitted 26 April, 2024; originally announced April 2024.

Comments: Short version accepted to CVPR 2024 Workshop on Generative Models for Computer Vision

Skewed Pivot-Blend Modeling with Applications to Semicontinuous Outcomes

Authors: Yiyuan She, Xiaoqiang Wu, Lizhu Tao, Debajyoti Sinha

Abstract: Skewness is a common occurrence in statistical applications. In recent years, various distribution families have been proposed to model skewed data by introducing unequal scales based on the median or mode. However, we argue that the point at which unbalanced scales occur may be at any quantile and cannot be reparametrized as an ordinary shift parameter in the presence of skewness. In this paper,… ▽ More Skewness is a common occurrence in statistical applications. In recent years, various distribution families have been proposed to model skewed data by introducing unequal scales based on the median or mode. However, we argue that the point at which unbalanced scales occur may be at any quantile and cannot be reparametrized as an ordinary shift parameter in the presence of skewness. In this paper, we introduce a novel skewed pivot-blend technique to create a skewed density family based on any continuous density, even those that are asymmetric and nonunimodal. Our framework enables the simultaneous estimation of scales, the pivotal point, and other location parameters, along with various extensions. We also introduce a skewed two-part model tailored for semicontinuous outcomes, which identifies relevant variables across the entire population and mitigates the additional skewness induced by commonly used transformations. Our theoretical analysis reveals the influence of skewness without assuming asymptotic conditions. Experiments on synthetic and real-life data demonstrate the excellent performance of the proposed method. △ Less

Submitted 9 January, 2024; originally announced January 2024.

Robustly Improving Bandit Algorithms with Confounded and Selection Biased Offline Data: A Causal Approach

Authors: Wen Huang, Xintao Wu

Abstract: This paper studies bandit problems where an agent has access to offline data that might be utilized to potentially improve the estimation of each arm's reward distribution. A major obstacle in this setting is the existence of compound biases from the observational data. Ignoring these biases and blindly fitting a model with the biased data could even negatively affect the online learning phase. In… ▽ More This paper studies bandit problems where an agent has access to offline data that might be utilized to potentially improve the estimation of each arm's reward distribution. A major obstacle in this setting is the existence of compound biases from the observational data. Ignoring these biases and blindly fitting a model with the biased data could even negatively affect the online learning phase. In this work, we formulate this problem from a causal perspective. First, we categorize the biases into confounding bias and selection bias based on the causal structure they imply. Next, we extract the causal bound for each arm that is robust towards compound biases from biased observational data. The derived bounds contain the ground truth mean reward and can effectively guide the bandit agent to learn a nearly-optimal decision policy. We also conduct regret analysis in both contextual and non-contextual bandit settings and show that prior causal bounds could help consistently reduce the asymptotic regret. △ Less

Submitted 19 December, 2023; originally announced December 2023.

ZeroQuant(4+2): Redefining LLMs Quantization with a New FP6-Centric Strategy for Diverse Generative Tasks

Authors: Xiaoxia Wu, Haojun Xia, Stephen Youn, Zhen Zheng, Shiyang Chen, Arash Bakhtiari, Michael Wyatt, Reza Yazdani Aminabadi, Yuxiong He, Olatunji Ruwase, Leon Song, Zhewei Yao

Abstract: This study examines 4-bit quantization methods like GPTQ in large language models (LLMs), highlighting GPTQ's overfitting and limited enhancement in Zero-Shot tasks. While prior works merely focusing on zero-shot measurement, we extend task scope to more generative categories such as code generation and abstractive summarization, in which we found that INT4 quantization can significantly underperf… ▽ More This study examines 4-bit quantization methods like GPTQ in large language models (LLMs), highlighting GPTQ's overfitting and limited enhancement in Zero-Shot tasks. While prior works merely focusing on zero-shot measurement, we extend task scope to more generative categories such as code generation and abstractive summarization, in which we found that INT4 quantization can significantly underperform. However, simply shifting to higher precision formats like FP6 has been particularly challenging, thus overlooked, due to poor performance caused by the lack of sophisticated integration and system acceleration strategies on current AI hardware. Our results show that FP6, even with a coarse-grain quantization scheme, performs robustly across various algorithms and tasks, demonstrating its superiority in accuracy and versatility. Notably, with the FP6 quantization, \codestar-15B model performs comparably to its FP16 counterpart in code generation, and for smaller models like the 406M it closely matches their baselines in summarization. Neither can be achieved by INT4. To better accommodate various AI hardware and achieve the best system performance, we propose a novel 4+2 design for FP6 to achieve similar latency to the state-of-the-art INT4 fine-grain quantization. With our design, FP6 can become a promising solution to the current 4-bit quantization methods used in LLMs. △ Less

Submitted 18 December, 2023; v1 submitted 13 December, 2023; originally announced December 2023.

Enhancing Polynomial Chaos Expansion Based Surrogate Modeling using a Novel Probabilistic Transfer Learning Strategy

Authors: Wyatt Bridgman, Uma Balakrishnan, Reese Jones, Jiefu Chen, Xuqing Wu, Cosmin Safta, Yueqin Huang, Mohammad Khalil

Abstract: In the field of surrogate modeling, polynomial chaos expansion (PCE) allows practitioners to construct inexpensive yet accurate surrogates to be used in place of the expensive forward model simulations. For black-box simulations, non-intrusive PCE allows the construction of these surrogates using a set of simulation response evaluations. In this context, the PCE coefficients can be obtained using… ▽ More In the field of surrogate modeling, polynomial chaos expansion (PCE) allows practitioners to construct inexpensive yet accurate surrogates to be used in place of the expensive forward model simulations. For black-box simulations, non-intrusive PCE allows the construction of these surrogates using a set of simulation response evaluations. In this context, the PCE coefficients can be obtained using linear regression, which is also known as point collocation or stochastic response surfaces. Regression exhibits better scalability and can handle noisy function evaluations in contrast to other non-intrusive approaches, such as projection. However, since over-sampling is generally advisable for the linear regression approach, the simulation requirements become prohibitive for expensive forward models. We propose to leverage transfer learning whereby knowledge gained through similar PCE surrogate construction tasks (source domains) is transferred to a new surrogate-construction task (target domain) which has a limited number of forward model simulations (training data). The proposed transfer learning strategy determines how much, if any, information to transfer using new techniques inspired by Bayesian modeling and data assimilation. The strategy is scrutinized using numerical investigations and applied to an engineering problem from the oil and gas industry. △ Less

Submitted 7 December, 2023; originally announced December 2023.

Decoding Social Sentiment in DAO: A Comparative Analysis of Blockchain Governance Communities

Authors: Yutong Quan, Xintong Wu, Wanlin Deng, Luyao Zhang

Abstract: Blockchain technology is leading a revolutionary transformation across diverse industries, with effective governance being critical for the success and sustainability of blockchain projects. Community forums, pivotal in engaging decentralized autonomous organizations (DAOs), significantly impact blockchain governance decisions. Concurrently, Natural Language Processing (NLP), particularly sentimen… ▽ More Blockchain technology is leading a revolutionary transformation across diverse industries, with effective governance being critical for the success and sustainability of blockchain projects. Community forums, pivotal in engaging decentralized autonomous organizations (DAOs), significantly impact blockchain governance decisions. Concurrently, Natural Language Processing (NLP), particularly sentiment analysis, provides powerful insights from textual data. While prior research has explored the potential of NLP tools in social media sentiment analysis, there is a gap in understanding the sentiment landscape of blockchain governance communities. The evolving discourse and sentiment dynamics on the forums of top DAOs remain largely unknown. This paper delves deep into the evolving discourse and sentiment dynamics on the public forums of leading DeFi projects: Aave, Uniswap, Curve DAO, Yearn.finance, Merit Circle, and Balancer, focusing primarily on discussions related to governance issues. Our study shows that participants in decentralized communities generally express positive sentiments during Discord discussions. Furthermore, there is a potential interaction between discussion intensity and sentiment dynamics; higher discussion volume may contribute to a more stable sentiment from code analysis. The insights gained from this study are valuable for decision-makers in blockchain governance, underscoring the pivotal role of sentiment analysis in interpreting community emotions and its evolving impact on the landscape of blockchain governance. This research significantly contributes to the interdisciplinary exploration of the intersection of blockchain and society, specifically emphasizing the decentralized blockchain governance ecosystem. We provide our data and code for replicability as open access on GitHub. △ Less

Submitted 25 May, 2024; v1 submitted 31 October, 2023; originally announced November 2023.

A unified consensus-based parallel ADMM algorithm for high-dimensional regression with combined regularizations

Authors: Xiaofei Wu, Zhimin Zhang, Zhenyu Cui

Abstract: The parallel alternating direction method of multipliers (ADMM) algorithm is widely recognized for its effectiveness in handling large-scale datasets stored in a distributed manner, making it a popular choice for solving statistical learning models. However, there is currently limited research on parallel algorithms specifically designed for high-dimensional regression with combined (composite) re… ▽ More The parallel alternating direction method of multipliers (ADMM) algorithm is widely recognized for its effectiveness in handling large-scale datasets stored in a distributed manner, making it a popular choice for solving statistical learning models. However, there is currently limited research on parallel algorithms specifically designed for high-dimensional regression with combined (composite) regularization terms. These terms, such as elastic-net, sparse group lasso, sparse fused lasso, and their nonconvex variants, have gained significant attention in various fields due to their ability to incorporate prior information and promote sparsity within specific groups or fused variables. The scarcity of parallel algorithms for combined regularizations can be attributed to the inherent nonsmoothness and complexity of these terms, as well as the absence of closed-form solutions for certain proximal operators associated with them. In this paper, we propose a unified constrained optimization formulation based on the consensus problem for these types of convex and nonconvex regression problems and derive the corresponding parallel ADMM algorithms. Furthermore, we prove that the proposed algorithm not only has global convergence but also exhibits linear convergence rate. Extensive simulation experiments, along with a financial example, serve to demonstrate the reliability, stability, and scalability of our algorithm. The R package for implementing the proposed algorithms can be obtained at https://github.com/xfwu1016/CPADMM. △ Less

Submitted 20 November, 2023; originally announced November 2023.

From Identifiable Causal Representations to Controllable Counterfactual Generation: A Survey on Causal Generative Modeling

Authors: Aneesh Komanduri, Xintao Wu, Yongkai Wu, Feng Chen

Abstract: Deep generative models have shown tremendous capability in data density estimation and data generation from finite samples. While these models have shown impressive performance by learning correlations among features in the data, some fundamental shortcomings are their lack of explainability, tendency to induce spurious correlations, and poor out-of-distribution extrapolation. To remedy such chall… ▽ More Deep generative models have shown tremendous capability in data density estimation and data generation from finite samples. While these models have shown impressive performance by learning correlations among features in the data, some fundamental shortcomings are their lack of explainability, tendency to induce spurious correlations, and poor out-of-distribution extrapolation. To remedy such challenges, recent work has proposed a shift toward causal generative models. Causal models offer several beneficial properties to deep generative models, such as distribution shift robustness, fairness, and interpretability. Structural causal models (SCMs) describe data-generating processes and model complex causal relationships and mechanisms among variables in a system. Thus, SCMs can naturally be combined with deep generative models. We provide a technical survey on causal generative modeling categorized into causal representation learning and controllable counterfactual generation methods. We focus on fundamental theory, methodology, drawbacks, datasets, and metrics. Then, we cover applications of causal generative models in fairness, privacy, out-of-distribution generalization, precision medicine, and biological sciences. Lastly, we discuss open problems and fruitful research directions for future work in the field. △ Less

Submitted 23 May, 2024; v1 submitted 17 October, 2023; originally announced October 2023.

Comments: Published in Transactions on Machine Learning Research (TMLR) (05/2024); 72 pages, 27 figures, 4 tables

Journal ref: Transactions on Machine Learning Research, 2024

Outlier Detection Using Generative Models with Theoretical Performance Guarantees

Authors: Jirong Yi, **gchao Gao, Tianming Wang, Xiaodong Wu, Weiyu Xu

Abstract: This paper considers the problem of recovering signals modeled by generative models from linear measurements contaminated with sparse outliers. We propose an outlier detection approach for reconstructing the ground-truth signals modeled by generative models under sparse outliers. We establish theoretical recovery guarantees for reconstruction of signals using generative models in the presence of o… ▽ More This paper considers the problem of recovering signals modeled by generative models from linear measurements contaminated with sparse outliers. We propose an outlier detection approach for reconstructing the ground-truth signals modeled by generative models under sparse outliers. We establish theoretical recovery guarantees for reconstruction of signals using generative models in the presence of outliers, giving lower bounds on the number of correctable outliers. Our results are applicable to both linear generator neural networks and the nonlinear generator neural networks with an arbitrary number of layers. We propose an iterative alternating direction method of multipliers (ADMM) algorithm for solving the outlier detection problem via $\ell_1$ norm minimization, and a gradient descent algorithm for solving the outlier detection problem via squared $\ell_1$ norm minimization. We conduct extensive experiments using variational auto-encoder and deep convolutional generative adversarial networks, and the experimental results show that the signals can be successfully reconstructed under outliers using our approach. Our approach outperforms the traditional Lasso and $\ell_2$ minimization approach. △ Less

Submitted 15 October, 2023; originally announced October 2023.

Comments: arXiv admin note: substantial text overlap with arXiv:1810.11335

CausalGPS: An R Package for Causal Inference With Continuous Exposures

Authors: Naeem Khoshnevis, Xiao Wu, Danielle Braun

Abstract: Quantifying the causal effects of continuous exposures on outcomes of interest is critical for social, economic, health, and medical research. However, most existing software packages focus on binary exposures. We develop the CausalGPS R package that implements a collection of algorithms to provide algorithmic solutions for causal inference with continuous exposures. CausalGPS implements a causal… ▽ More Quantifying the causal effects of continuous exposures on outcomes of interest is critical for social, economic, health, and medical research. However, most existing software packages focus on binary exposures. We develop the CausalGPS R package that implements a collection of algorithms to provide algorithmic solutions for causal inference with continuous exposures. CausalGPS implements a causal inference workflow, with algorithms based on generalized propensity scores (GPS) as the core, extending propensity scores (the probability of a unit being exposed given pre-exposure covariates) from binary to continuous exposures. As the first step, the package implements efficient and flexible estimations of the GPS, allowing multiple user-specified modeling options. As the second step, the package provides two ways to adjust for confounding: weighting and matching, generating weighted and matched data sets, respectively. Lastly, the package provides built-in functions to fit flexible parametric, semi-parametric, or non-parametric regression models on the weighted or matched data to estimate the exposure-response function relating the outcome with the exposures. The computationally intensive tasks are implemented in C++, and efficient shared-memory parallelization is achieved by OpenMP API. This paper outlines the main components of the CausalGPS R package and demonstrates its application to assess the effect of long-term exposure to PM2.5 on educational attainment using zip code-level data from the contiguous United States from 2000-2016. △ Less

Submitted 30 September, 2023; originally announced October 2023.

Comments: 22 pages, 8 figures

On Prediction Feature Assignment in the Heckman Selection Model

Authors: Huy Mai, Xintao Wu

Abstract: Under missing-not-at-random (MNAR) sample selection bias, the performance of a prediction model is often degraded. This paper focuses on one classic instance of MNAR sample selection bias where a subset of samples have non-randomly missing outcomes. The Heckman selection model and its variants have commonly been used to handle this type of sample selection bias. The Heckman model uses two separate… ▽ More Under missing-not-at-random (MNAR) sample selection bias, the performance of a prediction model is often degraded. This paper focuses on one classic instance of MNAR sample selection bias where a subset of samples have non-randomly missing outcomes. The Heckman selection model and its variants have commonly been used to handle this type of sample selection bias. The Heckman model uses two separate equations to model the prediction and selection of samples, where the selection features include all prediction features. When using the Heckman model, the prediction features must be properly chosen from the set of selection features. However, choosing the proper prediction features is a challenging task for the Heckman model. This is especially the case when the number of selection features is large. Existing approaches that use the Heckman model often provide a manually chosen set of prediction features. In this paper, we propose Heckman-FA as a novel data-driven framework for obtaining prediction features for the Heckman model. Heckman-FA first trains an assignment function that determines whether or not a selection feature is assigned as a prediction feature. Using the parameters of the trained function, the framework extracts a suitable set of prediction features based on the goodness-of-fit of the prediction model given the chosen prediction features and the correlation between noise terms of the prediction and selection equations. Experimental results on real-world datasets show that Heckman-FA produces a robust regression model under MNAR sample selection bias. △ Less

Submitted 22 April, 2024; v1 submitted 14 September, 2023; originally announced September 2023.

Comments: Full version of work accepted to IJCNN 2024

Reduced Order Modeling of a MOOSE-based Advanced Manufacturing Model with Operator Learning

Authors: Mahmoud Yaseen, Dewen Yushu, Peter German, Xu Wu

Abstract: Advanced Manufacturing (AM) has gained significant interest in the nuclear community for its potential application on nuclear materials. One challenge is to obtain desired material properties via controlling the manufacturing process during runtime. Intelligent AM based on deep reinforcement learning (DRL) relies on an automated process-level control mechanism to generate optimal design variables… ▽ More Advanced Manufacturing (AM) has gained significant interest in the nuclear community for its potential application on nuclear materials. One challenge is to obtain desired material properties via controlling the manufacturing process during runtime. Intelligent AM based on deep reinforcement learning (DRL) relies on an automated process-level control mechanism to generate optimal design variables and adaptive system settings for improved end-product properties. A high-fidelity thermo-mechanical model for direct energy deposition has recently been developed within the MOOSE framework at the Idaho National Laboratory (INL). The goal of this work is to develop an accurate and fast-running reduced order model (ROM) for this MOOSE-based AM model that can be used in a DRL-based process control and optimization method. Operator learning (OL)-based methods will be employed due to their capability to learn a family of differential equations, in this work, produced by changing process variables in the Gaussian point heat source for the laser. We will develop OL-based ROM using Fourier neural operator, and perform a benchmark comparison of its performance with a conventional deep neural network-based ROM. △ Less

Submitted 18 August, 2023; originally announced August 2023.

Comments: 10 Pages, 7 Figures, 2 Tables. arXiv admin note: text overlap with arXiv:2308.02462

Journal ref: In Proceedings of the 2023 International Conference on Mathematics and Computational Methods Applied to Nuclear Science and Engineering (M&C 2023)

An Efficient 1 Iteration Learning Algorithm for Gaussian Mixture Model And Gaussian Mixture Embedding For Neural Network

Authors: Weiguo Lu, Xuan Wu, Deng Ding, Gangnan Yuan

Abstract: We propose an Gaussian Mixture Model (GMM) learning algorithm, based on our previous work of GMM expansion idea. The new algorithm brings more robustness and simplicity than classic Expectation Maximization (EM) algorithm. It also improves the accuracy and only take 1 iteration for learning. We theoretically proof that this new algorithm is guarantee to converge regardless the parameters initialis… ▽ More We propose an Gaussian Mixture Model (GMM) learning algorithm, based on our previous work of GMM expansion idea. The new algorithm brings more robustness and simplicity than classic Expectation Maximization (EM) algorithm. It also improves the accuracy and only take 1 iteration for learning. We theoretically proof that this new algorithm is guarantee to converge regardless the parameters initialisation. We compare our GMM expansion method with classic probability layers in neural network leads to demonstrably better capability to overcome data uncertainty and inverse problem. Finally, we test GMM based generator which shows a potential to build further application that able to utilized distribution random sampling for stochastic variation as well as variation control. △ Less

Submitted 6 September, 2023; v1 submitted 18 August, 2023; originally announced August 2023.

Fast and Accurate Reduced-Order Modeling of a MOOSE-based Additive Manufacturing Model with Operator Learning

Authors: Mahmoud Yaseen, Dewen Yushu, Peter German, Xu Wu

Abstract: One predominant challenge in additive manufacturing (AM) is to achieve specific material properties by manipulating manufacturing process parameters during the runtime. Such manipulation tends to increase the computational load imposed on existing simulation tools employed in AM. The goal of the present work is to construct a fast and accurate reduced-order model (ROM) for an AM model developed wi… ▽ More One predominant challenge in additive manufacturing (AM) is to achieve specific material properties by manipulating manufacturing process parameters during the runtime. Such manipulation tends to increase the computational load imposed on existing simulation tools employed in AM. The goal of the present work is to construct a fast and accurate reduced-order model (ROM) for an AM model developed within the Multiphysics Object-Oriented Simulation Environment (MOOSE) framework, ultimately reducing the time/cost of AM control and optimization processes. Our adoption of the operator learning (OL) approach enabled us to learn a family of differential equations produced by altering process variables in the laser's Gaussian point heat source. More specifically, we used the Fourier neural operator (FNO) and deep operator network (DeepONet) to develop ROMs for time-dependent responses. Furthermore, we benchmarked the performance of these OL methods against a conventional deep neural network (DNN)-based ROM. Ultimately, we found that OL methods offer comparable performance and, in terms of accuracy and generalizability, even outperform DNN at predicting scalar model responses. The DNN-based ROM afforded the fastest training time. Furthermore, all the ROMs were faster than the original MOOSE model yet still provided accurate predictions. FNO had a smaller mean prediction error than DeepONet, with a larger variance for time-dependent responses. Unlike DNN, both FNO and DeepONet were able to simulate time series data without the need for dimensionality reduction techniques. The present work can help facilitate the AM optimization process by enabling faster execution of simulation tools while still preserving evaluation accuracy. △ Less

Submitted 4 August, 2023; originally announced August 2023.

Comments: 28 pages, 18 figures, 4 tables

Journal ref: Int J Adv Manuf Technol (2023)

Estimating causal quantile exposure response functions via matching

Authors: Luca Merlo, Francesca Dominici, Lea Petrella, Nicola Salvati, Xiao Wu

Abstract: We develop new matching estimators for estimating causal quantile exposure-response functions and quantile exposure effects with continuous treatments. We provide identification results for the parameters of interest and establish the asymptotic properties of the derived estimators. We introduce a two-step estimation procedure. In the first step, we construct a matched data set via generalized pro… ▽ More We develop new matching estimators for estimating causal quantile exposure-response functions and quantile exposure effects with continuous treatments. We provide identification results for the parameters of interest and establish the asymptotic properties of the derived estimators. We introduce a two-step estimation procedure. In the first step, we construct a matched data set via generalized propensity score matching, adjusting for measured confounding. In the second step, we fit a kernel quantile regression to the matched set. We also derive a consistent estimator of the variance of the matching estimators. Using simulation studies, we compare the introduced approach with existing alternatives in various settings. We apply the proposed method to Medicare claims data for the period 2012-2014, and we estimate the causal effect of exposure to PM$_{2.5}$ on the length of hospital stay for each zip code of the contiguous United States. △ Less

Submitted 3 August, 2023; originally announced August 2023.

Causal exposure-response curve estimation with surrogate confounders: a study of air pollution and children's health in Medicaid claims data

Authors: Jenny J. Lee, Xiao Wu, Francesca Dominici, Rachel C. Nethery

Abstract: In this paper, we undertake a case study in which interest lies in estimating a causal exposure-response function (ERF) for long-term exposure to fine particulate matter (PM$_{2.5}$) and respiratory hospitalizations in socioeconomically disadvantaged children using nationwide Medicaid claims data. New methods are needed to address the specific challenges the Medicaid data present. First, Medicaid… ▽ More In this paper, we undertake a case study in which interest lies in estimating a causal exposure-response function (ERF) for long-term exposure to fine particulate matter (PM$_{2.5}$) and respiratory hospitalizations in socioeconomically disadvantaged children using nationwide Medicaid claims data. New methods are needed to address the specific challenges the Medicaid data present. First, Medicaid eligibility criteria, which are largely based on family income for children, differ by state, creating socioeconomically distinct populations and leading to clustered data, where zip codes (our units of analysis) are nested within states. Second, Medicaid enrollees' individual-level socioeconomic status, which is known to be a confounder and an effect modifier of the exposure-response relationships under study, is not available. However, two useful surrogates are available: median household income of each enrollee's zip code of residence and state-level Medicaid family income eligibility thresholds for children. In this paper, we introduce a customized approach, called \textit{MedMatch}, that builds on generalized propensity score matching methods for estimating causal ERFs, adapting these approaches to leverage our two surrogate variables to account for potential confounding and/or effect modification by socioeconomic status. We conduct extensive simulation studies, consistently demonstrating the strong performance of \textit{MedMatch} relative to conventional approaches to handling the surrogate variables. We apply \textit{MedMatch} to estimate the causal ERF between long-term PM$_{2.5}$ exposure and first respiratory hospitalization among children in Medicaid from 2000 to 2012. We find a positive association, with a steeper curve at PM$_{2.5} \le 8$ $μ$g/m$^3$ that levels off at higher concentrations. △ Less

Submitted 1 August, 2023; originally announced August 2023.

Comments: 38 pages,5 figures

Functional PCA and Deep Neural Networks-based Bayesian Inverse Uncertainty Quantification with Transient Experimental Data

Authors: Ziyu Xie, Mahmoud Yaseen, Xu Wu

Abstract: Inverse UQ is the process to inversely quantify the model input uncertainties based on experimental data. This work focuses on develo** an inverse UQ process for time-dependent responses, using dimensionality reduction by functional principal component analysis (PCA) and deep neural network (DNN)-based surrogate models. The demonstration is based on the inverse UQ of TRACE physical model paramet… ▽ More Inverse UQ is the process to inversely quantify the model input uncertainties based on experimental data. This work focuses on develo** an inverse UQ process for time-dependent responses, using dimensionality reduction by functional principal component analysis (PCA) and deep neural network (DNN)-based surrogate models. The demonstration is based on the inverse UQ of TRACE physical model parameters using the FEBA transient experimental data. The measurement data is time-dependent peak cladding temperature (PCT). Since the quantity-of-interest (QoI) is time-dependent that corresponds to infinite-dimensional responses, PCA is used to reduce the QoI dimension while preserving the transient profile of the PCT, in order to make the inverse UQ process more efficient. However, conventional PCA applied directly to the PCT time series profiles can hardly represent the data precisely due to the sudden temperature drop at the time of quenching. As a result, a functional alignment method is used to separate the phase and amplitude information of the transient PCT profiles before dimensionality reduction. DNNs are then trained using PC scores from functional PCA to build surrogate models of TRACE in order to reduce the computational cost in Markov Chain Monte Carlo sampling. Bayesian neural networks are used to estimate the uncertainties of DNN surrogate model predictions. In this study, we compared four different inverse UQ processes with different dimensionality reduction methods and surrogate models. The proposed approach shows an improvement in reducing the dimension of the TRACE transient simulations, and the forward propagation of inverse UQ results has a better agreement with the experimental data. △ Less

Submitted 10 July, 2023; originally announced July 2023.

Comments: 31 pages, 21 figures

Precise Asymptotic Generalization for Multiclass Classification with Overparameterized Linear Models

Authors: David X. Wu, Anant Sahai

Abstract: We study the asymptotic generalization of an overparameterized linear model for multiclass classification under the Gaussian covariates bi-level model introduced in Subramanian et al.~'22, where the number of data points, features, and classes all grow together. We fully resolve the conjecture posed in Subramanian et al.~'22, matching the predicted regimes for generalization. Furthermore, our new… ▽ More We study the asymptotic generalization of an overparameterized linear model for multiclass classification under the Gaussian covariates bi-level model introduced in Subramanian et al.~'22, where the number of data points, features, and classes all grow together. We fully resolve the conjecture posed in Subramanian et al.~'22, matching the predicted regimes for generalization. Furthermore, our new lower bounds are akin to an information-theoretic strong converse: they establish that the misclassification rate goes to 0 or 1 asymptotically. One surprising consequence of our tight results is that the min-norm interpolating classifier can be asymptotically suboptimal relative to noninterpolating classifiers in the regime where the min-norm interpolating regressor is known to be optimal. The key to our tight analysis is a new variant of the Hanson-Wright inequality which is broadly useful for multiclass problems with sparse labels. As an application, we show that the same type of analysis can be used to analyze the related multilabel classification problem under the same bi-level ensemble. △ Less

Submitted 5 December, 2023; v1 submitted 22 June, 2023; originally announced June 2023.

Comments: NeurIPS 2023, 56 pages

Learning Causally Disentangled Representations via the Principle of Independent Causal Mechanisms

Authors: Aneesh Komanduri, Yongkai Wu, Feng Chen, Xintao Wu

Abstract: Learning disentangled causal representations is a challenging problem that has gained significant attention recently due to its implications for extracting meaningful information for downstream tasks. In this work, we define a new notion of causal disentanglement from the perspective of independent causal mechanisms. We propose ICM-VAE, a framework for learning causally disentangled representation… ▽ More Learning disentangled causal representations is a challenging problem that has gained significant attention recently due to its implications for extracting meaningful information for downstream tasks. In this work, we define a new notion of causal disentanglement from the perspective of independent causal mechanisms. We propose ICM-VAE, a framework for learning causally disentangled representations supervised by causally related observed labels. We model causal mechanisms using nonlinear learnable flow-based diffeomorphic functions to map noise variables to latent causal variables. Further, to promote the disentanglement of causal factors, we propose a causal disentanglement prior learned from auxiliary labels and the latent causal structure. We theoretically show the identifiability of causal factors and mechanisms up to permutation and elementwise reparameterization. We empirically demonstrate that our framework induces highly disentangled causal factors, improves interventional robustness, and is compatible with counterfactual generation. △ Less

Submitted 8 May, 2024; v1 submitted 1 June, 2023; originally announced June 2023.

Comments: Accepted to IJCAI 2024

Inverse Uncertainty Quantification by Hierarchical Bayesian Modeling and Application in Nuclear System Thermal-Hydraulics Codes

Authors: Chen Wang, Xu Wu, Tomasz Kozlowski

Abstract: Inverse Uncertainty Quantification (IUQ) method has been widely used to quantify the uncertainty of Physical Model Parameters (PMPs) in nuclear Thermal Hydraulics (TH) systems. This paper introduces a novel hierarchical Bayesian model which aims to mitigate two existing challenges in IUQ: the high variability of PMPs under varying experimental conditions, and unknown model discrepancies or outlier… ▽ More Inverse Uncertainty Quantification (IUQ) method has been widely used to quantify the uncertainty of Physical Model Parameters (PMPs) in nuclear Thermal Hydraulics (TH) systems. This paper introduces a novel hierarchical Bayesian model which aims to mitigate two existing challenges in IUQ: the high variability of PMPs under varying experimental conditions, and unknown model discrepancies or outliers causing over-fitting issues. The proposed hierarchical model is compared with the conventional single-level Bayesian model using TRACE code and the measured void fraction data in the BFBT benchmark. A Hamiltonian Monte Carlo Method - No U-Turn Sampler (NUTS) is used for posterior sampling. The results demonstrate the effectiveness of the proposed hierarchical model in providing better estimates of the posterior distributions of PMPs and being less prone to over-fitting. The proposed method also demonstrates a promising approach for generalizing IUQ to larger databases with broad ranges of experimental conditions. △ Less

Submitted 25 March, 2024; v1 submitted 26 May, 2023; originally announced May 2023.

Demystifying Oversmoothing in Attention-Based Graph Neural Networks

Authors: Xinyi Wu, Amir Ajorlou, Zihui Wu, Ali Jadbabaie

Abstract: Oversmoothing in Graph Neural Networks (GNNs) refers to the phenomenon where increasing network depth leads to homogeneous node representations. While previous work has established that Graph Convolutional Networks (GCNs) exponentially lose expressive power, it remains controversial whether the graph attention mechanism can mitigate oversmoothing. In this work, we provide a definitive answer to th… ▽ More Oversmoothing in Graph Neural Networks (GNNs) refers to the phenomenon where increasing network depth leads to homogeneous node representations. While previous work has established that Graph Convolutional Networks (GCNs) exponentially lose expressive power, it remains controversial whether the graph attention mechanism can mitigate oversmoothing. In this work, we provide a definitive answer to this question through a rigorous mathematical analysis, by viewing attention-based GNNs as nonlinear time-varying dynamical systems and incorporating tools and techniques from the theory of products of inhomogeneous matrices and the joint spectral radius. We establish that, contrary to popular belief, the graph attention mechanism cannot prevent oversmoothing and loses expressive power exponentially. The proposed framework extends the existing results on oversmoothing for symmetric GCNs to a significantly broader class of GNN models, including random walk GCNs, Graph Attention Networks (GATs) and (graph) transformers. In particular, our analysis accounts for asymmetric, state-dependent and time-varying aggregation operators and a wide range of common nonlinear activation functions, such as ReLU, LeakyReLU, GELU and SiLU. △ Less

Submitted 3 June, 2024; v1 submitted 25 May, 2023; originally announced May 2023.

Comments: NeurIPS 2023 spotlight. Fixed an error in the previous version; new results and remarks added

On the tightness of information-theoretic bounds on generalization error of learning algorithms

Authors: Xuetong Wu, Jonathan H. Manton, Uwe Aickelin, **gge Zhu

Abstract: A recent line of works, initiated by Russo and Xu, has shown that the generalization error of a learning algorithm can be upper bounded by information measures. In most of the relevant works, the convergence rate of the expected generalization error is in the form of $O(\sqrt{λ/n})$ where $λ$ is some information-theoretic quantities such as the mutual information or conditional mutual information… ▽ More A recent line of works, initiated by Russo and Xu, has shown that the generalization error of a learning algorithm can be upper bounded by information measures. In most of the relevant works, the convergence rate of the expected generalization error is in the form of $O(\sqrt{λ/n})$ where $λ$ is some information-theoretic quantities such as the mutual information or conditional mutual information between the data and the learned hypothesis. However, such a learning rate is typically considered to be ``slow", compared to a ``fast rate" of $O(λ/n)$ in many learning scenarios. In this work, we first show that the square root does not necessarily imply a slow rate, and a fast rate result can still be obtained using this bound under appropriate assumptions. Furthermore, we identify the critical conditions needed for the fast rate generalization error, which we call the $(η,c)$-central condition. Under this condition, we give information-theoretic bounds on the generalization error and excess risk, with a fast convergence rate for specific learning algorithms such as empirical risk minimization and its regularized version. Finally, several analytical examples are given to show the effectiveness of the bounds. △ Less

Submitted 26 March, 2023; originally announced March 2023.

Comments: 32 pages, 1 figure. arXiv admin note: substantial text overlap with arXiv:2205.03131

A Non-Asymptotic Analysis of Oversmoothing in Graph Neural Networks

Authors: Xinyi Wu, Zhengdao Chen, William Wang, Ali Jadbabaie

Abstract: Oversmoothing is a central challenge of building more powerful Graph Neural Networks (GNNs). While previous works have only demonstrated that oversmoothing is inevitable when the number of graph convolutions tends to infinity, in this paper, we precisely characterize the mechanism behind the phenomenon via a non-asymptotic analysis. Specifically, we distinguish between two different effects when a… ▽ More Oversmoothing is a central challenge of building more powerful Graph Neural Networks (GNNs). While previous works have only demonstrated that oversmoothing is inevitable when the number of graph convolutions tends to infinity, in this paper, we precisely characterize the mechanism behind the phenomenon via a non-asymptotic analysis. Specifically, we distinguish between two different effects when applying graph convolutions -- an undesirable mixing effect that homogenizes node representations in different classes, and a desirable denoising effect that homogenizes node representations in the same class. By quantifying these two effects on random graphs sampled from the Contextual Stochastic Block Model (CSBM), we show that oversmoothing happens once the mixing effect starts to dominate the denoising effect, and the number of layers required for this transition is $O(\log N/\log (\log N))$ for sufficiently dense graphs with $N$ nodes. We also extend our analysis to study the effects of Personalized PageRank (PPR), or equivalently, the effects of initial residual connections on oversmoothing. Our results suggest that while PPR mitigates oversmoothing at deeper layers, PPR-based architectures still achieve their best performance at a shallow depth and are outperformed by the graph convolution approach on certain graphs. Finally, we support our theoretical results with numerical experiments, which further suggest that the oversmoothing phenomenon observed in practice can be magnified by the difficulty of optimizing deep GNN models. △ Less

Submitted 28 February, 2023; v1 submitted 20 December, 2022; originally announced December 2022.

Comments: Accepted by the 11th International Conference on Learning Representations (ICLR 2023)

A comparative analysis of several multivariate zero-inflated and zero-modified models with applications in insurance

Authors: Pengcheng Zhang, David Pitt, Xueyuan Wu

Abstract: Claim frequency data in insurance records the number of claims on insurance policies during a finite period of time. Given that insurance companies operate with multiple lines of insurance business where the claim frequencies on different lines of business are often correlated, multivariate count modeling with dependence for claim frequency is therefore essential. Due in part to the operation of b… ▽ More Claim frequency data in insurance records the number of claims on insurance policies during a finite period of time. Given that insurance companies operate with multiple lines of insurance business where the claim frequencies on different lines of business are often correlated, multivariate count modeling with dependence for claim frequency is therefore essential. Due in part to the operation of bonus-malus systems, claims data in automobile insurance are often characterized by an excess of common zeros. This feature is referred to as multivariate zero-inflation. In this paper, we establish two ways of dealing with this feature. The first is to use a multivariate zero-inflated model, where we artificially augment the probability of common zeros based on standard multivariate count distributions. The other is to apply a multivariate zero-modified model, which deals with the common zeros and the number of claims incurred in each line, given that at least one claim occurs separately. A comprehensive comparative analysis of several models under these two frameworks is conducted using the data of an automobile insurance portfolio from a major insurance company in Spain. A less common situation in insurance is the absence of some common zeros resulting from incomplete records. This feature of these data is known as multivariate zero-deflation. In this case, our proposed multivariate zero-modified model still works, as shown by the second empirical study. △ Less

Submitted 2 December, 2022; originally announced December 2022.

Prediction and Uncertainty Quantification of SAFARI-1 Axial Neutron Flux Profiles with Neural Networks

Authors: Lesego E. Moloko, Pavel M. Bokov, Xu Wu, Kostadin N. Ivanov

Abstract: Artificial Neural Networks (ANNs) have been successfully used in various nuclear engineering applications, such as predicting reactor physics parameters within reasonable time and with a high level of accuracy. Despite this success, they cannot provide information about the model prediction uncertainties, making it difficult to assess ANN prediction credibility, especially in extrapolated domains.… ▽ More Artificial Neural Networks (ANNs) have been successfully used in various nuclear engineering applications, such as predicting reactor physics parameters within reasonable time and with a high level of accuracy. Despite this success, they cannot provide information about the model prediction uncertainties, making it difficult to assess ANN prediction credibility, especially in extrapolated domains. In this study, Deep Neural Networks (DNNs) are used to predict the assembly axial neutron flux profiles in the SAFARI-1 research reactor, with quantified uncertainties in the ANN predictions and extrapolation to cycles not used in the training process. The training dataset consists of copper-wire activation measurements, the axial measurement locations and the measured control bank positions obtained from the reactor's historical cycles. Uncertainty Quantification of the regular DNN models' predictions is performed using Monte Carlo Dropout (MCD) and Bayesian Neural Networks solved by Variational Inference (BNN VI). The regular DNNs, DNNs solved with MCD and BNN VI results agree very well among each other as well as with the new measured dataset not used in the training process, thus indicating good prediction and generalization capability. The uncertainty bands produced by MCD and BNN VI agree very well, and in general, they can fully envelop the noisy measurement data points. The developed ANNs are useful in supporting the experimental measurements campaign and neutronics code Verification and Validation (V&V). △ Less

Submitted 15 November, 2022; originally announced November 2022.

Comments: 34 pages, 12 figures

Alcohol Intake Differentiates AD and LATE: A Telltale Lifestyle from Two Large-Scale Datasets

Authors: Xinxing Wu, Chong Peng, Peter T. Nelson, Qiang Cheng

Abstract: Alzheimer's disease (AD), as a progressive brain disease, affects cognition, memory, and behavior. Similarly, limbic-predominant age-related TDP-43 encephalopathy (LATE) is a recently defined common neurodegenerative disease that mimics the clinical symptoms of AD. At present, the risk factors implicated in LATE and those distinguishing LATE from AD are largely unknown. We leveraged an integrated… ▽ More Alzheimer's disease (AD), as a progressive brain disease, affects cognition, memory, and behavior. Similarly, limbic-predominant age-related TDP-43 encephalopathy (LATE) is a recently defined common neurodegenerative disease that mimics the clinical symptoms of AD. At present, the risk factors implicated in LATE and those distinguishing LATE from AD are largely unknown. We leveraged an integrated feature selection-based algorithmic approach, to identify important factors differentiating subjects with LATE and/or AD from Control on significantly imbalanced data. We analyzed two datasets ROSMAP and NACC and discovered that alcohol consumption was a top lifestyle and environmental factor linked with LATE and AD and their associations were differential. In particular, we identified a specific subpopulation consisting of APOE e4 carriers. We found that, for this subpopulation, light-to-moderate alcohol intake was a protective factor against both AD and LATE, but its protective role against AD appeared stronger than LATE. The codes for our algorithms are available at https://github.com/xinxingwu-uk/PFV. △ Less

Submitted 25 August, 2022; originally announced September 2022.

Comments: 10 pages

Journal ref: AMIA 2022 Annual Symposium (AMIA 2022)

Stacked Autoencoder Based Multi-Omics Data Integration for Cancer Survival Prediction

Authors: Xing Wu, Qiulian Fang

Abstract: Cancer survival prediction is important for develo** personalized treatments and inducing disease-causing mechanisms. Multi-omics data integration is attracting widespread interest in cancer research for providing information for understanding cancer progression at multiple genetic levels. Many works, however, are limited because of the high dimensionality and heterogeneity of multi-omics data.… ▽ More Cancer survival prediction is important for develo** personalized treatments and inducing disease-causing mechanisms. Multi-omics data integration is attracting widespread interest in cancer research for providing information for understanding cancer progression at multiple genetic levels. Many works, however, are limited because of the high dimensionality and heterogeneity of multi-omics data. In this paper, we propose a novel method to integrate multi-omics data for cancer survival prediction, called Stacked AutoEncoder-based Survival Prediction Neural Network (SAEsurv-net). In the cancer survival prediction for TCGA cases, SAEsurv-net addresses the curse of dimensionality with a two-stage dimensionality reduction strategy and handles multi-omics heterogeneity with a stacked autoencoder model. The two-stage dimensionality reduction strategy achieves a balance between computation complexity and information exploiting. The stacked autoencoder model removes most heterogeneities such as data's type and size in the first group of autoencoders, and integrates multiple omics data in the second autoencoder. The experiments show that SAEsurv-net outperforms models based on a single type of data as well as other state-of-the-art methods. △ Less

Submitted 8 July, 2022; originally announced July 2022.

Rethinking Spatial Invariance of Convolutional Networks for Object Counting

Authors: Zhi-Qi Cheng, Qi Dai, Hong Li, **gKuan Song, Xiao Wu, Alexander G. Hauptmann

Abstract: Previous work generally believes that improving the spatial invariance of convolutional networks is the key to object counting. However, after verifying several mainstream counting networks, we surprisingly found too strict pixel-level spatial invariance would cause overfit noise in the density map generation. In this paper, we try to use locally connected Gaussian kernels to replace the original… ▽ More Previous work generally believes that improving the spatial invariance of convolutional networks is the key to object counting. However, after verifying several mainstream counting networks, we surprisingly found too strict pixel-level spatial invariance would cause overfit noise in the density map generation. In this paper, we try to use locally connected Gaussian kernels to replace the original convolution filter to estimate the spatial position in the density map. The purpose of this is to allow the feature extraction process to potentially stimulate the density map generation process to overcome the annotation noise. Inspired by previous work, we propose a low-rank approximation accompanied with translation invariance to favorably implement the approximation of massive Gaussian convolution. Our work points a new direction for follow-up research, which should investigate how to properly relax the overly strict pixel-level spatial invariance for object counting. We evaluate our methods on 4 mainstream object counting networks (i.e., MCNN, CSRNet, SANet, and ResNet-50). Extensive experiments were conducted on 7 popular benchmarks for 3 applications (i.e., crowd, vehicle, and plant counting). Experimental results show that our methods significantly outperform other state-of-the-art methods and achieve promising learning of the spatial position of objects. △ Less

Submitted 18 August, 2022; v1 submitted 10 June, 2022; originally announced June 2022.

Comments: Accepted to CVPR 2022, Code: https://github.com/zhiqic/Rethinking-Counting

Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models

Authors: Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza , et al. (426 additional authors not shown)

Abstract: Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Despite their potentially transformative impact, these new capabilities are as yet poorly characterized. In order to inform future research, prepare for disruptive new model capabilities, and ameliorate socially harmful effects, it is vital that we understand the present and near-futur… ▽ More Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Despite their potentially transformative impact, these new capabilities are as yet poorly characterized. In order to inform future research, prepare for disruptive new model capabilities, and ameliorate socially harmful effects, it is vital that we understand the present and near-future capabilities and limitations of language models. To address this challenge, we introduce the Beyond the Imitation Game benchmark (BIG-bench). BIG-bench currently consists of 204 tasks, contributed by 450 authors across 132 institutions. Task topics are diverse, drawing problems from linguistics, childhood development, math, common-sense reasoning, biology, physics, social bias, software development, and beyond. BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models. We evaluate the behavior of OpenAI's GPT models, Google-internal dense transformer architectures, and Switch-style sparse transformers on BIG-bench, across model sizes spanning millions to hundreds of billions of parameters. In addition, a team of human expert raters performed all tasks in order to provide a strong baseline. Findings include: model performance and calibration both improve with scale, but are poor in absolute terms (and when compared with rater performance); performance is remarkably similar across model classes, though with benefits from sparsity; tasks that improve gradually and predictably commonly involve a large knowledge or memorization component, whereas tasks that exhibit "breakthrough" behavior at a critical scale often involve multiple steps or components, or brittle metrics; social bias typically increases with scale in settings with ambiguous context, but this can be improved with prompting. △ Less

Submitted 12 June, 2023; v1 submitted 9 June, 2022; originally announced June 2022.

Comments: 27 pages, 17 figures + references and appendices, repo: https://github.com/google/BIG-bench

Journal ref: Transactions on Machine Learning Research, May/2022, https://openreview.net/forum?id=uyTL5Bvosj

Distributed Dynamic Safe Screening Algorithms for Sparse Regularization

Authors: Runxue Bao, Xidong Wu, Wenhan Xian, Heng Huang

Abstract: Distributed optimization has been widely used as one of the most efficient approaches for model training with massive samples. However, large-scale learning problems with both massive samples and high-dimensional features widely exist in the era of big data. Safe screening is a popular technique to speed up high-dimensional models by discarding the inactive features with zero coefficients. Neverth… ▽ More Distributed optimization has been widely used as one of the most efficient approaches for model training with massive samples. However, large-scale learning problems with both massive samples and high-dimensional features widely exist in the era of big data. Safe screening is a popular technique to speed up high-dimensional models by discarding the inactive features with zero coefficients. Nevertheless, existing safe screening methods are limited to the sequential setting. In this paper, we propose a new distributed dynamic safe screening (DDSS) method for sparsity regularized models and apply it on shared-memory and distributed-memory architecture respectively, which can achieve significant speedup without any loss of accuracy by simultaneously enjoying the sparsity of the model and dataset. To the best of our knowledge, this is the first work of distributed safe dynamic screening method. Theoretically, we prove that the proposed method achieves the linear convergence rate with lower overall complexity and can eliminate almost all the inactive features in a finite number of iterations almost surely. Finally, extensive experimental results on benchmark datasets confirm the superiority of our proposed method. △ Less

Submitted 22 April, 2022; originally announced April 2022.

An alternative Interpretation of residual feed intake by phenotypic recursive relationships in dairy cattle

Authors: Xiao-Lin Wu, Kristen L. Parker Gaddis, Javier Burchard, H. Duane Norman, Ezequiel Nicolazzi, Erin E. Connor, John B. Cole, Joao Durr

Abstract: There has been an increasing interest in residual feed intake (RFI) as a measure of net feed efficiency in dairy cattle. RFI phenotypes are obtained as residuals from linear regression encompassing relevant factors (i.e., energy sinks) to account for body tissue mobilization. However, fitting energy sink phenotypes as regression variables in standard linear regression was criticized because phenot… ▽ More There has been an increasing interest in residual feed intake (RFI) as a measure of net feed efficiency in dairy cattle. RFI phenotypes are obtained as residuals from linear regression encompassing relevant factors (i.e., energy sinks) to account for body tissue mobilization. However, fitting energy sink phenotypes as regression variables in standard linear regression was criticized because phenotypes are subject to measurement errors. Multiple-trait models have been proposed which derive RFI by follow-up partial regression. By re-arranging the single-trait linear regression, we showed a causal RFI interpretation underlying the linear regression for RFI. It postulates recursive effects in energy allocation from energy sinks on dry matter intake, but the feedback or simultaneous effects are assumed to be nonexistent. A Bayesian recursive structural equation model was proposed for directly predicting RFI and energy sinks and estimating relevant genetic parameters simultaneously. A simplified Markov chain Monte Carlo algorithm that implemented the Bayesian recursive model was described. The recursive model is asymptotically equivalent to one-step linear regression for RFI, yet extends the analytical capacity to multiple-trait analysis. It is equivalent to Bayesian-implemented, multiple-trait model reparameterized based on Cholesky decomposition of phenotypic (co)variance matrix for evaluating RFI, but varied in assumptions about relationships between energy sinks. △ Less

Submitted 17 March, 2022; originally announced March 2022.

Comments: 12 pages, 2 Tables, and 1 Figure

Daily milk yield correction factors: what are they?

Authors: Xiao-Lin Wu, George Wiggans, H. Duane Norman, Asha M. Miles, Curt Van Tassell, Ransom L. Baldwin VI, Javier Burchard, Joao Durr

Abstract: Cows are typically milked two or more times on a test day, but not all these milkings are sampled and weighed. Statistical methods have been proposed to estimate daily yields in dairy cows, centering on various yield correction factors in two broad categories. The initial approach estimated a test-day yield with doubled morning (AM) or evening (PM) yield in the AM-PM milking plans, assuming equal… ▽ More Cows are typically milked two or more times on a test day, but not all these milkings are sampled and weighed. Statistical methods have been proposed to estimate daily yields in dairy cows, centering on various yield correction factors in two broad categories. The initial approach estimated a test-day yield with doubled morning (AM) or evening (PM) yield in the AM-PM milking plans, assuming equal AM and PM milking intervals. However, AM and PM milking intervals can vary, and milk secretion rates may be different between day and night. Additive correction factors (ACF) are evaluated by the average differences between AM and PM milk yield for various milking interval classes (MIC). We show that an ACF model is equivalent to a regression model of daily yield on categorical regressor variables, and a continuous variable for AM or PM yield with a fixed regression coefficient. Similarly, a linear regression model can be implemented as an ACF model with the regression coefficient for AM or PM yield estimated from the data. Multiplicative correction factors (MCF) are ratio of daily yield to yield from single milkings, but their statistical interpretations vary. Overall, MCF were more accurate for estimating daily milk yield than ACF. MCF have biological and statistical challenges. An exponential regression model was proposed as an alternative model for estimating daily milk yield, which improved the accuracy in the present study. Characterization of ACF and MCF showed how ACF and MCF improved the accuracy compared to doubling AM or PM yield as the daily milk yield. The methods were explicitly described to estimate daily milk yield in AM and PM milking plans. Still, the principles are generally applicable to cows milked more than two times a day, and they apply similarly to the estimation of daily fat and protein yields with some necessary modifications. △ Less

Submitted 17 March, 2022; originally announced March 2022.

Comments: 14 pages, 2 tables, and 1 figure

When AUC meets DRO: Optimizing Partial AUC for Deep Learning with Non-Convex Convergence Guarantee

Authors: Dixian Zhu, Gang Li, Bokun Wang, Xiaodong Wu, Tianbao Yang

Abstract: In this paper, we propose systematic and efficient gradient-based methods for both one-way and two-way partial AUC (pAUC) maximization that are applicable to deep learning. We propose new formulations of pAUC surrogate objectives by using the distributionally robust optimization (DRO) to define the loss for each individual positive data. We consider two formulations of DRO, one of which is based o… ▽ More In this paper, we propose systematic and efficient gradient-based methods for both one-way and two-way partial AUC (pAUC) maximization that are applicable to deep learning. We propose new formulations of pAUC surrogate objectives by using the distributionally robust optimization (DRO) to define the loss for each individual positive data. We consider two formulations of DRO, one of which is based on conditional-value-at-risk (CVaR) that yields a non-smooth but exact estimator for pAUC, and another one is based on a KL divergence regularized DRO that yields an inexact but smooth (soft) estimator for pAUC. For both one-way and two-way pAUC maximization, we propose two algorithms and prove their convergence for optimizing their two formulations, respectively. Experiments demonstrate the effectiveness of the proposed algorithms for pAUC maximization for deep learning on various datasets. △ Less

Submitted 17 September, 2023; v1 submitted 28 February, 2022; originally announced March 2022.

Comments: 25 pages

Journal ref: Proceedings of the 39th International Conference on Machine Learning, 2022

A pilot study of the Earable device to measure facial muscle and eye movement tasks among healthy volunteers

Authors: Matthew F. Wipperman, Galen Pogoncheff, Katrina F. Mateo, Xuefang Wu, Yiziying Chen, Oren Levy, Andreja Avbersek, Robin R. Deterding, Sara C. Hamon, Tam Vu, Rinol Alaj, Olivier Harari

Abstract: Many neuromuscular disorders impair function of cranial nerve enervated muscles. Clinical assessment of cranial muscle function has several limitations. Clinician rating of symptoms suffers from inter-rater variation, qualitative or semi-quantitative scoring, and limited ability to capture infrequent or fluctuating symptoms. Patient-reported outcomes are limited by recall bias and poor precision.… ▽ More Many neuromuscular disorders impair function of cranial nerve enervated muscles. Clinical assessment of cranial muscle function has several limitations. Clinician rating of symptoms suffers from inter-rater variation, qualitative or semi-quantitative scoring, and limited ability to capture infrequent or fluctuating symptoms. Patient-reported outcomes are limited by recall bias and poor precision. Current tools to measure orofacial and oculomotor function are cumbersome, difficult to implement, and non-portable. Here, we show how Earable, a wearable device, can discriminate certain cranial muscle activities such as chewing, talking, and swallowing. We demonstrate using data from a pilot study of 10 healthy participants how Earable can be used to measure features from EMG, EEG, and EOG waveforms from subjects performing mock Performance Outcome Assessments (mock-PerfOs), utilized widely in clinical research. Our analysis pipeline provides a framework for how to computationally process and statistically rank features from the Earable device. Finally, we demonstrate that Earable data may be used to classify these activities. Our results, conducted in a pilot study of healthy participants, enable a more comprehensive strategy for the design, development, and analysis of wearable sensor data for investigating clinical populations. Additionally, the results from this study support further evaluation of Earable or similar devices as tools to objectively measure cranial muscle activity in the context of a clinical research setting. Future work will be conducted in clinical disease populations, with a focus on detecting disease signatures, as well as monitoring intra-subject treatment responses. Readily available quantitative metrics from wearable sensor devices like Earable support strategies for the development of novel digital endpoints, a hallmark goal of clinical research. △ Less

Submitted 31 January, 2022; originally announced February 2022.

Generalized Transitional Markov Chain Monte Carlo Sampling Technique for Bayesian Inversion

Authors: Han Lu, Mohammad Khalil, Thomas Catanach, Jiefu Chen, Xuqing Wu, Xin Fu, Cosmin Safta, Yueqin Huang

Abstract: In the context of Bayesian inversion for scientific and engineering modeling, Markov chain Monte Carlo sampling strategies are the benchmark due to their flexibility and robustness in dealing with arbitrary posterior probability density functions (PDFs). However, these algorithms been shown to be inefficient when sampling from posterior distributions that are high-dimensional or exhibit multi-moda… ▽ More In the context of Bayesian inversion for scientific and engineering modeling, Markov chain Monte Carlo sampling strategies are the benchmark due to their flexibility and robustness in dealing with arbitrary posterior probability density functions (PDFs). However, these algorithms been shown to be inefficient when sampling from posterior distributions that are high-dimensional or exhibit multi-modality and/or strong parameter correlations. In such contexts, the sequential Monte Carlo technique of transitional Markov chain Monte Carlo (TMCMC) provides a more efficient alternative. Despite the recent applicability for Bayesian updating and model selection across a variety of disciplines, TMCMC may require a prohibitive number of tempering stages when the prior PDF is significantly different from the target posterior. Furthermore, the need to start with an initial set of samples from the prior distribution may present a challenge when dealing with implicit priors, e.g. based on feasible regions. Finally, TMCMC can not be used for inverse problems with improper prior PDFs that represent lack of prior knowledge on all or a subset of parameters. In this investigation, a generalization of TMCMC that alleviates such challenges and limitations is proposed, resulting in a tempering sampling strategy of enhanced robustness and computational efficiency. Convergence analysis of the proposed sequential Monte Carlo algorithm is presented, proving that the distance between the intermediate distributions and the target posterior distribution monotonically decreases as the algorithm proceeds. The enhanced efficiency associated with the proposed generalization is highlighted through a series of test inverse problems and an engineering application in the oil and gas industry. △ Less

Submitted 3 December, 2021; originally announced December 2021.

Understanding Entropic Regularization in GANs

Authors: Daria Reshetova, Yikun Bai, Xiugang Wu, Ayfer Ozgur

Abstract: Generative Adversarial Networks are a popular method for learning distributions from data by modeling the target distribution as a function of a known distribution. The function, often referred to as the generator, is optimized to minimize a chosen distance measure between the generated and target distributions. One commonly used measure for this purpose is the Wasserstein distance. However, Wasse… ▽ More Generative Adversarial Networks are a popular method for learning distributions from data by modeling the target distribution as a function of a known distribution. The function, often referred to as the generator, is optimized to minimize a chosen distance measure between the generated and target distributions. One commonly used measure for this purpose is the Wasserstein distance. However, Wasserstein distance is hard to compute and optimize, and in practice entropic regularization techniques are used to improve numerical convergence. The influence of regularization on the learned solution, however, remains not well-understood. In this paper, we study how several popular entropic regularizations of Wasserstein distance impact the solution in a simple benchmark setting where the generator is linear and the target distribution is high-dimensional Gaussian. We show that entropy regularization promotes the solution sparsification, while replacing the Wasserstein distance with the Sinkhorn divergence recovers the unregularized solution. Both regularization techniques remove the curse of dimensionality suffered by Wasserstein distance. We show that the optimal generator can be learned to accuracy $ε$ with $O(1/ε^2)$ samples from the target distribution. We thus conclude that these regularization techniques can improve the quality of the generator learned from empirical data for a large class of distributions. △ Less

Submitted 2 November, 2021; originally announced November 2021.

Comments: 29 pages, 7 figures

Adaptive Differentially Private Empirical Risk Minimization

Authors: Xiaoxia Wu, Lingxiao Wang, Irina Cristali, Quanquan Gu, Rebecca Willett

Abstract: We propose an adaptive (stochastic) gradient perturbation method for differentially private empirical risk minimization. At each iteration, the random noise added to the gradient is optimally adapted to the stepsize; we name this process adaptive differentially private (ADP) learning. Given the same privacy budget, we prove that the ADP method considerably improves the utility guarantee compared t… ▽ More We propose an adaptive (stochastic) gradient perturbation method for differentially private empirical risk minimization. At each iteration, the random noise added to the gradient is optimally adapted to the stepsize; we name this process adaptive differentially private (ADP) learning. Given the same privacy budget, we prove that the ADP method considerably improves the utility guarantee compared to the standard differentially private method in which vanilla random noise is added. Our method is particularly useful for gradient-based algorithms with time-varying learning rates, including variants of AdaGrad (Duchi et al., 2011). We provide extensive numerical experiments to demonstrate the effectiveness of the proposed adaptive differentially private algorithm. △ Less

Submitted 24 October, 2021; v1 submitted 14 October, 2021; originally announced October 2021.

Kalman Bayesian Neural Networks for Closed-form Online Learning

Authors: Philipp Wagner, Xinyang Wu, Marco F. Huber

Abstract: Compared to point estimates calculated by standard neural networks, Bayesian neural networks (BNN) provide probability distributions over the output predictions and model parameters, i.e., the weights. Training the weight distribution of a BNN, however, is more involved due to the intractability of the underlying Bayesian inference problem and thus, requires efficient approximations. In this paper… ▽ More Compared to point estimates calculated by standard neural networks, Bayesian neural networks (BNN) provide probability distributions over the output predictions and model parameters, i.e., the weights. Training the weight distribution of a BNN, however, is more involved due to the intractability of the underlying Bayesian inference problem and thus, requires efficient approximations. In this paper, we propose a novel approach for BNN learning via closed-form Bayesian inference. For this purpose, the calculation of the predictive distribution of the output and the update of the weight distribution are treated as Bayesian filtering and smoothing problems, where the weights are modeled as Gaussian random variables. This allows closed-form expressions for training the network's parameters in a sequential/online fashion without gradient descent. We demonstrate our method on several UCI datasets and compare it to the state of the art. △ Less

Submitted 30 November, 2022; v1 submitted 3 October, 2021; originally announced October 2021.

Comments: 37th AAAI Conference on Artificial Intelligence (AAAI)

Hierarchical Gaussian Process Models for Regression Discontinuity/Kink under Sharp and Fuzzy Designs

Authors: Ximing Wu

Abstract: We propose nonparametric Bayesian estimators for causal inference exploiting Regression Discontinuity/Kink (RD/RK) under sharp and fuzzy designs. Our estimators are based on Gaussian Process (GP) regression and classification. The GP methods are powerful probabilistic machine learning approaches that are advantageous in terms of derivative estimation and uncertainty quantification, facilitating RK… ▽ More We propose nonparametric Bayesian estimators for causal inference exploiting Regression Discontinuity/Kink (RD/RK) under sharp and fuzzy designs. Our estimators are based on Gaussian Process (GP) regression and classification. The GP methods are powerful probabilistic machine learning approaches that are advantageous in terms of derivative estimation and uncertainty quantification, facilitating RK estimation and inference of RD/RK models. These estimators are extended to hierarchical GP models with an intermediate Bayesian neural network layer and can be characterized as hybrid deep learning models. Monte Carlo simulations show that our estimators perform comparably to and sometimes better than competing estimators in terms of precision, coverage and interval length. The hierarchical GP models considerably improve upon one-layer GP models. We apply the proposed methods to estimate the incumbency advantage of US house elections. Our estimations suggest a significant incumbency advantage in terms of both vote share and probability of winning in the next elections. Lastly we present an extension to accommodate covariate adjustment. △ Less

Submitted 28 February, 2022; v1 submitted 3 October, 2021; originally announced October 2021.

Estimating a Causal Exposure Response Function with a Continuous Error-Prone Exposure: A Study of Fine Particulate Matter and All-Cause Mortality

Authors: Kevin P. Josey, Priyanka deSouza, Xiao Wu, Danielle Braun, Rachel Nethery

Abstract: Numerous studies have examined the associations between long-term exposure to fine particulate matter (PM2.5) and adverse health outcomes. Recently, many of these studies have begun to employ high-resolution predicted PM2.5 concentrations, which are subject to measurement error. Previous approaches for exposure measurement error correction have either been applied in non-causal settings or have on… ▽ More Numerous studies have examined the associations between long-term exposure to fine particulate matter (PM2.5) and adverse health outcomes. Recently, many of these studies have begun to employ high-resolution predicted PM2.5 concentrations, which are subject to measurement error. Previous approaches for exposure measurement error correction have either been applied in non-causal settings or have only considered a categorical exposure. Moreover, most procedures have failed to account for uncertainty induced by error correction when fitting an exposure-response function (ERF). To remedy these deficiencies, we develop a multiple imputation framework that combines regression calibration and Bayesian techniques to estimate a causal ERF. We demonstrate how the output of the measurement error correction steps can be seamlessly integrated into a Bayesian additive regression trees (BART) estimator of the causal ERF. We also demonstrate how locally-weighted smoothing of the posterior samples from BART can be used to create a more accurate ERF estimate. Our proposed approach also properly propagates the exposure measurement error uncertainty to yield accurate standard error estimates. We assess the robustness of our proposed approach in an extensive simulation study. We then apply our methodology to estimate the effects of PM2.5 on all-cause mortality among Medicare enrollees in New England from 2000-2012. △ Less

Submitted 28 November, 2022; v1 submitted 30 September, 2021; originally announced September 2021.

Quantifying Grid Resilience Against Extreme Weather Using Large-Scale Customer Power Outage Data

Authors: Shixiang Zhu, Rui Yao, Yao Xie, Feng Qiu, Yueming, Qiu, Xuan Wu

Abstract: In recent decades, the weather around the world has become more irregular and extreme, often causing large-scale extended power outages. Resilience -- the capability of withstanding, adapting to, and recovering from a large-scale disruption -- has become a top priority for the power sector. However, the understanding of power grid resilience still stays on the conceptual level mostly or focuses on… ▽ More In recent decades, the weather around the world has become more irregular and extreme, often causing large-scale extended power outages. Resilience -- the capability of withstanding, adapting to, and recovering from a large-scale disruption -- has become a top priority for the power sector. However, the understanding of power grid resilience still stays on the conceptual level mostly or focuses on particular components, yielding no actionable results or revealing few insights on the system level. This study provides a quantitatively measurable definition of power grid resilience, using a statistical model inspired by patterns observed from data and domain knowledge. We analyze a large-scale quarter-hourly historical electricity customer outage data and the corresponding weather records, and draw connections between the model and industry resilience practice. We showcase the resilience analysis using three major service territories on the east coast of the United States. Our analysis suggests that cumulative weather effects play a key role in causing immediate, sustained outages, and these outages can propagate and cause secondary outages in neighboring areas. The proposed model also provides some interesting insights into grid resilience enhancement planning. For example, our simulation results indicate that enhancing the power infrastructure in a small number of critical locations can reduce nearly half of the number of customer power outages in Massachusetts. In addition, we have shown that our model achieves promising accuracy in predicting the progress of customer power outages throughout extreme weather events, which can be very valuable for system operators and federal agencies to prepare disaster response. △ Less

Submitted 4 September, 2022; v1 submitted 20 September, 2021; originally announced September 2021.

AdaLoss: A computationally-efficient and provably convergent adaptive gradient method

Authors: Xiaoxia Wu, Yuege Xie, Simon Du, Rachel Ward

Abstract: We propose a computationally-friendly adaptive learning rate schedule, "AdaLoss", which directly uses the information of the loss function to adjust the stepsize in gradient descent methods. We prove that this schedule enjoys linear convergence in linear regression. Moreover, we provide a linear convergence guarantee over the non-convex regime, in the context of two-layer over-parameterized neural… ▽ More We propose a computationally-friendly adaptive learning rate schedule, "AdaLoss", which directly uses the information of the loss function to adjust the stepsize in gradient descent methods. We prove that this schedule enjoys linear convergence in linear regression. Moreover, we provide a linear convergence guarantee over the non-convex regime, in the context of two-layer over-parameterized neural networks. If the width of the first-hidden layer in the two-layer networks is sufficiently large (polynomially), then AdaLoss converges robustly \emph{to the global minimum} in polynomial time. We numerically verify the theoretical results and extend the scope of the numerical experiments by considering applications in LSTM models for text clarification and policy gradients for control problems. △ Less

Submitted 16 September, 2021; originally announced September 2021.

Comments: arXiv admin note: text overlap with arXiv:1902.07111

Top-$k$ Regularization for Supervised Feature Selection

Authors: Xinxing Wu, Qiang Cheng

Abstract: Feature selection identifies subsets of informative features and reduces dimensions in the original feature space, hel** provide insights into data generation or a variety of domain problems. Existing methods mainly depend on feature scoring functions or sparse regularizations; nonetheless, they have limited ability to reconcile the representativeness and inter-correlations of features. In this… ▽ More Feature selection identifies subsets of informative features and reduces dimensions in the original feature space, hel** provide insights into data generation or a variety of domain problems. Existing methods mainly depend on feature scoring functions or sparse regularizations; nonetheless, they have limited ability to reconcile the representativeness and inter-correlations of features. In this paper, we introduce a novel, simple yet effective regularization approach, named top-$k$ regularization, to supervised feature selection in regression and classification tasks. Structurally, the top-$k$ regularization induces a sub-architecture on the architecture of a learning model to boost its ability to select the most informative features and model complex nonlinear relationships simultaneously. Theoretically, we derive and mathematically prove a uniform approximation error bound for using this approach to approximate high-dimensional sparse functions. Extensive experiments on a wide variety of benchmarking datasets show that the top-$k$ regularization is effective and stable for supervised feature selection. △ Less

Submitted 3 June, 2021; originally announced June 2021.

Comments: 12 pages

InfoFair: Information-Theoretic Intersectional Fairness

Authors: Jian Kang, Tiankai Xie, Xintao Wu, Ross Maciejewski, Hanghang Tong

Abstract: Algorithmic fairness is becoming increasingly important in data mining and machine learning. Among others, a foundational notation is group fairness. The vast majority of the existing works on group fairness, with a few exceptions, primarily focus on debiasing with respect to a single sensitive attribute, despite the fact that the co-existence of multiple sensitive attributes (e.g., gender, race,… ▽ More Algorithmic fairness is becoming increasingly important in data mining and machine learning. Among others, a foundational notation is group fairness. The vast majority of the existing works on group fairness, with a few exceptions, primarily focus on debiasing with respect to a single sensitive attribute, despite the fact that the co-existence of multiple sensitive attributes (e.g., gender, race, marital status, etc.) in the real-world is commonplace. As such, methods that can ensure a fair learning outcome with respect to all sensitive attributes of concern simultaneously need to be developed. In this paper, we study the problem of information-theoretic intersectional fairness (InfoFair), where statistical parity, a representative group fairness measure, is guaranteed among demographic groups formed by multiple sensitive attributes of interest. We formulate it as a mutual information minimization problem and propose a generic end-to-end algorithmic framework to solve it. The key idea is to leverage a variational representation of mutual information, which considers the variational distribution between learning outcomes and sensitive attributes, as well as the density ratio between the variational and the original distributions. Our proposed framework is generalizable to many different settings, including other statistical notions of fairness, and could handle any type of learning task equipped with a gradient-based optimizer. Empirical evaluations in the fair classification task on three real-world datasets demonstrate that our proposed framework can effectively debias the classification results with minimal impact to the classification accuracy. △ Less

Submitted 31 December, 2022; v1 submitted 23 May, 2021; originally announced May 2021.

Comments: IEEE Big Data 2022

Bayesian Inverse Uncertainty Quantification of a MOOSE-based Melt Pool Model for Additive Manufacturing Using Experimental Data

Authors: Ziyu Xie, Wen Jiang, Congjian Wang, Xu Wu

Abstract: Additive manufacturing (AM) technology is being increasingly adopted in a wide variety of application areas due to its ability to rapidly produce, prototype, and customize designs. AM techniques afford significant opportunities in regard to nuclear materials, including an accelerated fabrication process and reduced cost. High-fidelity modeling and simulation (M\&S) of AM processes is being develop… ▽ More Additive manufacturing (AM) technology is being increasingly adopted in a wide variety of application areas due to its ability to rapidly produce, prototype, and customize designs. AM techniques afford significant opportunities in regard to nuclear materials, including an accelerated fabrication process and reduced cost. High-fidelity modeling and simulation (M\&S) of AM processes is being developed in Idaho National Laboratory (INL)'s Multiphysics Object-Oriented Simulation Environment (MOOSE) to support AM process optimization and provide a fundamental understanding of the various physical interactions involved. In this paper, we employ Bayesian inverse uncertainty quantification (UQ) to quantify the input uncertainties in a MOOSE-based melt pool model for AM. Inverse UQ is the process of inversely quantifying the input uncertainties while kee** model predictions consistent with the measurement data. The inverse UQ process takes into account uncertainties from the model, code, and data while simultaneously characterizing the uncertain distributions in the input parameters--rather than merely providing best-fit point estimates. We employ measurement data on melt pool geometry (lengths and depths) to quantify the uncertainties in several melt pool model parameters. Simulation results using the posterior uncertainties have shown improved agreement with experimental data, as compared to those using the prior nominal values. The resulting parameter uncertainties can be used to replace expert opinions in future uncertainty, sensitivity, and validation studies. △ Less

Submitted 17 May, 2021; v1 submitted 11 May, 2021; originally announced May 2021.

Comments: 26 pages, 11 figures

A Bayesian Gaussian Process for Estimating a Causal Exposure Response Curve in Environmental Epidemiology

Authors: Boyu Ren, Xiao Wu, Danielle Braun, Natesh Pillai, Francesca Dominici

Abstract: Motivated by environmental policy questions, we address the challenges of estimation, change point detection, and uncertainty quantification of a causal exposure-response function (CERF). Under a potential outcome framework, the CERF describes the relationship between a continuously varying exposure (or treatment) and its causal effect on an outcome. We propose a new Bayesian approach that relies… ▽ More Motivated by environmental policy questions, we address the challenges of estimation, change point detection, and uncertainty quantification of a causal exposure-response function (CERF). Under a potential outcome framework, the CERF describes the relationship between a continuously varying exposure (or treatment) and its causal effect on an outcome. We propose a new Bayesian approach that relies on a Gaussian process (GP) model to estimate the CERF nonparametrically. To achieve the desired separation of design and analysis phases, we parametrize the covariance (kernel) function of the GP to mimic matching via a Generalized Propensity Score (GPS). The hyper-parameters as well as the form of the kernel function of the GP are chosen to optimize covariate balance. Our approach achieves automatic uncertainty evaluation of the CERF with high computational efficiency, and enables change point detection through inference on derivatives of the CERF. We provide theoretical results showing the correspondence between our Bayesian GP framework and traditional approaches in causal inference for estimating causal effects of a continuous exposure. We apply the methods to 520,711 ZIP-code-level observations to estimate the causal effect of long-term exposures to PM2.5, ozone, and NO2 on all-cause mortality among Medicare enrollees in the US. A computationally efficient implementation of the proposed GP models is provided in the GPCERF R package, which is available on CRAN. △ Less

Submitted 25 January, 2023; v1 submitted 7 May, 2021; originally announced May 2021.