Showing 1–33 of 33 results for author: Yue, M

A Geometric Unification of Distributionally Robust Covariance Estimators: Shrinking the Spectrum by Inflating the Ambiguity Set

Authors: Man-Chung Yue, Yves Rychener, Daniel Kuhn, Viet Anh Nguyen

Abstract: The state-of-the-art methods for estimating high-dimensional covariance matrices all shrink the eigenvalues of the sample covariance matrix towards a data-insensitive shrinkage target. The underlying shrinkage transformation is either chosen heuristically - without compelling theoretical justification - or optimally in view of restrictive distributional assumptions. In this paper, we propose a pri… ▽ More The state-of-the-art methods for estimating high-dimensional covariance matrices all shrink the eigenvalues of the sample covariance matrix towards a data-insensitive shrinkage target. The underlying shrinkage transformation is either chosen heuristically - without compelling theoretical justification - or optimally in view of restrictive distributional assumptions. In this paper, we propose a principled approach to construct covariance estimators without imposing restrictive assumptions. That is, we study distributionally robust covariance estimation problems that minimize the worst-case Frobenius error with respect to all data distributions close to a nominal distribution, where the proximity of distributions is measured via a divergence on the space of covariance matrices. We identify mild conditions on this divergence under which the resulting minimizers represent shrinkage estimators. We show that the corresponding shrinkage transformations are intimately related to the geometrical properties of the underlying divergence. We also prove that our robust estimators are efficiently computable and asymptotically consistent and that they enjoy finite-sample performance guarantees. We exemplify our general methodology by synthesizing explicit estimators induced by the Kullback-Leibler, Fisher-Rao, and Wasserstein divergences. Numerical experiments based on synthetic and real data show that our robust estimators are competitive with state-of-the-art estimators. △ Less

Submitted 30 May, 2024; originally announced May 2024.

Subdifferentially polynomially bounded functions and Gaussian smoothing-based zeroth-order optimization

Authors: Ming Lei, Ting Kei Pong, Shuqin Sun, Man-Chung Yue

Abstract: We introduce the class of subdifferentially polynomially bounded (SPB) functions, which is a rich class of locally Lipschitz functions that encompasses all Lipschitz functions, all gradient- or Hessian-Lipschitz functions, and even some non-smooth locally Lipschitz functions. We show that SPB functions are compatible with Gaussian smoothing (GS), in the sense that the GS of any SPB function is wel… ▽ More We introduce the class of subdifferentially polynomially bounded (SPB) functions, which is a rich class of locally Lipschitz functions that encompasses all Lipschitz functions, all gradient- or Hessian-Lipschitz functions, and even some non-smooth locally Lipschitz functions. We show that SPB functions are compatible with Gaussian smoothing (GS), in the sense that the GS of any SPB function is well-defined and satisfies a descent lemma akin to gradient-Lipschitz functions, with the Lipschitz constant replaced by a polynomial function. Leveraging this descent lemma, we propose GS-based zeroth-order optimization algorithms with an adaptive stepsize strategy for constrained minimization of SPB functions, and analyze their iteration complexity. An important instrument in our analysis, which could be of independent interest, is the quantification of Goldstein stationarity via the GS gradient. △ Less

Submitted 7 May, 2024; originally announced May 2024.

Multigrid method for nonlinear eigenvalue problems based on Newton iteration

Authors: Fei Xu, Manting Xie, Meiling Yue

Abstract: In this paper, a novel multigrid method based on Newton iteration is proposed to solve nonlinear eigenvalue problems. Instead of handling the eigenvalue $λ$ and eigenfunction $u$ separately, we treat the eigenpair $(λ, u)$ as one element in a product space $\mathbb R \times H_0^1(Ω)$. Then in the presented multigrid method, only one discrete linear boundary value problem needs to be solved for eac… ▽ More In this paper, a novel multigrid method based on Newton iteration is proposed to solve nonlinear eigenvalue problems. Instead of handling the eigenvalue $λ$ and eigenfunction $u$ separately, we treat the eigenpair $(λ, u)$ as one element in a product space $\mathbb R \times H_0^1(Ω)$. Then in the presented multigrid method, only one discrete linear boundary value problem needs to be solved for each level of the multigrid sequence. Because we avoid solving large-scale nonlinear eigenvalue problems directly, the overall efficiency is significantly improved. The optimal error estimate and linear computational complexity can be derived simultaneously. In addition, we also provide an improved multigrid method coupled with a mixing scheme to further guarantee the convergence and stability of the iteration scheme. More importantly, we prove convergence for the residuals after each iteration step. For nonlinear eigenvalue problems, such theoretical analysis is missing from the existing literatures on the mixing iteration scheme. △ Less

Submitted 29 April, 2024; originally announced April 2024.

A Max-Min-Max Algorithm for Large-Scale Robust Optimization

Authors: Kai Tu, Zhi Chen, Man-Chung Yue

Abstract: Robust optimization (RO) is a powerful paradigm for decision making under uncertainty. Existing algorithms for solving RO, including the reformulation approach and the cutting-plane method, do not scale well, hindering the application of RO to large-scale decision problems. In this paper, we devise a first-order algorithm for solving RO based on a novel max-min-max perspective. Our algorithm opera… ▽ More Robust optimization (RO) is a powerful paradigm for decision making under uncertainty. Existing algorithms for solving RO, including the reformulation approach and the cutting-plane method, do not scale well, hindering the application of RO to large-scale decision problems. In this paper, we devise a first-order algorithm for solving RO based on a novel max-min-max perspective. Our algorithm operates directly on the model functions and sets through the subgradient and projection oracles, which enables the exploitation of problem structures and is especially suitable for large-scale RO. Theoretically, we prove that the oracle complexity of our algorithm for attaining an $\varepsilon$-approximate optimal solution is $\mathcal{O}(\varepsilon^{-3})$ or $\mathcal{O}(\varepsilon^{-2})$, depending on the smoothness of the model functions. The algorithm and its theoretical results are then extended to RO with projection-unfriendly uncertainty sets. We also show via extensive numerical experiments that the proposed algorithm outperforms the reformulation approach, the cutting-plane method and two other recent first-order algorithms. △ Less

Submitted 8 April, 2024; originally announced April 2024.

An MILP-Based Solution Scheme for Factored and Robust Factored Markov Decision Processes

Authors: Huikang Liu, Wolfram Wiesemann, Man-Chung Yue

Abstract: Factored Markov decision processes (MDPs) are a prominent paradigm within the artificial intelligence community for modeling and solving large-scale MDPs whose rewards and dynamics decompose into smaller, loosely interacting components. Through the use of dynamic Bayesian networks and context-specific independence, factored MDPs can achieve an exponential reduction in the state space of an MDP and… ▽ More Factored Markov decision processes (MDPs) are a prominent paradigm within the artificial intelligence community for modeling and solving large-scale MDPs whose rewards and dynamics decompose into smaller, loosely interacting components. Through the use of dynamic Bayesian networks and context-specific independence, factored MDPs can achieve an exponential reduction in the state space of an MDP and thus scale to problem sizes that are beyond the reach of classical MDP algorithms. However, factored MDPs are typically solved using custom-designed algorithms that can require meticulous implementations and considerable fine-tuning. In this paper, we propose a mathematical programming approach to solving factored MDPs. In contrast to existing solution schemes, our approach leverages off-the-shelf solvers, which allows for a streamlined implementation and maintenance; it effectively capitalizes on the factored structure present in both state and action spaces; and it readily extends to the largely unexplored class of robust factored MDPs, whose transition kernels are only known to reside in a pre-specified ambiguity set. Our numerical experiments demonstrate the potential of our approach. △ Less

Submitted 2 April, 2024; originally announced April 2024.

Coverage-Validity-Aware Algorithmic Recourse

Authors: Ngoc Bui, Duy Nguyen, Man-Chung Yue, Viet Anh Nguyen

Abstract: Algorithmic recourse emerges as a prominent technique to promote the explainability, transparency and hence ethics of machine learning models. Existing algorithmic recourse approaches often assume an invariant predictive model; however, the predictive model is usually updated upon the arrival of new data. Thus, a recourse that is valid respective to the present model may become invalid for the fut… ▽ More Algorithmic recourse emerges as a prominent technique to promote the explainability, transparency and hence ethics of machine learning models. Existing algorithmic recourse approaches often assume an invariant predictive model; however, the predictive model is usually updated upon the arrival of new data. Thus, a recourse that is valid respective to the present model may become invalid for the future model. To resolve this issue, we propose a novel framework to generate a model-agnostic recourse that exhibits robustness to model shifts. Our framework first builds a coverage-validity-aware linear surrogate of the nonlinear (black-box) model; then, the recourse is generated with respect to the linear surrogate. We establish a theoretical connection between our coverage-validity-aware linear surrogate and the minimax probability machines (MPM). We then prove that by prescribing different covariance robustness, the proposed framework recovers popular regularizations for MPM, including the $\ell_2$-regularization and class-reweighting. Furthermore, we show that our surrogate pushes the approximate hyperplane intuitively, facilitating not only robust but also interpretable recourses. The numerical results demonstrate the usefulness and robustness of our framework. △ Less

Submitted 19 November, 2023; originally announced November 2023.

Numerical strategy on the grid orientation effect in the simulation for two-phase flow in porous media by using the adaptive artificial viscosity method

Authors: Xiao-Hong Wang, Meng-Chen Yue, Zhi-Feng Liu, Wei-Dong Cao, Yong Wang, Jun Hu, Chang-Hao Xiao, Yao-Yong Li

Abstract: It is a challenge to numerically solve nonlinear partial differential equations whose solution involves discontinuity. In the context of numerical simulators for multi-phase flow in porous media, there exists a long-standing issue known as Grid Orientation Effect (GOE), wherein different numerical solutions can be obtained when considering grids with different orientations under certain unfavorabl… ▽ More It is a challenge to numerically solve nonlinear partial differential equations whose solution involves discontinuity. In the context of numerical simulators for multi-phase flow in porous media, there exists a long-standing issue known as Grid Orientation Effect (GOE), wherein different numerical solutions can be obtained when considering grids with different orientations under certain unfavorable conditions. Our perspective is that GOE arises due to numerical instability near displacement fronts, where spurious oscillations accompanied by sharp fronts, if not adequately suppressed, lead to GOE. To reduce or even eliminate GOE, we propose augmenting adaptive artificial viscosity when solving the saturation equation. It has been demonstrated that appropriate artificial viscosity can effectively reduce or even eliminate GOE. The proposed numerical method can be easily applied in practical engineering problems. △ Less

Submitted 13 August, 2023; originally announced August 2023.

Comments: 10 pages

On Approximating the Dynamic Response of Synchronous Generators via Operator Learning: A Step Towards Building Deep Operator-based Power Grid Simulators

Authors: Christian Moya, Guang Lin, Tianqiao Zhao, Meng Yue

Abstract: This paper designs an Operator Learning framework to approximate the dynamic response of synchronous generators. One can use such a framework to (i) design a neural-based generator model that can interact with a numerical simulator of the rest of the power grid or (ii) shadow the generator's transient response. To this end, we design a data-driven Deep Operator Network~(DeepONet) that approximates… ▽ More This paper designs an Operator Learning framework to approximate the dynamic response of synchronous generators. One can use such a framework to (i) design a neural-based generator model that can interact with a numerical simulator of the rest of the power grid or (ii) shadow the generator's transient response. To this end, we design a data-driven Deep Operator Network~(DeepONet) that approximates the generators' infinite-dimensional solution operator. Then, we develop a DeepONet-based numerical scheme to simulate a given generator's dynamic response over a short/medium-term horizon. The proposed numerical scheme recursively employs the trained DeepONet to simulate the response for a given multi-dimensional input, which describes the interaction between the generator and the rest of the system. Furthermore, we develop a residual DeepONet numerical scheme that incorporates information from mathematical models of synchronous generators. We accompany this residual DeepONet scheme with an estimate for the prediction's cumulative error. We also design a data aggregation (DAgger) strategy that allows (i) employing supervised learning to train the proposed DeepONets and (ii) fine-tuning the DeepONet using aggregated training data that the DeepONet is likely to encounter during interactive simulations with other grid components. Finally, as a proof of concept, we demonstrate that the proposed DeepONet frameworks can effectively approximate the transient model of a synchronous generator. △ Less

Submitted 29 January, 2023; originally announced January 2023.

Approximate Secular Equations for the Cubic Regularization Subproblem

Authors: Yihang Gao, Man-Chung Yue, Michael K. Ng

Abstract: The cubic regularization method (CR) is a popular algorithm for unconstrained non-convex optimization. At each iteration, CR solves a cubically regularized quadratic problem, called the cubic regularization subproblem (CRS). One way to solve the CRS relies on solving the secular equation, whose computational bottleneck lies in the computation of all eigenvalues of the Hessian matrix. In this paper… ▽ More The cubic regularization method (CR) is a popular algorithm for unconstrained non-convex optimization. At each iteration, CR solves a cubically regularized quadratic problem, called the cubic regularization subproblem (CRS). One way to solve the CRS relies on solving the secular equation, whose computational bottleneck lies in the computation of all eigenvalues of the Hessian matrix. In this paper, we propose and analyze a novel CRS solver based on an approximate secular equation, which requires only some of the Hessian eigenvalues and is therefore much more efficient. Two approximate secular equations (ASEs) are developed. For both ASEs, we first study the existence and uniqueness of their roots and then establish an upper bound on the gap between the root and that of the standard secular equation. Such an upper bound can in turn be used to bound the distance from the approximate CRS solution based ASEs to the true CRS solution, thus offering a theoretical guarantee for our CRS solver. A desirable feature of our CRS solver is that it requires only matrix-vector multiplication but not matrix inversion, which makes it particularly suitable for high-dimensional applications of unconstrained non-convex optimization, such as low-rank recovery and deep learning. Numerical experiments with synthetic and real data-sets are conducted to investigate the practical performance of the proposed CRS solver. Experimental results show that the proposed solver outperforms two state-of-the-art methods. △ Less

Submitted 27 September, 2022; originally announced September 2022.

Comments: Accepted to NeurIPS 2022

Variance Reduced Random Relaxed Projection Method for Constrained Finite-sum Minimization Problems

Authors: Zhichun Yang, Fu-quan Xia, Kai Tu, Man-Chung Yue

Abstract: For many applications in signal processing and machine learning, we are tasked with minimizing a large sum of convex functions subject to a large number of convex constraints. In this paper, we devise a new random projection method (RPM) to efficiently solve this problem. Compared with existing RPMs, our proposed algorithm features two useful algorithmic ideas. First, at each iteration, instead of… ▽ More For many applications in signal processing and machine learning, we are tasked with minimizing a large sum of convex functions subject to a large number of convex constraints. In this paper, we devise a new random projection method (RPM) to efficiently solve this problem. Compared with existing RPMs, our proposed algorithm features two useful algorithmic ideas. First, at each iteration, instead of projecting onto the subset defined by one of the constraints, our algorithm only requires projecting onto a half-space approximation of the subset, which significantly reduces the computational cost as it admits a closed-form formula. Second, to exploit the structure that the objective is a sum, variance reduction is incorporated into our algorithm to further improve the performance. As theoretical contributions, under a novel error bound condition and other standard assumptions, we prove that the proposed RPM converges to an optimal solution and that both optimality and feasibility gaps vanish at a sublinear rate. In particular, via a new analysis framework, we show that our RPM attains a faster convergence rate in optimality gap than existing RPMs when the objective function has a Lipschitz continuous gradient, capitalizing the benefit of the variance reduction. We also provide sufficient conditions for the error bound condition to hold. Experiments on a beamforming problem and a robust classification problem are also presented to demonstrate the superiority of our RPM over existing ones. △ Less

Submitted 5 April, 2024; v1 submitted 27 June, 2022; originally announced June 2022.

DeepONet-Grid-UQ: A Trustworthy Deep Operator Framework for Predicting the Power Grid's Post-Fault Trajectories

Authors: Christian Moya, Shiqi Zhang, Meng Yue, Guang Lin

Abstract: This paper proposes a new data-driven method for the reliable prediction of power system post-fault trajectories. The proposed method is based on the fundamentally new concept of Deep Operator Networks (DeepONets). Compared to traditional neural networks that learn to approximate functions, DeepONets are designed to approximate nonlinear operators. Under this operator framework, we design a DeepON… ▽ More This paper proposes a new data-driven method for the reliable prediction of power system post-fault trajectories. The proposed method is based on the fundamentally new concept of Deep Operator Networks (DeepONets). Compared to traditional neural networks that learn to approximate functions, DeepONets are designed to approximate nonlinear operators. Under this operator framework, we design a DeepONet to (1) take as inputs the fault-on trajectories collected, for example, via simulation or phasor measurement units, and (2) provide as outputs the predicted post-fault trajectories. In addition, we endow our method with a much-needed ability to balance efficiency with reliable/trustworthy predictions via uncertainty quantification. To this end, we propose and compare two methods that enable quantifying the predictive uncertainty. First, we propose a \textit{Bayesian DeepONet} (B-DeepONet) that uses stochastic gradient Hamiltonian Monte-Carlo to sample from the posterior distribution of the DeepONet parameters. Then, we propose a \textit{Probabilistic DeepONet} (Prob-DeepONet) that uses a probabilistic training strategy to equip DeepONets with a form of automated uncertainty quantification, at virtually no extra computational cost. Finally, we validate the predictive power and uncertainty quantification capability of the proposed B-DeepONet and Prob-DeepONet using the IEEE 16-machine 68-bus system. △ Less

Submitted 14 February, 2022; originally announced February 2022.

Distributionally Robust Fair Principal Components via Geodesic Descents

Authors: Hieu Vu, Toan Tran, Man-Chung Yue, Viet Anh Nguyen

Abstract: Principal component analysis is a simple yet useful dimensionality reduction technique in modern machine learning pipelines. In consequential domains such as college admission, healthcare and credit approval, it is imperative to take into account emerging criteria such as the fairness and the robustness of the learned projection. In this paper, we propose a distributionally robust optimization pro… ▽ More Principal component analysis is a simple yet useful dimensionality reduction technique in modern machine learning pipelines. In consequential domains such as college admission, healthcare and credit approval, it is imperative to take into account emerging criteria such as the fairness and the robustness of the learned projection. In this paper, we propose a distributionally robust optimization problem for principal component analysis which internalizes a fairness criterion in the objective function. The learned projection thus balances the trade-off between the total reconstruction error and the reconstruction error gap between subgroups, taken in the min-max sense over all distributions in a moment-based ambiguity set. The resulting optimization problem over the Stiefel manifold can be efficiently solved by a Riemannian subgradient descent algorithm with a sub-linear convergence rate. Our experimental results on real-world datasets show the merits of our proposed method over state-of-the-art baselines. △ Less

Submitted 7 February, 2022; originally announced February 2022.

Comments: International Conference on Learning Representations (ICLR) 2022

Short-step Methods Are Not Strongly Polynomial-Time

Authors: Manru Zong, Yin Tat Lee, Man-Chung Yue

Abstract: Short-step methods are an important class of algorithms for solving convex constrained optimization problems. In this short paper, we show that under very mild assumptions on the self-concordant barrier and the width of the $\ell_2$-neighbourhood, any short-step interior-point method is not strongly polynomial-time. Short-step methods are an important class of algorithms for solving convex constrained optimization problems. In this short paper, we show that under very mild assumptions on the self-concordant barrier and the width of the $\ell_2$-neighbourhood, any short-step interior-point method is not strongly polynomial-time. △ Less

Submitted 8 January, 2022; originally announced January 2022.

Sequential Domain Adaptation by Synthesizing Distributionally Robust Experts

Authors: Bahar Taskesen, Man-Chung Yue, Jose Blanchet, Daniel Kuhn, Viet Anh Nguyen

Abstract: Least squares estimators, when trained on a few target domain samples, may predict poorly. Supervised domain adaptation aims to improve the predictive accuracy by exploiting additional labeled training samples from a source distribution that is close to the target distribution. Given available data, we investigate novel strategies to synthesize a family of least squares estimator experts that are… ▽ More Least squares estimators, when trained on a few target domain samples, may predict poorly. Supervised domain adaptation aims to improve the predictive accuracy by exploiting additional labeled training samples from a source distribution that is close to the target distribution. Given available data, we investigate novel strategies to synthesize a family of least squares estimator experts that are robust with regard to moment conditions. When these moment conditions are specified using Kullback-Leibler or Wasserstein-type divergences, we can find the robust estimators efficiently using convex optimization. We use the Bernstein online aggregation algorithm on the proposed family of robust experts to generate predictions for the sequential stream of target test samples. Numerical experiments on real data show that the robust strategies may outperform non-robust interpolations of the empirical least squares estimators. △ Less

Submitted 1 June, 2021; originally announced June 2021.

Small errors in random zeroth-order optimization are imaginary

Authors: Wouter Jongeneel, Man-Chung Yue, Daniel Kuhn

Abstract: Most zeroth-order optimization algorithms mimic a first-order algorithm but replace the gradient of the objective function with some gradient estimator that can be computed from a small number of function evaluations. This estimator is constructed randomly, and its expectation matches the gradient of a smooth approximation of the objective function whose quality improves as the underlying smoothin… ▽ More Most zeroth-order optimization algorithms mimic a first-order algorithm but replace the gradient of the objective function with some gradient estimator that can be computed from a small number of function evaluations. This estimator is constructed randomly, and its expectation matches the gradient of a smooth approximation of the objective function whose quality improves as the underlying smoothing parameter $δ$ is reduced. Gradient estimators requiring a smaller number of function evaluations are preferable from a computational point of view. While estimators based on a single function evaluation can be obtained by use of the divergence theorem from vector calculus, their variance explodes as $δ$ tends to $0$. Estimators based on multiple function evaluations, on the other hand, suffer from numerical cancellation when $δ$ tends to $0$. To combat both effects simultaneously, we extend the objective function to the complex domain and construct a gradient estimator that evaluates the objective at a complex point whose coordinates have small imaginary parts of the order $δ$. As this estimator requires only one function evaluation, it is immune to cancellation. In addition, its variance remains bounded as $δ$ tends to $0$. We prove that zeroth-order algorithms that use our estimator offer the same theoretical convergence guarantees as the state-of-the-art methods. Numerical experiments suggest, however, that they often converge faster in practice. △ Less

Submitted 19 March, 2024; v1 submitted 9 March, 2021; originally announced March 2021.

Comments: Final version (33 pages), to appear in the SIAM Journal on Optimization

MSC Class: 65D25; 65G50; 65K05; 65Y04; 65Y20; 90C56

A Unified Approach to Synchronization Problems over Subgroups of the Orthogonal Group

Authors: Huikang Liu, Man-Chung Yue, Anthony Man-Cho So

Abstract: The problem of synchronization over a group $\mathcal{G}$ aims to estimate a collection of group elements $G^*_1, \dots, G^*_n \in \mathcal{G}$ based on noisy observations of a subset of all pairwise ratios of the form $G^*_i {G^*_j}^{-1}$. Such a problem has gained much attention recently and finds many applications across a wide range of scientific and engineering areas. In this paper, we consid… ▽ More The problem of synchronization over a group $\mathcal{G}$ aims to estimate a collection of group elements $G^*_1, \dots, G^*_n \in \mathcal{G}$ based on noisy observations of a subset of all pairwise ratios of the form $G^*_i {G^*_j}^{-1}$. Such a problem has gained much attention recently and finds many applications across a wide range of scientific and engineering areas. In this paper, we consider the class of synchronization problems in which the group is a closed subgroup of the orthogonal group. This class covers many group synchronization problems that arise in practice. Our contribution is fivefold. First, we propose a unified approach for solving this class of group synchronization problems, which consists of a suitable initialization step and an iterative refinement step based on the generalized power method, and show that it enjoys a strong theoretical guarantee on the estimation error under certain assumptions on the group, measurement graph, noise, and initialization. Second, we formulate two geometric conditions that are required by our approach and show that they hold for various practically relevant subgroups of the orthogonal group. The conditions are closely related to the error-bound geometry of the subgroup -- an important notion in optimization. Third, we verify the assumptions on the measurement graph and noise for standard random graph and random matrix models. Fourth, based on the classic notion of metric entropy, we develop and analyze a novel spectral-type estimator. Finally, we show via extensive numerical experiments that our proposed non-convex approach outperforms existing approaches in terms of computational speed, scalability, and/or estimation error. △ Less

Submitted 16 June, 2023; v1 submitted 16 September, 2020; originally announced September 2020.

A Matrix Generalization of the Hardy-Littlewood-Pólya Rearrangement Inequality and Its Applications

Authors: Man-Chung Yue

Abstract: We prove a generalization of the Hardy-Littlewood-Pólya rearrangement inequality to positive definite matrices. The inequality can be seen as a commutation principle in the sense of Iusem and Seeger. An important instrument in the proof is a first-order perturbation formula for a certain spectral function, which could be of independent interests. The inequality is then extended to rectangular matr… ▽ More We prove a generalization of the Hardy-Littlewood-Pólya rearrangement inequality to positive definite matrices. The inequality can be seen as a commutation principle in the sense of Iusem and Seeger. An important instrument in the proof is a first-order perturbation formula for a certain spectral function, which could be of independent interests. The inequality is then extended to rectangular matrices. Using our main results, we derive new inequalities for several distance-like functions encountered in various signal processing or machine learning applications. △ Less

Submitted 14 May, 2024; v1 submitted 15 June, 2020; originally announced June 2020.

On Linear Optimization over Wasserstein Balls

Authors: Man-Chung Yue, Daniel Kuhn, Wolfram Wiesemann

Abstract: Wasserstein balls, which contain all probability measures within a pre-specified Wasserstein distance to a reference measure, have recently enjoyed wide popularity in the distributionally robust optimization and machine learning communities to formulate and solve data-driven optimization problems with rigorous statistical guarantees. In this technical note we prove that the Wasserstein ball is wea… ▽ More Wasserstein balls, which contain all probability measures within a pre-specified Wasserstein distance to a reference measure, have recently enjoyed wide popularity in the distributionally robust optimization and machine learning communities to formulate and solve data-driven optimization problems with rigorous statistical guarantees. In this technical note we prove that the Wasserstein ball is weakly compact under mild conditions, and we offer necessary and sufficient conditions for the existence of optimal solutions. We also characterize the sparsity of solutions if the Wasserstein ball is centred at a discrete reference measure. In comparison with the existing literature, which has proved similar results under different conditions, our proofs are self-contained and shorter, yet mathematically rigorous, and our necessary and sufficient conditions for the existence of optimal solutions are easily verifiable in practice. △ Less

Submitted 6 June, 2021; v1 submitted 15 April, 2020; originally announced April 2020.

Guaranteed lower eigenvalue bounds for Steklov operators using conforming finite element methods

Authors: Taiga Nakano, Qin Li, Meiling Yue, Xuefeng Liu

Abstract: For the eigenvalue problem of the Steklov differential operator, by following Liu's approach, an algorithm utilizing the conforming finite element method (FEM) is proposed to provide guaranteed lower bounds for the eigenvalues. The proposed method requires the a priori error estimation for FEM solution to nonhomogeneous Neumann problems, which is solved by constructing the hypercircle for the corr… ▽ More For the eigenvalue problem of the Steklov differential operator, by following Liu's approach, an algorithm utilizing the conforming finite element method (FEM) is proposed to provide guaranteed lower bounds for the eigenvalues. The proposed method requires the a priori error estimation for FEM solution to nonhomogeneous Neumann problems, which is solved by constructing the hypercircle for the corresponding FEM spaces and boundary conditions. Numerical examples are also shown to confirm the efficiency of our proposed method. △ Less

Submitted 5 February, 2023; v1 submitted 23 January, 2020; originally announced January 2020.

Comments: 26 pages, 15 figures, 7 tables

MSC Class: 65N30; 65N25; 65L15; 65B99

Review on Set-Theoretic Methods for Safety Verification and Control of Power System

Authors: Yichen Zhang, Yan Li, Kevin Tomsovic, Seddik Djouadi, Meng Yue

Abstract: Increasing penetration of renewable energy introduces significant uncertainty into power systems. Traditional simulation-based verification methods may not be applicable due to the unknown-but-bounded feature of the uncertainty sets. Emerging set-theoretic methods have been intensively investigated to tackle this challenge. The paper comprehensively reviews these methods categorized by underlying… ▽ More Increasing penetration of renewable energy introduces significant uncertainty into power systems. Traditional simulation-based verification methods may not be applicable due to the unknown-but-bounded feature of the uncertainty sets. Emerging set-theoretic methods have been intensively investigated to tackle this challenge. The paper comprehensively reviews these methods categorized by underlying mathematical principles, that is, set operation-based methods and passivity-based methods. Set operation-based methods are more computationally efficient, while passivity-based methods provide semi-analytical expression of reachable sets, which can be readily employed for control. Other features between different methods are also discussed and illustrated by numerical examples. A benchmark example is presented and solved by different methods to verify consistency. △ Less

Submitted 21 February, 2020; v1 submitted 31 December, 2019; originally announced January 2020.

An accelerated first-order method with complexity analysis for solving cubic regularization subproblems

Authors: Rujun Jiang, Man-Chung Yue, Zhishuo Zhou

Abstract: We propose a first-order method to solve the cubic regularization subproblem (CRS) based on a novel reformulation. The reformulation is a constrained convex optimization problem whose feasible region admits an easily computable projection. Our reformulation requires computing the minimum eigenvalue of the Hessian. To avoid the expensive computation of the exact minimum eigenvalue, we develop a sur… ▽ More We propose a first-order method to solve the cubic regularization subproblem (CRS) based on a novel reformulation. The reformulation is a constrained convex optimization problem whose feasible region admits an easily computable projection. Our reformulation requires computing the minimum eigenvalue of the Hessian. To avoid the expensive computation of the exact minimum eigenvalue, we develop a surrogate problem to the reformulation where the exact minimum eigenvalue is replaced with an approximate one. We then apply first-order methods such as the Nesterov's accelerated projected gradient method (APG) and projected Barzilai-Borwein method to solve the surrogate problem. As our main theoretical contribution, we show that when an $ε$-approximate minimum eigenvalue is computed by the Lanczos method and the surrogate problem is approximately solved by APG, our approach returns an $ε$-approximate solution to CRS in $\tilde O(ε^{-1/2})$ matrix-vector multiplications (where $\tilde O(\cdot)$ hides the logarithmic factors). Numerical experiments show that our methods are comparable to and outperform the Krylov subspace method in the easy and hard cases, respectively. We further implement our methods as subproblem solvers of adaptive cubic regularization methods, and numerical results show that our algorithms are comparable to the state-of-the-art algorithms. △ Less

Submitted 1 June, 2021; v1 submitted 28 November, 2019; originally announced November 2019.

Optimistic Distributionally Robust Optimization for Nonparametric Likelihood Approximation

Authors: Viet Anh Nguyen, Soroosh Shafieezadeh-Abadeh, Man-Chung Yue, Daniel Kuhn, Wolfram Wiesemann

Abstract: The likelihood function is a fundamental component in Bayesian statistics. However, evaluating the likelihood of an observation is computationally intractable in many applications. In this paper, we propose a non-parametric approximation of the likelihood that identifies a probability measure which lies in the neighborhood of the nominal measure and that maximizes the probability of observing the… ▽ More The likelihood function is a fundamental component in Bayesian statistics. However, evaluating the likelihood of an observation is computationally intractable in many applications. In this paper, we propose a non-parametric approximation of the likelihood that identifies a probability measure which lies in the neighborhood of the nominal measure and that maximizes the probability of observing the given sample point. We show that when the neighborhood is constructed by the Kullback-Leibler divergence, by moment conditions or by the Wasserstein distance, then our \textit{optimistic likelihood} can be determined through the solution of a convex optimization problem, and it admits an analytical expression in particular cases. We also show that the posterior inference problem with our optimistic likelihood approximation enjoys strong theoretical performance guarantees, and it performs competitively in a probabilistic classification task. △ Less

Submitted 23 October, 2019; originally announced October 2019.

Calculating Optimistic Likelihoods Using (Geodesically) Convex Optimization

Authors: Viet Anh Nguyen, Soroosh Shafieezadeh-Abadeh, Man-Chung Yue, Daniel Kuhn, Wolfram Wiesemann

Abstract: A fundamental problem arising in many areas of machine learning is the evaluation of the likelihood of a given observation under different nominal distributions. Frequently, these nominal distributions are themselves estimated from data, which makes them susceptible to estimation errors. We thus propose to replace each nominal distribution with an ambiguity set containing all distributions in its… ▽ More A fundamental problem arising in many areas of machine learning is the evaluation of the likelihood of a given observation under different nominal distributions. Frequently, these nominal distributions are themselves estimated from data, which makes them susceptible to estimation errors. We thus propose to replace each nominal distribution with an ambiguity set containing all distributions in its vicinity and to evaluate an \emph{optimistic likelihood}, that is, the maximum of the likelihood over all distributions in the ambiguity set. When the proximity of distributions is quantified by the Fisher-Rao distance or the Kullback-Leibler divergence, the emerging optimistic likelihoods can be computed efficiently using either geodesic or standard convex optimization techniques. We showcase the advantages of working with optimistic likelihoods on a classification problem using synthetic as well as empirical data. △ Less

Submitted 17 October, 2019; originally announced October 2019.

Universal Barrier is $n$-Self-Concordant

Authors: Yin Tat Lee, Man-Chung Yue

Abstract: This paper shows that the self-concordance parameter of the universal barrier on any $n$-dimensional proper convex domain is upper bounded by $n$. This bound is tight and improves the previous $O(n)$ bound by Nesterov and Nemirovski. The key to our main result is a pair of new, sharp moment inequalities for $s$-concave distributions, which could be of independent interest. This paper shows that the self-concordance parameter of the universal barrier on any $n$-dimensional proper convex domain is upper bounded by $n$. This bound is tight and improves the previous $O(n)$ bound by Nesterov and Nemirovski. The key to our main result is a pair of new, sharp moment inequalities for $s$-concave distributions, which could be of independent interest. △ Less

Submitted 14 April, 2022; v1 submitted 9 September, 2018; originally announced September 2018.

Phase Retrieval via Sensor Network Localization

Authors: Sherry Xue-Ying Ni, Man-Chung Yue, Kam-Fung Cheung, Anthony Man-Cho So

Abstract: The problem of phase retrieval is revisited and studied from a fresh perspective. In particular, we establish a connection between the phase retrieval problem and the sensor network localization problem, which allows us to utilize the vast theoretical and algorithmic literature on the latter to tackle the former. Leveraging this connection, we develop a two-stage algorithm for phase retrieval that… ▽ More The problem of phase retrieval is revisited and studied from a fresh perspective. In particular, we establish a connection between the phase retrieval problem and the sensor network localization problem, which allows us to utilize the vast theoretical and algorithmic literature on the latter to tackle the former. Leveraging this connection, we develop a two-stage algorithm for phase retrieval that can provably recover the desired signal. In both sparse and dense settings, our proposed algorithm improves upon prior approaches simultaneously in the number of required measurements for recovery and the reconstruction time. We present numerical results to corroborate our theory and to demonstrate the efficiency of the proposed algorithm. As a side result, we propose a new form of phase retrieval problem and connect it to the complex rigidity theory proposed by Gortler and Thurston. △ Less

Submitted 21 March, 2018; originally announced March 2018.

On the Quadratic Convergence of the Cubic Regularization Method under a Local Error Bound Condition

Authors: Man-Chung Yue, Zirui Zhou, Anthony Man-Cho So

Abstract: In this paper we consider the cubic regularization (CR) method for minimizing a twice continuously differentiable function. While the CR method is widely recognized as a globally convergent variant of Newton's method with superior iteration complexity, existing results on its local quadratic convergence require a stringent non-degeneracy condition. We prove that under a local error bound (EB) cond… ▽ More In this paper we consider the cubic regularization (CR) method for minimizing a twice continuously differentiable function. While the CR method is widely recognized as a globally convergent variant of Newton's method with superior iteration complexity, existing results on its local quadratic convergence require a stringent non-degeneracy condition. We prove that under a local error bound (EB) condition, which is much weaker a requirement than the existing non-degeneracy condition, the sequence of iterates generated by the CR method converges at least Q-quadratically to a second-order critical point. This indicates that adding a cubic regularization not only equips Newton's method with remarkable global convergence properties but also enables it to converge quadratically even in the presence of degenerate solutions. As a byproduct, we show that without assuming convexity, the proposed EB condition is equivalent to a quadratic growth condition, which could be of independent interest. To demonstrate the usefulness and relevance of our convergence analysis, we focus on two concrete nonconvex optimization problems that arise in phase retrieval and low-rank matrix recovery, respectively, and prove that with overwhelming probability, the sequence of iterates generated by the CR method for solving these two problems converges at least Q-quadratically to a global minimizer. We also present numerical results of the CR method when applied to solve these two problems to support and complement our theoretical development. △ Less

Submitted 29 January, 2018; originally announced January 2018.

Energy Error Estimates of Subspace Method and Multigrid Algorithm for Eigenvalue Problems

Authors: Yunhui He, Qichen Hong, Hehu Xie, Meiling Yue, Chunguang You

Abstract: This paper is to give a new understanding and applications of the subspace projection method for selfadjoint eigenvalue problems. A new error estimate in the energy norm, which is induced by the stiff matrix, of the subspace projection method for eigenvalue problems is given. The relation between error estimates in $L^2$-norm and energy norm is also deduced. Based on this relation, a new type of i… ▽ More This paper is to give a new understanding and applications of the subspace projection method for selfadjoint eigenvalue problems. A new error estimate in the energy norm, which is induced by the stiff matrix, of the subspace projection method for eigenvalue problems is given. The relation between error estimates in $L^2$-norm and energy norm is also deduced. Based on this relation, a new type of inverse power method is designed for eigenvalue problems and the corresponding convergence analysis is also provided. Then we present the analysis of the geometric and algebraic multigrid methods for eigenvalue problems based on the convergence result of the new inverse power method. △ Less

Submitted 23 August, 2017; v1 submitted 8 May, 2017; originally announced May 2017.

Comments: 19 pages, 0 figure

MSC Class: 65N30; 65N25; 65L15; 65B99

A Family of Inexact SQA Methods for Non-Smooth Convex Minimization with Provable Convergence Guarantees Based on the Luo-Tseng Error Bound Property

Authors: Man-Chung Yue, Zirui Zhou, Anthony Man-Cho So

Abstract: We propose a new family of inexact sequential quadratic approximation (SQA) methods, which we call the inexact regularized proximal Newton ($\textsf{IRPN}$) method, for minimizing the sum of two closed proper convex functions, one of which is smooth and the other is possibly non-smooth. Our proposed method features strong convergence guarantees even when applied to problems with degenerate solutio… ▽ More We propose a new family of inexact sequential quadratic approximation (SQA) methods, which we call the inexact regularized proximal Newton ($\textsf{IRPN}$) method, for minimizing the sum of two closed proper convex functions, one of which is smooth and the other is possibly non-smooth. Our proposed method features strong convergence guarantees even when applied to problems with degenerate solutions while allowing the inner minimization to be solved inexactly. Specifically, we prove that when the problem possesses the so-called Luo-Tseng error bound (EB) property, $\textsf{IRPN}$ converges globally to an optimal solution, and the local convergence rate of the sequence of iterates generated by $\textsf{IRPN}$ is linear, superlinear, or even quadratic, depending on the choice of parameters of the algorithm. Prior to this work, such EB property has been extensively used to establish the linear convergence of various first-order methods. However, to the best of our knowledge, this work is the first to use the Luo-Tseng EB property to establish the superlinear convergence of SQA-type methods for non-smooth convex minimization. As a consequence of our result, $\textsf{IRPN}$ is capable of solving regularized regression or classification problems under the high-dimensional setting with provable convergence guarantees. We compare our proposed $\textsf{IRPN}$ with several empirically efficient algorithms by applying them to the $\ell_1$-regularized logistic regression problem. Experiment results show the competitiveness of our proposed method. △ Less

Submitted 26 January, 2018; v1 submitted 24 May, 2016; originally announced May 2016.

A Multigrid Method for the Ground State Solution of Bose-Einstein Condensates Based on Newton Iteration

Authors: Hehu Xie, Fei Xu, Meiling Yue

Abstract: In this paper, a new kind of multigrid method is proposed for the ground state solution of Bose-Einstein condensates based on Newton iteration method. Instead of treating eigenvalue $λ$ and eigenvector $u$ respectively, we regard the eigenpair $(λ, u)$ as one element in the composite space $\R \times H_0^1(Ω)$ and then Newton iteration method is adopted for the nonlinear problem. Thus in this mult… ▽ More In this paper, a new kind of multigrid method is proposed for the ground state solution of Bose-Einstein condensates based on Newton iteration method. Instead of treating eigenvalue $λ$ and eigenvector $u$ respectively, we regard the eigenpair $(λ, u)$ as one element in the composite space $\R \times H_0^1(Ω)$ and then Newton iteration method is adopted for the nonlinear problem. Thus in this multigrid scheme, we only need to solve a linear discrete boundary value problem in every refined space, which can improve the overall efficiency for the simulation of Bose-Einstein condensations. △ Less

Submitted 18 April, 2016; originally announced April 2016.

Comments: 17 Pages, 4 figures

MSC Class: 65N30; 65N25; 65L15; 65B99

On the Estimation Performance and Convergence Rate of the Generalized Power Method for Phase Synchronization

Authors: Huikang Liu, Man-Chung Yue, Anthony Man-Cho So

Abstract: An estimation problem of fundamental interest is that of phase synchronization, in which the goal is to recover a collection of phases using noisy measurements of relative phases. It is known that in the Gaussian noise setting, the maximum likelihood estimator (MLE) has an expected squared $\ell_2$-estimation error that is on the same order as the Cramér-Rao lower bound. Moreover, even though the… ▽ More An estimation problem of fundamental interest is that of phase synchronization, in which the goal is to recover a collection of phases using noisy measurements of relative phases. It is known that in the Gaussian noise setting, the maximum likelihood estimator (MLE) has an expected squared $\ell_2$-estimation error that is on the same order as the Cramér-Rao lower bound. Moreover, even though the MLE is an optimal solution to a non-convex quadratic optimization problem, it can be found with high probability using semidefinite programming (SDP), provided that the noise power is not too large. In this paper, we study the estimation and convergence performance of a recently-proposed low-complexity alternative to the SDP-based approach, namely, the generalized power method (GPM). Our contribution is twofold. First, we bound the rate at which the estimation error decreases in each iteration of the GPM and use this bound to show that all iterates---not just the MLE---achieve an estimation error that is on the same order as the Cramér-Rao bound. Our result holds under the least restrictive assumption on the noise power and gives the best provable bound on the estimation error known to date. It also implies that one can terminate the GPM at any iteration and still obtain an estimator that has a theoretical guarantee on its estimation error. Second, we show that under the same assumption on the noise power as that for the SDP-based method, the GPM will converge to the MLE at a linear rate with high probability. This answers a question raised in [3] and shows that the GPM is competitive in terms of both theoretical guarantees and numerical efficiency with the SDP-based method. At the heart of our convergence rate analysis is a new error bound for the non-convex quadratic optimization formulation of the phase synchronization problem, which could be of independent interest. △ Less

Submitted 1 November, 2016; v1 submitted 1 March, 2016; originally announced March 2016.

Fully Computable Error Bounds for Eigenvalue Problem

Authors: Hehu Xie, Meiling Yue, Ning Zhang

Abstract: This paper is concerned with the computable error estimates for the eigenvalue problem which is solved by the general conforming finite element methods on the general meshes. Based on the computable error estimate, we can give an asymptotically lower bound of the general eigenvalues. Furthermore, we also give a guaranteed upper bound of the error estimates for the first eigenfunction approximation… ▽ More This paper is concerned with the computable error estimates for the eigenvalue problem which is solved by the general conforming finite element methods on the general meshes. Based on the computable error estimate, we can give an asymptotically lower bound of the general eigenvalues. Furthermore, we also give a guaranteed upper bound of the error estimates for the first eigenfunction approximation and a guaranteed lower bound of the first eigenvalue based on computable error estimator. Some numerical examples are presented to validate the theoretical results deduced in this paper. △ Less

Submitted 19 June, 2016; v1 submitted 7 January, 2016; originally announced January 2016.

Comments: 21 pages, 11 figures. In this version, we obtain a sharper estimate for the eigenvalue approximations

MSC Class: 65N30; 65N25; 65L15; 65B99

The Lie conformal algebra of a Block type Lie algebra

Authors: Ming Gao Ying Xu Xiaoqing Yue

Abstract: Let $L$ be a Lie algebra of Block type over $\C$ with basis $\{L_{α,i}\,|\,α,i\in\Z\}$ and brackets $[L_{α,i},L_{β,j}]=(β(i+1)-α(j+1))L_{α+β,i+j}$. In this paper, we shall construct a formal distribution Lie algebra of $L$. Then we decide its conformal algebra $B$ with $\C[\partial]$-basis $\{L_α(w)\,|\,α\in\Z\}$ and $λ$-brackets $[L_α(w)_λL_β(w)]=(α\partial+(α+β)λ)L_{α+β}(w)$. Finally, we give a… ▽ More Let $L$ be a Lie algebra of Block type over $\C$ with basis $\{L_{α,i}\,|\,α,i\in\Z\}$ and brackets $[L_{α,i},L_{β,j}]=(β(i+1)-α(j+1))L_{α+β,i+j}$. In this paper, we shall construct a formal distribution Lie algebra of $L$. Then we decide its conformal algebra $B$ with $\C[\partial]$-basis $\{L_α(w)\,|\,α\in\Z\}$ and $λ$-brackets $[L_α(w)_λL_β(w)]=(α\partial+(α+β)λ)L_{α+β}(w)$. Finally, we give a classification of free intermediate series $B$-modules. △ Less

Submitted 23 October, 2012; originally announced October 2012.

Comments: 17

MSC Class: 17B68; 17B69

A Perturbation Inequality for the Schatten-$p$ Quasi-Norm and Its Applications to Low-Rank Matrix Recovery

Authors: Man-Chung Yue, Anthony Man-Cho So

Abstract: In this paper, we establish the following perturbation result concerning the singular values of a matrix: Let $A,B \in \mathbb{R}^{m\times n}$ be given matrices, and let $f:\mathbb{R}_+\rightarrow\mathbb{R}_+$ be a concave function satisfying $f(0)=0$. Then, we have $$ \sum_{i=1}^{\min\{m,n\}} \big| f(σ_i(A)) - f(σ_i(B)) \big| \le \sum_{i=1}^{\min\{m,n\}} f(σ_i(A-B)), $$ where $σ_i(\cdot)$ denotes… ▽ More In this paper, we establish the following perturbation result concerning the singular values of a matrix: Let $A,B \in \mathbb{R}^{m\times n}$ be given matrices, and let $f:\mathbb{R}_+\rightarrow\mathbb{R}_+$ be a concave function satisfying $f(0)=0$. Then, we have $$ \sum_{i=1}^{\min\{m,n\}} \big| f(σ_i(A)) - f(σ_i(B)) \big| \le \sum_{i=1}^{\min\{m,n\}} f(σ_i(A-B)), $$ where $σ_i(\cdot)$ denotes the $i$--th largest singular value of a matrix. This answers an open question that is of interest to both the compressive sensing and linear algebra communities. In particular, by taking $f(\cdot)=(\cdot)^p$ for any $p \in (0,1]$, we obtain a perturbation inequality for the so--called Schatten $p$--quasi--norm, which allows us to confirm the validity of a number of previously conjectured conditions for the recovery of low--rank matrices via the popular Schatten $p$--quasi--norm heuristic. We believe that our result will find further applications, especially in the study of low--rank matrix recovery. △ Less

Submitted 27 June, 2014; v1 submitted 3 September, 2012; originally announced September 2012.