-
Closing Duality Gaps of SDPs through Perturbation
Authors:
Takashi Tsuchiya,
Bruno F. Lourenço,
Masakazu Muramatsu,
Takayuki Okuno
Abstract:
Let $({\bf P},{\bf D})$ be a primal-dual pair of SDPs with a nonzero finite duality gap. Under such circumstances, ${\bf P}$ and ${\bf D}$ are weakly feasible and if we perturb the problem data to recover strong feasibility, the (common) optimal value function $v$ as a function of the perturbation is not well-defined at zero (unperturbed data) since there are ``two different optimal values''…
▽ More
Let $({\bf P},{\bf D})$ be a primal-dual pair of SDPs with a nonzero finite duality gap. Under such circumstances, ${\bf P}$ and ${\bf D}$ are weakly feasible and if we perturb the problem data to recover strong feasibility, the (common) optimal value function $v$ as a function of the perturbation is not well-defined at zero (unperturbed data) since there are ``two different optimal values'' $v({\bf P})$ and $v({\bf D})$, where $v({\bf P})$ and $v({\bf D})$ are the optimal values of ${\bf P}$ and ${\bf D}$ respectively. Thus, continuity of $v$ is lost at zero though $v$ is continuous elsewhere. Nevertheless, we show that a limiting version ${v_a}$ of $v$ is a well-defined monotone decreasing continuous bijective function connecting $v({\bf P})$ and $v({\bf D})$ with domain $[0, π/2]$ under the assumption that both ${\bf P}$ and ${\bf D}$ have singularity degree one. The domain $[0,π/2]$ corresponds to directions of perturbation defined in a certain manner. Thus, ${v_a}$ ``completely fills'' the nonzero duality gap under a mild regularity condition. Our result is tight in that there exists an instance with singularity degree two for which ${v_a}$ is not continuous.
△ Less
Submitted 10 April, 2023;
originally announced April 2023.
-
Analysis of the primal-dual central path for nonlinear semidefinite optimization without the nondegeneracy condition
Authors:
Takayuki Okuno
Abstract:
We study properties of the central path underlying a nonlinear semidefinite optimization problem, called NSDP for short. The latest radical work on this topic was contributed by Yamashita and Yabe (2012): they proved that the Jacobian of a certain equation-system derived from the Karush-Kuhn-Tucker (KKT) conditions of the NSDP is nonsingular at a KKT point under the second-order sufficient conditi…
▽ More
We study properties of the central path underlying a nonlinear semidefinite optimization problem, called NSDP for short. The latest radical work on this topic was contributed by Yamashita and Yabe (2012): they proved that the Jacobian of a certain equation-system derived from the Karush-Kuhn-Tucker (KKT) conditions of the NSDP is nonsingular at a KKT point under the second-order sufficient condition (SOSC), the strict complementarity condition (SC), and the nondegeneracy condition (NC). This yields uniqueness and existence of the central path through the implicit function theorem. In this paper, we consider the following three assumptions on a KKT point: the strong SOSC, the SC, and the Mangasarian-Fromovitz constraint qualification. Under the absence of the NC, the Lagrange multiplier set is not necessarily a singleton and the nonsingularity of the above-mentioned Jacobian is no longer valid. Nonetheless, we establish that the central path exists uniquely, and moreover prove that the dual component of the path converges to the so-called analytic center of the Lagrange multiplier set. As another notable result, we clarify a region around the central path where Newton's equations relevant to primal-dual interior point methods are uniquely solvable.
△ Less
Submitted 21 February, 2024; v1 submitted 3 October, 2022;
originally announced October 2022.
-
Riemannian Levenberg-Marquardt Method with Global and Local Convergence Properties
Authors:
Sho Adachi,
Takayuki Okuno,
Akiko Takeda
Abstract:
We extend the Levenberg-Marquardt method on Euclidean spaces to Riemannian manifolds. Although a Riemannian Levenberg-Marquardt (RLM) method was produced by Peeters in 1993, to the best of our knowledge, there has been no analysis of theoretical guarantees for global and local convergence properties. As with the Euclidean LM method, how to update a specific parameter known as the dam** parameter…
▽ More
We extend the Levenberg-Marquardt method on Euclidean spaces to Riemannian manifolds. Although a Riemannian Levenberg-Marquardt (RLM) method was produced by Peeters in 1993, to the best of our knowledge, there has been no analysis of theoretical guarantees for global and local convergence properties. As with the Euclidean LM method, how to update a specific parameter known as the dam** parameter has significant effects on its performances. We propose a trust-region-like approach for determining the parameter. We evaluate the worst-case iteration complexity to reach an epsilon-stationary point, and also prove that it has desirable local convergence properties under the local error-bound condition. Finally, we demonstrate the efficiency of our proposed algorithm by numerical experiments.
△ Less
Submitted 17 July, 2023; v1 submitted 1 October, 2022;
originally announced October 2022.
-
Accelerated-gradient-based generalized Levenberg--Marquardt method with oracle complexity bound and local quadratic convergence
Authors:
Naoki Marumo,
Takayuki Okuno,
Akiko Takeda
Abstract:
Minimizing the sum of a convex function and a composite function appears in various fields. The generalized Levenberg--Marquardt (LM) method, also known as the prox-linear method, has been developed for such optimization problems. The method iteratively solves strongly convex subproblems with a dam** term. This study proposes a new generalized LM method for solving the problem with a smooth comp…
▽ More
Minimizing the sum of a convex function and a composite function appears in various fields. The generalized Levenberg--Marquardt (LM) method, also known as the prox-linear method, has been developed for such optimization problems. The method iteratively solves strongly convex subproblems with a dam** term. This study proposes a new generalized LM method for solving the problem with a smooth composite function. The method enjoys three theoretical guarantees: iteration complexity bound, oracle complexity bound, and local convergence under a Hölderian growth condition. The local convergence results include local quadratic convergence under the quadratic growth condition; this is the first to extend the classical result for least-squares problems to a general smooth composite function. In addition, this is the first LM method with both an oracle complexity bound and local quadratic convergence under standard assumptions. These results are achieved by carefully controlling the dam** parameter and solving the subproblems by the accelerated proximal gradient method equipped with a particular termination condition. Experimental results show that the proposed method performs well in practice for several instances, including classification with a neural network and nonnegative matrix factorization.
△ Less
Submitted 10 January, 2024; v1 submitted 25 April, 2022;
originally announced April 2022.
-
Stable Linear System Identification with Prior Knowledge by Riemannian Sequential Quadratic Optimization
Authors:
Mitsuaki Obara,
Kazuhiro Sato,
Hiroki Sakamoto,
Takayuki Okuno,
Akiko Takeda
Abstract:
We consider an identification method for a linear continuous time-invariant autonomous system from noisy state observations. In particular, we focus on the identification to satisfy the asymptotic stability of the system with some prior knowledge. To this end, we propose to model this identification problem as a Riemannian nonlinear optimization (RNLO) problem, where the stability is ensured throu…
▽ More
We consider an identification method for a linear continuous time-invariant autonomous system from noisy state observations. In particular, we focus on the identification to satisfy the asymptotic stability of the system with some prior knowledge. To this end, we propose to model this identification problem as a Riemannian nonlinear optimization (RNLO) problem, where the stability is ensured through a certain Riemannian manifold and the prior knowledge is expressed as nonlinear constraints defined on this manifold. To solve this RNLO, we apply the Riemannian sequential quadratic optimization (RSQO) that was proposed by Obara, Okuno, and Takeda (2022) most recently. RSQO performs quite well with theoretical guarantee to find a point satisfying the Karush-Kuhn-Tucker conditions of RNLO. In this paper, we demonstrate that the identification problem can be indeed solved by RSQO more effectively than competing algorithms.
△ Less
Submitted 15 September, 2023; v1 submitted 28 December, 2021;
originally announced December 2021.
-
Unified Smoothing Approach for Best Hyperparameter Selection Problem Using a Bilevel Optimization Strategy
Authors:
Jan Harold Alcantara,
Chieu Thanh Nguyen,
Takayuki Okuno,
Akiko Takeda,
Jein-Shan Chen
Abstract:
Strongly motivated from use in various fields including machine learning, the methodology of sparse optimization has been developed intensively so far. Especially, the recent advance of algorithms for solving problems with nonsmooth regularizers is remarkable. However, those algorithms suppose that weight parameters of regularizers, called hyperparameters hereafter, are pre-fixed, and it is a cruc…
▽ More
Strongly motivated from use in various fields including machine learning, the methodology of sparse optimization has been developed intensively so far. Especially, the recent advance of algorithms for solving problems with nonsmooth regularizers is remarkable. However, those algorithms suppose that weight parameters of regularizers, called hyperparameters hereafter, are pre-fixed, and it is a crucial matter how the best hyperparameter should be selected. In this paper, we focus on the hyperparameter selection of regularizers related to the $\ell_p$ function with $0<p\le 1$ and apply a bilevel programming strategy, wherein we need to solve a bilevel problem, whose lower-level problem is nonsmooth, possibly nonconvex and non-Lipschitz. Recently, for solving a bilevel problem for hyperparameter selection of the pure $\ell_p\ (0<p \le 1)$ regularizer Okuno et al. discovered new necessary optimality conditions, called SB(scaled bilevel)-KKT conditions, and further proposed a smoothing-type algorithm using a certain smoothing function. However, this optimality measure is loose in the sense that there could be many points that satisfy the SB-KKT conditions. In this work, we propose new bilevel KKT conditions, which are new necessary optimality conditions tighter than the ones proposed by Okuno et al. Moreover, we propose a unified smoothing approach using smoothing functions that belong to the Chen-Mangasarian class, and then prove that generated iteration points accumulate at \alert{bilevel KKT points under milder constraint qualifications. Another contribution is that our approach and analysis are applicable to a wider class of regularizers. Numerical comparisons demonstrate which smoothing functions work well for hyperparameter optimization via bilevel optimization approach.
△ Less
Submitted 19 April, 2023; v1 submitted 25 October, 2021;
originally announced October 2021.
-
Complexity analysis of interior-point methods for second-order stationary points of nonlinear semidefinite optimization problems
Authors:
Shun Arahata,
Takayuki Okuno,
Akiko Takeda
Abstract:
We propose a primal-dual interior-point method (IPM) with convergence to second-order stationary points (SOSPs) of nonlinear semidefinite optimization problems, abbreviated as NSDPs. As far as we know, the current algorithms for NSDPs only ensure convergence to first-order stationary points such as Karush-Kuhn-Tucker points, but without a worst-case iteration complexity. The proposed method genera…
▽ More
We propose a primal-dual interior-point method (IPM) with convergence to second-order stationary points (SOSPs) of nonlinear semidefinite optimization problems, abbreviated as NSDPs. As far as we know, the current algorithms for NSDPs only ensure convergence to first-order stationary points such as Karush-Kuhn-Tucker points, but without a worst-case iteration complexity. The proposed method generates a sequence approximating SOSPs while minimizing a primal-dual merit function for NSDPs by using scaled gradient directions and directions of negative curvature. Under some assumptions, the generated sequence accumulates at an SOSP with a worst-case iteration complexity. This result is also obtained for a primal IPM with a slight modification. Finally, our numerical experiments show the benefits of using directions of negative curvature in the proposed method.
△ Less
Submitted 16 June, 2023; v1 submitted 26 March, 2021;
originally announced March 2021.
-
Sequential Quadratic Optimization for Nonlinear Optimization Problems on Riemannian Manifolds
Authors:
Mitsuaki Obara,
Takayuki Okuno,
Akiko Takeda
Abstract:
We consider optimization problems on Riemannian manifolds with equality and inequality constraints, which we call Riemannian nonlinear optimization (RNLO) problems. Although they have numerous applications, the existing studies on them are limited especially in terms of algorithms. In this paper, we propose Riemannian sequential quadratic optimization (RSQO) that uses a line-search technique with…
▽ More
We consider optimization problems on Riemannian manifolds with equality and inequality constraints, which we call Riemannian nonlinear optimization (RNLO) problems. Although they have numerous applications, the existing studies on them are limited especially in terms of algorithms. In this paper, we propose Riemannian sequential quadratic optimization (RSQO) that uses a line-search technique with an ell_1 penalty function as an extension of the standard SQO algorithm for constrained nonlinear optimization problems in Euclidean spaces to Riemannian manifolds. We prove its global convergence to a Karush-Kuhn-Tucker point of the RNLO problem by means of parallel transport and the exponential map**. Furthermore, we establish its local quadratic convergence by analyzing the relationship between sequences generated by RSQO and the Riemannian Newton method. Ours is the first algorithm that has both global and local convergence properties for constrained nonlinear optimization on Riemannian manifolds. Empirical results show that RSQO finds solutions more stably and with higher accuracy compared with the existing Riemannian penalty and augmented Lagrangian methods.
△ Less
Submitted 15 June, 2021; v1 submitted 15 September, 2020;
originally announced September 2020.
-
Local convergence of primal-dual interior point methods for nonlinear semidefinite optimization using the Monteiro-Tsuchiya family of search directions
Authors:
Takayuki Okuno
Abstract:
The recent advance of algorithms for nonlinear semi-definite optimization problems, called NSDPs, is remarkable. Yamashita et al. first proposed a primal-dual interior point method (PDIPM) for solving NSDPs using the family of Monteiro-Zhang (MZ) search directions. Since then, various kinds of PDIPMs have been proposed for NSDPs, but, as far as we know, all of them are based on the MZ family. In t…
▽ More
The recent advance of algorithms for nonlinear semi-definite optimization problems, called NSDPs, is remarkable. Yamashita et al. first proposed a primal-dual interior point method (PDIPM) for solving NSDPs using the family of Monteiro-Zhang (MZ) search directions. Since then, various kinds of PDIPMs have been proposed for NSDPs, but, as far as we know, all of them are based on the MZ family. In this paper, we present a PDIPM equipped with the family of Monteiro-Tsuchiya (MT) directions, which were originally devised for solving linear semi-definite optimization problems as were the MZ family. We further prove local superlinear convergence to a Karush-Kuhn-Tucker point of the NSDP in the presence of certain general assumptions on scaling matrices, which are used in producing the MT scaling directions.
△ Less
Submitted 25 January, 2024; v1 submitted 7 September, 2020;
originally announced September 2020.
-
Majorization-Minimization-Based Levenberg--Marquardt Method for Constrained Nonlinear Least Squares
Authors:
Naoki Marumo,
Takayuki Okuno,
Akiko Takeda
Abstract:
A new Levenberg--Marquardt (LM) method for solving nonlinear least squares problems with convex constraints is described. Various versions of the LM method have been proposed, their main differences being in the choice of a dam** parameter. In this paper, we propose a new rule for updating the parameter so as to achieve both global and local convergence even under the presence of a convex constr…
▽ More
A new Levenberg--Marquardt (LM) method for solving nonlinear least squares problems with convex constraints is described. Various versions of the LM method have been proposed, their main differences being in the choice of a dam** parameter. In this paper, we propose a new rule for updating the parameter so as to achieve both global and local convergence even under the presence of a convex constraint set. The key to our results is a new perspective of the LM method from majorization-minimization methods. Specifically, we show that if the dam** parameter is set in a specific way, the objective function of the standard subproblem in LM methods becomes an upper bound on the original objective function under certain standard assumptions.
Our method solves a sequence of the subproblems approximately using an (accelerated) projected gradient method. It finds an $ε$-stationary point after $O(ε^{-2})$ computation and achieves local quadratic convergence for zero-residual problems under a local error bound condition. Numerical results on compressed sensing and matrix factorization show that our method converges faster in many cases than existing methods.
△ Less
Submitted 14 December, 2022; v1 submitted 17 April, 2020;
originally announced April 2020.
-
A Limiting Analysis on Regularization of Singular SDP and its Implication to Infeasible Interior-point Algorithms
Authors:
Takashi Tsuchiya,
Bruno F. Lourenco,
Masakazu Muramatsu,
Takayuki Okuno
Abstract:
We consider primal-dual pairs of semidefinite programs and assume that they are ill-posed, i.e., both primal and dual are either weakly feasible or weakly infeasible. Under such circumstances, strong duality may break down and the primal and dual might have a nonzero duality gap. Nevertheless, there are arbitrary small perturbations to the problem data which makes the perturbed primal-dual pair st…
▽ More
We consider primal-dual pairs of semidefinite programs and assume that they are ill-posed, i.e., both primal and dual are either weakly feasible or weakly infeasible. Under such circumstances, strong duality may break down and the primal and dual might have a nonzero duality gap. Nevertheless, there are arbitrary small perturbations to the problem data which makes the perturbed primal-dual pair strongly feasible thus zeroing the duality gap. In this paper, we conduct an asymptotic analysis of the optimal value as the perturbation is driven to zero. Specifically, we fix two positive definite matrices (typically the identity matrices), and shift the associated affine spaces of the primal and dual slightly in the direction of the two positive definite matrices possibly in a different proportion so that the perturbed problems have interior feasible solutions, and analyze the behavior of the optimal value of the perturbed problem when the perturbation is reduced to zero kee** the proportion. A key feature of our analysis is that no further assumptions such as compactness or constraint qualifications are ever made. It will be shown that the optimal value of the perturbed problem converges to a value between the primal and dual optimal values of the original problem. Finally, the analysis leads us to the relatively surprising consequence that the infeasible interior-point algorithms for SDP generates a sequence converging to a number between the primal and dual optimal values, even in the presence of a nonzero duality gap. We expect that this property might be particularly useful in solving mixed integer SDPs with infeasible interior-point methods.
△ Less
Submitted 24 October, 2022; v1 submitted 20 December, 2019;
originally announced December 2019.
-
A stabilized sequential quadratic semidefinite programming method for degenerate nonlinear semidefinite programs
Authors:
Yuya Yamakawa,
Takayuki Okuno
Abstract:
In this paper, we propose a new sequential quadratic semidefinite programming (SQSDP) method for solving degenerate nonlinear semidefinite programs (NSDPs), in which we produce iteration points by solving a sequence of stabilized quadratic semidefinite programming (QSDP) subproblems, which we derive from the minimax problem associated with the NSDP. Unlike the existing SQSDP methods, the proposed…
▽ More
In this paper, we propose a new sequential quadratic semidefinite programming (SQSDP) method for solving degenerate nonlinear semidefinite programs (NSDPs), in which we produce iteration points by solving a sequence of stabilized quadratic semidefinite programming (QSDP) subproblems, which we derive from the minimax problem associated with the NSDP. Unlike the existing SQSDP methods, the proposed one allows us to solve those QSDP subproblems inexactly, and each QSDP is feasible. One more remarkable point of the proposed method is that constraint qualifications (CQs) or boundedness of Lagrange multiplier sequences are not required in the global convergence analysis. Specifically, without assuming such conditions, we prove the global convergence to a point satisfying any of the following: the stationary conditions for the feasibility problem, the approximate-Karush-Kuhn-Tucker (AKKT) conditions, and the trace-AKKT conditions. Finally, we conduct some numerical experiments to examine the efficiency of the proposed method.
△ Less
Submitted 22 April, 2021; v1 submitted 30 September, 2019;
originally announced September 2019.
-
Extension of the LP-Newton method to SOCPs via semi-infinite representation
Authors:
Takayuki Okuno,
Mirai Tanaka
Abstract:
The LP-Newton method solves the linear programming problem (LP) by repeatedly projecting a current point onto a certain relevant polytope. In this paper, we extend the algorithmic framework of the LP-Newton method to the second-order cone programming problem (SOCP) via a linear semi-infinite programming (LSIP) reformulation of the given SOCP. In the extension, we produce a sequence by projection o…
▽ More
The LP-Newton method solves the linear programming problem (LP) by repeatedly projecting a current point onto a certain relevant polytope. In this paper, we extend the algorithmic framework of the LP-Newton method to the second-order cone programming problem (SOCP) via a linear semi-infinite programming (LSIP) reformulation of the given SOCP. In the extension, we produce a sequence by projection onto polyhedral cones constructed from LPs obtained by finitely relaxing the LSIP. We show the global convergence property of the proposed algorithm under mild assumptions, and investigate its efficiency through numerical experiments comparing the proposed approach with the primal-dual interior-point method for the SOCP.
△ Less
Submitted 3 February, 2019;
originally announced February 2019.
-
Primal-dual path following method for nonlinear semi-infinite programs with semi-definite constraints
Authors:
Takayuki Okuno,
Masao Fukushima
Abstract:
In this paper, we propose two algorithms for nonlinear semi-infinite semi-definite programs with infinitely many convex inequality constraints, called SISDP for short. A straightforward approach to the SISDP is to use classical methods for semi-infinite programs such as discretization and exchange methods and solve a sequence of (nonlinear) semi-definite programs (SDPs). However, it is often too d…
▽ More
In this paper, we propose two algorithms for nonlinear semi-infinite semi-definite programs with infinitely many convex inequality constraints, called SISDP for short. A straightforward approach to the SISDP is to use classical methods for semi-infinite programs such as discretization and exchange methods and solve a sequence of (nonlinear) semi-definite programs (SDPs). However, it is often too demanding to find exact solutions of SDPs.
Our first approach does not rely on solving SDPs but on approximately following {a path leading to a solution, which is formed on the intersection of the semi-infinite region and the interior of the semi-definite region. We show weak* convergence of this method to a Karush-Kuhn-Tucker point of the SISDP under some mild assumptions and further provide with sufficient conditions for strong convergence. Moreover, as the second method, to achieve fast local convergence, we integrate a two-step sequential quadratic programming method equipped with Monteiro-Zhang scaling technique into the first method. We particularly prove two-step superlinear convergence of the second method using Alizadeh-Hareberly-Overton-like, Nesterov-Todd, and Helmberg-Rendle-Vanderbei-Wolkowicz/Kojima-Shindoh-Hara/Monteiro scaling directions. Finally, we conduct some numerical experiments to demonstrate the efficiency of the proposed method through comparison with a discretization method that solves SDPs obtained by finite relaxation of the SISDP.
△ Less
Submitted 30 September, 2018;
originally announced October 2018.
-
An oracle-based projection and rescaling algorithm for linear semi-infinite feasibility problems and its application to SDP and SOCP
Authors:
Masakazu Muramatsu,
Tomonari Kitahara,
Bruno F. Lourenço,
Takayuki Okuno,
Takashi Tsuchiya
Abstract:
We point out that Chubanov's oracle-based algorithm for linear programming [5] can be applied almost as it is to linear semi-infinite programming (LSIP). In this note, we describe the details and prove the polynomial complexity of the algorithm based on the real computation model proposed by Blum, Shub and Smale (the BSS model) which is more suitable for floating point computation in modern comput…
▽ More
We point out that Chubanov's oracle-based algorithm for linear programming [5] can be applied almost as it is to linear semi-infinite programming (LSIP). In this note, we describe the details and prove the polynomial complexity of the algorithm based on the real computation model proposed by Blum, Shub and Smale (the BSS model) which is more suitable for floating point computation in modern computers. The adoption of the BBS model makes our description and analysis much simpler than the original one by Chubanov [5]. Then we reformulate semidefinite programming (SDP) and second-order cone programming (SOCP) into LSIP, and apply our algorithm to obtain new complexity results for computing interior feasible solutions of homogeneous SDP and SOCP.
△ Less
Submitted 27 September, 2018;
originally announced September 2018.
-
An interior point sequential quadratic programming-type method for log-determinant semi-infinite programs
Authors:
Takayuki Okuno,
Masao Fukushima
Abstract:
In this paper, we consider a nonlinear semi-infinite program that minimizes a function including a log-determinant (logdet) function over positive definite matrix constraints and infinitely many convex inequality constraints, called SIPLOG for short. The main purpose of the paper is to develop an algorithm for computing a Karush-Kuhn-Tucker (KKT) point for the SIPLOG efficiently. More specifically…
▽ More
In this paper, we consider a nonlinear semi-infinite program that minimizes a function including a log-determinant (logdet) function over positive definite matrix constraints and infinitely many convex inequality constraints, called SIPLOG for short. The main purpose of the paper is to develop an algorithm for computing a Karush-Kuhn-Tucker (KKT) point for the SIPLOG efficiently. More specifically, we propose an interior point sequential quadratic programming-type method that inexactly solves a sequence of semi-infinite quadratic programs approximating the SIPLOG. Furthermore, to generate a search direction in the dual matrix space associated with the semi-definite constraint, we solve scaled Newton equations {that yield} the family of Monteiro-Zhang directions. We prove that the proposed method weakly* converges to a KKT point under some mild assumptions. Finally, we conduct some numerical experiments to demonstrate the efficiency of the proposed method.
△ Less
Submitted 24 September, 2018;
originally announced September 2018.
-
On $\ell_p$-hyperparameter Learning via Bilevel Nonsmooth Optimization
Authors:
Takayuki Okuno,
Akiko Takeda,
Akihiro Kawana,
Motokazu Watanabe
Abstract:
We propose a bilevel optimization strategy for selecting the best hyperparameter value for the nonsmooth $\ell_p$ regularizer with $0<p\le 1$. The concerned bilevel optimization problem has a nonsmooth, possibly nonconvex, $\ell_p$-regularized problem as the lower-level problem. Despite the recent popularity of nonconvex $\ell_p$-regularizer and the usefulness of bilevel optimization for selecting…
▽ More
We propose a bilevel optimization strategy for selecting the best hyperparameter value for the nonsmooth $\ell_p$ regularizer with $0<p\le 1$. The concerned bilevel optimization problem has a nonsmooth, possibly nonconvex, $\ell_p$-regularized problem as the lower-level problem. Despite the recent popularity of nonconvex $\ell_p$-regularizer and the usefulness of bilevel optimization for selecting hyperparameters, algorithms for such bilevel problems have not been studied because of the difficulty of $\ell_p$-regularizer.
Our contribution is the proposal of the first algorithm equipped with a theoretical guarantee for finding the best hyperparameter of $\ell_p$-regularized supervised learning problems. Specifically, we propose a smoothing-type algorithm for the above mentioned bilevel optimization problems and provide a theoretical convergence guarantee for the algorithm. Indeed, since optimality conditions are not known for such bilevel optimization problems so far, new necessary optimality conditions, which are called the SB-KKT conditions, are derived and it is shown that a sequence generated by the proposed algorithm actually accumulates at a point satisfying the SB-KKT conditions under some mild assumptions. The proposed algorithm is simple and scalable as our numerical comparison to Bayesian optimization and grid search indicates.
△ Less
Submitted 20 September, 2021; v1 submitted 5 June, 2018;
originally announced June 2018.
-
A new approach for solving mixed integer DC programs using a continuous relaxation with no integrality gap and smoothing techniques
Authors:
Takayuki Okuno,
Yoshiko T. Ikebe
Abstract:
In this paper, we consider a class of mixed integer programming problems (MIPs) whose objective functions are DC functions, that is, functions representable in terms of the difference of two convex functions. These MIPs contain a very wide class of computationally difficult nonconvex MIPs since the DC functions have powerful expressability. Recently, Maehara, Marumo, and Murota provided a continuo…
▽ More
In this paper, we consider a class of mixed integer programming problems (MIPs) whose objective functions are DC functions, that is, functions representable in terms of the difference of two convex functions. These MIPs contain a very wide class of computationally difficult nonconvex MIPs since the DC functions have powerful expressability. Recently, Maehara, Marumo, and Murota provided a continuous reformulation without integrality gaps for discrete DC programs having only integral variables. They also presented a new algorithm to solve the reformulated problem. Our aim is to extend their results to MIPs and give two specific algorithms to solve them. First, we propose an algorithm based on DCA originally proposed by Pham Dinh and Le Thi, where convex MIPs are solved iteratively. Next, to handle nonsmooth functions efficiently, we incorporate a smoothing technique into the first method. We show that sequences generated by the two methods converge to stationary points under some mild assumptions.
△ Less
Submitted 2 February, 2017;
originally announced February 2017.
