Showing 1–10 of 10 results for author: Alkousa, M S

Subgradient methods with variants of Polyak step-size for quasi-convex optimization with inequality constraints for analogues of sharp minima

Authors: S. M. Puchinin, E. R. Korolkov, F. S. Stonyakin, M. S. Alkousa, A. A Vyguzov

Abstract: In this paper, we consider two variants of the concept of sharp minimum for mathematical programming problems with quasiconvex objective function and inequality constraints. It investigated the problem of describing a variant of a simple subgradient method with switching along productive and non-productive steps, for which, on a class of problems with Lipschitz functions, it would be possible to g… ▽ More In this paper, we consider two variants of the concept of sharp minimum for mathematical programming problems with quasiconvex objective function and inequality constraints. It investigated the problem of describing a variant of a simple subgradient method with switching along productive and non-productive steps, for which, on a class of problems with Lipschitz functions, it would be possible to guarantee convergence with the rate of geometric progression to the set of exact solutions or its vicinity. It is important that to implement the proposed method there is no need to know the sharp minimum parameter, which is usually difficult to estimate in practice. To overcome this problem, the authors propose to use a step djustment procedure similar to that previously proposed by B.~T.~Polyak. △ Less

Submitted 28 December, 2023; v1 submitted 12 December, 2023; originally announced December 2023.

Comments: in Russian language

Adaptive Methods or Variational Inequalities with Relatively Smooth and Reletively Strongly Monotone Operators

Authors: S. S. Ablaev, F. S. Stonyakin, M. S. Alkousa, D. A. Pasechnyuk

Abstract: The article is devoted to some adaptive methods for variational inequalities with relatively smooth and relatively strongly monotone operators. Starting from the recently proposed proximal variant of the extragradient method for this class of problems, we investigate in detail the method with adaptively selected parameter values. An estimate of the convergence rate of this method is proved. The re… ▽ More The article is devoted to some adaptive methods for variational inequalities with relatively smooth and relatively strongly monotone operators. Starting from the recently proposed proximal variant of the extragradient method for this class of problems, we investigate in detail the method with adaptively selected parameter values. An estimate of the convergence rate of this method is proved. The result is generalized to a class of variational inequalities with relatively strongly monotone generalized smooth variational inequality operators. Numerical experiments have been performed for the problem of ridge regression and variational inequality associated with box-simplex games. △ Less

Submitted 1 August, 2023; originally announced August 2023.

Comments: in Russian language

Adaptive Variant of the Frank-Wolfe Algorithm for Convex Optimization Problems

Authors: G. V. Aivazian, F. S. Stonyakin, D. A. Pasechnyuk, M. S. Alkousa, A. M. Raigorodskii

Abstract: Some variant of the Frank-Wolfe method for convex optimization problems with adaptive selection of the step parameter corresponding to information about the smoothness of the objective function (the Lipschitz constant of the gradient). Theoretical estimates of the quality of the solution provided by the method are obtained in terms of adaptively selected parameters L_k. An important feature of the… ▽ More Some variant of the Frank-Wolfe method for convex optimization problems with adaptive selection of the step parameter corresponding to information about the smoothness of the objective function (the Lipschitz constant of the gradient). Theoretical estimates of the quality of the solution provided by the method are obtained in terms of adaptively selected parameters L_k. An important feature of the obtained result is the elaboration of a situation in which it is possible to guarantee, after the completion of the iteration, a reduction of the discrepancy in the function by at least 2 times. At the same time, using of adaptively selected parameters in theoretical estimates makes it possible to apply the method for both smooth and nonsmooth problems, provided that the exit criterion from the iteration is met. For smooth problems, this can be proved, and the theoretical estimates of the method are guaranteed to be optimal up to multiplication by a constant factor. Computational experiments were performed, and a comparison with two other algorithms was carried out, during which the efficiency of the algorithm was demonstrated for a number of both smooth and non-smooth problems. △ Less

Submitted 29 July, 2023; originally announced July 2023.

Comments: in Russian language

Subgradient methods for non-smooth optimization problems with some relaxation of sharp minimum

Authors: S. S. Ablaev, D. V. Makarenko, F. S. Stonyakin, M. S. Alkousa, I. V. Baran

Abstract: In this paper we propose a generalized condition for a sharp minimum, somewhat similar to the inexact oracle proposed recently by Devolder-Glineur-Nesterov. The proposed approach makes it possible to extend the class of applicability of subgradient methods with the Polyak step-size, to the situation of inexact information about the value of the minimum, as well as the unknown Lipschitz constant of… ▽ More In this paper we propose a generalized condition for a sharp minimum, somewhat similar to the inexact oracle proposed recently by Devolder-Glineur-Nesterov. The proposed approach makes it possible to extend the class of applicability of subgradient methods with the Polyak step-size, to the situation of inexact information about the value of the minimum, as well as the unknown Lipschitz constant of the objective function. Moreover, the use of local analogs of the global characteristics of the objective function makes it possible to apply the results of this type to wider classes of problems. We show the possibility of applying the proposed approach to strongly convex non-smooth problems, also, we make an experimental comparison with the known optimal subgradient method for such a class of problems. Moreover, there were obtained some results connected to the applicability of the proposed technique to some types of problems with convexity relaxations: the recently proposed notion of weak $β$-quasi-convexity and ordinary quasi-convexity. Also in the paper, we study a generalization of the described technique to the situation with the assumption that the $δ$-subgradient of the objective function is available instead of the usual subgradient. For one of the considered methods, conditions are found under which, in practice, it is possible to escape the projection of the considered iterative sequence onto the feasible set of the problem. △ Less

Submitted 12 December, 2022; originally announced December 2022.

Comments: in Russian language

Gradient-Type Methods for Optimization Problems with Polyak-Łojasiewicz Condition: Early Stop** and Adaptivity to Inexactness Parameter

Authors: Ilya A. Kuruzov, Fedor S. Stonyakin, Mohammad S. Alkousa

Abstract: Due to its applications in many different places in machine learning and other connected engineering applications, the problem of minimization of a smooth function that satisfies the Polyak-Łojasiewicz condition receives much attention from researchers. Recently, for this problem, the authors of recent work proposed an adaptive gradient-type method using an inexact gradient. The adaptivity took pl… ▽ More Due to its applications in many different places in machine learning and other connected engineering applications, the problem of minimization of a smooth function that satisfies the Polyak-Łojasiewicz condition receives much attention from researchers. Recently, for this problem, the authors of recent work proposed an adaptive gradient-type method using an inexact gradient. The adaptivity took place only with respect to the Lipschitz constant of the gradient. In this paper, for problems with the Polyak-Łojasiewicz condition, we propose a full adaptive algorithm, which means that the adaptivity takes place with respect to the Lipschitz constant of the gradient and the level of the noise in the gradient. We provide a detailed analysis of the convergence of the proposed algorithm and an estimation of the distance from the starting point to the output point of the algorithm. Numerical experiments and comparisons are presented to illustrate the advantages of the proposed algorithm in some examples. △ Less

Submitted 8 December, 2022; originally announced December 2022.

Some Adaptive First-order Methods for Variational Inequalities with Relatively Strongly Monotone Operators and Generalized Smoothness

Authors: A. A. Titov, S. S. Ablaev, M. S. Alkousa, F. S. Stonyakin, A. V. Gasnikov

Abstract: In this paper, we introduce some adaptive methods for solving variational inequalities with relatively strongly monotone operators. Firstly, we focus on the modification of the recently proposed, in smooth case [1], adaptive numerical method for generalized smooth (with Hölder condition) saddle point problem, which has convergence rate estimates similar to accelerated methods. We provide the motiv… ▽ More In this paper, we introduce some adaptive methods for solving variational inequalities with relatively strongly monotone operators. Firstly, we focus on the modification of the recently proposed, in smooth case [1], adaptive numerical method for generalized smooth (with Hölder condition) saddle point problem, which has convergence rate estimates similar to accelerated methods. We provide the motivation for such an approach and obtain theoretical results of the proposed method. Our second focus is the adaptation of widespread recently proposed methods for solving variational inequalities with relatively strongly monotone operators. The key idea in our approach is the refusal of the well-known restart technique, which in some cases causes difficulties in implementing such algorithms for applied problems. Nevertheless, our algorithms show a comparable rate of convergence with respect to algorithms based on the above-mentioned restart technique. Also, we present some numerical experiments, which demonstrate the effectiveness of the proposed methods. [1] **, Y., Sidford, A., & Tian, K. (2022). Sharper rates for separable minimax and finite sum optimization via primal-dual extragradient methods. arXiv preprint arXiv:2202.04640. △ Less

Submitted 28 October, 2022; v1 submitted 19 July, 2022; originally announced July 2022.

Some Methods for Relatively Strongly Monotone Variational Inequalities

Authors: F. S. Stonyakin, A. A. Titov, D. V. Makarenko, M. S. Alkousa

Abstract: The article is devoted to the development of numerical methods for solving variational inequalities with relatively strongly monotone operators. We consider two classes of variational inequalities related to some analogs of the Lipschitz condition of the operator that appeared several years ago. One of these classes is associated with the relative boundedness of the operator, and the other one wit… ▽ More The article is devoted to the development of numerical methods for solving variational inequalities with relatively strongly monotone operators. We consider two classes of variational inequalities related to some analogs of the Lipschitz condition of the operator that appeared several years ago. One of these classes is associated with the relative boundedness of the operator, and the other one with the analog of the Lipschitz condition (namely, relative smoothness). For variational inequalities with relatively bounded and relatively strongly monotone operators, we introduce a modification of the Mirror Descent method, which optimizes the convergence rate. We also propose the adaptive Proximal Pirror algorithm and its restarted version with a linear convergence rate for problems with relatively smooth and relatively strongly monotone operators. △ Less

Submitted 24 May, 2022; v1 submitted 7 September, 2021; originally announced September 2021.

Comments: in Russian

On Modification of an Adaptive Stochastic Mirror Descent Algorithm for Convex Optimization Problems with Functional Constraints

Authors: Mohammad S. Alkousa

Abstract: This paper is devoted to a new modification of a recently proposed adaptive stochastic mirror descent algorithm for constrained convex optimization problems in the case of several convex functional constraints. Algorithms, standard and its proposed modification, are considered for the type of problems with non-smooth Lipschitz-continuous convex objective function and convex functional constraints.… ▽ More This paper is devoted to a new modification of a recently proposed adaptive stochastic mirror descent algorithm for constrained convex optimization problems in the case of several convex functional constraints. Algorithms, standard and its proposed modification, are considered for the type of problems with non-smooth Lipschitz-continuous convex objective function and convex functional constraints. Both algorithms, with an accuracy $\varepsilon$ of the approximate solution to the problem, are optimal in the terms of lower bounds of estimates and have the complexity $O\left( \varepsilon^{-2} \right)$. In both algorithms, the precise first-order information, which connected with (sub)gradient of the objective function and functional constraints, is replaced with its unbiased stochastic estimates. This means that in each iteration, we can still use the value of the objective function and functional constraints at the research point, but instead of their (sub)gradient, we calculate their stochastic (sub)gradient. Due to the consideration of not all functional constraints on non-productive steps, the proposed modification allows saving the running time of the algorithm. Estimates for the rate of convergence of the proposed modified algorithm is obtained. The results of numerical experiments demonstrating the advantages and the efficient of the proposed modification for some examples are also given. △ Less

Submitted 21 January, 2020; v1 submitted 20 April, 2019; originally announced April 2019.

Comments: 17 pages, 2 figures, 2 tables

On Some Adaptive Mirror Descent Algorithms for Convex and Strongly Convex Optimization Problems with Functional Constraints

Authors: F. S. Stonyakin, M . S. Alkousa, A. A. Titov

Abstract: In this paper some adaptive mirror descent algorithms for problems of minimization convex objective functional with several convex Lipschitz (generally, non-smooth) functional constraints are considered. It is shown that the methods are applicable to the objective functionals of various level of smoothness: the Lipschitz condition is valid either for the objective functional itself or for its grad… ▽ More In this paper some adaptive mirror descent algorithms for problems of minimization convex objective functional with several convex Lipschitz (generally, non-smooth) functional constraints are considered. It is shown that the methods are applicable to the objective functionals of various level of smoothness: the Lipschitz condition is valid either for the objective functional itself or for its gradient or Hessian (and the functional may not satisfy the Lipschitz condition). By using the restart technique methods for strongly convex minimization problems are proposed. Estimates of the rate of convergence of the considered algorithms are obtained depending on the level of smoothness of the objective functional. Numerical experiments illustrating the advantages of the proposed methods for some examples are presented. △ Less

Submitted 18 December, 2018; originally announced December 2018.

Comments: in Russian

Adaptive algorithms for mirror descent in convex programming problems with Lipschitz constraints

Authors: Fedor S. Stonyakin, Mohammad S. Alkousa, Alexey N. Stepanov, Maxim A. Barinov

Abstract: The paper is devoted to new modifications of recently proposed adaptive methods of Mirror Descent for convex minimization problems in the case of several convex functional constraints. Methods for problems of two classes are considered. The first type of problems with Lipschitz-continuous objective (generally speaking, nonsmooth) functional. The second one is for problems with a Lipschitz-continuo… ▽ More The paper is devoted to new modifications of recently proposed adaptive methods of Mirror Descent for convex minimization problems in the case of several convex functional constraints. Methods for problems of two classes are considered. The first type of problems with Lipschitz-continuous objective (generally speaking, nonsmooth) functional. The second one is for problems with a Lipschitz-continuous gradient of the objective smooth functional. We consider the class of problems with a non-smooth objective functional equal to the maximum of smooth functionals with a Lipschitz-continuous gradient. Note that functional constraints, generally speaking, are non-smooth and Lipschitz-contionuous. The proposed modifications allow saving the algorithm running time due to consideration of not all functional constraints on non-productive steps. Estimates for the rate of convergence of the methods under consideration are obtained. The methods proposed are optimal from the point of view of lower oracle estimates. The results of numerical experiments illustrating the advantages of the proposed procedure for some examples are given. △ Less

Submitted 27 May, 2018; originally announced May 2018.

Comments: in Russian