Showing 1–20 of 20 results for author: Stonyakin, F S

Subgradient methods with variants of Polyak step-size for quasi-convex optimization with inequality constraints for analogues of sharp minima

Authors: S. M. Puchinin, E. R. Korolkov, F. S. Stonyakin, M. S. Alkousa, A. A Vyguzov

Abstract: In this paper, we consider two variants of the concept of sharp minimum for mathematical programming problems with quasiconvex objective function and inequality constraints. It investigated the problem of describing a variant of a simple subgradient method with switching along productive and non-productive steps, for which, on a class of problems with Lipschitz functions, it would be possible to g… ▽ More In this paper, we consider two variants of the concept of sharp minimum for mathematical programming problems with quasiconvex objective function and inequality constraints. It investigated the problem of describing a variant of a simple subgradient method with switching along productive and non-productive steps, for which, on a class of problems with Lipschitz functions, it would be possible to guarantee convergence with the rate of geometric progression to the set of exact solutions or its vicinity. It is important that to implement the proposed method there is no need to know the sharp minimum parameter, which is usually difficult to estimate in practice. To overcome this problem, the authors propose to use a step djustment procedure similar to that previously proposed by B.~T.~Polyak. △ Less

Submitted 28 December, 2023; v1 submitted 12 December, 2023; originally announced December 2023.

Comments: in Russian language

Adaptive Methods or Variational Inequalities with Relatively Smooth and Reletively Strongly Monotone Operators

Authors: S. S. Ablaev, F. S. Stonyakin, M. S. Alkousa, D. A. Pasechnyuk

Abstract: The article is devoted to some adaptive methods for variational inequalities with relatively smooth and relatively strongly monotone operators. Starting from the recently proposed proximal variant of the extragradient method for this class of problems, we investigate in detail the method with adaptively selected parameter values. An estimate of the convergence rate of this method is proved. The re… ▽ More The article is devoted to some adaptive methods for variational inequalities with relatively smooth and relatively strongly monotone operators. Starting from the recently proposed proximal variant of the extragradient method for this class of problems, we investigate in detail the method with adaptively selected parameter values. An estimate of the convergence rate of this method is proved. The result is generalized to a class of variational inequalities with relatively strongly monotone generalized smooth variational inequality operators. Numerical experiments have been performed for the problem of ridge regression and variational inequality associated with box-simplex games. △ Less

Submitted 1 August, 2023; originally announced August 2023.

Comments: in Russian language

Adaptive Variant of the Frank-Wolfe Algorithm for Convex Optimization Problems

Authors: G. V. Aivazian, F. S. Stonyakin, D. A. Pasechnyuk, M. S. Alkousa, A. M. Raigorodskii

Abstract: Some variant of the Frank-Wolfe method for convex optimization problems with adaptive selection of the step parameter corresponding to information about the smoothness of the objective function (the Lipschitz constant of the gradient). Theoretical estimates of the quality of the solution provided by the method are obtained in terms of adaptively selected parameters L_k. An important feature of the… ▽ More Some variant of the Frank-Wolfe method for convex optimization problems with adaptive selection of the step parameter corresponding to information about the smoothness of the objective function (the Lipschitz constant of the gradient). Theoretical estimates of the quality of the solution provided by the method are obtained in terms of adaptively selected parameters L_k. An important feature of the obtained result is the elaboration of a situation in which it is possible to guarantee, after the completion of the iteration, a reduction of the discrepancy in the function by at least 2 times. At the same time, using of adaptively selected parameters in theoretical estimates makes it possible to apply the method for both smooth and nonsmooth problems, provided that the exit criterion from the iteration is met. For smooth problems, this can be proved, and the theoretical estimates of the method are guaranteed to be optimal up to multiplication by a constant factor. Computational experiments were performed, and a comparison with two other algorithms was carried out, during which the efficiency of the algorithm was demonstrated for a number of both smooth and non-smooth problems. △ Less

Submitted 29 July, 2023; originally announced July 2023.

Comments: in Russian language

Gradient-Type Method for Optimization Problems with Polyak-Lojasiewicz Condition: Relative Inexactness in Gradient and Adaptive Parameters Setting

Authors: Sergei M. Puchinin, Fedor S. Stonyakin

Abstract: We consider minimization problems with the well-known Polya-Lojasievich condition and Lipshitz-continuous gradient. Such problem occurs in different places in machine learning and related fields. Furthermore, we assume that a gradient is available with some relative inexactness. We propose some adaptive gradient-type algorithm, where the adaptivity took place with respect to the smoothness paramet… ▽ More We consider minimization problems with the well-known Polya-Lojasievich condition and Lipshitz-continuous gradient. Such problem occurs in different places in machine learning and related fields. Furthermore, we assume that a gradient is available with some relative inexactness. We propose some adaptive gradient-type algorithm, where the adaptivity took place with respect to the smoothness parameter and the level of the gradient inexactness. The theoretical estimate of the the quality of the output point is obtained and backed up by experimental results. △ Less

Submitted 10 December, 2023; v1 submitted 26 July, 2023; originally announced July 2023.

Comments: in Russian language

Stop** Rules for Gradient Method for Saddle Point Problems with Twoside Polyak-Lojasievich Condition

Authors: A. Ya. Muratidi, F. S. Stonyakin

Abstract: The paper considers approaches to saddle point problems with a two-sided variant of the Polyak-Lojasievich condition based on the gradient method with inexact information and proposes a stop** rule based on the smallness of the norm of the inexact gradient of the external subproblem. Achieving this rule in combination with a suitable accuracy of solving the auxiliary subproblem ensures that the… ▽ More The paper considers approaches to saddle point problems with a two-sided variant of the Polyak-Lojasievich condition based on the gradient method with inexact information and proposes a stop** rule based on the smallness of the norm of the inexact gradient of the external subproblem. Achieving this rule in combination with a suitable accuracy of solving the auxiliary subproblem ensures that the quality of the original saddle point problem is acceptable. The results of numerical experiments for various saddle point problems are discussed to illustrate the effectiveness of the proposed method, including the comparison with proven convergence rate estimates. △ Less

Submitted 26 July, 2023; v1 submitted 19 July, 2023; originally announced July 2023.

Comments: in Russian language

Subgradient methods for non-smooth optimization problems with some relaxation of sharp minimum

Authors: S. S. Ablaev, D. V. Makarenko, F. S. Stonyakin, M. S. Alkousa, I. V. Baran

Abstract: In this paper we propose a generalized condition for a sharp minimum, somewhat similar to the inexact oracle proposed recently by Devolder-Glineur-Nesterov. The proposed approach makes it possible to extend the class of applicability of subgradient methods with the Polyak step-size, to the situation of inexact information about the value of the minimum, as well as the unknown Lipschitz constant of… ▽ More In this paper we propose a generalized condition for a sharp minimum, somewhat similar to the inexact oracle proposed recently by Devolder-Glineur-Nesterov. The proposed approach makes it possible to extend the class of applicability of subgradient methods with the Polyak step-size, to the situation of inexact information about the value of the minimum, as well as the unknown Lipschitz constant of the objective function. Moreover, the use of local analogs of the global characteristics of the objective function makes it possible to apply the results of this type to wider classes of problems. We show the possibility of applying the proposed approach to strongly convex non-smooth problems, also, we make an experimental comparison with the known optimal subgradient method for such a class of problems. Moreover, there were obtained some results connected to the applicability of the proposed technique to some types of problems with convexity relaxations: the recently proposed notion of weak $β$-quasi-convexity and ordinary quasi-convexity. Also in the paper, we study a generalization of the described technique to the situation with the assumption that the $δ$-subgradient of the objective function is available instead of the usual subgradient. For one of the considered methods, conditions are found under which, in practice, it is possible to escape the projection of the considered iterative sequence onto the feasible set of the problem. △ Less

Submitted 12 December, 2022; originally announced December 2022.

Comments: in Russian language

Gradient-Type Methods for Optimization Problems with Polyak-Łojasiewicz Condition: Early Stop** and Adaptivity to Inexactness Parameter

Authors: Ilya A. Kuruzov, Fedor S. Stonyakin, Mohammad S. Alkousa

Abstract: Due to its applications in many different places in machine learning and other connected engineering applications, the problem of minimization of a smooth function that satisfies the Polyak-Łojasiewicz condition receives much attention from researchers. Recently, for this problem, the authors of recent work proposed an adaptive gradient-type method using an inexact gradient. The adaptivity took pl… ▽ More Due to its applications in many different places in machine learning and other connected engineering applications, the problem of minimization of a smooth function that satisfies the Polyak-Łojasiewicz condition receives much attention from researchers. Recently, for this problem, the authors of recent work proposed an adaptive gradient-type method using an inexact gradient. The adaptivity took place only with respect to the Lipschitz constant of the gradient. In this paper, for problems with the Polyak-Łojasiewicz condition, we propose a full adaptive algorithm, which means that the adaptivity takes place with respect to the Lipschitz constant of the gradient and the level of the noise in the gradient. We provide a detailed analysis of the convergence of the proposed algorithm and an estimation of the distance from the starting point to the output point of the algorithm. Numerical experiments and comparisons are presented to illustrate the advantages of the proposed algorithm in some examples. △ Less

Submitted 8 December, 2022; originally announced December 2022.

Some Adaptive First-order Methods for Variational Inequalities with Relatively Strongly Monotone Operators and Generalized Smoothness

Authors: A. A. Titov, S. S. Ablaev, M. S. Alkousa, F. S. Stonyakin, A. V. Gasnikov

Abstract: In this paper, we introduce some adaptive methods for solving variational inequalities with relatively strongly monotone operators. Firstly, we focus on the modification of the recently proposed, in smooth case [1], adaptive numerical method for generalized smooth (with Hölder condition) saddle point problem, which has convergence rate estimates similar to accelerated methods. We provide the motiv… ▽ More In this paper, we introduce some adaptive methods for solving variational inequalities with relatively strongly monotone operators. Firstly, we focus on the modification of the recently proposed, in smooth case [1], adaptive numerical method for generalized smooth (with Hölder condition) saddle point problem, which has convergence rate estimates similar to accelerated methods. We provide the motivation for such an approach and obtain theoretical results of the proposed method. Our second focus is the adaptation of widespread recently proposed methods for solving variational inequalities with relatively strongly monotone operators. The key idea in our approach is the refusal of the well-known restart technique, which in some cases causes difficulties in implementing such algorithms for applied problems. Nevertheless, our algorithms show a comparable rate of convergence with respect to algorithms based on the above-mentioned restart technique. Also, we present some numerical experiments, which demonstrate the effectiveness of the proposed methods. [1] **, Y., Sidford, A., & Tian, K. (2022). Sharper rates for separable minimax and finite sum optimization via primal-dual extragradient methods. arXiv preprint arXiv:2202.04640. △ Less

Submitted 28 October, 2022; v1 submitted 19 July, 2022; originally announced July 2022.

Stop** Rules for Gradient Methods for Non-Convex Problems with Additive Noise in Gradient

Authors: Boris T. Polyak, Ilia A. Kuruzov, Fedor S. Stonyakin

Abstract: We study the gradient method under the assumption that an additively inexact gradient is available for, generally speaking, non-convex problems. The non-convexity of the objective function, as well as the use of an inexactness specified gradient at iterations, can lead to various problems. For example, the trajectory of the gradient method may be far enough away from the starting point. On the oth… ▽ More We study the gradient method under the assumption that an additively inexact gradient is available for, generally speaking, non-convex problems. The non-convexity of the objective function, as well as the use of an inexactness specified gradient at iterations, can lead to various problems. For example, the trajectory of the gradient method may be far enough away from the starting point. On the other hand, the unbounded removal of the trajectory of the gradient method in the presence of noise can lead to the removal of the trajectory of the method from the desired exact solution. The results of investigating the behavior of the trajectory of the gradient method are obtained under the assumption of the inexactness of the gradient and the condition of gradient dominance. It is well known that such a condition is valid for many important non-convex problems. Moreover, it leads to good complexity guarantees for the gradient method. A rule of early stop** of the gradient method is proposed. Firstly, it guarantees achieving an acceptable quality of the exit point of the method in terms of the function. Secondly, the stop** rule ensures a fairly moderate distance of this point from the chosen initial position. In addition to the gradient method with a constant step, its variant with adaptive step size is also investigated in detail, which makes it possible to apply the developed technique in the case of an unknown Lipschitz constant for the gradient. Some computational experiments have been carried out which demonstrate effectiveness of the proposed stop** rule for the investigated gradient methods. △ Less

Submitted 11 December, 2022; v1 submitted 16 May, 2022; originally announced May 2022.

Adaptation to Inexactness for some Gradient-type Methods

Authors: Fedor S. Stonyakin

Abstract: We introduce a notion of inexact model of a convex objective function, which allows for errors both in the function and in its gradient. For this situation, a gradient method with an adaptive adjustment of some parameters of the model is proposed and an estimate for the convergence rate is found. This estimate is optimal on a class of sufficiently smooth problems in the presence of errors. We cons… ▽ More We introduce a notion of inexact model of a convex objective function, which allows for errors both in the function and in its gradient. For this situation, a gradient method with an adaptive adjustment of some parameters of the model is proposed and an estimate for the convergence rate is found. This estimate is optimal on a class of sufficiently smooth problems in the presence of errors. We consider a special class of convex nonsmooth optimization problems. In order to apply the proposed technique to this class, an artificial error should be introduced. We show that the method can be modified for such problems to guarantee a convergence in the function with a nearly optimal rate on the class of convex nonsmooth optimization problems. An adaptive gradient method is proposed for objective functions with some relaxation of the Lipschitz condition for the gradient that satisfy the Polyak--Lojasievicz gradient dominance condition. Here, the objective function and its gradient can be given inexactly. The adaptive choice of the parameters is performed during the operation of the method with respect to both the Lipschitz constant of the gradient and a value corresponding to the error of the gradient and the objective function. The linear convergence of the method is justified up to a value associated with the errors. △ Less

Submitted 11 October, 2021; originally announced October 2021.

Comments: in Russian

Some Methods for Relatively Strongly Monotone Variational Inequalities

Authors: F. S. Stonyakin, A. A. Titov, D. V. Makarenko, M. S. Alkousa

Abstract: The article is devoted to the development of numerical methods for solving variational inequalities with relatively strongly monotone operators. We consider two classes of variational inequalities related to some analogs of the Lipschitz condition of the operator that appeared several years ago. One of these classes is associated with the relative boundedness of the operator, and the other one wit… ▽ More The article is devoted to the development of numerical methods for solving variational inequalities with relatively strongly monotone operators. We consider two classes of variational inequalities related to some analogs of the Lipschitz condition of the operator that appeared several years ago. One of these classes is associated with the relative boundedness of the operator, and the other one with the analog of the Lipschitz condition (namely, relative smoothness). For variational inequalities with relatively bounded and relatively strongly monotone operators, we introduce a modification of the Mirror Descent method, which optimizes the convergence rate. We also propose the adaptive Proximal Pirror algorithm and its restarted version with a linear convergence rate for problems with relatively smooth and relatively strongly monotone operators. △ Less

Submitted 24 May, 2022; v1 submitted 7 September, 2021; originally announced September 2021.

Comments: in Russian

Adaptive Gradient-type Methods for Convex Optimization Problems with Relative Accuracy and Sharp Minimum

Authors: Fedor S. Stonyakin, Seydamet S. Ablaev, Inna V. Baran

Abstract: In this paper, we consider gradient-type methods for convex positively homogeneous optimization problems with relative accuracy. An analogue of the accelerated universal gradient-type method for positively homogeneous optimization problems with relative accuracy is investigated. The second approach is related to subgradient methods with B. T. Polyak stepsize. Result on the linear convergence rate… ▽ More In this paper, we consider gradient-type methods for convex positively homogeneous optimization problems with relative accuracy. An analogue of the accelerated universal gradient-type method for positively homogeneous optimization problems with relative accuracy is investigated. The second approach is related to subgradient methods with B. T. Polyak stepsize. Result on the linear convergence rate for some methods of this type with adaptive step adjustment is obtained for some class of non-smooth problems. Some generalization to a special class of non-convex non-smooth problems is also considered. △ Less

Submitted 12 December, 2021; v1 submitted 31 March, 2021; originally announced March 2021.

Comments: in Russian

Adaptive Mirror Descent Methods for Convex Programming Problems with delta-subgradients

Authors: Fedor S. Stonyakin

Abstract: We propose some adaptive mirror descent dethods for convex programming problems with delta-subgradients and prove some theoretical results. We propose some adaptive mirror descent dethods for convex programming problems with delta-subgradients and prove some theoretical results. △ Less

Submitted 23 December, 2020; originally announced December 2020.

Comments: in Russian

Some adaptive proximal method for a special class of variational inequalities and related problems

Authors: Fedor S. Stonyakin

Abstract: An adaptive proximal method for a special class of variational inequalities and related problems is proposed. For example, the so-called mixed variational inequalities and composite saddle problems are considered. Some estimates of the necessary number of iterations are obtained to achieve a given quality of the solution. An adaptive proximal method for a special class of variational inequalities and related problems is proposed. For example, the so-called mixed variational inequalities and composite saddle problems are considered. Some estimates of the necessary number of iterations are obtained to achieve a given quality of the solution. △ Less

Submitted 23 August, 2020; v1 submitted 9 January, 2019; originally announced January 2019.

Comments: in Russian

One Method for Minimization a Convex Lipschitz-Continuous Function of 2 Variables on a Fixed Square

Authors: Dmitry A. Pasechnyuk, Fedor S. Stonyakin

Abstract: In the article we have obtained some estimates of the rate of convergence for the recently proposed by Yu.E. Nesterov method of minimization of a convex Lipschitz-continuous function of two variables on a square with a fixed side. The method consists in solving auxiliary problems of one-dimensional minimization along the separating segments and does not imply the calculation of the exact value of… ▽ More In the article we have obtained some estimates of the rate of convergence for the recently proposed by Yu.E. Nesterov method of minimization of a convex Lipschitz-continuous function of two variables on a square with a fixed side. The method consists in solving auxiliary problems of one-dimensional minimization along the separating segments and does not imply the calculation of the exact value of the gradient of the objective functional. Experiments have shown that the method under consideration can achieve the desired accuracy of solving the problem in less time than the other methods (gradient descent and ellipsoid method) considered, both in the case of a known exact solution and using estimates of the convergence rate of the methods. △ Less

Submitted 13 January, 2020; v1 submitted 26 December, 2018; originally announced December 2018.

Comments: in Russian

On Some Adaptive Mirror Descent Algorithms for Convex and Strongly Convex Optimization Problems with Functional Constraints

Authors: F. S. Stonyakin, M . S. Alkousa, A. A. Titov

Abstract: In this paper some adaptive mirror descent algorithms for problems of minimization convex objective functional with several convex Lipschitz (generally, non-smooth) functional constraints are considered. It is shown that the methods are applicable to the objective functionals of various level of smoothness: the Lipschitz condition is valid either for the objective functional itself or for its grad… ▽ More In this paper some adaptive mirror descent algorithms for problems of minimization convex objective functional with several convex Lipschitz (generally, non-smooth) functional constraints are considered. It is shown that the methods are applicable to the objective functionals of various level of smoothness: the Lipschitz condition is valid either for the objective functional itself or for its gradient or Hessian (and the functional may not satisfy the Lipschitz condition). By using the restart technique methods for strongly convex minimization problems are proposed. Estimates of the rate of convergence of the considered algorithms are obtained depending on the level of smoothness of the objective functional. Numerical experiments illustrating the advantages of the proposed methods for some examples are presented. △ Less

Submitted 18 December, 2018; originally announced December 2018.

Comments: in Russian

Some Analogue of Quadratic Interpolation for a Special Class of Non-Smooth Functionals and One Application to Adaptive Mirror Descent for Constrained Optimization Problems

Authors: Fedor S. Stonyakin

Abstract: Theoretical estimates of the convergence rate of many well-known gradient-type optimization methods are based on quadratic interpolation, provided that the Lipschitz condition for the gradient is satisfied. In this article we obtain a possibility of constructing an analogue of such interpolation in the class of locally Lipschitz quasi-convex functionals with the special conditions of non-smoothnes… ▽ More Theoretical estimates of the convergence rate of many well-known gradient-type optimization methods are based on quadratic interpolation, provided that the Lipschitz condition for the gradient is satisfied. In this article we obtain a possibility of constructing an analogue of such interpolation in the class of locally Lipschitz quasi-convex functionals with the special conditions of non-smoothness (Lipshitz-continuous subgradient) introduced in this paper. As an application, estimates are obtained for the rate of convergence of the previously proposed adaptive mirror descent method for the problems of minimizing a quasi-convex locally Lipschitz functional with several convex functional constraints. △ Less

Submitted 16 December, 2018; v1 submitted 11 December, 2018; originally announced December 2018.

Comments: in Russian

Adaptive algorithms for mirror descent in convex programming problems with Lipschitz constraints

Authors: Fedor S. Stonyakin, Mohammad S. Alkousa, Alexey N. Stepanov, Maxim A. Barinov

Abstract: The paper is devoted to new modifications of recently proposed adaptive methods of Mirror Descent for convex minimization problems in the case of several convex functional constraints. Methods for problems of two classes are considered. The first type of problems with Lipschitz-continuous objective (generally speaking, nonsmooth) functional. The second one is for problems with a Lipschitz-continuo… ▽ More The paper is devoted to new modifications of recently proposed adaptive methods of Mirror Descent for convex minimization problems in the case of several convex functional constraints. Methods for problems of two classes are considered. The first type of problems with Lipschitz-continuous objective (generally speaking, nonsmooth) functional. The second one is for problems with a Lipschitz-continuous gradient of the objective smooth functional. We consider the class of problems with a non-smooth objective functional equal to the maximum of smooth functionals with a Lipschitz-continuous gradient. Note that functional constraints, generally speaking, are non-smooth and Lipschitz-contionuous. The proposed modifications allow saving the algorithm running time due to consideration of not all functional constraints on non-productive steps. Estimates for the rate of convergence of the methods under consideration are obtained. The methods proposed are optimal from the point of view of lower oracle estimates. The results of numerical experiments illustrating the advantages of the proposed procedure for some examples are given. △ Less

Submitted 27 May, 2018; originally announced May 2018.

Comments: in Russian

Some adaptive analog of Yu. E. Nesterov's method for variational inequalities with a strongly monotone operator

Authors: Fedor S. Stonyakin

Abstract: An adaptive analogue of the Yu. E. Nesterov method for variational inequalities with a strongly monotone operator is proposed. Some estimates are obtained for the parameters determining the quality of the solution of the variational inequality depending on the number of iterations. An adaptive analogue of the Yu. E. Nesterov method for variational inequalities with a strongly monotone operator is proposed. Some estimates are obtained for the parameters determining the quality of the solution of the variational inequality depending on the number of iterations. △ Less

Submitted 16 December, 2018; v1 submitted 11 March, 2018; originally announced March 2018.

Comments: in Russian

One Mirror Descent Algorithm for Convex Constrained Optimization Problems with non-standard growth properties

Authors: Fedor S. Stonyakin, Alexander A. Titov

Abstract: The paper is devoted to a special Mirror Descent algorithm for problems of convex minimization with functional constraints. The objective function may not satisfy the Lipschitz condition, but it must necessarily have the Lipshitz-continuous gradient. We assume, that the functional constraint can be non-smooth, but satisfying the Lipschitz condition. In particular, such functionals appear in the we… ▽ More The paper is devoted to a special Mirror Descent algorithm for problems of convex minimization with functional constraints. The objective function may not satisfy the Lipschitz condition, but it must necessarily have the Lipshitz-continuous gradient. We assume, that the functional constraint can be non-smooth, but satisfying the Lipschitz condition. In particular, such functionals appear in the well-known Truss Topology Design problem. Also we have applied the technique of restarts in the mentioned version of Mirror Descent for strongly convex problems. Some estimations for a rate of convergence are investigated for considered Mirror Descent algorithms. △ Less

Submitted 15 April, 2018; v1 submitted 4 March, 2018; originally announced March 2018.

Comments: 12 pages