-
A Full Adagrad algorithm with O(Nd) operations
Authors:
Antoine Godichon-Baggioni,
Wei Lu,
Bruno Portier
Abstract:
A novel approach is given to overcome the computational challenges of the full-matrix Adaptive Gradient algorithm (Full AdaGrad) in stochastic optimization. By develo** a recursive method that estimates the inverse of the square root of the covariance of the gradient, alongside a streaming variant for parameter updates, the study offers efficient and practical algorithms for large-scale applicat…
▽ More
A novel approach is given to overcome the computational challenges of the full-matrix Adaptive Gradient algorithm (Full AdaGrad) in stochastic optimization. By develo** a recursive method that estimates the inverse of the square root of the covariance of the gradient, alongside a streaming variant for parameter updates, the study offers efficient and practical algorithms for large-scale applications. This innovative strategy significantly reduces the complexity and resource demands typically associated with full-matrix methods, enabling more effective optimization processes. Moreover, the convergence rates of the proposed estimators and their asymptotic efficiency are given. Their effectiveness is demonstrated through numerical studies.
△ Less
Submitted 3 May, 2024;
originally announced May 2024.
-
Online estimation of the inverse of the Hessian for stochastic optimization with application to universal stochastic Newton algorithms
Authors:
Antoine Godichon-Baggioni,
Wei Lu,
Bruno Portier
Abstract:
This paper addresses second-order stochastic optimization for estimating the minimizer of a convex function written as an expectation. A direct recursive estimation technique for the inverse Hessian matrix using a Robbins-Monro procedure is introduced. This approach enables to drastically reduces computational complexity. Above all, it allows to develop universal stochastic Newton methods and inve…
▽ More
This paper addresses second-order stochastic optimization for estimating the minimizer of a convex function written as an expectation. A direct recursive estimation technique for the inverse Hessian matrix using a Robbins-Monro procedure is introduced. This approach enables to drastically reduces computational complexity. Above all, it allows to develop universal stochastic Newton methods and investigate the asymptotic efficiency of the proposed approach. This work so expands the application scope of secondorder algorithms in stochastic optimization.
△ Less
Submitted 15 January, 2024;
originally announced January 2024.
-
A mixture of ellipsoidal densities for 3D data modelling
Authors:
Denis Brazey,
Antoine Godichon-Baggioni,
Bruno Portier
Abstract:
In this paper, we propose a new ellipsoidal mixture model. This model is based a new probability density function belonging to the family of elliptical distributions and designed to model points spread around an ellipsoidal surface. Then, we consider a mixture model based on this density, whose parameters are estimated with the help of an EM algorithm. The properties of the estimates are studied t…
▽ More
In this paper, we propose a new ellipsoidal mixture model. This model is based a new probability density function belonging to the family of elliptical distributions and designed to model points spread around an ellipsoidal surface. Then, we consider a mixture model based on this density, whose parameters are estimated with the help of an EM algorithm. The properties of the estimates are studied theoretically and empirically. The algorithm is compared to a state of the art ellipse fitting method and experimented on 3D data.
△ Less
Submitted 21 September, 2023;
originally announced September 2023.
-
Strong uniform consistency and asymptotic normality of a kernel based error density estimator in functional autoregressive models
Authors:
Nadine Hilgert,
Bruno Portier
Abstract:
Estimating the innovation probability density is an important issue in any regression analysis. This paper focuses on functional autoregressive models. A residual-based kernel estimator is proposed for the innovation density. Asymptotic properties of this estimator depend on the average prediction error of the functional autoregressive function. Sufficient conditions are studied to provide stron…
▽ More
Estimating the innovation probability density is an important issue in any regression analysis. This paper focuses on functional autoregressive models. A residual-based kernel estimator is proposed for the innovation density. Asymptotic properties of this estimator depend on the average prediction error of the functional autoregressive function. Sufficient conditions are studied to provide strong uniform consistency and asymptotic normality of the kernel density estimator.
△ Less
Submitted 6 May, 2010; v1 submitted 14 May, 2009;
originally announced May 2009.
