An adaptively inexact first-order method for bilevel optimization with application to hyperparameter learning
Authors:
Mohammad Sadegh Salehi,
Subhadip Mukherjee,
Lindon Roberts,
Matthias J. Ehrhardt
Abstract:
Various tasks in data science are modeled utilizing the variational regularization approach, where manually selecting regularization parameters presents a challenge. The difficulty gets exacerbated when employing regularizers involving a large number of hyperparameters. To overcome this challenge, bilevel learning can be employed to learn such parameters from data. However, neither exact function…
▽ More
Various tasks in data science are modeled utilizing the variational regularization approach, where manually selecting regularization parameters presents a challenge. The difficulty gets exacerbated when employing regularizers involving a large number of hyperparameters. To overcome this challenge, bilevel learning can be employed to learn such parameters from data. However, neither exact function values nor exact gradients with respect to the hyperparameters are attainable, necessitating methods that only rely on inexact evaluation of such quantities. State-of-the-art inexact gradient-based methods a priori select a sequence of the required accuracies and cannot identify an appropriate step size since the Lipschitz constant of the hypergradient is unknown. In this work, we propose an algorithm with backtracking line search that only relies on inexact function evaluations and hypergradients and show convergence to a stationary point. Furthermore, the proposed algorithm determines the required accuracy dynamically rather than manually selected before running it. Our numerical experiments demonstrate the efficiency and feasibility of our approach for hyperparameter estimation on a range of relevant problems in imaging and data science such as total variation and field of experts denoising and multinomial logistic regression. Particularly, the results show that the algorithm is robust to its own hyperparameters such as the initial accuracies and step size.
△ Less
Submitted 10 April, 2024; v1 submitted 19 August, 2023;
originally announced August 2023.
Perspective: Non-conventional Force Fields for Applications in Spectroscopy and Chemical Reaction Dynamics
Authors:
D. Koner,
M. S. Salehi,
P. Mondal,
M. Meuwly
Abstract:
Extensions and improvements of empirical force fields are discussed in view of applications to computational vibrational spectroscopy and reactive molecular dynamics simulations. Particular focus is on quantitative studies which make contact with experiments and provide complementary information for a molecular-level understanding of processes in the gas phase and in solution. Methods range from i…
▽ More
Extensions and improvements of empirical force fields are discussed in view of applications to computational vibrational spectroscopy and reactive molecular dynamics simulations. Particular focus is on quantitative studies which make contact with experiments and provide complementary information for a molecular-level understanding of processes in the gas phase and in solution. Methods range from including multipolar charge distributions to reproducing kernel Hilbert space approaches and machine learned energy functions based on neural networks.
△ Less
Submitted 1 April, 2020;
originally announced April 2020.
