Understanding the stochastic dynamics of sequential decision-making processes: A path-integral analysis of multi-armed bandits
Authors:
Bo Li,
Chi Ho Yeung
Abstract:
The multi-armed bandit (MAB) model is one of the most classical models to study decision-making in an uncertain environment. In this model, a player chooses one of $K$ possible arms of a bandit machine to play at each time step, where the corresponding arm returns a random reward to the player, potentially from a specific unknown distribution. The target of the player is to collect as many rewards…
▽ More
The multi-armed bandit (MAB) model is one of the most classical models to study decision-making in an uncertain environment. In this model, a player chooses one of $K$ possible arms of a bandit machine to play at each time step, where the corresponding arm returns a random reward to the player, potentially from a specific unknown distribution. The target of the player is to collect as many rewards as possible during the process. Despite its simplicity, the MAB model offers an excellent playground for studying the trade-off between exploration versus exploitation and designing effective algorithms for sequential decision-making under uncertainty. Although many asymptotically optimal algorithms have been established, the finite-time behaviors of the stochastic dynamics of the MAB model appear much more challenging to analyze, due to the intertwine between the decision-making and the rewards being collected. In this paper, we employ techniques in statistical physics to analyze the MAB model, which facilitates the characterization of the distribution of cumulative regrets at a finite short time, the central quantity of interest in an MAB algorithm, as well as the intricate dynamical behaviors of the model. Our analytical results, in good agreement with simulations, point to the emergence of an interesting multimodal regret distribution, with large regrets resulting from excess exploitation of sub-optimal arms due to an initial unlucky output from the optimal one.
△ Less
Submitted 10 June, 2023; v1 submitted 11 August, 2022;
originally announced August 2022.
The reinforcing influence of recommendations on global diversification
Authors:
An Zeng,
Chi Ho Yeung,
Mingsheng Shang,
Yi-Cheng Zhang
Abstract:
Recommender systems are promising ways to filter the overabundant information in modern society. Their algorithms help individuals to explore decent items, but it is unclear how they allocate popularity among items. In this paper, we simulate successive recommendations and measure their influence on the dispersion of item popularity by Gini coefficient. Our result indicates that local diffusion an…
▽ More
Recommender systems are promising ways to filter the overabundant information in modern society. Their algorithms help individuals to explore decent items, but it is unclear how they allocate popularity among items. In this paper, we simulate successive recommendations and measure their influence on the dispersion of item popularity by Gini coefficient. Our result indicates that local diffusion and collaborative filtering reinforce the popularity of hot items, widening the popularity dispersion. On the other hand, the heat conduction algorithm increases the popularity of the niche items and generates smaller dispersion of item popularity. Simulations are compared to mean-field predictions. Our results suggest that recommender systems have reinforcing influence on global diversification.
△ Less
Submitted 1 June, 2011;
originally announced June 2011.
