-
A Critical Look At Tokenwise Reward-Guided Text Generation
Authors:
Ahmad Rashid,
Ruotian Wu,
Julia Grosse,
Agustinus Kristiadi,
Pascal Poupart
Abstract:
Large language models (LLMs) can significantly be improved by aligning to human preferences -- the so-called reinforcement learning from human feedback (RLHF). However, the cost of fine-tuning an LLM is prohibitive for many users. Due to their ability to bypass LLM finetuning, tokenwise reward-guided text generation (RGTG) methods have recently been proposed. They use a reward model trained on ful…
▽ More
Large language models (LLMs) can significantly be improved by aligning to human preferences -- the so-called reinforcement learning from human feedback (RLHF). However, the cost of fine-tuning an LLM is prohibitive for many users. Due to their ability to bypass LLM finetuning, tokenwise reward-guided text generation (RGTG) methods have recently been proposed. They use a reward model trained on full sequences to score partial sequences during a tokenwise decoding, in a bid to steer the generation towards sequences with high rewards. However, these methods have so far been only heuristically motivated and poorly analyzed. In this work, we show that reward models trained on full sequences are not compatible with scoring partial sequences. To alleviate this issue, we propose to explicitly train a Bradley-Terry reward model on partial sequences, and autoregressively sample from the implied tokenwise policy during decoding time. We study the property of this reward model and the implied policy. In particular, we show that this policy is proportional to the ratio of two distinct RLHF policies. We show that our simple approach outperforms previous RGTG methods and achieves similar performance as strong offline baselines but without large-scale LLM finetuning.
△ Less
Submitted 11 June, 2024;
originally announced June 2024.
-
Optimistic Optimization of Gaussian Process Samples
Authors:
Julia Grosse,
Cheng Zhang,
Philipp Hennig
Abstract:
Bayesian optimization is a popular formalism for global optimization, but its computational costs limit it to expensive-to-evaluate functions. A competing, computationally more efficient, global optimization framework is optimistic optimization, which exploits prior knowledge about the geometry of the search space in form of a dissimilarity function. We investigate to which degree the conceptual a…
▽ More
Bayesian optimization is a popular formalism for global optimization, but its computational costs limit it to expensive-to-evaluate functions. A competing, computationally more efficient, global optimization framework is optimistic optimization, which exploits prior knowledge about the geometry of the search space in form of a dissimilarity function. We investigate to which degree the conceptual advantages of Bayesian Optimization can be combined with the computational efficiency of optimistic optimization. By map** the kernel to a dissimilarity, we obtain an optimistic optimization algorithm for the Bayesian Optimization setting with a run-time of up to $\mathcal{O}(N \log N)$. As a high-level take-away we find that, when using stationary kernels on objectives of relatively low evaluation cost, optimistic optimization can be strongly preferable over Bayesian optimization, while for strongly coupled and parametric models, good implementations of Bayesian optimization can perform much better, even at low evaluation cost. We argue that there is a new research domain between geometric and probabilistic search, i.e. methods that run drastically faster than traditional Bayesian optimization, while retaining some of the crucial functionality of Bayesian optimization.
△ Less
Submitted 2 September, 2022;
originally announced September 2022.
-
Probabilistic DAG Search
Authors:
Julia Grosse,
Cheng Zhang,
Philipp Hennig
Abstract:
Exciting contemporary machine learning problems have recently been phrased in the classic formalism of tree search -- most famously, the game of Go. Interestingly, the state-space underlying these sequential decision-making problems often posses a more general latent structure than can be captured by a tree. In this work, we develop a probabilistic framework to exploit a search space's latent stru…
▽ More
Exciting contemporary machine learning problems have recently been phrased in the classic formalism of tree search -- most famously, the game of Go. Interestingly, the state-space underlying these sequential decision-making problems often posses a more general latent structure than can be captured by a tree. In this work, we develop a probabilistic framework to exploit a search space's latent structure and thereby share information across the search tree. The method is based on a combination of approximate inference in jointly Gaussian models for the explored part of the problem, and an abstraction for the unexplored part that imposes a reduction of complexity ad hoc. We empirically find our algorithm to compare favorably to existing non-probabilistic alternatives in Tic-Tac-Toe and a feature selection application.
△ Less
Submitted 16 June, 2021;
originally announced June 2021.
-
MathPSfrag 2: Convenient LaTeX Labels in Mathematica
Authors:
Johannes Große
Abstract:
This article introduces the next version of MathPSfrag. MathPSfrag is a Mathematica package that during export automatically replaces all expressions in a plot by corresponding LaTeX commands. The new version can also produce LaTeX independent images; e.g., PDF files for inclusion in pdfLaTeX. Moreover from these files a preview is generated and shown within Mathematica.
This article introduces the next version of MathPSfrag. MathPSfrag is a Mathematica package that during export automatically replaces all expressions in a plot by corresponding LaTeX commands. The new version can also produce LaTeX independent images; e.g., PDF files for inclusion in pdfLaTeX. Moreover from these files a preview is generated and shown within Mathematica.
△ Less
Submitted 15 January, 2008;
originally announced January 2008.
-
MathPSfrag: Creating Publication-Quality Labels in Mathematica Plots
Authors:
J. Grosse
Abstract:
This article introduces a Mathematica package providing a graphics export function that automatically replaces Mathematica expressions in a graphic by the corresponding LaTeX constructs and positions them correctly. It thus facilitates the creation of publication-quality Enscapulated PostScript (EPS) graphics.
This article introduces a Mathematica package providing a graphics export function that automatically replaces Mathematica expressions in a graphic by the corresponding LaTeX constructs and positions them correctly. It thus facilitates the creation of publication-quality Enscapulated PostScript (EPS) graphics.
△ Less
Submitted 31 October, 2005;
originally announced October 2005.
