-
Evaluating Language Models for Mathematics through Interactions
Authors:
Katherine M. Collins,
Albert Q. Jiang,
Simon Frieder,
Lionel Wong,
Miri Zilka,
Umang Bhatt,
Thomas Lukasiewicz,
Yuhuai Wu,
Joshua B. Tenenbaum,
William Hart,
Timothy Gowers,
Wenda Li,
Adrian Weller,
Mateja Jamnik
Abstract:
There is much excitement about the opportunity to harness the power of large language models (LLMs) when building problem-solving assistants. However, the standard methodology of evaluating LLMs relies on static pairs of inputs and outputs, and is insufficient for making an informed decision about which LLMs and under which assistive settings can they be sensibly used. Static assessment fails to a…
▽ More
There is much excitement about the opportunity to harness the power of large language models (LLMs) when building problem-solving assistants. However, the standard methodology of evaluating LLMs relies on static pairs of inputs and outputs, and is insufficient for making an informed decision about which LLMs and under which assistive settings can they be sensibly used. Static assessment fails to account for the essential interactive element in LLM deployment, and therefore limits how we understand language model capabilities. We introduce CheckMate, an adaptable prototype platform for humans to interact with and evaluate LLMs. We conduct a study with CheckMate to evaluate three language models (InstructGPT, ChatGPT, and GPT-4) as assistants in proving undergraduate-level mathematics, with a mixed cohort of participants from undergraduate students to professors of mathematics. We release the resulting interaction and rating dataset, MathConverse. By analysing MathConverse, we derive a taxonomy of human behaviours and uncover that despite a generally positive correlation, there are notable instances of divergence between correctness and perceived helpfulness in LLM generations, amongst other findings. Further, we garner a more granular understanding of GPT-4 mathematical problem-solving through a series of case studies, contributed by expert mathematicians. We conclude with actionable takeaways for ML practitioners and mathematicians: models that communicate uncertainty respond well to user corrections, and are more interpretable and concise may constitute better assistants. Interactive evaluation is a promising way to navigate the capability of these models; humans should be aware of language models' algebraic fallibility and discern where they are appropriate to use.
△ Less
Submitted 5 November, 2023; v1 submitted 2 June, 2023;
originally announced June 2023.
-
Interleaved group products
Authors:
W. T. Gowers,
Emanuele Viola
Abstract:
Let $G$ be the special linear group $\mathrm{SL}(2,q)$. We show that if $(a_1,\ldots,a_t)$ and $(b_1,\ldots,b_t)$ are sampled uniformly from large subsets $A$ and $B$ of $G^t$ then their interleaved product $a_1 b_1 a_2 b_2 \cdots a_t b_t$ is nearly uniform over $G$. This extends a result of the first author, which corresponds to the independent case where $A$ and $B$ are product sets. We obtain a…
▽ More
Let $G$ be the special linear group $\mathrm{SL}(2,q)$. We show that if $(a_1,\ldots,a_t)$ and $(b_1,\ldots,b_t)$ are sampled uniformly from large subsets $A$ and $B$ of $G^t$ then their interleaved product $a_1 b_1 a_2 b_2 \cdots a_t b_t$ is nearly uniform over $G$. This extends a result of the first author, which corresponds to the independent case where $A$ and $B$ are product sets. We obtain a number of other results. For example, we show that if $X$ is a probability distribution on $G^m$ such that any two coordinates are uniform in $G^2$, then a pointwise product of $s$ independent copies of $X$ is nearly uniform in $G^m$, where $s$ depends on $m$ only. Extensions to other groups are also discussed.
We obtain closely related results in communication complexity, which is the setting where some of these questions were first asked by Miles and Viola. For example, suppose party $A_i$ of $k$ parties $A_1,\dots,A_k$ receives on its forehead a $t$-tuple $(a_{i1},\dots,a_{it})$ of elements from $G$. The parties are promised that the interleaved product $a_{11}\dots a_{k1}a_{12}\dots a_{k2}\dots a_{1t}\dots a_{kt}$ is equal either to the identity $e$ or to some other fixed element $g\in G$, and their goal is to determine which of the two the product is equal to. We show that for all fixed $k$ and all sufficiently large $t$ the communication is $Ω(t \log |G|)$, which is tight. Even for $k=2$ the previous best lower bound was $Ω(t)$. As an application, we establish the security of the leakage-resilient circuits studied by Miles and Viola in the "only computation leaks" model.
△ Less
Submitted 25 April, 2018;
originally announced April 2018.
-
A fully automatic problem solver with human-style output
Authors:
M. Ganesalingam,
W. T. Gowers
Abstract:
This paper describes a program that solves elementary mathematical problems, mostly in metric space theory, and presents solutions that are hard to distinguish from solutions that might be written by human mathematicians. The program is part of a more general project, which we also discuss.
This paper describes a program that solves elementary mathematical problems, mostly in metric space theory, and presents solutions that are hard to distinguish from solutions that might be written by human mathematicians. The program is part of a more general project, which we also discuss.
△ Less
Submitted 17 September, 2013;
originally announced September 2013.
