Branch-Solve-Merge Improves Large Language Model
Evaluation and Generation

For a given task, BSM defines a controller as an algorithm that lays out the transition logic between the modules. Let us denote the three modules with their functional forms: $\texttt{branch}(\cdot)$, $\texttt{solve}(\cdot)$, and $\texttt{merge}(\cdot)$. Then the program is defined as $\texttt{Prog}:(x,\texttt{branch}(\cdot),\texttt{solve}(\cdot),\texttt{merge}(\cdot))\rightarrow y$, taking as input a task instance $x$, along with the module implementations and generating an output $y$.

Given a task, the branch module generates multiple sub-tasks where each sub-task is represented by a unique branch. Branching into sub-problems allows task decomposition such that each part can be solved independently in parallel, at which point the partial solutions are combined. Formally, given a task input $x$, we define a ‘branch’ prompt $\texttt{prompt}_{\texttt{branch}}(x)$ that can be wrapped around $x$ with branching instructions and some demonstrations (if available). Conditioning on the prompt, the LLM $p_{\theta}$ generates a set of $k$ sub-problems $X=\{x^{(1)},x^{(2)},\cdot\cdot\cdot,x^{(k)}\}$, where $k$ is referred to as the branching factor. The sub-problems are generated auto-regressively as a sequence of tokens: $X\sim p_{\theta}(X|\texttt{prompt}_{\texttt{branch}}(x))$. Importantly, the flexibility of our method comes from the fact that for a given problem, the LLM itself decides (generates) the sub-problems and the corresponding branching factor.

The solve module solves the task at hand by generating an output $y^{(i)}$ for a branch task input $x^{(i)}$. Similar to the branch prompt, we define a ‘solve’ prompt $\texttt{prompt}_{\texttt{solve}}(x^{(i)})$, conditioning on which the LLM generates a solution $y^{(i)}\sim p_{\theta}(y^{(i)}|\texttt{prompt}_{\texttt{solve}}(x^{(i)}))$ for each branch.

The merge module fuses the solutions to the sub-problems to generate a global solution to the main problem. This is done through a ‘merge’ prompt $\texttt{prompt}_{\texttt{merge}}(Y)$ that generates a merged solution $y\sim p_{\theta}(y|\texttt{prompt}_{\texttt{merge}}(Y))$, conditioning on a set of sub-solutions $Y=\{y^{(1)},y^{(2)},\cdot\cdot\cdot,y^{(k)}\}$. Conceptually, the merge module learns an aggregator function that could aggregate a set of values (using an aggregation operator) or fuse pieces of text, depending on the task.

Next, we motivate and conduct case studies of BSM with two challenging NLP tasks: LLM evaluation and constrained generation.

We consider the task of evaluating LLM-based chat assistants. Formally, given an open-ended question and a pair of responses from two LLM agents, the task requires producing a preference judgement of which response is better or if it is a tie (see Fig. 1). Evaluating LLM responses is challenging for many reasons:

1. 1.

   Long-form answers to arbitrary questions. With the goal of providing a general-purpose assistant, the user asks arbitrary questions from any domain, and the LLM responds with long-form answers (Zheng et al., 2023). Based on the initial model response, the user can ask follow-up questions. Depending on the type of question, the evaluation process must consider the intent of the question, what is expected from an ideal response, and what criteria to evaluate on.

2. 2.

   LLM evaluators are prone to biases. LLM-based evaluators are not reliable and are prone to different biases including (a) Position Bias: evaluation changes based on the encoding order of the responses, (b) Length Bias: tendency to favor longer responses, (c) Self-enhancement Bias: the LLM-evaluator favoring its own responses (Zheng et al., 2023).

3. 3.

   GPT-4 as evaluator is expensive. While API-based models like GPT-4 are fairly good evaluators (Zheng et al., 2023), these models are proprietary and charge users per token generated. Current open-source alternatives correlate less well with humans and are much more susceptible to the aforementioned biases.

4. 4.

   Hand-designing evaluation plans is not scalable. A robust evaluator should generalize well, capable of evaluating responses to arbitrary questions and hence, hand-designing the evaluation plan for every task is not desirable (Liu et al., 2023). For example, see Fig. 1, where evaluating responses to a ‘writing’ question requires considering factors like ‘Relevance’, ‘Clarity’, etc whereas if the question is a ‘coding’ question (see Fig. 2 in the Appendix), one should evaluate for ‘Code Correctness’, ‘Code Readability’, etc.

Hence, given the multi-faceted nature of this evaluation task, we develop a version of BSM, as described below. For this study, we focus on evaluating two-turn conversational questions although our method is generally applicable for any number of turns. Let us denote the first question as $q_{1}$ and the follow-up question as $q_{2}$. Let the responses from the two LLMs $A$ and $B$ be $r_{1}^{(A)}$ and $r_{1}^{(B)}$ for $q_{1}$, and $r_{2}^{(A)}$ and $r_{2}^{(B)}$ for $q_{2}$.

It generates an evaluation plan i.e., a set of evaluation criteria that the response will be evaluated against. The branch module only conditions on the input question and for turn-1 questions, is defined as as branch($q_{1}$), while for turn-2 questions, it conditions on both turn-1 and turn-2 questions, represented as branch($q_{1},q_{2}$). The output is a set of evaluation criteria, $\texttt{branch}(q)\rightarrow\{c_{i}\}_{i=1}^{k}$, where each $c_{i}$ is the title of the criterion (e.g., ‘Relevance’) and a short description of how to evaluate for it (e.g., ‘Assess how well the response aligns with the user’s question … and must-see attractions in Hawaii.’). See Fig. 1 and 2 for examples of generated branches for different questions.

It compares and evaluates the responses based on a specific criterion. The output of the evaluation is a pair of scores (within a specified range, according to the solving instruction, e.g., 1-5) for each of the responses. For example, given an evaluation criterion $c$, we denote the solve module for a question $q$ as: $\texttt{solve}(q,r_{1}^{(A)},r_{1}^{(B)},c)\rightarrow(s^{(A)},s^{(B)})$, where $s^{(A)}$ and $s^{(B)}$ are the evaluation scores assigned to the two assistant responses. Note that the solve module is not symmetric i.e., the encoding order of the two responses is important (and we address this below in our LLM program). The module additionally generates explanations along with the scores. Fig. 1 shows example generations from the solve module with a LLaMA-2-70B-chat model.

We develop two variants of the merge module. A simple non-neural variant sums up the scores across all branches. We also develop a neural LLM variant that conditions on the individual evaluations and generates the final verdict with a model-decided aggregation strategy, denoted as $\texttt{merge}(q,\{c_{i}\}_{i=1}^{k},\{s_{i}^{(A)}\}_{i=1}^{k},\{s_{i}^{(B)}\}_{i=1}^{k})\rightarrow y$, where the evaluation criteria $\{c_{i}\}_{i=1}^{k}$ are the outputs of the branch module and $s_{i}^{(A)}$ and $s_{i}^{(B)}$ are the criterion-wise evaluations (scores and explanations) of the two assistant responses generated from the solve module. The final verdict is $y\in\{A,B,tie\}$.

The overall LLM program pseudocode is given in Algorithm 1. To account for position bias, the program executes two independent runs of BSM by swap** the encoding order of the responses in the ‘solve’ module. The final judgment is either ‘A’ or ‘B’ if and only if the judgement is consistent for both orders, otherwise it is a ‘tie’.

Our next case study shows the general applicability of BSM by applying it to a completely different task, that of LLM generation. We consider a constrained story generation task – given a set of concepts $l$, the task is to generate a coherent story $y$ by including all concepts in it (see Fig. 3 in the Appendix). When the number of concepts is large, LLMs tend to either leave out some concepts or generate text that is incoherent. The task requires composition incorporating the various constraints.

The branch module $\texttt{branch}(l)\rightarrow(l_{1},l_{2},t)$ proposes a story generation plan, consisting of (1) two subsets of concepts $l_{1}$ and $l_{2}$ and (2) a story topic $t$. The two subsets represent sub-problems of the original task with a smaller number of concepts. The story topic ensures that all sub-stories generated as part of BSM belong to the same topic.

The solve module $\texttt{solve}(l_{i},t)\rightarrow y_{i}$ conditions on a subset of concepts $l_{i}$ and the story topic $t$ to generate a story $y_{i}$ on that topic, while also including all concepts in $l_{i}$. Intuitively, ‘solving’ the constrained generation task is easier with a smaller number of concepts.

The merge module $\texttt{merge}(y_{1},y_{2})\rightarrow y$ conditions on two intermediate stories and fuses them together to generate the final story $y$. Since both intermediate stories belong to the same high-level topic, the fusion can lead to a final coherent story. Overall, BSM ensures better constraint satisfaction by solving sub-problems and maintains coherency through a top-level plan that includes a story topic.