Smurfs: Leveraging Multiple Proficiency Agents with Context-Efficiency for Tool Planning

The tool documents about the tools that Smurfs can utilize in the completion of a complex task are denoted as $D=\{n_{i},d_{i},p_{i}\}^{|d|}_{i=1}$, where n represents the tool name, d represents tool usage description, p represents parameter description and $|d|$ represents the amount of the available tools. The available tool list is denoted as $\tau=\{n_{i},d_{i}\}^{|\tau|}_{i=1}$. $\tau_{t}$ denotes the tool list Smurfs can utilize at time t.

The memory of the agent system at time t is the history of the task-solving process before t, denoted as $M=(m_{1},m_{2},...,m_{t-1})$ and $m_{i}=(\gamma_{i},a_{i})$, where $m_{i}$ represents memory element at time i and $\gamma_{i}$, $a_{i}$ represents thought and answer generated by the system at time i. There are two types of memory in Smurfs: local memory and global memory. the local memory is used to record the ongoing solution trajectory and to generate the next action in the current trajectory. The global memory, meanwhile, records all trajectories and is used to generate the sub-problem’s answer by combining all trajectory records when the maximum number of retries is exceeded. This local-global combined memory system ensures that the planning of the current solution trajectory is not influenced by the context of erroneous trajectories. It also generates an answer that combines all trajectories when the verifier agent cannot determine task completion within the maximum number of planned steps. This memory system ensures context efficiency during the task-solving process.

The primary responsibility of the Planning Agent is doing macro-level planning through task decomposition to prevent premature termination. The inference process of the Planning Agent is:

Where $p_{i}$ represents sub-problem of the original query q, PA represents the Planning Agent. After the task decomposition, the agent system will use Executor Agent, Answer Agent an Verifier Agent to solve each sub-problem using DFSDT collaboratively in a sequential order. To utilize the answer of the previous sub-problem when solving subsequent sub-problem, the strategy known as least-to-most prompting Zhou et al. (2023) is used.

To address the instability of the rollback mechanism in DFSDT, we propose a rollback mechanism based on rules. Whenever an error occurs while using a tool $\tau_{t,i}$ at time t, the tool list at t $\tau_{t}$ will pop $\tau_{t,i}$ out and reperform tool selection and tool planning (ensuring that the faulty tool is not selected again). If, at time t, the tool list becomes empty, it signifies that after the system choosing tool $\tau_{t-1,j}$ at time t-1, no subsequent trajectory can solve the problem. In this case, the agent system will roll back to time t-1, meaning that the local memory M will pop out the memory element $m_{t-1}$ at time t-1, and the tool list at time t-1 $\tau_{t-1}$ will pop out tool $\tau_{t-1,j}$. The agent system will then set the time t=t-1 and continue planning. This rule-based rollback mechanism, compared to the original model-based rollback mechanism of DFSDT, is less flexible and might reduce rollback efficiency. However, it is more stable, ensuring the correctness of deep first search and enabling models with weaker capabilities to utilize DFSDT on tool planning.

The Executor Agent is responsible for choosing and executing the tools to solve the sub-tasks. At each time t, the agent can invoke one tool to tackle the given task:

Where p is the sub-problem from Planning Agent, h is the hint from the Verifier Agent, $\tau$ is the tool list, M is local memory, $D[\alpha]$ means the tool document of tool $\alpha$. The agent, using the ReACT format Yao et al. (2022) to choose the tool and arguments, then execute the tool. Noticed that each inference process only uses the relevant part from the local memory and tool list to reduce the context redundancy. More detailed information of the Executor Agent can be found in Figure 6.

To mitigate the performance degradation caused by lengthy contexts, we introduce the Answer Agent role, designed to extract crucial content for each step and sub-problem:

Where q is sub-problem from the Planning Agent, r is response from the Executor Agent, M is the local memory (or global memory if max retry reaches). As the ‘lost-in-the-middle’ theory described in section 2.1, retaining all information may not always be beneficial, particularly in cases where the solution path is challenging to discern. Therefore, the primary role of the Answer Agent is to succinctly summarize the generated answers and tool responses to maintain the memory efficiency.

The Verifier Agent serves as an early-stop** and reflection mechanism, allowing for a balance between effectiveness and efficiency:

Where q denotes the sub-problems from the Planning Agent, a denotes the answer from the answer agent, h and c denotes hint and check status respectively. If check status generated is 0, that means the Verifier Agent thinks the sub-problem isn’t completed, the system will add the thought and answer of this time to the local and global memory, set t=t+1 and continue the inference procedure.If check status is 1, the sub-problems is thought to be solved and the system will start to deal with the next sub-problem.

Following the original paper that introduced the benchmark, we adopt ReACT (CoT) Wei et al. (2023) and DFSDT Touvron et al. (2023) as baseline methods for comparison. Additionally, we include the backbones used in the paper: gpt-3.5-turbo-0613 (GPT-3.5 Turbo) OpenAI and gpt-4-turbo-preview (GPT-4 Turbo). To explore the adaptability of the tool-planning methods, we also include Mistral-7B-Instruct-v0.2 (Mistral-7B) Jiang et al. (2023) as one of the selected backbones in our experiments.

To minimize the influence of varying tool APIs on experimental results, we conducted all experiments using the same API cache Guo et al. (2024). For a fair comparison among the candidate methods and to reduce variability, each model was executed once and evaluated three times, with results averaged. Other settings follow those specified in the original benchmark paper.

Table 3 displays the results on StableToolBench. For the untrained LLM, Mistral-7B, existing agent frameworks did not improve its performance in tool planning tasks; Mistral-7B failed these tasks when integrated with the ReACT and DFSDT frameworks ^***Experiment results show that Mistral-7B failed to correctly execute the ‘finish’ action during inference, resulting in invalid responses.. However, Smurfs exhibited exceptional performance: when combined with Mistral-7B, Smurfs achieved competitive scores among the baselines. Through its task decomposition mechanism, Smurfs transforms long-context tasks into simpler ones, enabling the untrained model to effectively utilize external tools for managing complex tasks. Regarding closed-source models, specifically GPT4 in these experiments, Smurfs also demonstrated outstanding performance on the benchmark compared to other agent frameworks and achieved state-of-the-art results on the benchmark. Its high success rate suggests that Smurfs is more effective at finding optimal solution paths compared to ChatGPT.

We conducted a detailed analysis of the token costs associated with each tool planning method for the tasks, a critical evaluation aspect for multihop reasoning tasks. As shown in Table 1 (detailed in Appendix E), the average token costs per question and API request are evaluated for ReACT, DFSDT, and Smurfs on StableToolBench. The analysis reveals that DFSDT generally requires about 20,000 tokens per question, encompassing both prompt and completion tokens. This is nearly three times the token cost compared to ReACT and twice as much as Smurfs. Despite this higher token cost, DFSDT does not demonstrate commensurate effectiveness improvements over other methods. These findings underscore the cost-efficiency of the proposed MAS framework, Smurfs, which not only reduces token expenditure in solving multihop planning tasks but also delivers outstanding performance in evaluations.

The compared baselines include CoT Wei et al. (2023), ReActYao et al. (2022), ChameleonLu et al. (2023), Reflexion Shinn et al. (2023), BOLAA Liu et al. (2023), ReWOO Xu et al. (2023), FireAct Chen et al. (2023a), AutoActQiao et al. (2024).

Following the settings in  Qiao et al. (2024), we use open-source Llama-2 models Touvron et al. (2023) and Mistral-7B Jiang et al. (2023) as the backbones of each agent to evaluate the performance of Smurfs. The evaluation metrics is $\texttt{reward}\in[0,1]$, defined as the F1 score grading between the prediction and ground-truth answer. For more details about the experiment, see Appendix C.

Smurfs, as an untrained MAS system, not only comprehensively outperforms untrained agents but also achieves and even surpasses the accuracy of trained agents across most backbone models. This sufficiently demonstrates that the mechanism of smurfs ensures strong generalization capabilities while maintaining high effectiveness.

Observations indicate that the performance of LLama-2-13b-chat on smurfs-related tasks is suboptimal, likely due to its limited capabilities in tool arguments generation. Specifically, the primary issue identified is that, when the Executor agent successfully selects relevant tool, it tends to produce hallucination arguments that can’t be used by the tools. This indicates that LLama-2-13b-chat may need further training for usage of tools. The experimental results may substantiate this viewpoint, demonstrating that the untrained methods of llama-2-13b-chat generally exhibit significantly lower accuracy compared to the trained methods. Nevertheless, Smurfs achieves the second highest accuracy among the untrained methods, only slightly behind reflexion, which still attests to Smurfs’ capability.