Allies: Prompting Large Language Model with Beam Search (2023)

Abstract: With the advance of large language models (LLMs), the research field of LLM applications becomes more and more popular and the idea of constructing pipelines to accomplish complex tasks by stacking LLM API calls come true. However, this kind of methods face two limitations: narrow information coverage and low fault tolerance. In this work, we propose a novel method called ALLIES. Given an input query, ALLIES leverages LLMs to iteratively generate new queries related to the original query, enabling an iterative reasoning process. By iteratively refining and expanding the scope of the original query, ALLIES captures and utilizes hidden knowledge that may not be directly obtainable through retrieval. We take zero-shot open-domain question answering (ODQA) as an application scene and evaluate ALLIES on the widely-used benchmarks, such as NQ, WebQ and TriviaQA. The experimental results demonstrate that ALLIES significantly outperforms other zero-shot baselines, indicating its effectiveness in tackling those challenges. Our code is available in https://github.com/microsoft/SimXNS/tree/main/ALLIES.

## introduction with the emergence of large language models (llms) @xcite @xcite , researchers have explored their potential to generate responses, including answering queries with the in-context learning method @xcite . in that method, the models are prompted with demonstrations such as human-selected query-response pairs @xcite @xcite . in this field, open-domain question answering @xcite @xcite is an important and representative task that usually requires * this work was done during internship at msra. ## main idea the main idea of allies is an interactive and iterative process based on the widely-used search algorithm, beam search foot_0 . we use a tuple with five slots to represent a state, which is the element of a beam. each state ⟨q, q, e, r, s⟩ consists of the original query q, the set of historical query completions q, the set of historical external evidences e, the current response r, and the estimated score s according to the current state. assume the maximum search depth is d, as illustrated in figure 2 , there are four main stages of allies. ## detailed approach for odqa in this section, we present the application of al-lies in odqa, whose algorithm is illustrated in algorithm 1. there are four key functions used in allies, each serving a specific purpose. the corresponding prompts are illustrated in appendix c.