Coupling Large Language Models with Logic Programming for Robust and General Reasoning from Text (2023)

Abstract: While large language models (LLMs), such as GPT-3, appear to be robust and general, their reasoning ability is not at a level to compete with the best models trained for specific natural language reasoning problems. In this study, we observe that a large language model can serve as a highly effective few-shot semantic parser. It can convert natural language sentences into a logical form that serves as input for answer set programs, a logic-based declarative knowledge representation formalism. The combination results in a robust and general system that can handle multiple question-answering tasks without requiring retraining for each new task. It only needs a few examples to guide the LLM’s adaptation to a specific task, along with reusable ASP knowledge modules that can be applied to multiple tasks. We demonstrate that this method achieves state-of-the-art performance on several NLP benchmarks, including bAbI, StepGame, CLUTRR, and gSCAN. Additionally, it successfully tackles robot planning tasks that an LLM alone fails to solve.

## introduction a typical way to handle a question-answering task is to train a neural network model on large training data and test it on similar data. such models work well with linguistic variability and ambiguity but often learn statistical features and correlations rather than true reasoning @xcite , which makes them not robust, lack generalization, and difficult to interpret. alternatively, transformer-based large language models (llms) have recently shown wide success on many downstream tasks, demonstrating general reasoning capability on diverse tasks without being retrained. however, when we restrict our attention to individual nlp reasoning benchmarks, they usually do not perform as well as state-of-the-art models despite various efforts to improve accuracy through prompt engineering @xcite . similarly, llms gained attention for plan generation for robots due to the rich semantic knowledge they acquired about the world @xcite @xcite . however, llms are known to perform shallow reasoning and cannot find complex plans @xcite . in another context, @xcite note that while llms are good at system-1 thinking, their outputs are often inconsistent and incoherent. this is because llms are trained to predict subsequent words in a sequence and do not appear to have a deep understanding of concepts such as cause and effect, logic, and probability, which are important for reasoning. nevertheless, we note that the rich semantic knowledge that llms possess makes them effective general-purpose few-shot semantic parsers that can convert linguistically variable natural language sentences into atomic facts that serve as input to logic programs. we also note that the fully declarative nature of answer set programs @xcite makes them a good pair with the llm semantic parsers, providing interpretable and explainable reasoning on the parsed result of the llms using background knowledge. combining large language models and answer set programs leads to an attractive dual-process, neuro-symbolic reasoning that works across multiple qa tasks without retraining for individual tasks. we tested this idea with several nlp benchmarks, babi @xcite , stepgame @xcite , clutrr @xcite , and gscan @xcite , by applying the same dual-system model and achieved state-of-the-art performance in all of them. furthermore, the high accuracy and transparency allow us to easily identify the source of errors, making our system a useful data set validation tool as well. in particular, we found a significant amount of errors in the original clutrr dataset that are hard to detect manually. while the new version of @xcite ) (text-davinci-003) shows improvement over its predecessors, we observe that it also retains critical limitations. in the process, we develop prompt methods for semantic parsing to overcome some of them. the implementation of our method is publicly available online at https://github.com/ azreasoners/llm-asp . ## our method we refer to our framework as [llm]+asp where [llm] denotes a large pre-trained network such as gpt-3, which we use as a semantic parser to generate input to the asp reasoner. specifically, we assume data instances of the form ⟨s, q, a⟩, where s is a context story in natural language, q is a natural language query associated with s, and a is the answer. we use an llm to convert a problem description (that is, context s and query q) into atomic facts, which are then fed into the asp solver along with background knowledge encoded as asp rules. the output of the asp solver is interpreted as the prediction for the given data instance. figure 1 illustrates the inference flow in the context of stepgame. the pipeline is simple but general enough to bke applied to various tasks without the need for retraining. it only requires replacing the few-shot prompts to the llm and the asp background knowledge with those suitable for the new tasks. by combining llms and asp in this manner, we enable robust symbolic reasoning that can handle diverse and unprocessed textual input. the asp knowledge modules remain unaffected by the diverse forms of input text that express the same facts. our method does not rely on training datasets. instead, a few examples that turn natural language sentences into atomic facts are sufficient to build a semantic parser due to the learned representations in llms. furthermore, asp knowledge modules can be reused for different tasks. ## experiments we apply the method in the previous section to four datasets. 2 as a reminder, our approach involves few-shot in-context learning and does not require training. we use the same pipeline as shown in figure 1 , but with different prompts and knowledge modules for each dataset. for more detailed information about the experimental settings, please refer to the appendix. ## conclusion symbolic logic programming was previously considered limited in its ability to reason from text due 5193 to its inability to handle various and ambiguous linguistic expressions. however, combining it with a large language model that has learned distributed representations helps alleviate this problem. the method not only achieves high accuracy but also produces interpretable results, as the source of the errors can be identified. it is also general; by using pre-trained networks with few-shot prompts and reusable knowledge modules, adapting to a new domain does not require extensive training. the knowledge modules used in our experiments are reusable. for the above experiments, the modules are relatively simple to write, as are the prompts for parsing natural language for llms. however, acquiring this kind of knowledge on a massive scale is also an important line of research @xcite @xcite that needs to be combined. in addition, it is possible to use llm's code generation capability @xcite to generate logic program rules, which we leave for future work. one may think that the logic rules are too rigid. however, there are many weighted or probabilistic rules that can be defeated @xcite @xcite . they could be used for more realistic settings, but for the benchmark problems above, they were not needed.