ZEBRA : Zero-Shot Example-Based Retrieval Augmentation for Commonsense Question Answering (2024)

Abstract: Current Large Language Models (LLMs) have shown strong reasoning capabilities in commonsense question answering benchmarks, but the process underlying their success remains largely opaque. As a consequence, recent approaches have equipped LLMs with mechanisms for knowledge retrieval, reasoning and introspection, not only to improve their capabilities but also to enhance the interpretability of their outputs. However, these methods require additional training, hand-crafted templates or human-written explanations. To address these issues, we introduce ZEBRA, a zero-shot question answering framework that combines retrieval, case-based reasoning and introspection and dispenses with the need for additional training of the LLM. Given an input question, ZEBRA retrieves relevant question-knowledge pairs from a knowledge base and generates new knowledge by reasoning over the relationships in these pairs. This generated knowledge is then used to answer the input question, improving the model’s performance and interpretability. We evaluate our approach across 8 well-established commonsense reasoning benchmarks, demonstrating that ZEBRA consistently outperforms strong LLMs and previous knowledge integration approaches, achieving an average accuracy improvement of up to 4.5 points.

## introduction over recent years, the research community has explored how to improve the reasoning capabilities of language models and the interpretability of their predictions, with many approaches relying on knowledge augmentation @xcite @xcite . for instance, @xcite introduced casebased reasoning @xcite into the training process of a knowledge-based question answering system to guide its reasoning. at * equal contribution. figure 1 : performance benefits of using zebra against standard retrieval augmentation methods for commonsense reasoning across four large language models. the same time, @xcite and @xcite investigated how to prompt llms to generate useful knowledge via hand-crafted templates or human-written explanations, while @xcite introduced an approach to teach an llm to reason over a knowledge base of commonsense knowledge, which is accessed via a retriever. furthermore, @xcite introduced the concept of knowledge introspection, which is the process of generating contextually relevant knowledge in response to given questions. however, some of these approaches rely on commonsense knowledge bases, which are finite by definition and, therefore, may not include the exact information needed to augment the input for downstream tasks. moreover, most of the aforementioned approaches require the user to train -or, at least, fine-tune -existing llms; not only is this process resource-intensive but it also leads to taskspecific models, which may result in sub-optimal performance outside the domain of their training datasets. to address these issues, we introduce zebra, a zero-shot framework for commonsense reasoning and question answering that aims to combine the benefits of knowledge retrieval, case-based reasoning, and introspection without fine-tuning the underlying llm. zebra stems from two observations: first, direct retrieval of commonsense facts may provide useful hints, but it is limited by the finite nature of knowledge bases and the noise therein; second, introspection can generate contextually relevant knowledge which is tailored to the input question, but this is limited to what the llm already "knows". in contrast, rather than directly retrieving or generating knowledge for the specific input question, zebra: i) retrieves one or more examples -also referred to as cases in the literature -that elicit a correct reasoning process for the input question based on commonsense, ii) generates commonsense knowledge tailored for the input question by following the relationship in the question-knowledge pairs contained in the retrieved examples, and iii) uses the generated knowledge to answer the input question. we can summarize the contributions of this work as follows: we believe that zebra represents a significant step forward for improving the capabilities of llms in commonsense question answering tasks, as outlined in figure 1 . we release our software and data at https://github.com/sapienzanlp/ zebra . ## related work over the years, the research community has proposed several approaches for the integration of commonsense knowledge into language models. ## zebra in this section, we introduce zebra, our novel zero-shot example-based retrieval augmentation framework for commonsense reasoning that combines the benefits of knowledge retrieval and introspection while dropping the need for additional training of the llm. zebra's pipeline is composed of three main steps, as illustrated in figure 2 : i) example retrieval (section 3.1), ii) exampleguided knowledge generation (section 3.2), and iii) knowledge-informed reasoning (section 3.3). ## experimental setup in this section, we describe the experimental setup used to train and evaluate zebra. more specifically, we provide an overview on the training process of the retriever (section 4.1), the datasets used to evaluate our framework (section 4.2), the process to generate zebra-kb (section 4.3), as well as the models that we consider for our experiments (section 4.4). ## results in this section, we present and discuss the results of zebra on the 8 commonsense question-answering benchmarks introduced in section 4.2 using the models listed in section 4.4. ## conclusions in this paper we presented zebra, a novel zeroshot example-based retrieval augmentation framework for commonsense reasoning. zebra is composed of two main components: an example retriever that fetches examples that are relevant to the input question from a dedicated knowledge base, and an llm that generates relevant knowledge for the input question by following the relationship in the questions and knowledge provided in the examples. this allows zebra to tackle the limitations of previous methods such as commonsense knowledge retrievers, unsupervised methods relying on human intervention and introspective reasoners, providing a simple and effective way to improve the performance of llms on commonsense reasoning and question answering tasks. since a knowledge base of curated examples may not always be available, we also introduced zebra-kb, a novel knowledge base of examples with silver knowledge explanations for each question, showing how zebra can be used in conjunction with zebra-kb to achieve strong zero-shot results on 8 well-established qa datasets for commonsense reasoning. our experiments support the effectiveness of zebra over other state-of-the-art methods for knowledge augmented reasoning, such as specialized commonsense retrievers and introspective reasoners. finally, we investigated the interpretability of the answers provided by the llms through a human evaluation of the knowledge generated by zebra. the results demonstrate that the knowledge is often perceived as reasonable by humans, aiding their understanding of the model's reasoning process leading to the final answer.