Retrieval Augmentation for Commonsense Reasoning: A Unified Approach (2022)

Abstract: A common thread of retrieval-augmented methods in the existing literature focuses on retrieving encyclopedic knowledge, such as Wikipedia, which facilitates well-defined entity and relation spaces that can be modeled. However, applying such methods to commonsense reasoning tasks faces two unique challenges, i.e., the lack of a general large-scale corpus for retrieval and a corresponding effective commonsense retriever. In this paper, we systematically investigate how to leverage commonsense knowledge retrieval to improve commonsense reasoning tasks. We proposed a unified framework of retrieval-augmented commonsense reasoning (called RACo), including a newly constructed commonsense corpus with over 20 million documents and novel strategies for training a commonsense retriever. We conducted experiments on four different commonsense reasoning tasks. Extensive evaluation results showed that our proposed RACo can significantly outperform other knowledge-enhanced method counterparts, achieving new SoTA performance on the CommonGen and CREAK leaderboards.

## introduction recent work has shown that scaling language models with considerably more data and parameters, such as gpt3-175b @xcite , could drive significant advances in commonsense reasoning tasks. nevertheless, such models make predictions by only "looking up information" stored in their parameters, making it difficult to determine what knowledge is stored or has been already forgotten by the neural network @xcite . besides, storage space is limited by the size of the neural network. in order to memorize more world knowledge, one must train ever-larger networks, which can be prohibitively expensive and slow. the solution that may seem obvious at first glance is to grant language models free access to open-world sources of commonsense knowledge in a plug-and-play manner, instead of memorizing all world knowledge. to achieve this capability, language models must be able to retrieve relevant commonsense knowledge from an unbounded set of situations. then, the language models can leverage the input text, as well as the retrieved information to produce the desired output. compared with the large-scale language model counterparts, e.g., unicorn @xcite , retrieval-augmented methods have three remarkable advantages: first, the knowledge is not stored implicitly in the model parameters, but is explicitly acquired in a plug-and-play manner, leading to great scalability; second, the paradigm generates text based on some retrieved references, which alleviates the difficulty of generating from scratch @xcite ; third, knowledge corpus can be constantly edited and updated by experts, making the model aware of the latest information. besides, compared with knowledge graph inference model counterparts, e.g., qa-gnn @xcite , retrieval-augmented methods allow more flexibility in accessing and using knowledge from different sources, because of the nature of commonsense knowledge, which cannot all be contained in a single knowledge graph defined by a certain schema @xcite . a common thread of retrieval-augmented methods in the existing literature focuses on retrieving encyclopedic knowledge such as wikipedia, which lends itself to a well-defined space of entities and relations that can be modeled @xcite @xcite . however, retrieval-augmented methods for commonsense reasoning have been rarely studied in the literature. in this paper, we propose a unified frame-raco aristoroberta re-t5 kfcnet opencsr (this work) @xcite @xcite number of corpus types 3 1 1 1 1 number of commonsense tasks 4 1 1 1 1 number of docs for retrieval 20m 5k 0.8m 0.8m 1m the first challenge to address is what commonsense knowledge to retrieve. different from encyclopedic knowledge used in open-domain qa tasks, commonsense knowledge is very diverse, containing everyday events and their effects, facts about beliefs and desires, and properties of objects in human's daily life. since commonsense involves various aspects including human interaction and object properties in everyday life, we collected a over 20 million commonsense documents collection from both open-domain knowledge sources (e.g., omcs) that cover multiple domains of commonsense, and domain-specific sources (e.g., atomic) that focus on particular commonsense types. the second challenge is to address how to retrieve relevant commonsense knowledge from the corpus. different from training a dense retriever on wikipedia @xcite , the heuristic of taking "documents containing correct answers" as positive candidates cannot be used because the output answer in commonsense reasoning tasks is usually not a substring of retrieved documents. for example, in binary question answering, the answer is true or false but it does not appear in the retrieved documents. therefore, we propose novel strategies to construct question-document pairs for commonsense dense retriever training. overall, our main contributions in this work can be summarized as follows: ## related work though large-scale language models yield state-ofthe-art performance on many commonsense reasoning tasks, their pre-training objectives do not explicitly guide the models to reason with commonsense knowledge such as the relation and composition of daily concepts in our lives @xcite , leading to unsatisfactory performance in many real-world scenarios @xcite . existing work has mainly explored two directions to improve their commonsense reasoning ability. the first is to pre-train or post-train a language model on commonsense corpora @xcite @xcite . when the commonsense materials are appropriately selected, this simple strategy could demonstrate significantly superior performance than vanilla pre-trained language models @xcite . notable methods include comet @xcite , calm @xcite , unicorn @xcite , etc. nonetheless, these methods still suffer from the same drawbacks as the pre-trained language models introduced in §1. the second is to explicitly introduce external knowledge from commonsense knowledge graphs to augment the limited textual information. @xcite . a kg often provides comprehensive and rich entity features and relations so models can easily traverse links to discover how entities are interconnected to express certain commonsense knowledge. notable methods include kagnet @xcite , grf @xcite , qa-gnn @xcite , greaselm @xcite , etc. however, commonsense knowledge lies at an unbounded set of facts and situations that usually cannot be covered by a single knowledge graph defined by a cer-tain schema. reasoning over multiple knowledge graphs is a challenging task. retrieval-augmented method is a new learning paradigm that fuses pre-trained language models and traditional information retrieval techniques @xcite . a few recent methods have explored retrieving in-domain commonsense documents from a task-relevant corpus to improve commonsense reasoning performance @xcite @xcite . we provide a detailed comparison in ## proposed method in this section, we elaborate on how to leverage commonsense knowledge retrieval from a largescale corpus to improve various commonsense reasoning tasks, including commonsense corpus construction ( §3.1), commonsense document retriever ( §3.2) and commonsense document reader ( §3.3). the architecture of raco is shown in figure 1 . ## epilogue conclusions. retrieval-augmented methods have been widely used in many nlp tasks such as open-domain question answering. however, applying this approach to commonsense reasoning has been neglected in the existing literature. in this paper, we systematically investigate how to leverage commonsense knowledge retrieval to improve commonsense reasoning tasks. we collected a commonsense corpus containing over 20 million documents, and proposed novel strategies for training a commonsense retriever. extensive experiments demonstrate our method can effectively improve the performance of various commonsense reasoning tasks, achieving new state-of-the-art performance on the commongen and creak leaderboards. future work. a natural extension of this work is to leverage heterogeneous knowledge to improve commonsense reasoning tasks, such as combining structured (i.e., knowledge graph) and unstructured (i.e., retrieved text) knowledge. such a model will require building a graph reasoning module and a textual reasoning module, and merging the knowledge learned from both, which is a challenging task. the second future direction is to learn a commonsense dense retriever without question-document pairs. for example, in binary question answering, the labels are true / false that cannot be used to train a commonsense retriever. limitations. there are two main limitations. first, raco retrieves documents from a large-scale corpus, then leverage the retrieved documents to make predictions. so, compared with baseline methods such as t5 and unicorn, raco consumes more time and computing resources. second, due to the diversity and multi-source nature of commonsense knowledge, the retrieved evidence might contain noisy information that can even hurt model performance. a fine-grained filtering or re-ranking module could be a future work. csqa2.0 dataset. first, compared to t5, our model can improve by 8.3% accuracy on all dev data (shown in the first column). however, on different statement types, the model performance is different. for example, from the predicted results of t5, the performance on "comparison" state-ments and "condition" statements is below-average. by introducing the retrieved commonsense knowledge, raco demonstrated significantly better performance on these two sub-categories, achieving 15.3% and 18.5% improvement, which is significantly higher than the average 8.3% improvement. nevertheless, we also observe the retrieved evidence might provide noisy information, resulting in performance degradation, such as "reason" related statements. we show an example in statements under these categories are often descriptions or comparisons of factual commonsense, the retrieved documents can thus well complement the necessary knowledge of a given statement. however, some statements require the model to reason in a given scenario, so making correct predictions requires the model to use commonsense knowledge to understand the local contexts. in these statements, retrieved knowledge might even contradict the assumptions, hurting the model performance.