Complementary Roles of Inference and Language Models in QA (2023)

Abstract: Answering open-domain questions through unsupervised methods poses challenges for both machine-reading (MR) and language model (LM) -based approaches. The MR-based approach suffers from sparsity issues in extracted knowledge graphs (KGs), while the performance of the LM-based approach significantly depends on the quality of the retrieved context for questions. In this paper, we compare these approaches and propose a novel methodology that leverages directional predicate entailment (inference) to address these limitations. We use entailment graphs (EGs), with natural language predicates as nodes and entailment as edges, to enhance parsed KGs by inferring unseen assertions, effectively mitigating the sparsity problem in the MR-based approach. We also show EGs improve context retrieval for the LM-based approach. Additionally, we present a Boolean QA task, demonstrating that EGs exhibit comparable directional inference capabilities to large language models (LLMs). Our results highlight the importance of inference in open-domain QA and the improvements brought by leveraging EGs.

## introduction unsupervised open-domain question answering (qa), the task of learning knowledge from a large collection of documents of diversified topics to answer questions, has been a long-standing challenge in nlp, information retrieval and related fields @xcite @xcite . the traditional machine-reading (mr) approach first extracts a knowledge graph (kg) from an opendomain corpus and then uses the kg for qa @xcite @xcite . this approach offers explainability, since the information in kgs is directly supported by the text. however, the relevant assertions need to be exactly stored in the extracted kg, which is often not the case because assertions can be stated in many different ways, while usually only a small subset of them are available in the kg. on the other hand, language models have been claimed to be capable of performing a wide range of nlp tasks when used in zero-shot or fewshot prompting mode, including open-domain qa, where they have been argued to act as a latent kg over the pretraining data for querying @xcite @xcite @xcite @xcite . advocates of lms argue that traditional mr approaches relying on kgs built by open relation extraction are prone to errors arising from components like open information extraction and entity linking. in addition to querying lms directly, it is shown that when relevant context is available and added to the query, the lms' performance increases significantly @xcite @xcite . however, while lms have performed impressively in answering questions on the basis of manually selected contextual documents, their practical usage is limited since automatic retrieval methods do not always return relevant contextual documents to the query. in this paper, we show that we can leverage directional predicate entailment effectively to alleviate the limitations of both unsupervised approaches to qa. the contributions of this paper are as follows: semantic parsing is an efficient open-domain information extraction approach for large corpora @xcite . @xcite propose an effective pipeline that extracts facts by utilizing a localized update algorithm, which transfers sentences into syntax structures and generates kgs incrementally. these mr-based approaches are explainable for qa because every answer is supported by source sentences in the text. however, kgs built in this way are limited to exact match between the question form and the triples in the graph. for example, if a triple (amon bazira, be assassinated in, kenya) is extracted from the sentence "amon bazira was assassinated in kenya", the kg would not provide an answer to the question "where did amon bazira die?" because the training corpus lacks any sentence constituting an exact match, such as "amon bazira died in kenya". as a result, the parsed kg exhibits high precision but low recall on the task. using pre-trained lms as latent kg. @xcite claim that pre-trained lms encode the knowledge presented in large amounts of texts. they query lms using "fill-in-the-blank" cloze statements, such as "amon bazira was assassinated in [mask]". they report results on masked language models (mlms) such as bert, which are optimized to predict the next word in a sequence or fill in masked words. they show promising performance on cloze-style qa tasks. @xcite propose a method for fact extraction based on bert, using the bert sentence-encoding algorithm on a corpus already annotated for named entities. additionally, @xcite al. (2020) propose approaches to train prompt-learning models with supervised datasets, using the generated prompts to enhance lm performance on opendomain qa. larger lm models, as shown in the works of @xcite , demonstrate better performance. these results suggest that lms could work as latent kgs by memorizing vast corpora. however, llms are expensive to train, and impractical to update for tasks like questions involving recent news events. smaller neural lms are faster to retrain, but fail when natural language inference from limited context is required @xcite . attempts to fine-tune these lms with supervision from natural language inference (nli) datasets tend to pick up artifacts and show little evidence of learning directional common-sense inferences, such as that, "be assassinated in" entails "die in" but not the reverse @xcite . in this paper, we query lms for factual knowledge in a zero-shot setting, but show how the lm-based approach could benefit from the mr-based approach and predicate entailment. ## method in §3.1, we propose an unsupervised mr-based method that consists of three key steps: a) constructing a kg by semantic parsing ( §3.1.1), b) constructing egs from text ( §3.1.2), and c) augmenting the kg with egs in an unsupervised way to infer latent knowledge ( §3.1.3). we then further augment the kg with lm backoff ( §3.1.4). in §3.2, we discuss the lm-based approach and propose a method to enhance the performance by extracting highly-relevant contexts using egs ( §3.2.1). ## conclusion in this paper, we have conducted a comprehensive analysis of the limitations of machine-reading and lm-based approaches in qa. we propose a novel method that utilizes entailment graphs to infer directional relations, addressing the sparsity issue and low relevance of retrieved contexts. additionally, we have introduced an open-domain boolean qa task to evaluate the capabilities of directional inference. in boolean qa, the entailment graphs present stronger capabilities in directional inference than bert and roberta, achieving comparable performance to gpt-3.5. these results demonstrate the effectiveness of the entailment graphs in enhancing performance under both unsupervised approaches, by making common-sense inference available to open-domain qa.