R eason BERT : P re-trained to Reason with Distant Supervision (2021)

Abstract: We present ReasonBert, a pre-training method that augments language models with the ability to reason over long-range relations and multiple, possibly hybrid contexts. Unlike existing pre-training methods that only harvest learning signals from local contexts of naturally occurring texts, we propose a generalized notion of distant supervision to automatically connect multiple pieces of text and tables to create pre-training examples that require long-range reasoning. Different types of reasoning are simulated, including intersecting multiple pieces of evidence, bridging from one piece of evidence to another, and detecting unanswerable cases. We conduct a comprehensive evaluation on a variety of extractive question answering datasets ranging from single-hop to multi-hop and from text-only to table-only to hybrid that require various reasoning capabilities and show that ReasonBert achieves remarkable improvement over an array of strong baselines. Few-shot experiments further demonstrate that our pre-training method substantially improves sample efficiency.

## introduction recent advances in pre-trained language models (lms) have remarkably transformed the landscape of natural language processing. pre-trained with self-supervised objectives such as autoregressive language modeling @xcite @xcite and masked language modeling (mlm) @xcite @xcite , lms encode a great deal of knowledge about language and significantly boost model performance on a wide range of downstream tasks @xcite ranging from spell checking @xcite to sentiment analysis @xcite and semantic parsing @xcite , just to name a few. existing self-supervised objectives for lm pretraining primarily focus on consecutive, naturally occurring text. for example, mlm enables lms to correctly predict the missing word "daughters" in the sentence "obama has two __ , malia and sasha." based on the local context and the knowledge stored in the parameters. however, many tasks require reasoning beyond local contexts: multi-hop question answering (qa) @xcite and fact verification @xcite require reasoning over multiple pieces of evidence, hybrid qa @xcite requires simultaneously reasoning over unstructured text and structured tables, and dialogue systems require reasoning over the whole dialogue history to accurately understand the current user utterance @xcite . to address this limitation in existing lm pretraining, we propose reasonbert, a pre-training method to augment lms for explicitly reasoning over long-range relations and multiple contexts. reasonbert pairs a query sentence with multiple relevant pieces of evidence drawn from possibly different places and defines a new lm pre-training objective, span reasoning, to recover entity spans that are masked out from the query sentence by jointly reasoning over the query sentence and the relevant evidence (figure 1 ). in addition to text, we also include tables as evidence to further empower lms to reason over hybrid contexts. one major challenge in developing reasonbert lies in how to create a large set of query-evidence pairs for pre-training. unlike existing self-supervised pre-training methods, examples with complex reasoning cannot be easily harvested from naturally occurring texts. instead, we draw inspiration from distant supervision @xcite , which assumes figure 1 : examples of our pre-training data acquired via distant supervision, which covers a wide range of topics with both textual and tabular evidence. for each query sentence (in black), we first select two pairs of entities (underlined) to find two pieces of evidence (in grey) via distant supervision. we then randomly mask one entity from each selected pair and aim to recover it by reasoning over the evidence. note that the two selected pairs may share a common entity; in case this entity is masked, we can mimic different types of multi-hop reasoning, e.g., intersection (ex. 1) and bridging (ex. 2). to simulate unanswerable cases, we additionally mask one entity (in blue) that does not exist in the evidence. figure best viewed in color. we first replace the masked entities in a query sentence with the [question] tokens. the new pre-training objective, span reasoning, then extracts the masked entities from the provided evidence. we augment existing lms like bert @xcite and roberta @xcite by continuing to train them with the new objective, which leads to reasonbert, a new lm with better reasoning capabilities. then query sentence and textual evidence are encoded via the lm. when tabular evidence is present, we use the structure-aware transformer tapas @xcite as the encoder to capture the table structure. we evaluate reasonbert on the extractive qa task, which is arguably the most representative task requiring reasoning about world knowledge. we conduct a comprehensive evaluation using a variety of popular datasets: mrqa @xcite , a single-hop qa benchmark including six datasets from different domains; hotpotqa @xcite , a multi-hop qa dataset; nqtables, a subset of the natural questions dataset @xcite where answers can be found in tables; and hybridqa @xcite , a hybrid multi-hop qa dataset that requires reasoning over both tables and text. under the few-shot setting, reasonbert substantially outperforms the baselines in almost all datasets, demonstrating that the reasoning ability learned from pre-training can easily transfer to downstream qa tasks and generalize well across domains. under the full-data setting, reasonbert obtains substantial gains in multihop and hybrid qa datasets. despite its simple model architecture, reasonbert achieves similar or better performance compared with more sophisticated state-of-the-art models for each dataset. existing pretraining objectives such as mlms @xcite tend to implicitly memorize the learned knowledge in the parameters of the underlying neural network. in this work, we aim to augment pre-training by encouraging a model to reason about (instead of memorizing) world knowledge over the given contexts. extractive question answering. to measure a model's reasoning ability about world knowledge, we select extractive qa as a downstream task, which is perhaps one of the most representative tasks for this purpose. given a question q and provided evidence e, an extractive qa model p θ (a|q, e) aims to select a contiguous span a from e that answers the question, or output a special token if e is not sufficient to answer the question. our approach, reasonbert, is inspired by this formulation and extends it to language model pretraining. the challenge in defining such a selfsupervised task is in the creation of questionevidence pairs from unlabeled data. moreover, we aim for a generic approach that works for a wide range of extractive qa settings including singlehop and multi-hop reasoning, hybrid contexts with both unstructured texts and structured tables, as well as few-shot settings. we discuss how to address the challenge and achieve this goal in the next two sections. ## distant supervision (ds) for pre-training we use english wikipedia as our data source for pre-training. we first extract sentences and tables from wikipedia pages and then identify salient spans (such as named entities) from them. we apply the idea of distant supervision and match the sentences and tables to form query-evidence pairs, which are used to create pre-training examples. ## pre-training 4.1 encoder for the text-only setting, we use the standard transformer encoder in bert @xcite . for settings where the input contains tables, we adopt the transformer variant recently introduced in tapas @xcite , which uses extra token-type embeddings (indicating the row/column position of a token) to model the table structure. ## ablation study we further conduct ablation studies on hotpotqa to verify our design choices, summarized in our work is most related to sspt, but uses distant supervision to collect query-evidence pairs and thus obviate the need for a retriever. meanwhile, to encourage the model to learn complex reasoning, we mimic different types of reasoning chains by masking multiple entities, including unanswerable ones, and simultaneously inferring them from disjoint pieces of evidence. our method also works with heterogeneous sources including both text and tables, while most existing work considers only text-based question answering. ## conclusion and future work we propose reasonbert, a novel pre-training method to enhance the reasoning ability of language models. the resulting model obtains substantial improvements on multi-hop and hybrid qa tasks that require complex reasoning, and demonstrates superior few-shot performance. in the future, we plan to use our query-evidence pairs collected by distant supervision to improve the retrieval performance for open-domain qa, as well as empower reasonbert to handle more types of reasoning, like comparison and numeric reasoning, in natural language understanding. mike mintz, steven bills, rion snow, and daniel jurafsky. 2009a. distant supervision for relation extraction without labeled data. in proceedings of the joint conference of the 47th annual meeting of the acl and the 4th international joint conference on natural language processing of the afnlp, pages 1003-1011, suntec, singapore. association for computational linguistics. mike mintz, steven bills, rion snow, and daniel jurafsky. 2009b. distant supervision for relation extraction without labeled data. in proceedings of the joint conference of the 47th annual meeting of the acl and the 4th international joint conference on natural language processing of the afnlp, pages 1003-1011, suntec, singapore. association for computational linguistics. liangming pan, wenhu chen, wenhan xiong, min-yen kan, and william yang wang. 2021. unsupervised multi-hop question answering by question generation. in proceedings of the 2021 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 5866-5880, online. association for computational linguistics. matthew e. peters, mark neumann, robert logan, roy schwartz, vidur joshi, sameer singh, and noah a. smith. 2019. knowledge enhanced contextual word representations. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 43-54, hong kong, china. association for computational linguistics. alec radford and karthik narasimhan. 2018. improving language understanding by generative pretraining. alec radford, jeffrey wu, r. child, david luan, dario amodei, and ilya sutskever. 2019. language models are unsupervised multitask learners. pranav rajpurkar, jian zhang, konstantin lopyrev, and percy liang. 2016. squad: 100,000+ questions for machine comprehension of text. in proceedings of the 2016 conference on empirical methods in natural language processing, pages 2383-2392, austin, texas. association for computational linguistics. ori ram, yuval kirstain, jonathan berant, amir globerson, and omer levy. 2021. few-shot question answering by pretraining span selection. in proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 1: long papers), pages 3066-3079, online. association for computational linguistics.