Pre-training Language Models for Comparative Reasoning (2023)

Abstract: Comparative reasoning is a process of comparing objects, concepts, or entities to draw conclusions, which constitutes a fundamental cognitive ability. In this paper, we propose a novel framework to pre-train language models for enhancing their abilities of comparative reasoning over texts. While there have been approaches for NLP tasks that require comparative reasoning, they suffer from costly manual data labeling and limited generalizability to different tasks. Our approach introduces a novel method of collecting scalable data for text-based entity comparison, which leverages both structured and unstructured data. Moreover, we present a framework of pre-training language models via three novel objectives on comparative reasoning. Evaluation on downstream tasks including comparative question answering, question generation, and summarization shows that our pre-training framework significantly improves the comparative reasoning abilities of language models, especially under low-resource conditions. This work also releases the first integrated benchmark for comparative reasoning.

## introduction comparative reasoning constitutes a fundamental cognitive ability that plays a crucial role in decisionmaking. it refers to comparing objects, concepts, or entities to draw conclusions or make informed decisions. for example, consumers often compare products on their features such as price, quality, and user reviews before placing an order. policymakers weigh the advantages and disadvantages of different policy proposals to address pressing issues. regarding textual documents, comparative reasoning is commonly needed in identifying differences between research studies, contrasting news articles from different sources, or synthesizing arguments of opposing viewpoints in a debate. recent research has developed models for a few nlp tasks related to comparing texts, including identifying comparative sentences @xcite , mining comparable entities @xcite , identifying comparative aspects from a set of questions @xcite , extracting comparative summaries @xcite , and summarizing different opinions @xcite . yet, the data collection for these tasks relies on expensive and time-consuming manual annotation. as a result, low-resource scenarios are common when it comes to new comparative tasks @xcite . moreover, the task-specific design of such models limits their general comparative reasoning abilities. meanwhile, pre-trained language models (plms) such as bart @xcite and t5 @xcite ) exhibit generalizability on several nlp tasks. however, existing pre-training methods such as masked language modeling and span in-filling fail to empower language models (lms) with strong comparative reasoning abilities due to the lack of explicit training on comparisons. to address these challenges, we propose a novel pre-training framework to enhance the comparative reasoning abilities of lms. specifically, it trains lms to capture the comparison information between entities from paired documents. our approach pilots around a scalable, labor-free data collection method that gathers documents as entity descriptions and a wealth of facts for entity comparison by combining structured (i.e., wikidata) and unstructured data (i.e., news and wikipedia). we represent these comparisons of facts as quintuples, which consist of a pair of entities and the corresponding values of their shared property. to empower lms with comparative reasoning abilities on such data, given two comparable entities and their corresponding descriptive documents, we design three novel pre-training tasks including the generation of comparative answers, question-answer pairs, and summaries. pre-training data of these tasks are obtained through automatic textualization of fac- the show, with a book by the screenwriter diablo cody ("juno") and staging by director diane paulus ("waitress"), takes on the good work we are always asking new musicals to do: the work of singing about real things. to comprehensively evaluate the comparative reasoning abilities of lms, we introduce a new benchmark with a suite of comparative reasoning tasks. it contains: (1) comparative question answering (qa), sourced from subsets of hotpotqa and 2wikiqa datasets @xcite ; (2) comparative question generation (qg), including hotpotqg and 2wikiqg which are converted from the qa datasets; (3) comparative summarization, including the diffen dataset that we crawled and the existing cocotrip dataset @xcite . with this benchmark, we conduct extensive experiments with vanilla plms (i.e., bart and t5) and their counterparts trained by our proposed framework. results demonstrate a notable improvement in the performance of these plms on different comparative reasoning scenarios, especially under low-resource settings. specifically, under the fewshot setting, the bart model pre-trained with our framework outperforms the vanilla bart by an average of 6.17 points on all datasets. under the zero-shot setting, the improvement becomes as high as 13.99 points on average. these results highlight the effectiveness of our pre-training framework which empowers lms with impressive abilities of comparative reasoning even when zero or few examples are available. besides, we analyze the effect of the pre-training data size on the model performance, and provide a case study to better understand the benefits of our pre-training. our contributions are summarized as follows: as identifying comparative sentences @xcite , extracting comparable entities @xcite , and classifying components of comparison @xcite . recent work focused more on text generation tasks such as generating arguments to answer comparative questions @xcite , generating comparable questions from news @xcite , and summarizing comparative opinions @xcite . the existing techniques were designed for specific tasks and could not generalize across all types of comparative reasoning tasks. moreover, they suffered from the scarcity of labelled data in low-resource settings. our approach aims to address these two challenges. ## pre-training framework to enhance the ability of lms in comparative reasoning, we introduce a novel framework for pretraining lms on a collected corpus of comparative entities. specifically, lms are given a pair of documents, each describing an entity, and are trained to generate target sequences which require comparison between the entities. we consider three types of target sequences: an answer to a comparative question, a question-answer pair that requires comparative reasoning, and a comparative summary of entities. correspondingly, we design three text-totext pre-training tasks that require the lms to simultaneously attend to both documents and extract information for pairwise comparison. this framework enables them to handle various downstream scenarios that require comparative reasoning. to collect data for large-scale pre-training, we extract comparable entity pairs with their properties by combining structured and unstructured data. the extracted data are first formulated as quintuples (elaborated in §3.1 to show their comparative nature), which are later used for text-to-text pretraining. ## conclusion in this paper, we presented a novel framework for pre-training language models for comparative reasoning. it obtained quintuples for entity comparison by combining structured and unstructured data, converted the quintuples into textual components, and employed them in three novel sequence-tosequence pre-training tasks. we demonstrated the effects of the pre-training tasks on six downstream datasets, especially in limited-resource scenarios. d 1 : airsoft is a team game in which participants eliminate opposing players by tagging them out of play with spherical plastic projectiles shot with mock air weapons called airsoft guns. ... (446 words left) d 2 : paintball is a competitive team shooting sport in which players eliminate opponents from play by hitting them with spherical dye-filled gelatin capsules called paintballs that break upon impact. ... (472 words left) gold: airsoft is a popular combat simulation game where participants are eliminated when hit by pellets launched from guns that resemble real firearms. in paintball participants try to hit each other with paintballs launched from a special paintball marker/gun. while airsoft is :::::: cheaper :::: and :::::::: provides :: a ::::: more :::::::: realistic ::::::: warfare ::::::::: experience, paintball is :::: more :::::::: popular, :::::: more ::::::::: organized ::: bart (r-l: 18.66, r-2: 4.39) airsoft is a team shooting sport in which participants eliminate opponents by hitting them with airsoft guns. airsoft guns are shaped like basketballs or baseball bats and are equipped with a series of round-shaped projectiles called paintballs. bart+cmp (r-l: 19.17, r-2: 8.62) airsoft and paintball are two of the most popular shooting sports of all time. airsoft is a shooting sport that involves hitting opponents :::: with :::::: airsoft ::::: guns, while paintball is a more aggressive game that :::: uses : a :::::: softer, ::::: more :::::::::: aggressive, :::::::::: ball-shaped ::::::::: paintball.