Mining Knowledge for Natural Language Inference from W ikipedia Categories (2020)

Abstract: Accurate lexical entailment (LE) and natural language inference (NLI) often require large quantities of costly annotations. To alleviate the need for labeled data, we introduce WikiNLI: a resource for improving model performance on NLI and LE tasks. It contains 428,899 pairs of phrases constructed from naturally annotated category hierarchies in Wikipedia. We show that we can improve strong baselines such as BERT and RoBERTa by pretraining them on WikiNLI and transferring the models on downstream tasks. We conduct systematic comparisons with phrases extracted from other knowledge bases such as WordNet and Wikidata to find that pretraining on WikiNLI gives the best performance. In addition, we construct WikiNLI in other languages, and show that pretraining on them improves performance on NLI tasks of corresponding languages.

## introduction natural language inference (nli) is the task of classifying the relationship, such as entailment or contradiction, between sentences. it has been found useful in downstream tasks, such as summarization @xcite and long-form text generation @xcite . nli involves rich natural language understanding capabilities, many of which relate to world knowledge. to acquire such knowledge, researchers have found benefit from external knowledge bases like wordnet @xcite , framenet @xcite , wikidata @xcite , and large-scale human-annotated datasets @xcite @xcite . creating these resources generally requires expensive human annotation. in this work, we are interested in automatically generating a large-scale dataset from wikipedia categories that can improve performance on both nli and lexical entailment (le) tasks. one key component of nli tasks is recognizing lexical and phrasal hypernym relationships. for example, vehicle is a hypernym of car. in this paper, we take advantage of the naturally-annotated wikipedia category graph, where we observe that most of the parent-child category pairs are entailment relationships, i.e., a child category entails a parent category. compared to wordnet and wikidata, the wikipedia category graph has more fine-grained connections, which could be helpful for training models. inspired by this observation, we construct wikinli, a dataset for training nli models constructed automatically from the wikipedia category graph, by automatic filtering from the wikipedia category graph. the dataset has 428,899 pairs of phrases and contains three categories that correspond to the entailment and neutral relationships in nli datasets. to empirically demonstrate the usefulness of wikinli, we pretrain bert and roberta on wikinli, wordnet, and wikidata, before finetuning on various le and nli tasks. our experimental results show that wikinli gives the best performance averaging over 8 tasks for both bert and roberta. we perform an in-depth analysis of approaches to handling the wikipedia category graph and the effects of pretraining with wikinli and other data sources under different configurations. we find that wikinli brings consistent improvements in a low resource nli setting where there are limited amounts of training data, and the improvements plateau as the number of training instances increases; more wikinli instances for pretraining are beneficial for downstream finetuning tasks with pretraining on a fourway variant of wikinli showing more significant gains for the task requiring higher-level conceptual knowledge; wikinli also introduces additional knowledge related to lexical relations benefiting finer-grained le and nli tasks. we also construct wikinli in other languages and benchmark several resources on xnli @xcite , showing that wikinli benefits performance on nli tasks in the corresponding languages. ## related work we build on a rich body of literature on leveraging specialized resources (such as knowledge bases) to enhance model performance. these works either (1) pretrain the model on datasets extracted from such resources, or (2) use the resources directly by changing the model itself. the first approach aims to improve performance at test time by designing useful signals for pretraining, for instance using hyperlinks @xcite or document structure in wikipedia @xcite , knowledge bases @xcite , and discourse markers @xcite . here, we focus on using category hierarchies in wikipedia. there are some previous works that also use category relations derived from knowledge bases @xcite , but they are used in a particular form of distant supervision in which they are matched with an additional corpus to create noisy labels. in contrast, we use the category relations directly without requiring such additional steps. @xcite use the direct parent categories of hyperlinks for training entity linking systems. within this first approach, there have been many efforts aimed at harvesting inference rules from raw text @xcite @xcite @xcite @xcite . since wikinli uses category pairs in which one is a hyponym of the other, it is more closely related to work in extracting hyponym-hypernym pairs from text @xcite @xcite @xcite . @xcite automatically generate a large-scale phrase pair dataset with several relationships by training classifiers on a relatively small amount of human-annotated data. however, new year's eve firework events entertainment events … new year's eve public holiday day … wikipedia categories wikidata new year's eve day calendar day … wordnet holidays days … figure 1: example hierarchies obtained from wikipedia categories, wikidata, and wordnet. the second approach aims to enable the model to leverage knowledge resources during prediction, for instance by computing attention weights over lexical relations in wordnet @xcite or linking to reference entities in knowledge bases within the transformer block @xcite . while effective, this approach requires nontrivial and domain-specific modifications of the model itself. in contrast, we develop a simple pretraining method to leverage knowledge bases that can likewise improve the performance of already strong baselines such as bert without requiring such complex model modifications. there are some additional related works that focus on the category information of wikipedia. @xcite and @xcite extract knowledge of entities from the wikipedia category graphs using predefined rules. @xcite build a dataset based on wikipedia article or category titles as well as the relations between categories and pages ("wikinet"), but they do not empirically validate the usefulness of the dataset. in a similarly non-empirical vein, @xcite analyze the differences between the graphs from wordnet and the ones from wikipedia categories. instead, we address the empirical benefits of leveraging the category information in the modern setting of pretrained text representations. ## wikinli we now describe how the wikinli dataset is constructed from wikipedia and its principal characteristics. each wikipedia article is associated with crowd-sourced categories that correspond to topics or concepts covered by that article. wikipedia or-ganizes these categories into a directed graph that models their hierarchical relations. for instance, the category "days" is a parent node of the category "holidays" in this graph. the central observation underlying wikinli is that this category hierarchy resembles the concept hierarchies and ontologies found in knowledge bases, such as wikidata and wordnet. while there are similarities between the three resources, the wikipedia category hierarchy contains more diverse connections between parent and child concepts. figure 1 shows an example category "new year's eve" and its ancestors under these resources. all resources include a path that corresponds to the generalization of new year's eve as a regular day, but wikipedia additionally includes a path that corresponds to the generalization as a celebration or entertainment event. thus the wikipedia hierarchy provides more abstract and fine-grained generalization that can be useful for nli tasks. in this example, the commonsense knowledge that new year's eve implies entertainment is only directly captured by the wikipedia hierarchy. wikinli is a dataset of category pairs extracted from this wikipedia hierarchy to be used as an auxiliary task for pretraining nli models. specifically, wikinli contains three types of category pairs based on their relations in the wikipedia hierarchy: child-parent ("child"), parent-child ("parent"), and other pairs ("neutral"). the motivation is that childparent resembles entailment; parent-child resembles reverse entailment; and other pairs resemble a neutral relationship. we find that this simple definition of relations is effective in practice; we also report an exploration with other types of relations such as siblings in experiments. for the following experiments, unless otherwise specified, we only use 100,000 samples from wikinli as training data and 5,000 as the development set since we find larger training set does not lead to performance gains (see sec. 6.3 for more details). wikinli is available at https: //github.com/zeweichu/wikinli. ## approach to demonstrate the effectiveness of wikinli, we pretrain bert and roberta on wikinli and other resources, and then finetune them on several nli and le tasks. we assume that if a pretraining resource is better aligned with downstream tasks, it will lead to better downstream performance of the models pretrained on it. ## analysis we perform several kinds of analysis, including using bert to compare the effects of different settings. due to the submission constraints of the glue leaderboard, we will report dev set results (medians of 5 runs) for the tables in this section, except for break which is only a test set. ## multilingual wikinli wikipedia has different languages, which naturally motivates us to extend wikinli to other languages. we mostly follow the same procedures as english wikinli to construct a multilingual version of wikinli from wikipedia in other languages, except that (1) we filter out instances that contain english words for arabic, urdu, and chinese; and (2) we translate the keywords into chinese when filtering the chinese wikinli. we will refer to this version of wikinli as "mwikinli". as a baseline, we also consider "trwikinli", where we translate the english wikinli into other languages using google translate. we benchmark these resources on xnli in four languages: french (fr), arabic (ar), urdu (ur), and chinese (zh). when reporting these results, we pretrain multilingual bert (mbert; @xcite on the corresponding resources, finetune it on 3000 instances of the training set, perform early stopping on the development set, and test it on the test set. we always use xnli from the corresponding language. in addition, we pretrain mbert on english wikinli, wikidata, and wordnet, finetune and evaluate them on other languages using the same language-specific 3000 nli pairs mentioned earlier. we note that when pretraining on mwikinli or trwikinli, we use the versions of these datasets with the same languages as the test sets. the accuracy differences between mwikinli and wikinli could be partly attributed to domain differences across languages. to measure the differences, we compile a list of the top 20 most frequent words in the chinese mwikinli, shown in future research will be required to obtain a richer understanding of how training on wikinli benefits non-english languages more than training on the language-specific mwikinli resources. one possibility is the presence of emergent crosslingual structure in mbert @xcite . nonetheless, we believe mwikinli and our training setup offer a useful framework for further research into multilingual learning with pretrained models. ## conclusion we constructed wikinli, a large-scale naturallyannotated dataset for improving model performance on nli and le tasks. empirically, we wikinli chinese mwikinli albums 中国(china) songs 中华人民共和国(p. r. c.) players 行政区划(administrative division) male 人(man) people 政治(politics) american 人物(people) british 各国(countries) writers 组织(organization) (band) 各省(provinces) female 建筑物(building)