Cross-lingual Inference with A C hinese Entailment Graph (2022)

Abstract: Predicate entailment detection is a crucial task for question-answering from text, where previous work has explored unsupervised learning of entailment graphs from typed open relation triples. In this paper, we present the first pipeline for building Chinese entailment graphs, which involves a novel high-recall open relation extraction (ORE) method and the first Chinese fine-grained entity typing dataset under the FIGER type ontology. Through experiments on the Levy-Holt dataset, we verify the strength of our Chinese entailment graph, and reveal the cross-lingual complementarity: on the parallel Levy-Holt dataset, an ensemble of Chinese and English entailment graphs outperforms both monolingual graphs, and raises unsupervised SOTA by 4.7 AUC points.

## introduction predicate entailment detection is important for many tasks of natural language understanding (nlu), including reading comprehension and semantic parsing. suppose we wish to answer a question by finding a relation v between entities a and b. often, v cannot be found directly from the reference passage or database, but another relation u can be found between a and b, where u entails v (for instance, suppose u is buy, v is own). if we can identify this with predicate entailment detection, we can then answer the question. to detect predicate entailments, previous work has explored unsupervised learning of typed entailment graphs @xcite @xcite @xcite . entailment graphs are directed graphs, where each node represents the predicate of a relation, and an edge from node u to node v denotes "u entails v". entailment graphs are built based on the distributional inclusion hypothesis (dih) @xcite @xcite . predicates are disambiguated according to their arguments' types, predicates taking the same types of arguments go into one subgraph. while previous work on entailment graphs has mostly been limited to english, building entailment graphs in other languages is interesting and challenging. the importance is two-fold: for that language, a native entailment graph would facilitate nlu in it; for cross-lingual inference, entailment graphs in different languages host exploitable complementary information. in particular, we argue that by jointly consulting strong entailment graphs in multiple languages, improvements can be gained for inference in all participant languages. in this paper, we choose chinese as our target language to build entailment graphs, as it is distant enough from english to exhibit rich complementarity, while relatively high-resource. the main challenge for building chinese entailment graphs, is to extract reliable typed relation triples from raw corpora as strong input. this involves open relation extraction (ore) and fine-grained entity typing (fet), which we discuss below. ore extracts predicate-argument triples from sentences, where previous work has used rulebased methods over syntactic parsers either directly @xcite @xcite , or for distant supervision @xcite @xcite . the challenge in ore can be largely attributed to the poor definition of "open relations". the situation worsens in chinese, as the parts of speech are more ambiguous and many linguistic indicators of relations are poorly represented. previous work on chinese ore @xcite has defined narrow sets of open relations, failing to identify many relational constructions. conversely, we propose a novel dependency-based ore method, which we claim provides comprehensive coverage of relational constructions. fet assigns types to the arguments of extracted relations, so that word-senses of predicates can be disambiguated. the challenge in chinese fet lies mainly in the lack of datasets over a suitable type ontology: too coarse a type set would be insufficient for disambiguation, too granular a type set would result in disastrous sparsity in the entailment graph. following @xcite , we use the popular figer type set @xcite , and build cfiger, the first figer-labelled chinese fet dataset via label mapping. entity typing models built on this dataset show satisfactory accuracy and are helpful for predicate disambiguation. we evaluate our chinese entailment graph on the levy-holt entailment dataset @xcite via translation. results show that our chinese entailment graph outperforms baselines by large margins, and is comparable to the english graph. we verify our cross-lingual complementarity hypothesis with an ensemble between english and chinese graphs, where we show a clear advantage over both monolingual graphs foot_0 , and set a new sota for unsupervised predicate entailment detection. our contributions are as follows: 1) we present a novel chinese ore method sensitive to a much wider range of relations than previous sota, and a chinese fet dataset, the first under the figer type ontology; 2) we construct the first chinese entailment graph, comparable to its english counterpart; 3) we reveal the cross-lingual complementarity of entailment graphs with an ensemble. foot_1 2 background and related work predicate entailment detection has been an area of active research. @xcite ; @xcite ; @xcite proposed various count-based entailment scores; @xcite proposed to "globalize" typed entailment graphs by closing them with transitivity constraints; @xcite proposed a more scalable global learning approach with soft transitivity constraints; @xcite further refined the entailment scores with standard and contextual link prediction. our work is closely related to @xcite , with key adaptations for chinese in ore and fet. their ore method is based on a ccg parser @xcite , while ours is based on a dependency parser @xcite ; their fet is done by linking entities to wikipedia entries, while we use neural entity typing for the task. dependency parses are less informative than ccg parses, and require heavier adaptation. however, chinese dependency parsers are currently more reliable than ccg parsers @xcite . previous chinese ore methods @xcite are based on dependency parsers, but they omit many common constructions essential to ore. in §3, we present the most comprehensive chinese ore method so far. linking-based entity-typing can be more accurate than neural methods, since the type labels are exact as long as linking is correct. however, current chinese entity linking methods require either translation @xcite or search logs @xcite . both hurt linking accuracy, and the latter grows prohibitively expensive with scale. on the other hand, since the seminal work of @xcite , neural fine-grained entity typing has developed rapidly @xcite @xcite , with a common interest in figer type set. for chinese, @xcite built an ultra-fine-grained entity typing dataset, based on which we build our cfiger dataset via label mapping. @xcite aligned english and german entailment graphs, and showed that the english graph can help with german entailment detection. yet it was uncertain whether this effect comes from genuine complementarity or the mere fact that the english graph is stronger. we take one step further, and show that complementarity can be exploited in both directions: for english, the higher resource language, entailment detection can also benefit from the ensemble to reach new heights. as a related resource, ganitkevitch et al. (2013) created a multi-lingual database for symmetric paraphrases; in contrast, entailment graphs are directional. more recently, @xcite proposed to fine-tune language models on predicate entailment datasets via prompt learning. in contrast to our entailment graphs, their approach is supervised, which carries the danger of overfitting to dataset artifacts @xcite . another related strand of research, e.g. snli @xcite , is concerned with the more general nli task, including hypernymy detection and logic reasoning like a ∧ @xmath0, but rarely covers the cases requiring external knowledge of predicate entailment @xcite . conversely, entailment graphs aim to serve as a robust resource for directional predicate entailments induced from textual corpora. ## chinese open relation extraction we build our ore method based on ddparser @xcite , a sota chinese dependency parser. we mine relation triples from its output by identifying patterns in the dependency paths. depending on the semantics of the head verb, instances of a dependency pattern can range from being highly felicitous to marginally acceptable as a relation. motivated by our downstream task of entailment graph construction, we go for higher recall and take them in based on the relation frequency assumption: less felicitous relations occur less frequently, and are less likely to take part in entailments when they do occur, thus they are negligible. due to the lack of a commonly accepted benchmark or criterion for "relations", we did not perform an intrinsic evaluation for our chinese ore method; its significant benefit to our eg zh graph, as shown in §7, should suffice to demonstrate its strength. ## chinese fine-grained entity typing as shown in previous work @xcite , the types of a predicate's arguments are helpful for disambiguating a predicate in context. to this end, we need a fine-grained entity typing model to classify the arguments into sufficiently discriminative yet populous types. @xcite presented cfet dataset, an ultra-fine-grained entity typing dataset in chinese. they labelled entities in sentence-level context, into around 6,000 free-form types and 10 general types. unfortunately, their free-form types are too fragmented for predicate disambiguation, and their general types are too ambiguous. we turn to figer @xcite , a commonly used type set: we re-annotate the cfet dataset with figer types through label mapping. given that there are around 6,000 ultra-fine-grained types and only 112 figer types (49 in the first layer), we can reasonably assume that each ultrafine-grained type can be unambiguously mapped to a single figer type. for instance, the ultra-finegrained type "湖 (lake)" is unambiguously mapped to the figer label "location / body_of_water". based on this assumption, we manually create a mapping between the two, and re-annotate cfet dataset with the mapping. we call the re-annotated dataset cfiger, as it is the first in chinese with figer labels. as with cfet, this dataset consists of 4.8k crowd-annotated data (equally divided into macro f1 (%) dev test cfet with cfet dataset -24.9 cfet with cfiger dataset 75.7 75.7 hiertype with figer dataset -82.6 hiertype with cfiger dataset 74.8 74. 5 for training set we combine the crowd-train and wikipedia subsets; for dev and test sets we use crowd-dev and crowd-test respectively. we train two baseline models: cfet, the baseline model with cfet dataset; hiertype @xcite , a sota english entity typing model. results are shown in for downstream applications, we nevertheless employ the hiertype model, as empirically it generalizes better to our news corpora. as shown in later sections, the resulting fet model can substantially help with predicate disambiguation. ## the chinese entailment graph we construct the chinese entailment graph from the webhose corpus foot_6 , a multi-source news corpus of 316k news articles, crawled from 133 news websites in october 2016. similarly to the newsspike corpus used in @xcite , the webhose corpus contains multi-source non-fiction articles from a short period of time. this means it is also rich in reliable and diverse relation triples over a focused set of events, ideal for building entailment graphs. we cut the articles into sentences by punctuations, limiting the maximum sentence length to 500 characters (the maximum sequence length for chinese bert). we discard the sentences shorter than 5 characters, and the articles whose sentences are all shorter than 5 characters. after applying the filter, we are left with 313,718 articles, as shown in for these 314k valid articles in webhose, we get their corenlp pos tags and feed them into our ore method in §3, to extract the open relation triples. then, with hiertype model @xcite in §4, we type all arguments of the extracted relations; we type each predicate with its subjectobject type pair, such as person-event or food-law; following previous work, we consider only the firstlayer figer types; when multiple type labels are outputted, we consider all combinations as valid types for that predicate. we finally employ the entailment graph construction method in @xcite , taking in only binary relations 10 . the detailed statistics of our chinese entailment graph are shown in we have also considered using another larger corpus, the clue corpus, for building the chinese entailment graphs, but couldn't finish due to limits on computational resources foot_7 . the larger corpus is built by @xcite , which is eight times the size of the webhose corpus and is originally intended for training chinese language models. we provide the typed relation triples extracted from the clue corpus as a part of our release, and encour-10 we encourage interested readers to also check appendix d for a brief introduction to @xcite . ## results and discussions to measure the performance of our constructed chinese entailment graphs, we follow previous work in reporting the precision-recall (p-r) curves plotted for successively lower confidence thresholds, and their area under curves (auc), for the range @xmath1. a language-specific lemma baseline sets the left boundary of recall, by exact match over the lemmatized premise / hypothesis. for our chinese entailment graph (eg zh ) and its baselines, the boundary is set by chinese lemma baseline. for the ensembles, in order to get commensurable auc values with previous work instead of being overoptimistic, we use the english lemma baseline. ## conclusion we have presented a pipeline for building chinese entailment graphs. along the way, we proposed a novel high-recall open relation extraction method, and built a fine-grained entity-typing dataset via label mapping. as our main result, we have shown that: our chinese entailment graph is comparable with english graphs, where unsupervised bert baseline did poorly; an ensemble between chinese and english entailment graphs substantially outperforms both monolingual graphs, and sets a new sota for unsupervised entailment detection. directions for future work include multilingual entailment graph alignment and alternative approaches for predicate disambiguation. a a brief summary of @xcite in in each box, at top is an example sentence, presented in chinese and its english metaphrase (inflection ignored); below are the relations they extract.