Event-Guided Denoising for Multilingual Relation Learning (2020)

Abstract: General purpose relation extraction has recently seen considerable gains in part due to a massively data-intensive distant supervision technique from Soares et al. (2019) that produces state-of-the-art results across many benchmarks. In this work, we present a methodology for collecting high quality training data for relation extraction from unlabeled text that achieves a near-recreation of their zero-shot and few-shot results at a fraction of the training cost. Our approach exploits the predictable distributional structure of date-marked news articles to build a denoised corpus – the extraction process filters out low quality examples. We show that a smaller multilingual encoder trained on this corpus performs comparably to the current state-of-the-art (when both receive little to no fine-tuning) on few-shot and standard relation benchmarks in English and Spanish despite using many fewer examples (50k vs. 300mil+).

## introduction multilingual relation extraction is an important problem in nlp, facilitating a diverse set of downstream tasks from the autopopulation of knowledge graphs (e.g., @xcite ) to question answering (e.g., @xcite ). while early efforts in relation extraction used supervised methods that rely on a fixed set of predetermined relations, research has since shifted to the identification of arbitrary unseen relations in any language. in this paper, we present a method for extracting high quality relation training examples from date-marked news articles. this technique leverages the predictable distributional structure of such articles to build a corpus that is denoised (i.e., where sentences with the same entities only express the same relation, see figure 1 ). we use this corpus to learn general purpose relation representations and evaluate their quality on few-shot and standard relation extraction benchmarks in english and spanish with little to no task-specific fine-tuning, achieving comparable results to a significantly more data-intensive approach that is the current state-of-the-art. the current state-of-the-art model, "matching the blanks" or mtb @xcite , is a distant supervision @xcite technique that provides large gains on many relation extraction benchmarks and builds on harris' distributional hypothesis @xcite and its extensions @xcite . @xcite assume that the informational redundancy of very large text corpora (e.g., wikipedia) results in sentences that contain the same pair of entities generally expressing the same relation. thus, an encoder trained to collocate such sentences can be used to identify the relation between entities in any sentence s by finding the labeled relation example whose embedding is closest to s. while @xcite achieve state-of-the-art on fewrel @xcite and semeval 2010 task 8 @xcite , their approach relies on a huge amount of data, making it difficult to retrain in english or any other language with standard computational resources: they fine-tune bert large @xcite , which has 340mil parameters, on 300mil+ relation pair statements with a batch size of 2, 048 for 1mil steps. in contrast our method, with only 50k relations statements and a language-model one-third the size, achieves comparable performance when fine-tuned on little to no task-specific data. figure 1 : event-guided denoising: using date-marked events from wikipedia to select a diverse group of sentences that describe the same relation () from news articles, filtering out noisy examples () our main contribution is a distant supervision approach in which we assume that sections of news corpora exhibit even more informational redundancy than wikipedia. specifically, news in the days following an event (e.g., the 2006 world cup) frequently re-summarizes the event before adding new details. as a result, news exhibits a strong form of local consistency over short rolling time windows where otherwise fluid relations between entities remain fixed. for example, the relation between italy and france as expressed in a random piece of text is dynamic and context-dependent, spanning a wide range of possibilities that include "enemies", "neighbors" and "allies". but, in the news coverage following the 2006 world cup, it is static -they are sporting competitors. therefore, by considering only sentences around specific events, we extract groups of statements that express the same relation and are relatively free of noise (i.e. statements that express a different relation, see figure 1 ). training multilingual bert (mbert) on our denoised corpus yields relation representations that adapt well to resource-constrained downstream tasks: we evaluate their quality on fewrel and semeval 2010 task 8, producing near state-of-the-art results when finetuned on little to no task-specific data. in addition to the strong performance of our approach in english, it is easily generalizable to other languages, requiring only news corpora and event descriptions from wikipedia to build a high-quality training corpus. we evaluate this in spanish and find our method outperforms mbert on the tac kbp 2016 relation corpus @xcite . we share our code to allow other researchers to apply our approach to news corpora of their own. foot_0 ## methods we focus on the task of learning to produce high quality general-purpose relation representations from relation statements in english and spanish. we adopt the common definition of a relation statement in the literature: a triple @xmath0, s 1 , s 2 ) where @xmath1... x n ] is a sequence of tokens and s 1 = (i, j) and s 2 = (k, l) are the indices of special start and end identifier tokens that demarcate two entity mentions in x. our goal is to learn a function f (r) = h that maps a relation statement to a fixed-length vector @xmath2. the vector h represents the relation between the entity mentions identified by s 1 and s 2 as expressed in x. the cosine similarity between f (r) and f (r o ) should be close to 1 if and only if r and r o express the same relation. that is to say, f should collocate sentences that exhibit similar relations. ## conclusion we present an event-guided denoising approach for relation extraction corpus creation that, when used with the current state-of-the-art training procedure, achieves comparable results in english under a lowresource regime for only a fraction of the training cost. it also performs well in spanish, demonstrating its adaptability to resource-constrained relation extraction tasks in non-english languages. our technique affords the broader research community the ability to approximate the current state-ofthe-art in relation extraction by significantly lowering its associated training costs. however, it requires a fairly large date-marked news corpus which may not be available in low resource languages. we leave an exploration of broader language coverage and minimal required corpus size for future work. one promising direction for expanding language coverage is cross-lingual learning via "codeswitched" examples and other language modeling losses (e.g. (lample and conneau, 2019)). we hypothesize that such methods could help knowledge transfer among languages and improve results on downstream tasks. finally, we note that since our approach extracts relation statements from news corpora, it is likely that the resulting distribution of underlying relation types is different than the distribution found in wikipedia. for example, wikipedia may contain more expressions of standard ontological relations (e.g., father-of, born-in) characteristic of factoids. despite this hypothesized difference, our approach performs well on both fewrel and semeval 2010 task 8, both of which include a subset of such relation types. in the future we intend to investigate these differences and their implications more closely.