D i S - R e X : A Multilingual Dataset for Distantly Supervised Relation Extraction (2022)

Abstract: Our goal is to study the novel task of distant supervision for multilingual relation extraction (Multi DS-RE). Research in Multi DS-RE has remained limited due to the absence of a reliable benchmarking dataset. The only available dataset for this task, RELX-Distant (Köksal and Özgür, 2020), displays several unrealistic characteristics, leading to a systematic overestimation of model performance. To alleviate these concerns, we release a new benchmark dataset for the task, named DiS-ReX. We also modify the widely-used bag attention models using an mBERT encoder and provide the first baseline results on the proposed task. We show that DiS-ReX serves as a more challenging dataset than RELX-Distant, leaving ample room for future research in this domain.

## introduction relation extraction (re) identifies the relation r between a pair of entities (e 1 , e 2 ) given some text mentioning both of them. to avoid large manual annotation, re is often trained via distant supervision (ds-re) @xcite . ds-re uses facts r(e 1 , e 2 ) in an existing kb to associate a label r with the bag containing all sentences that mention e 1 and e 2 . research in ds-re has been mostly monolingual and limited to english. our goal is to study multilingual re via distant supervision (multi ds-re). we expect multilingual re models to have several benefits over monolingual re. first, training data from multiple languages may be pooled to create a large dataset, enabling cross-lingual knowledge transfer @xcite . second, it may encourage re models to be consistent across languages @xcite , e.g., extraction of a fact already seen in one language should be easier in another. to the best of our knowledge, relx-distant @xcite ) is currently the only * equal contribution dataset available for multi ds-re, but even so, it has never been evaluated as a benchmark for the task. our analysis reveals that it suffers from a poor selection of relation classes. firstly, there are no examples of na class (sentences with no relation between the two entities). therefore, a model trained on relx-distant would find limited utility in any real world setting. secondly, its choice of relation classes is highly disjoint, resulting in an absence of instances with multiple labels (unusual for a ds-re dataset). finally, it is highly imbalanced -even though it has 24 relation classes, over 50% bags belong to just one "country" relation. owing to these attributes, we observe that models trained on relx-distant end up classifying the instances of the minority class based on just the entity type information. due to high skew, such mistakes have negligible impact on evaluation scores and the model achieves an auc of 0.99 after only 5 training epochs. such numbers are unheard of, especially when compared to benchmarking datasets in mono-lingual re (mono-lingual variant of the same architecture obtains an auc of 0.83 when trained and tested on the gds dataset @xcite . in response, we contribute a more realistic benchmark dataset for the task called dis-rex. our dataset has over 1.8 million sentences in four languages: english, french, spanish and german. it has 37 relation types including 1 no-relation (na) class and also has instances with multiple labels similar to the widely-used new york times (nyt) dataset for english ds-re @xcite , thus comparing favorably to relx-distant. we also adopt state-of-the-art ds-re models in the multilingual setting by using the mbert encoder @xcite , to create a strong baseline for this task. we achieve an auc of 0.82 and a micro-f1 of 0.76, suggesting that the dataset is not trivial to optimize on, and could act as a good benchmark supervised re datasets such as ace05 @xcite and klue @xcite are generally small, owing to the supervision needs per relation. distant supervision @xcite is a popular alternative to large-scale human annotation, but necessitate more complex models to handle dataset noise. the standard english ds-re dataset is new york times (nyt) corpus @xcite , which has served as the benchmark for research over the years. ds-re models have evolved to use multi-instance learning @xcite , multi-label learning @xcite , corrections for false negatives @xcite , and neural models such as piecewise cnns @xcite , intra-bag attention @xcite , and reinforcement learning @xcite . @xcite and @xcite propose extensions of bag-attention models for bilingual (english-mandarin) datasets. however, their adoption to multiple languages has been lacking, due to absence of a reliable multilingual dataset. although relx-distant is the only multilingual ds-re dataset so far, it wasn't originally used for multi ds-re task but to pre-train a model that gets fine-tuned for supervised re task. contemporary to our work, other multilingual re datasets and methods are being developed. these include a dataset for joint entity and relation extraction @xcite , a model for multilingual kb completion @xcite , and an approach for automatic construction of crosslingual training data for open ie @xcite . our proposed dataset, dis-rex, has already been used for further research on the multilingual ds-re task (rathore et al., 2022). 1 https://github.com/dair-iitd/dis-rex ## dataset curation all distant supervision datasets are curated by aligning known kb facts with sentences in a large corpus. we follow the same for dis-rex, while paying attention to cross-lingual normalization, and overall data and language statistics. first, we harvest a large number of sentences from english, french, spanish and german wikipedias. 2 we use dbpedia language editions @xcite for our kb -this gives us good coverage of entities that are local to different language speakers. dbpedia entities are associated with wikidata ids, which are normalized across languages. this enables us to fuse these dbpedia kbs and establish equivalence between entities like usa and estados unidos de américa. next, we use a language-specific ner tagger, (we use the md variant of spacy @xcite ner taggers for each language), returning a rich set of sentences. in contrast, relx-distant finds entity mentions using wikipedia hyperlinks. this severely limits its pool of sentences, since often only the first mention of an entity in a wiki document has a hyperlink while others do not. linking each mention with its entity can be challenging, due to unavailability of high-quality entity linking software for every language. we take the pragmatic approach of using simple string matching, but only on the subset of entities that have an unambiguous surface form (or alias) in our fused kb. this maintains scalability to many languages, while ensuring high enough precision of linking. for each entity-pair, we create a languagespecific bag of all sentences that mention both. we also search for all relations between them in our fused kb. we associate the bag with all those relation labels, or "na", if no relation is found. our next steps select a balanced subset of this dataset, so that it can serve as a good benchmark for multi ds-re. we first select the subset of relations that have at least 50 bags in all languages. this yields the 36 positive relation types used in our data. for each relation type, we limit the number of bags in a language to a max of 10,000. this helps curb the skew due to highly frequent relations such as country and birthp lace. during this filtering, we ensure that bags of entity pairs common across more than one language are not removed, so that we have an abundant number of cross-lingual bags. models can take advantage of such bags for establishing representation consistency across languages @xcite . finally, we add bags of entity pairs that have no relation between them. similar to nyt dataset, "na" is the majority class in dis-rex (kept at roughly 70%). hence, we obtain a dataset with over 1.8 million sentences, and over 250,000 (non-na) bags (see we estimate the percentage of bags satisfying "at-least one" assumption by manually labelling sentences across 50 randomly selected bags. we find that 82% of the bags satisfy "at-least one" assumption. for the test set of nyt corpus, this percentage is close to 62% @xcite finally, we create train-dev-test splits by splitting the bags in the ratio 70 : 10 : 20. while splitting we ensure that entity-pairs in three sets are mutually exclusive, so the model does not extract by memorizing a fact. ## conclusion we propose dis-rex, a novel dataset for multi ds-re in 4 languages. we show that it is a more realistic and challenging benchmark compared to the existing dataset. dis-rex has a fairly wellrepresented distribution of relation types, includes instances with no-relation between entity-pairs and the relation-types selected show several real-world characteristics like inverse relations, different relations with high overlap, etc. we also publish first baseline numbers on the task of multi ds-re by extending existing state-of-the-art models. a detailed analysis of model performance suggests several research challenges for future: (1) learning language-agnostic sentence embeddings, (2) robustness to related relations (inverse; overlapping but semantically different), and (3) handling multi-label entity-pairs. recently, rathore et al. (2022) develop a multilingual ds-re model named pare, which reports improved performance on the dis-rex dataset.