The RELX Dataset and Matching the Multilingual Blanks for Cross-Lingual Relation Classification (2020)

Abstract: Relation classification is one of the key topics in information extraction, which can be used to construct knowledge bases or to provide useful information for question answering. Current approaches for relation classification are mainly focused on the English language and require lots of training data with human annotations. Creating and annotating a large amount of training data for low-resource languages is impractical and expensive. To overcome this issue, we propose two cross-lingual relation classification models: a baseline model based on Multilingual BERT and a new multilingual pretraining setup, which significantly improves the baseline with distant supervision. For evaluation, we introduce a new public benchmark dataset for cross-lingual relation classification in English, French, German, Spanish, and Turkish, called RELX. We also provide the RELX-Distant dataset, which includes hundreds of thousands of sentences with relations from Wikipedia and Wikidata collected by distant supervision for these languages. Our code and data are available at: https://github.com/boun-tabi/RELX

## introduction extracting useful information from unstructured text is one of the most essential topics in natural language processing (nlp). relation classification can help achieving this objective by enabling the automatic construction of knowledge bases and by providing useful information for question answering models @xcite . given an entity pair (e1, e2) and a sentence s that contains these entities, the goal of relation classification is to predict the relation @xmath0, which may include 'no relation' as well. for example, with the help of relation classification, we can create semantic triples such as @xcite ) from a sentence like "rocky mountain high school opened at its current location in 1973 and was expanded in 1994.", where @xcite are the given entities and 'founded' is the relation between them based on this sample sentence. traditionally, relation classification methods rely on hand-crafted features @xcite . lately, pretrained word embeddings @xcite with rnn-lstm architecture @xcite or transformers based models @xcite have gained more attention in this domain. although non-english content on the web is estimated as over 40% @xcite and the number of multilingual text-corpora is increasing @xcite , recent studies on relation classification have generally focused on the english language. these supervised approaches for relation classification are not easily adaptable to other languages, since they require large annotated training datasets, which are both costly and time-consuming to create. the challenge of creating manually labeled training datasets for different languages can be alleviated through cross-lingual nlp approaches. in cross-lingual relation classification, the objective is to predict the relations in a sentence in a target language, while the model is trained with a dataset in a source language, which may be different from the target language. for example, a cross-lingual relation classification model should be able to extract semantic triples such as (cd laredo, founded, 1927) from a spanish sentence like "cd laredo fue fundado en 1927 con el nombre de sociedad deportiva charlestón. foot_0 " for the given entities 'cd @xcite when the annotated training data is solely in english. thanks to multilingual pretrained transformer models like multilingual bert (mbert) @xcite and xlm @xcite , cross-lingual models have been studied in depth for several nlp tasks such as question answering @xcite @xcite , natural language inference @xcite @xcite , and named entity recognition @xcite . in this paper, we first propose a baseline crosslingual model for relation classification based on the pretrained mbert model 2 . then, we introduce an approach called matching the multilingual blanks to improve the relation classification ability of mbert in different languages with the help of a considerable number of relation pairs collected by distant supervision. prior works on cross-lingual relation classification use additional resources in the target language such as aligned corpora @xcite , machine translation systems @xcite , or bilingual dictionaries @xcite . our mbert baseline model does not require any additional resources in the target language. the matching the multilingual blanks model improves mbert by utilizing the already available wikipedia and wikidata resources with distant supervision. we present two new datasets for cross-lingual relation classification, namely relx and relx-distant. relx has been developed by selecting a subset of the commonly-used kbp-37 english relation classification dataset @xcite and generating human translations and annotations in the french, german, spanish, and turkish languages. the resulting dataset contains 502 parallel test sentences in five different languages with 37 relation classes. to our knowledge, relx is the first parallel relation classification dataset, which we believe will serve as a useful benchmark for evaluating cross-lingual relation classification methods. relx-distant is a multilingual relation classification dataset collected from wikipedia and wikidata through distant supervision for the aforementioned five languages. we gather from 50 thousand upto 800 thousand sentences, whose entities have been labeled by the editors of wikipedia. the relations among these entities are extracted from wikidata. our contributions can be summarized as follows: the rest of the paper is organized as follows. the related work is discussed in section 2. the details about the datasets are presented in section 3. our mbert baseline model and the matching the multilingual blanks (mtmb) procedure are described in section 4. the experimental results for the mbert model and mtmb are presented in section 5. finally, we draw conclusions and discuss future work in section 6. ## related work in monolingual relation classification, traditional methods generally depend on hand-crafted features @xcite . after the introduction of word embeddings @xcite , many relation classification models used pretrained word embeddings with the rnn @xcite or cnn @xcite architectures. with the strong performance of transformer networks for various nlp tasks @xcite @xcite before the introduction of multilingual transformers @xcite @xcite , cross-lingual word embeddings have been widely used in zero-shot crosslingual transfer with word embedding alignments for different tasks such as named entity recognition @xcite and natural language inference @xcite . this approach has also been utilized for cross-lingual relation classification @xcite . however, recently, multilingual deep transformers have attracted lots of attention in several cross-lingual tasks such as question answering @xcite @xcite , natural language inference @xcite @xcite , and named entity recognition @xcite . to the best of our knowledge, we present the first transformer based approach for the task of cross-lingual relation classification. in addition, we introduce a multilingual distant supervision method to improve the baseline transformer model. @xcite use a similar approach for monolingual relation classification, called matching the blanks. for the pretraining process, they collect pairs of english sentences based on the shared entities, annotated by an entity linking system. on the other hand, we propose a multilingual approach that utilizes wikipedia and wikidata, which are already available for many languages and have been successfully used for tasks such as multilingual question answering @xcite and named entity recognition @xcite . most cross-lingual relation classification studies rely on parallel corpora, machine translation systems, or bilingual dictionaries. in @xcite , english labeled data are projected to korean with parallel corpora to train relation classification models in korean. @xcite apply a machine translation system to translate the sentence in a target language to a source language, so that a relation classification model trained with the source language can be used. @xcite make use of a generative adversarial network (gan) to transfer the feature representations from the source language to the target language with the help of machine translation systems. @xcite employ bilingual word embedding mappings trained with bilingual dictionaries to develop a cross-lingual relation classification model. in many studies, the multilingual ace05 @xcite relation classification dataset has been treated as cross-lingual for evaluation. ace05 includes data in english, arabic, and chinese; however, it is not freely available, and the number of relations is rather small, which is 6. in @xcite , a relation classification dataset for 6 languages with 53 relation types has been used, yet this dataset is not publicly available. in this paper, we release the relx dataset created with human annotations and the relx- ## the relx and relx-distant datasets in this work, the training set of kbp-37 @xcite is used as a source in the english language for training. for evaluation, we introduce and make publicly available the relx dataset in english, french, german, spanish, and turkish. we also present relx-distant, which we use for the pretraining procedure in the developed mtmb (matching the multilingual blanks) approach, explained in section 4.2. ## results we compare our monolingual relation classification results using kbp-37 and the cross-lingual results using relx. we report our results by taking the average scores of 10 runs to decrease the effect of high variance between different runs in bert as stated in @xcite . evaluation metric: we use (18+1)-way evaluation by taking directionality into account as used in @xcite . first, the f1 score of a relation is calculated by taking the micro average of f1's of both directions. then, the macro average of f1 scores of 18 relations is considered as our final score. ## conclusion in this paper, we addressed the cross-lingual relation classification task. first, we introduced two publicly available datasets: relx, a crosslingual relation classification benchmark for english, french, german, spanish, and turkish with parallel sentences and relx-distant, a multilingual dataset containing a large number of sentences with relations from wikipedia and wikidata collected via distant supervision. second, we proposed a baseline model with mbert and a new multilingual pretraining scheme with distant supervision called matching the multilingual blanks (mtmb). our experiments showed that mtmb significantly outperforms the mbert baseline on the monolingual and cross-lingual datasets. the improvement obtained by mtmb is higher in the lowresource settings for the source language. we also showed that better cross-lingual relation classification performance is obtained for target languages which are typologically similar to the source language. the performance for spanish is comparable to english (the source language in this study), while the lowest f1 scores are obtained for turkish. mtmb can be easily adopted to other languages by using our provided scripts foot_1 . the only requirement is the availability of wikipedia articles in the new target language. as future work, we plan to extend relx-distant to all the available languages in wikipedia. we will also investigate the effect of mtmb in different cross-lingual tasks such as natural language inference, named entity recognition, and question answering by using the extended relx-distant dataset. ramazan pala, and taha küc ¸ükkatırcı for their contributions on the translation and eltur.co for their quality check service. tubitak-bideb 2211-a scholarship program (to a.k.), and tuba-gebip award of the turkish science academy (to a.o.) are gratefully acknowledged.