Multilingual Entity and Relation Extraction Dataset and Model (2021)

Abstract: We present a novel dataset and model for a multilingual setting to approach the task of Joint Entity and Relation Extraction. The SMiLER dataset consists of 1.1 M annotated sentences, representing 36 relations, and 14 languages. To the best of our knowledge, this is currently both the largest and the most comprehensive dataset of this type. We introduce HERBERTa, a pipeline that combines two independent BERT models: one for sequence classification, and the other for entity tagging. The model achieves micro F1 81.49 for English on this dataset, which is close to the current SOTA on CoNLL, SpERT.

## introduction the majority of the -constantly growing -amount of openly accessible knowledge is locked in unstructured text, and hence inefficiently utilized by any systems. the nlp tasks related to this problem include information extraction, relation extraction, named entity retrieval, as well as joint entity and relation extraction. our contribution is twofold. first, we present smiler (samsung multilingual entity and relation extraction dataset): an open-domain corpus of annotated sentences, created for the joint entity and relation extraction task. with 1.1 m sentences, 36 relation types, and 14 languages, smiler seems to be both the largest and the most diversified corpus for the task in existence, to the best of our knowledge. the corpus was semi-automatically created from wikipedia and dbpedia, and partly checked by linguists. our second contribution is herberta -hybrid entity and relation extraction bert for a multi- * work done while at samsung r&d institute poland. lingual setting that consists of two independently pretrained bert models @xcite . the first one classifies the input sequence as belonging to one of our 36 relations (including no_relation). its output -the relation -is then fed to the second bert, together with the same input sequence. the second model performs entity tagging and outputs two spans for the two entities selected from the input sequence. our model is close to spert @xcite , the current state-of-the-art for joint entity and relation extraction, achieving micro f 1 81.49 for english on the presented dataset. the smiler corpus and the source code is available at https://github.com/samsungnlp/smiler/ . we believe that presenting a new, large and diversified dataset will be a valuable contribution to the joint task of entity and relation extractionboth for english and for the multilingual setting. information extraction information extraction systems collect knowledge that is locked in unstructured text, such as in our wikipedia articles. a nota-ble example is the never-ending language learner (nell), which was reading the web for almost 10 years, 2010-2019 @xcite . this semisupervised system was retrained continuously, with the use of the current knowledge, to collect new instances of a pre-defined set of entity types and relations, which also constitutes the final goal of our system. another well-known knowledge base is knowledge vault @xcite . the system extracts information from the internet (text, tabular data, page structure, human annotations) and combines it with information from freebase @xcite . a probabilistic inference system computes calibrated probabilities of fact correctness. instead of utilizing pre-defined entity and relation types, some systems use syntactic analysis: reverb @xcite , minie @xcite . another approach is to create a taxonomy with just one relation: isa, as in probase @xcite . relation extraction current models (heberta included) for relation extraction are based on bert @xcite , a multi-layer bidirectional transformer encoder @xcite . its main contribution is pre-training deep bidirectional representations from unlabeled text by joint conditioning on both left and right context, in all the layers. the current state-of-the-art in relation extraction are redn (on semeval-2010 task 8, nyt, and webnlg) @xcite and matching the blanks (on fewrel) @xcite . redn extracts the token embeddings (bert) from two different layers to represent the head and tail entities separately, in order to enhance learning reversible relations. next, it calculates a parameterized asymmetric kernel inner product matrix between all the head and tail embeddings of each token in a sequence. on the other hand, the model constructed by @xcite combines bert with extensions of harris' distributional hypothesis to relations, to build task-agnostic relation representations solely from entity-linked text. ## data our smiler corpus consists of 1.1 m annotated wikipedia sentences, and was created specifically for the task of entity and relation extraction. it consists of annotated sentences in 14 languages, with each language representing a subset of 36 relations. as far as we know, this is both the biggest and the most comprehensive corpus for these nlp tasks. figure 1 shows the number of sentences for each relation, for each language. relation sentence has-child [bill]e1 married hillary on october 11, 1975, and their only child, [chelsea]e2, was born on february 27, 1980. headquarters [amc airlines]e1 è una compagnia aerea egiziana con sede al [cairo]e2, esegue voli charter da sharm el-sheikh, hurghada, il cairo verso le maggiori capitali europee dal maggio 2006. moviehasdirector [lili]e1 ist ein us-amerikanischer spielfilm des regisseurs [charles walters]e2 aus dem jahr 1953. ## model our architecture herberta (figure 3 ) uses two pre-trained bert models. it solves the problem of entity and relation extraction with the use of an unconventional pipeline. in the first step it is trained for relation extraction, while the entities are retrieved in the second stage. the first model is bert, fine-tuned for the sequence classification task. its input is a tokenized sequence, while its output consists of a sequence output and a pooled output that represents the overall sentence context (from the [cls] token). the latter is passed to a softmax for classifying the relation. the second model is our implementation of bert for entity tagging. it is based on bert for question answering, and its inputs are: as a result, the input sequence is as follows: ## results for evaluating the models, we use the micro f 1 score on the entities and relations together, ignoring no_relation. below, this measure is referred to as combined. when comparing with the state-ofthe-art, we also show the relation-only micro f 1 , as well as the entity-only micro f 1 (for a single entity and for the pair of entities). we use such metrics because they are typically calculated for these tasks; see for instance @xcite . ## conclusions we have described our approach to solving the task of multilingual joint entity and relation extraction, by training our novel, herberta model on smiler -our large, comprehensive dataset. the model combines two independent bert models: one for sequence classification, and the other for entity tagging. regarding f1 measure our model is close to spert (the current state-of-the-art for conll04), the difference being 0.2 percentage point. what is more, the smiler dataset appears to be currently the largest (1.1 m annotated sentences) dataset for this task, as well as the most comprehensive one (14 languages, 36 relation types). we observe that our multilingual models achieve higher or similar results, compared to the models trained for each language separately. languages with less data can benefit the most from such multilingual models. the victim here is english, as it seems to be a giver of f1 to other languages, especially when the number of sentences for this language is significantly higher than for other languages. on the one hand, due to the large amount of data for english, a model is well trained for patterns existing in this language and thus other languages with less data can benefit from it (because of some similarities between languages). therefore, we observe increased results for less resourced languages. on the other hand, a model tries to accommodate the nuances of the less resourced languages. their features are noticeable for the model, which reduces the dominant role of english. as we can see from the results, each non-english language follows a slightly different (error) path. this does not seem to be a general rule that can be applied. we would like to point out though that this is exactly the reason why we have created this dataset in the first place. we wanted to observe the performance of models on different languages on the relation extraction task and now, thanks to our dataset, this is possible. the fact that there is no simple explanation for the difference in model performance shows that deeper analysis for each language is necessary. another observation is that we obtain significantly higher f 1 for entity 1 than for entity 2 , which suggests that entity 1 is simpler. this seems to be true, because entity 1 typically occurs at the beginning of the sentence, while entity 2 does not have any consistent location. in the future, we plan to train our model on the en-full dataset and to predict multiple plausible entity pairs for the same sentence. we would also like to extend the dataset to include entity types and sentences containing multiple relations. another promising direction is data augmentation by "cross-matching" entities and relations in the dataset with sentences in the dataset. this crossmatch could search for two cases. in both cases, the sentence could be added to the dataset multiple times.