M ultiˆ2 OIE : Multilingual Open Information Extraction Based on Multi-Head Attention with BERT (2020)

Abstract: In this paper, we propose Multi ^2 OIE, which performs open information extraction (open IE) by combining BERT with multi-head attention. Our model is a sequence-labeling system with an efficient and effective argument extraction method. We use a query, key, and value setting inspired by the Multimodal Transformer to replace the previously used bidirectional long short-term memory architecture with multi-head attention. Multi ^2 OIE outperforms existing sequence-labeling systems with high computational efficiency on two benchmark evaluation datasets, Re-OIE2016 and CaRB. Additionally, we apply the proposed method to multilingual open IE using multilingual BERT. Experimental results on new benchmark datasets introduced for two languages (Spanish and Portuguese) demonstrate that our model outperforms other multilingual systems without training data for the target languages.

## introduction open information extraction (open ie) @xcite aims to extract a set of arguments and their corresponding relationship phrases from natural language text. for example, an open ie system could derive the relational tuple (was elected; the republican candidate; president) from the given sentence "the republican candidate was elected president." because the extractions generated by open ie are considered as useful intermediate representations of the source text @xcite , this method has been applied to various downstream tasks @xcite @xcite . bidirectional-lstm bidirectional-lstm figure 1 : comparison between existing extractors and the proposed method. we use bert for feature embedding layers and as a predicate extractor. predicate information is reflected through multi-head attention instead of simple concatenation. @xcite @xcite , most recent open ie research has focused on deep-neural-network-based supervised learning models. such systems are typically based on bidirectional long short-term memory (bilstm) and are formulated for two categories: sequence labeling @xcite @xcite and sequence generation @xcite @xcite . the latter enables flexible extraction; however, it is more computationally expensive than the former. additionally, generation methods are not suitable for non-english text owing to a lack of training data because they are heavily dependent on in-language supervision @xcite . therefore, we adopted the sequence labeling method to maximize scalability by using (multilingual) bert @xcite and multi-head attention @xcite . the main advantages of our approach can be summarized as follows: experimental results on two english benchmark datasets called re-oie2016 @xcite and carb @xcite show that our model yields the best performance among the available sequence-labeling systems. additionally, it is demonstrated that the computational efficiency of multi 2 oie is far greater than that of sequence generation systems. for a multilingual experiment, we introduce multilingual open ie benchmarks (spanish and portuguese) constructed by translating and re-annotating the re-oie2016 dataset. experimental results demonstrate that the proposed multi 2 oie outperforms other multilingual systems without additional training data for non-english languages. to the best of our knowledge, ours is the first approach using bert for multilingual open ie foot_0 . the code and related resources can be found in https: //github.com/youngbin-ro/multi2oie. ## proposed method multi 2 oie extracts relational tuples from a given sentence in two steps. the first step is to find all predicates in the sentence. the second step is to extract the arguments associated with each identified predicate. the architecture of the proposed model is presented in figure 2 . ## multilingual performance as mentioned in section 2.2, we trained a multilingual version of multi 2 oie using multilingual bert and the same training dataset as the english version. we assumed that data for non-english languages were not available and tested the model's zero-shot performance. evaluations were conducted using a dataset generated based on the re-oie2016 dataset. ## conclusion in this paper, we propose multi 2 oie, which exploits bert and multi-head attention for the open ie task. multi-head attention has the advantage of fusing sentence and predicate features, which adequately reflect predicate information throughout a sentence. our model achieved the best performance among sequence labeling models. multi 2 oie also exhibited superior computational efficiency with competitive performance compared to the stateof-the-art sequence generation systems. additionally, a multi 2 oie model trained using multilingual bert, outperformed the baseline models without training on any non-english languages. however, some types of extractions, such as nominal relations, conjunctions in arguments, and contextual information, are not considered in multi 2 oie. future work could investigate how to apply multi 2 oie to these cases. for multilingual open ie, performance evaluations and further study on non-alphabetic languages that were not considered in this study can be conducted.