PENELOPIE : Enabling Open Information Extraction for the G reek Language through Machine Translation (2021)

Abstract: In this work, we present a methodology that aims at bridging the gap between high and low-resource languages in the context of Open Information Extraction, showcasing it on the Greek language. The goals of this paper are twofold: First, we build Neural Machine Translation (NMT) models for English-to-Greek and Greek-to-English based on the Transformer architecture. Second, we leverage these NMT models to produce English translations of Greek text as input for our NLP pipeline, to which we apply a series of pre-processing and triple extraction tasks. Finally, we back-translate the extracted triples to Greek. We conduct an evaluation of both our NMT and OIE methods on benchmark datasets and demonstrate that our approach outperforms the current state-of-the-art for the Greek natural language.

## introduction open information extraction (oie) techniques generally shine in high-resource languages (e.g. english, german) for which either linguistic principles leading to triple extraction have been identified or large annotated corpora and pretrained language models can be used. for lowresource languages like modern greek however, there is a relative sparsity of raw textual resources and annotated corpora that could lead to the development of similar systems. on the bright side, the need for multilingual resources (e.g. movie subtitles, applications, web content) has fueled several projects of compiling parallel corpora (i.e. collections of texts translated into one or more other languages than the original) over the last years. in this work, we propose a methodology that aims at enabling oie for low-resource languages, focusing on the greek oie use case. to achieve this, we rely on neural machine translation (nmt) as an intermediate step to translate the texts to english, in order to exploit the plethora of methods that exist for transforming english text to its structured representation. we present penelopie (parallel en-el open information extraction), a pipeline for information extraction from greek corpora. an overview of our methodology is given in figure 1 . the code and related resources can be found in https://github.com/lighteternal/pene lopie . our study has the following objectives: ## background in this section, we provide background information on neural machine translation and open information extraction approaches. ## methodology the nmt architecture used in this paper for english-to-greek (en2el) translation and greekto-english (el2en) back-translation is a variant of the transformer model @xcite , driven by the fact that self-attentional networks tend to perform distinctly better than other architectures on translation tasks @xcite . both the encoder and the decoder are composed of stacked, multi-head, self-attention and fully-connected layers. one key difference between the two implementations is that ours includes a fully connected feed-forward network with an inner-layer dimensionality of 𝑑 𝑓𝑓 = 1024, as opposed to the original one that uses a hidden layer with 𝑑 𝑓𝑓 = 2048, in an effort to reduce computational cost, as our training testbed had limited memory capabilities. with regard to vocabulary construction, we relied on subword units extracted with bpe, experimenting with two different configurations of merge operations our approach for efficient open information extraction on translated texts combines a series of distinct modules for in-place coreference resolution, extractive summarization and parallel triple extraction with the following specifications: coreference resolution: we rely on a variant of the pretrained end-to-end coreference resolution model from @xcite using span-bert embeddings @xcite , trained on the ontonotes 5.0 dataset. each translated sequence is pre-processed by the in-place coreference resolution component, where all noun phrases (mentions) referring to the same entity are substituted with that entity. summarization: extractive text summarization is used on the coreference-resolved text to reduce the original documents' length by omitting peripheral information while highlighting key features that are appropriate for triple extraction. we use the transformer-based implementation from @xcite where all sentences are embedded into the multi-dimensional space using 1 https://github.com/moses-smt/mosesdecoder bert embeddings. k-means clustering is then used on the sentence representations to identify those closest to the cluster's centroids for summary selection. parallel triple extraction: here we combine three popular oie systems, relying both on rulebased (handcrafted extraction heuristics and clauses) and learning-based (semantic role labelling and sequence bio tagging) systems, relying on the complementarity between the different approaches to ensure maximum recall. we provide additional information regarding the technical implementation of the described information extraction pipeline in the following section. ## results and discussion we provide results both for the el-en nmt tasks and for the oie task on greek corpora, since the former can be evaluated independently. ## conclusions and future work we have presented the use of nmt models integrated in an oie pipeline to achieve triple extraction for low-resource languages, showcasing our approach on the greek language. to this end, we trained 4 models (2 en2el and 2 el2en) that outperform the state-of-the-art by a significant margin (>10 bleu) and made them publicly available. we leveraged these along with a set of preprocessing and triple extraction tools to construct the penelopie pipeline aiming at information extraction from greek texts. we demonstrated the efficiency of our methodology via a benchmark framework and obtained significantly better results (+116% in f1-score) compared to the best multilingual oie system currently available. for future work, we will focus more on wordlevel alignment to improve the quality of our extractions. we would also like to explore transfer learning approaches to create an end-to-end oie system for greek without relying on annotated datasets.