Alignment-Augmented Consistent Translation for Multilingual Open Information Extraction (2022)

Abstract: Progress with supervised Open Information Extraction (OpenIE) has been primarily limited to English due to the scarcity of training data in other languages. In this paper, we explore techniques to automatically convert English text for training OpenIE systems in other languages. We introduce the Alignment-Augmented Constrained Translation (AACTrans) model to translate English sentences and their corresponding extractions consistently with each other — with no changes to vocabulary or semantic meaning which may result from independent translations. Using the data generated with AACTrans, we train a novel two-stage generative OpenIE model, which we call Gen2OIE, that outputs for each sentence: 1) relations in the first stage and 2) all extractions containing the relation in the second stage. Gen2OIE increases relation coverage using a training data transformation technique that is generalizable to multiple languages, in contrast to existing models that use an English-specific training loss. Evaluations on 5 languages — Spanish, Portuguese, Chinese, Hindi and Telugu — show that the Gen2OIE with AACTrans data outperforms prior systems by a margin of 6-25% in F1.

## introduction open information extraction (openie) is the task of converting unstructured text to semi-structured tuples of the @xmath0; relation; object>, where these three components are textual phrases, broadly extracted from the original text @xcite . openie tuples have shown utility in various downstream tasks @xcite like question answering @xcite , machine reading @xcite * denotes equal contribution 1 code and models released at github.com:dair-iitd/moie 2010), multi-document summarization @xcite , schema induction @xcite , and knowledge base construction @xcite . with widespread adoption of deep learning in nlp, open information extraction (openie) systems have gone through a paradigm shift from using rule-based, statistical systems to supervised neural models. however, both types of systems have been limited to only a few languages -earlier systems required language-specific openie insights, and current systems require annotated training corpus that pose a barrier, particularly for lowresource languages. related tasks such as semantic role labeling face similar challenges in extending to multiple languages. x-srl @xcite addresses this by automatic translation of english sentences to the target language followed by label projection to infer the semantic role labels in the translated sentence. however, translating the sentence alone may be insufficient for openie because the generated tuples (also referred to as extractions) can include additional words absent in the sentence or require some changes to the word morphology used in the sentence. although less prevalent in english, these characteristics need to be addressed in other languages. x-srl approach may be extended such that each extraction can also be automatically translated and subject, relation, object labels projected from english extractions. however, independent translation of sentence and extraction may introduce unwanted lexical (e.g. synonyms) or semantic (e.g., change in gender) variations between the translations, as shown in to maintain consistency between translations of a sentence and its extractions, both the trans-lexical inconsistency english sentence english extraction spanish sentence spanish extraction (indp) spanish extraction (const) the shield of athena parthenos, sculpted by phideas, depicts a fallen @xmath1, sculptado por phideas, representa un amazonas @xmath2 the discovery was remarkable as the skeleton was almost identical to a modern @xmath3: openie examples transferred from english to spanish, using both independent (indp) and consistent (const) translations. independent translation results in inconsistencies which may have the same meaning (by using synonyms, fallecido vs. caído) or may change the meaning (changing gender from male to female, moderno to moderna). consistent translation avoids these issues, resulting in better quality of training data. our major contributions are that we: ## related work our work is in line with the recent trend of extending ie and knowledge-based nlp systems to multiple languages. recent works have explored distantly supervised relation extraction @xcite , knowledge-base completion @xcite , and fact linking @xcite . our focus is openie. many of the prior openie systems, both nonneural @xcite , openie-5 @xcite , clausie @xcite ) and neural (rnnoie @xcite , openie-6 @xcite ) have been deployed for english. moreover, ope-nie systems built for other languages often work only for a single language due to their reliance on language-specific resources. for example, @xcite al. (2018); rahat and talebpour (2018); romadhony et al. (2018); guarasci et al. (2020 ); papadopoulos et al. (2021) focus on german, persian, indonesian, italian, and greek, respectively. claro et al. (2019) present the importance of and various challenges involved with building multilingual openie systems. neural models like logician @xcite and @xcite use language-specific training data. reliance on manually-annotated data or languagespecific resources makes it infeasible to develop systems for the plurality of languages in the world, due to the cost and effort involved. however, our automated data conversion method can handle even low-resource languages like telugu. non-neural systems such as predpatt @xcite and argoe @xcite work for multiple languages by using conll-x and universal dependency parses respectively, to extract predicate-argument structures. owing to their pipelined nature, their performance is below that of neural systems like multi foot_0 oie @xcite . multi 2 oie is a twostage labeling model that works for english, spanish and portuguese. gen2oie extends this 2stage design to the generative paradigm which allows for better modeling of the openie task. the underlying mbert encoder in multi 2 oie allows for cross-lingual generalization across various languages even after training with only english supervised data. however, dependence on zero-shot generalization limits the performance of the model. two types of methods have been proposed for constraining the outputs of the machine translation systems: 1) altering the decoding algorithm @xcite , or 2) modifying the training methodology @xcite . we follow the second approach for constraining translations by aactrans to be consistent to that of a reference sentence. unlike prior work which focuses on constraining translations of few words, our task requires constraining the entire translation. we make use of awesome-align @xcite , an unsupervised word alignment technique @xcite , that outputs the alignment between words in sentences of two languages. awesome-align is trained using only parallel set of sentences in the two languages and generates aligned target words for each source word. transferring linguistic annotations from source to target language has been pioneered by @xcite and has been used in context of semantic role labeling @xcite and pos-tagging @xcite . after consistent translation, we make use of crosslingual projection @xcite , to transfer openie tags. ## notation for the transfer of openie data from one language to another, we represent the source language 2 as e and the target language as f. further, we use sent e and ext e to represent a sentence and extraction in the source language and aact-sent f and aactext f to represent the transferred sentence and extraction in the target language. to aid in the translation of extractions, we create a sub-sentence from each extraction by concatenating the phrases in all the fields of the extraction. the order of concatenation is such that the formed sub-sentence is grammatically valid. we refer to this sub-sentence as an ext-sentence and represent it as es l , where the subscript l represents its language. for most english extractions, the ext-sentence corresponds to concatenating the fields in the order of subject, relation and object. however, other languages may follow a different order or allow for multiple orders. we rely on the output of system that translates the english extsentence to determine the ext-sentence in other languages. moreover, each extraction can be seen as a labeling over the words of ext-sentence with either the subject, relation or object tags. tags for each word in the ext-sentence can also be regarded as the extraction. ## crosslingual data transfer in this section we describe the technique used to convert openie training data from source language e to a target language f. the source sentence, sent e , and all its corresponding extsentences, es e , are consistently translated to language f (section 4.1), and then, for each extraction in language e, ext e , the s, r or o labels are projected to the translated ext-sentence, es f , to form the extraction, ext f , in language f (section 4.2). figure 1 describes the pipeline with the help of an example. ## gen2oie model to train openie systems in multiple languages, we use a novel gen2oie model that extends the 2stage design of multi 2 oie @xcite to a generative paradigm. the first stage generates all possible relations and the second stage generates all extractions that contain a given relation. gen2oie can produce overlapping relations and multiple extractions containing the same relation, thus overcoming the limitations of multi 2 oie model. moreover, due to its generative nature, gen2oie can add new words or introduce changes in morphology that may be necessary for producing correct extractions, which cannot be achieved by labeling models. both the stages of the gen2oie (shown in figure 2 ) use seq2seq models as follows: stage-1 seq2seq: the input sentence is passed to the encoder and decoder generates a string formed by concatenating the set of relations from all the extractions, separated by an [sep] token. during training, the target relations are concatenated in the order in which they occur in the sentence. we find that a deterministic order is important for adding stability to the model training. ## confidence scoring the word log probabilities assigned by the stage-2 decoder can be summed up to be used as confidence score for the extractions generated by gen2oie. we experiment with using a separate model for obtaining the confidence scores. a sequence-labeling model is trained on each language's extractions with ext-sentence as input and s, r, o labels over the ext-sentence as the output. the log probabilities given by the sequencelabeling model to the labels predicted by the gen2oie model are summed up to get the new confidence scores. ## experimental setting we train openie systems in 5 languages, spanish (es), portuguese (pt), chinese (zh), hindi (hi) and telugu (te), by using the training data transferred from english to the respective language. for training the seq2seq models used in the data generation pipeline and the openie systems based on the gen2oie architecture, we choose either the mbart @xcite or mt5 @xcite model depending on the particular language. both of them are pre-trained multilingual seq2seq models that are trained with a span denoising objective on a large corpus of text containing many languages. mbart is pre-trained on cc25 and mt5 is pre-trained on mc4 corpus which contain text in 25 and 101 languages, respectively. since mbart does not support portuguese and telugu, we use mt5 for these two languages and mbart for the remaining 3 languages. we use the default hyperparameters recommended for these models and they are reported in appendix f. training datasets: for training the aactrans model, we make use of parallel english, language f sentences available in standard translation corpora using the method described in section 4. for spanish we use parallel sentences from europarl corpus @xcite , and for portuguese we use a subset of the paracrawl corpus @xcite , as chosen by @xcite . for hindi we use the iit-b corpus @xcite , and for telugu we use the samanantar corpus @xcite . for chinese we use the data released for wmt19 @xcite . we list the bleu scores of the various systems in appendix c. we use the oie4 training corpus from @xcite and transfer it to other languages for training openie systems. evaluation datasets and metrics: for evaluating translation systems we use the test sets available in the respective corpora and use sacrebleu @xcite as the metric. 3 for evaluating different ope-nie systems we use the optimal f1 and area under curve (auc) as computed by the carb (bhardwaj et al., 2019) scoring function. for spanish, portuguese openie we use test sets provided in @xcite . for chinese openie, we randomly choose 10% of the saoke dataset @xcite . in order to evaluate our method on medium and low resource languages, we release new openie test sets in hindi and telugu. human annotators who are fluent in both the language and are knowledgeable about the openie task translated about 300 randomly chosen sentences and their corresponding extractions from carb test set. they were paid $2.5 per sentence. ## experiments we perform experiments to answer the questions: the final performance of systems trained using it and with metrics defined for evaluating consistency? ## conclusion we develop a novel aactrans+clp pipeline for consistently transferring english openie examples to other languages and present a novel two-stage generative model, gen2oie, for training openie systems in various languages. we show improvements over the existing baseline of multi