Cross-Lingual Semantic Role Labeling with High-Quality Translated Training Corpus (2020)

Abstract: Many efforts of research are devoted to semantic role labeling (SRL) which is crucial for natural language understanding. Supervised approaches have achieved impressing performances when large-scale corpora are available for resource-rich languages such as English. While for the low-resource languages with no annotated SRL dataset, it is still challenging to obtain competitive performances. Cross-lingual SRL is one promising way to address the problem, which has achieved great advances with the help of model transferring and annotation projection. In this paper, we propose a novel alternative based on corpus translation, constructing high-quality training datasets for the target languages from the source gold-standard SRL annotations. Experimental results on Universal Proposition Bank show that the translation-based method is highly effective, and the automatic pseudo datasets can improve the target-language SRL performances significantly.

## introduction semantic role labeling (srl), which aims to capture the high-level meaning of a sentence, such as who did what to whom, is an underlying task for facilitating a broad range of natural language processing (nlp) tasks @xcite @xcite @xcite . currently, the majority of research work on srl is dedicated to the english language, due to the availability of large quantity of labeled data. with this regard, cross-lingual srl, especially the one transferring the advantage of the source language with affluent amount of resources (e.g., english) to the target language where the labeled data is scarce or even not available, is of great importance our method is in the dotted blue box. @xcite @xcite . previous work on cross-lingual srl can generally be divided into two categories: model transferring and annotation projection. the former builds cross-lingual models on language-independent features such as cross-lingual word representations and universal pos tags which can be transferred into target languages directly @xcite @xcite . the latter bases on a large-scale parallel corpus between the source and target languages where the source-side sentences are annotated with srl tags automatically by a source srl labeler, and then the source annotations are projected onto the target-side sentences in accordance of word alignments @xcite @xcite @xcite . in addition, the annotation projection can be combined with model transferring naturally. particularly, the projected srl tags in annotation projection could contain much noise because of the source-side automatic annotations. a straightforward solution is the translation-based approach, which has been demonstrated effective for cross-lingual dependency parsing @xcite @xcite . the key idea is to translate the gold-standard source training data into target language side by translation directly, avoiding the problem of the low-quality source annotations. fortunately, due to recent great advances of neural machine translation (nmt) @xcite , this approach could have great potentials for cross-lingual transferring. to this end, in this paper, we study the translation-based method for cross-lingual srl. figure 1 illustrates the differences between previous approaches. sentences of the source language training corpus are translated into the target language, and then the source srl annotations are projected into the target side, resulting in a set of high-quality target language srl corpus, which is used to train the target srl model. further, we merge the gold-standard source corpus and the translated target together, which can be regarded as a combination of the translation-based method and the model transferring. our baseline is a simple bilstm crf model by using multilingual contextualized word representations @xcite . for a better exploration of the blended corpus, we adopt a parameter generation network (pgn) to enhance the bilstm module, which can capture the language differences effectively @xcite . we conduct experiments based on universal proposition bank corpus (v1.0) @xcite over seven languages. first, we verify the effectiveness of our method on the single-source srl transferring, where the english language is adopted as the source language and the remaining are used as the target languages. results show that the translation-based method is highly effective for cross-lingual srl, and the performances are further improved when pgn-bilstm is used. further, we conduct experiments on the multi-source srl transferring, where for each target language all the remaining six languages are used as the source languages. the same tendencies as the single-source setting can be observed. we conduct detailed analysis work for both settings to understand our proposed method comprehensively. in summary, we make the following two main contributions in this work: all codes and datasets are released publicly available for the research purpose foot_0 . ## related work there exists extensive work for cross-lingual transfer learning @xcite @xcite @xcite @xcite @xcite . model transferring and annotation projection are two mainstream categories for the goal. the first category aims to build a model based on the source language corpus, and then adapt it to the target languages @xcite @xcite . the second category attempts to produce a set of automatic training instances for the target language by a source language model and a number of parallel sentences, and then train a target model on the dataset @xcite @xcite @xcite . for cross-lingual srl, annotation projection has received the most attention @xcite . a range of strategies have been proposed to enhance the srl performance of the target language, including improving the projection quality @xcite , joint learning of syntax and semantics @xcite , iterative bootstrapping to reduce the influence of noise target corpus @xcite , and joint translation and srl @xcite . our work is mainly inspired by the recent work of treebank translation of cross-lingual dependency parsing @xcite @xcite @xcite @xcite , which is referred to as the translation-based approaches. these approaches directly project the gold-standard annotation into the target side, alleviating the problem of erroneous source annotations in standard annotation projection. in addition, we combine the approach with model transferring, which has been concerned little for cross-lingual srl. the model transferring benefits much from the recent advance of crosslingual contextualized word representations @xcite . the development of universal annotation schemes for a variety of nlp tasks can greatly facilitate cross-lingual srl, including pos tagging @xcite , dependency parsing @xcite , morphology @xcite and srl @xcite . our work makes use of the publicly available universal proposition bank (upb) @xcite , which annotates the predicate and semantic roles following the frame and role schemes of the english proposition bank 3.0 @xcite . supervised srl models are also closely related to our work @xcite @xcite . a great deal of work attempts for an endto-end solution with sophistical neural networks, detecting the predicates as well as the corresponding argument roles in one shot @xcite @xcite . also there exist a number of studies which aims for adapting various powerful features for the task @xcite . in this work, we exploit a multilingual pgn-bilstm model @xcite with contextualized word representations @xcite , which can obtain state-of-the-art performance for cross-lingual srl. ## srl translation we induce automatic target data from the goldstandard source data by full translation and then project the srl predicates and arguments into their corresponding words by aligning, producing the final translated srl corpus for the target language automatically. the method has been demonstrated effective for cross-lingual dependency parsing @xcite @xcite . compared with annotation projection, we can ensure the annotation quality at the source side, thus higher quality target corpus is also expected. in addition, dependency-based srl could benefit more by this method, as only predicate words and their arguments are required to be projected into the target side, while dependency parsing should concern all sentential words. the overall process is accomplished by two steps: translating and projecting. translating. first, we use a state-of-the-art translation system to produce the target translations for the sentences of the source srl data. give a source sentence e 1 • • • e n , we translate it into f 1 • • • f m of the target language. it is worth noting that the recent impressive advances in nmt @xcite facilitate our work greatly, which enables our method to have highquality translations. projecting. then we incrementally project the corresponding predicates or arguments of a source sentence e 1 • • • e n to its target f 1 • • • f m . we adopt two kinds of information to assist the projection: we focus on srl-related words of the source sentence only, and perform the process gradually at the predicate level. for each predicate in a sentence, we collect the predicate word as well as its role words, and then project their role labels into the target sentence. formally, for each of these words (i.e., e i ), we have the srl role tag r e i as well as its pos tag t e i , both of which have been already annotated in the upb. first, we find its target word f j with the highest alignment probability, regarding the word f j as the corresponding projection carrying the semantic role r e i . then we calculate the confidence score of this projection by the following formula: the one-one target-source alignment 2(a) is the ideal condition of the projection. however, there could be many-to-one cases for the given words, leading to semantic role conflicts at the target language words. for these cases, we take precedence for the predicate projections, and otherwise keep only the highest confidence projections. figure 2(b ) shows a predicate-argument conflict example, where the predicate projection is reserved, and figure 2(c ) shows an argument-argument conflict example where the projection with the higher confidence score is reserved. finally, we set a threshold value α to remove low confidence projections. if the confidence score of a predicate projection is below α, all the roles of this predicate are removed as well. for the argument projections whose confidence is below α, we remove the single arguments directly, with no influence on the other projections. ## the srl model in this work, we focus on dependency-based srl, recognizing semantic roles for a given predicate @xcite . the task can be treated as a standard sequence labeling problem, and a simple multi-layer bilstm-crf model is exploited here, which has archived state-of-the-art performance with contextualized word representations @xcite @xcite . in particular, we adapt the model to better support multilingual inputs by using a pgn module on the bilstm @xcite . figure 3 shows the overall architecture. ## conclusion we proposed a translation-based alternative for cross-lingual srl. the key idea is to construct high-quality datasets for the target languages by corpus translation from the gold-standard srl annotations of the source languages. in addition, we combined the gold-standard source srl corpora and the pseudo translated target corpora together to enhance the cross-lingual srl models. we investigated cross-lingual srl models with different kinds of multilingual word representations. further, we presented a pgn-bilstm encoder to better exploit the mixture corpora of different languages. experimental results on the upb v1.0 dataset show that the translation-based method is an effective method for cross-lingual srl transferring. significant improvements can be achieved by using the translated datasets for all selected languages, including both single-source and multi-source transfer. experiment analysis is offered to understand the proposed method in depth.