X - SRL : A Parallel Cross-Lingual Semantic Role Labeling Dataset (2020)

Abstract: Even though SRL is researched for many languages, major improvements have mostly been obtained for English, for which more resources are available. In fact, existing multilingual SRL datasets contain disparate annotation styles or come from different domains, hampering generalization in multilingual learning. In this work we propose a method to automatically construct an SRL corpus that is parallel in four languages: English, French, German, Spanish, with unified predicate and role annotations that are fully comparable across languages. We apply high-quality machine translation to the English CoNLL-09 dataset and use multilingual BERT to project its high-quality annotations to the target languages. We include human-validated test sets that we use to measure the projection quality, and show that projection is denser and more precise than a strong baseline. Finally, we train different SOTA models on our novel corpus for mono- and multilingual SRL, showing that the multilingual annotations improve performance especially for the weaker languages.

## introduction semantic role labeling (srl) is the task of extracting semantic predicate-argument structures from sentences. one of the most widely used labeling schemes for this task is based on propbank @xcite . it comes in two variants: span-based labeling, where arguments are characterized as word-spans @xcite , and head-based labeling, which only labels the syntactic head @xcite . in this work we focus on head-based labeling, as it is applied in the multilingual conll-09 shared task dataset, comprising 7 languages. the performance of english srl has considerably improved in recent years through continuous figure 1 : method to create x-srl. we automatically translate the english conll-09 corpus, use a fast label projection method for train-dev and get human annotators to select the appropriate head words on the target sentences to obtain gold annotations for the test sets. in this paper we offer a multilingual parallel srl corpus -x-srlfor english (en), german (de), french (fr) and spanish (es) that is based on english gold annotations and shares the same labeling scheme across languages. 1 our corpus has two major advantages compared to existing datasets: first, since it is a parallel corpus, all sen-tences are semantically equivalent allowing us to analyze performance at the sentence-level and to better understand the reasons for srl score divergences across languages foot_1 ; second, we expect that models trained jointly on multiple languages annotated with a homogeneous labeling style will be able to better generalize across languages foot_2 . moreover, by minimizing the need of specialized human annotators, our parallel corpus construction method is lightweight and portable, since it is built on three main components: i) a high-quality annotated dataset in the source language, in this case the english conll-09 corpus @xcite , ii) a high-quality sota machine translation system, we are using deepl pro foot_3 ; and iii) multilingual contextual word representations, in this case multilingual bert (mbert) @xcite . the situation for these multilingual resources is improving with each day, and thus our method, in perspective, could be followed for producing training data for more lower-resource languages. importantly, although we automatically project labels from english to the newly available corpora in the different languages for train and dev sections, we also provide test sets on which humans assess the quality of the automatic translations and select the valid predicates as well as the appropriate target head words for the target sentences. having a humanvalidated test set ensures solid benchmarking for srl systems, and additionally allows us to assess the validity of the proposed automatic projection for the rest of the data. our projection method works as follows (see also figure 1 ): we obtain automatic translations of the english conll-09 corpus into each of our target languages; then, we automatically label them without applying any language-pair specific label projection model, but use mbert with additional filters as a means for alignment. we show that by following this approach we obtain a more densely annotated dataset compared to an existing sota label projection method @xcite . in short, our contributions are: ## related work semi-automatic annotation projection has been applied to different srl frameworks. @xcite ; @xcite proposed a projection method for framenet @xcite semantic roles that searches for the best alignment of source and target constituent trees, and also created a small human-validated test set for benchmarking. a number of propbank resources are available for different languages: the benchmark datasets conll-09 @xcite and @xcite are well-established, however, a direct cross-lingual comparison of srl performance across the covered languages is not possible. the reason being that the language-specific datasets come from different sources and were not conceived for such a comparison. on the other hand, @xcite attested the validity of english propbank labels for french and directly applied them on french data. this motivated srl projection methods such as @xcite and @xcite , which aim to generate a common label set across languages. a known issue with this approach is the need for good quality parallel and sentence-level filtered data. for this reason they used existing parallel corpora, europarl @xcite and un @xcite , automatically labeled the english side with srl annotations and transferred them to the corresponding translations. the major issue with this is that evaluation against groundtruth and detailed error analysis on the target languages are not possible, since all annotations are automatic and come from noisy sources. likewise, the universal proposition bank foot_4 @xcite , adds an srl layer on top of the universal dependencies (de marneffe et al., 2014) corpus, which covers eight different languages. however, i) the original corpora come from independent sources and are not parallel, ii) the source sentences were automatically labeled containing noise even before the alignment step, and iii) the test sets also contain automatic projections without human validation of the labels. in contrast, we present a corpus that transfers english high-quality labels to the target side, thus projecting the same labeling style to the other languages; more importantly, we conceive of this corpus, from the very beginning, as a parallel resource with translation equivalence with the source and target languages at the sentence-level. in addition, we create a human-validated test set to allow for proper benchmarking in each language. the use of synthetic data generated by automatic translation has proven to improve performance for mt @xcite and argumentation mining @xcite . we similarly create a parallel corpus using automatic translation, however, to our knowledge, we are the first to create a directly comparable multilingual srl dataset using automatic translation with a manually validated test set, minimizing human labour. another attempt to close the gap between languages is by training multilingual srl systems. @xcite propose a biaffine scorer with syntax rules to prune of candidates, achieving sota independently in all languages from conll-09. @xcite and @xcite train a single model using input data from different languages and obtain modest improvements, especially for languages where less monolingual training data is available. in this sense, our x-srl corpus contributes with more compatible training data across languages, and aims to improve the performance of jointly trained multilingual models. ## building x-srl in this section we first explain our method for translating the english conll-09 srl dataset ( §3.1) into three target languages (de, es, fr) foot_5 . in §3.2 we describe how the human-validated labels (only for the test sets) were obtained in an efficient way, and report annotator agreement statistics in §3.3. the details of how we perform (automatic) label projection enhanced with filtering for train/dev are given in §3.4. with this we achieve big annotated srl datasets for three new languages (cf. when building the x-srl dataset, in line with the current propbank srl data available in different languages, we focus on verbal predicates only. note that the english conll-09 data includes both verbal and nominal predicate annotations, yet this is due to the nombank project @xcite being available for that language. by contrast, the remaining languages with prop-bank srl training data (including the conll-09 non-english data) only provide annotations for verbal predicates. while we could attempt projecting the english nominal predicate annotations and create an x-srl dataset that includes nominal srl for all target languages -which would mean a big advance over the current situation -admitting nominal and verbal srl annotations in a multilingual setting would confront us with many translation shifts. we could try to capture these for the manually curated test set, however we would run a risk of generating noisy or scarce target annotations when projecting them for the train/dev sections. foot_6 ## conclusions in this paper, we present the first fully parallel srl dataset with homogeneous annotations for four different languages. we included human-validated test sets where we address the linguistic difficulties that emerge when transferring labels across languages -despite transferring gold labels from the source. we introduce and evaluate a novel effective and portable automatic method to transfer srl labels that relies on the robustness of machine translation and multilingual bert and therefore could be straightforwardly applied to produce srl data in other languages. finally, we included an extrinsic evaluation where we train srl models using our data and obtain consistent results that showcase the generalization capacities emerging from our new 4-way multilingual dataset. future work should address the application of our method to more and typologically more divergent languages.