Alignment-free Cross-lingual Semantic Role Labeling (2020)

Abstract: Cross-lingual semantic role labeling (SRL) aims at leveraging resources in a source language to minimize the effort required to construct annotations or models for a new target language. Recent approaches rely on word alignments, machine translation engines, or preprocessing tools such as parsers or taggers. We propose a cross-lingual SRL model which only requires annotations in a source language and access to raw text in the form of a parallel corpus. The backbone of our model is an LSTM-based semantic role labeler jointly trained with a semantic role compressor and multilingual word embeddings. The compressor collects useful information from the output of the semantic role labeler, filtering noisy and conflicting evidence. It lives in a multilingual embedding space and provides direct supervision for predicting semantic roles in the target language. Results on the Universal Proposition Bank and manually annotated datasets show that our method is highly effective, even against systems utilizing supervised features.

## introduction semantic role labeling (srl) is the task of identifying the arguments of semantic predicates in a sentence and labeling them with a set of predefined relations (e.g., "who" did "what" to "whom," "when," and "where"). it has emerged as an important technology for a wide spectrum of applications ranging from machine translation @xcite to information extraction @xcite , and summarization @xcite . there have been considerable efforts on developing annotated resources for semantic role labeling @xcite which in turn have greatly facilitated the development of the various models designed to automatically predict semantic roles. recent years have seen the successful application of neural network models to srl @xcite @xcite which forego the need for extensive feature engineering. despite recent advances in representational learning, a perennial problem with building slr systems lies in the paucity of training data since semantic role annotations are available for only a handful of the world's languages. as a result, much previous work has focused on cross-lingual srl which aims at leveraging existing resources in a source language to minimize the effort required to construct a model or annotations for a new target language. annotation projection is a popular approach which transfers annotations from a source to a target language via automatic word alignments @xcite @xcite . although very intuitive, it is sensitive to the quality of the parallel data, the performance of the source-language srl model, and the accuracy of alignment tools, all of which introduce noise. translation-based approaches @xcite @xcite aim to alleviate the noise brought by the the source-side labeler by directly translating the gold-standard data into the target language. a third alternative is model transfer where a source-language model is modified in a way that it can be directly applied to a new language, e.g., by employing cross-lingual word representations @xcite @xcite and universal pos tags @xcite . word alignment noise poses serious problems for both annotation-projection and translationbased methods (the latter still rely on alignment tools to transfer word-level labels from source to target). for example, there could be many-to-one alignments, leading to semantic role conflicts in the target language. some form of filtering is often introduced to reduce the impact of this noise, e.g., parallel sentence pairs are discarded according to projection density @xcite or alignment confidence @xcite . in addition, translation-based approaches rely on high performance translation engines, which are often trained on large-scale parallel corpora. unfortunately, neither adequate mt nor high-quality parallel data can be guaranteed when dealing with low-resource languages. model transfer is an appealing alternative, however, it relies on accurate features based on lemmas, pos tags, and syntactic parse trees @xcite which are themselves obtained with access to additional annotation. it is not realistic to assume that treebank-style resources will be available for low-resource languages. in this paper, we propose a novel method for cross-lingual srl which does not rely on word alignments, machine translation or pre-processing tools such as parsers or taggers. aside from semantic role annotations in the source language, we only assume access to raw text in the form of a parallel corpus. the backbone of our model is an lstm-based semantic role labeler jointly trained with multi-lingual word embeddings and a semantic role compressor. the compressor distills useful information pertaining to arguments, predicates and their roles from the output of the semantic role labeler (e.g., by automatically filtering unrelated or conflicting information). importantly, the compressor lives in a multilingual space and can provide direct supervision for predicting semantic roles in the target language, sidestepping intermediaries like word-level alignments and machine translation. for evaluation, we make use of several multilingual benchmarks. these include the universal proposition bank (upb; @xcite , a recently released resource which contains semiautomatically created annotations under a unified labeling scheme for several languages, and a french corpus @xcite which follows propbank-style annotations @xcite . we also release two additional manually labeled resources in chinese and german, which we hope will be useful for future research. 2 ex-perimental results show that our method is highly effective across languages and annotation schemes, even compared against systems making use of supervised features. our contributions can be summarizes as follows: (a) we propose a knowledge-lean model which does not rely on alignments, machine translation or sophisticated linguistic preprocessing; (b) we introduce the concept semantic role compressor which is important at filtering noisy information and can be potentially useful for other crosslingual tasks (e.g., dependency parsing); (3) we release two manually annotated datasets which will further advance cross-lingual semantic role labeling complementing previous work @xcite which reports result on semi-automatically created annotations). ## model figure 1 provides a schematic overview of our model. we assume we have access to semantic role annotations in a source language (e.g., english) and a parallel corpus of source-target sentences (e.g., english-french). our model is jointly trained to predict semantic roles in the source and target languages. it has two main components, namely a semantic role labeler, and a semantic role compressor. the role labeler consists of: while the semantic role compressor consists of: in the following sections we describe these two components more formally. ## related work there has been a great deal of interest in crosslingual transfer learning for srl @xcite @xcite @xcite @xcite . the majority of previous work has focused on two types of approaches, namely annotation projection and model transfer. a variety of methods have been proposed to improve the quality of annotation projections due to alignment noise. these range from word and argument filtering techniques @xcite , to learning syntax and semantics jointly @xcite , and iterative bootstrap-ping @xcite . in an attempt to reduce the reliance on supervised lexico-syntactic features for the target language, aminian et al. (2019 ) make use of word and character features, and filter projected annotations according to projection density. model transfer does not require parallel corpora or word alignment tools; nevertheless, it relies on accurate features such as pos tags @xcite or syntactic parse trees @xcite to enhance the ability to generalize across languages. adversarial training is commonly used to extract language-agnostic features thereby improving the performance of cross-lingual systems @xcite . translation-based approaches have been gaining popularity in cross-lingual dependency parsing @xcite @xcite and have recently been applied to srl @xcite . @xcite propose a cross-lingual encoder-decoder model that simultaneously translates and generates sentences with semantic role annotations in a resource-poor target language. rather than creating annotations or models for a target language, other work aims to exploit the similarities between languages. mulcaire et al. (2018) combine resources for multiple languages to create polyglot semantic role labelers and show that polyglot training can result in better labeling accuracy than a monolingual labeler. an obstacle for developing cross-lingual srl models is the absence of a unified annotation scheme for all languages. although the conll-09 shared task @xcite provides annotations for seven languages, the labeling schemes and role sets are not shared. to this end, van der plas et al. (2010) build a french srl dataset, following an annotation scheme similar to conll-09 for english. some recent cross-lingual srl models @xcite @xcite ) make use of the publicly available universal proposition bank (upb; @xcite , which annotates predicates and semantic roles following the english proposition bank 3.0 @xcite . since annotation projection is involved in the construction of upb, the quality of upb is also influenced by the quality of the parallel data, the performance of the source-language srl model, and the accuracy of alignment tools. ## conclusions in this paper we developed a cross-lingual srl model and demonstrated it can effectively leverage unlabeled parallel data without relying on word alignments or any other external tools. we have also contributed two quality controlled datasets (compatible with propbank-style guidelines) which we hope will be useful for the development of crosslingual models. directions for future work are many and varied. although our focus has been on dependency-based srl, our model can be easily adapted to span-based annotations @xcite . in this case, the semantic role compressor could be modified to represent entire spans rather than just head words while decompression would remain unchanged (it would still output a probability distribution for each word over all semantic roles). we also plan to extend our framework to semi-supervised learning, where a small number of annotations might also be available in the target language. we adopt these new positional features for two reasons. firstly, during cross-lingual training, the length of parallel sentences s s and s t is usually different. more importantly, for i-th word w i in s s , its correspondence w j in s t is not the i-th word in s t in most cases. when performing cross-lingual training, it is important that w i and w j have the same position embeddings, so that they can obtain similar result after decompression. as shown in figure 2 , he ("il" in french) appears twice in the english sentence, and its french counterpart shares the same positional features. experimental results show that positional features can effectively improve cross-lingual training. however, there are still cases when the word order changes dramatically after translation and our position features do not work. the only solution seems to be to use contextualized embeddings like multilingual bert or multilingual elmo, where every word in a sentence will be assigned unique embeddings.