CLAR : A Cross-Lingual Argument Regularizer for Semantic Role Labeling (2020)

Abstract: Semantic role labeling (SRL) identifies predicate-argument structure(s) in a given sentence. Although different languages have different argument annotations, polyglot training, the idea of training one model on multiple languages, has previously been shown to outperform monolingual baselines, especially for low resource languages. In fact, even a simple combination of data has been shown to be effective with polyglot training by representing the distant vocabularies in a shared representation space. Meanwhile, despite the dissimilarity in argument annotations between languages, certain argument labels do share common semantic meaning across languages (e.g. adjuncts have more or less similar semantic meaning across languages). To leverage such similarity in annotation space across languages, we propose a method called Cross-Lingual Argument Regularizer (CLAR). CLAR identifies such linguistic annotation similarity across languages and exploits this information to map the target language arguments using a transformation of the space on which source language arguments lie. By doing so, our experimental results show that CLAR consistently improves SRL performance on multiple languages over monolingual and polyglot baselines for low resource languages.

## introduction semantic role labeling (srl) is the task of labeling each predicate and its corresponding arguments in a given sentence. srl provides a more stable meaning representation across syntactically different sentences and has been seen to help a wide range of nlp applications such as question answering @xcite and machine translation @xcite . recent end-to-end deep neural networks for srl, though performing well for languages with large training data @xcite @xcite , are much less effective for low resources languages due to very limited annotated data for these languages. methods such as polyglot training @xcite seek to make these models perform better on low resource languages by combining supervision from multiple languages. the key idea in polyglot training is to combine the training data from multiple languages by using multilingual word embeddings from a shared space and a common encoder model (e.g. an lstm). the argument sets for the languages are kept separate by using different classification layers. the arguments sets are kept separate because the semantic label spaces are usually language-specific @xcite . however, despite the dissimilarity in argument annotations between languages, certain argument labels do share common semantic meaning across languages. fig. 1 shows three different sentences from chinese, german, and english, respectively, with defined predicate-argument structures. although the predicates are essentially the same, their arguments are labeled differently across languages in the training data. for instance, all sentences contain words representing the same underlying semantic meaning that is temporal but with different argument labels (tmp in chinese, a4 in german, am-tmp in english). we hypothesize that we can improve the srl performance of low resource languages during cross-lingual transfer by identifying such arguments with similar semantic meaning across languages and representing them close to each other in the feature space. this requires: (1) detecting the correspondence between the labels in different languages; and (2) representing arguments with similar semantic meaning in the feature space for better srl performance. we propose a method called cross-lingual argument regularizer (clar) with a two-step process: step 1: pair matching: detecting a number of label pairs between the source and target languages during polyglot training. we call these arguments common arguments. given the multilingual embedding already used in polyglot training, clar does not require additional cross-lingual alignments on parallel data. step 2: regularization: given the common arguments identified, find a transformation to bring the paired arguments close together. this transformation is learned and used in the poloyglot training process so that the knowledge on the labels in the source language can be better transferred to knowledge in the corresponding labels in the target language. we evaluate clar on the srl portion of the conll 2009 dataset @xcite foot_0 and compare its performance against baseline and polyglot training methods. the main contributions of this work are: the rest of the paper is organized as follows: sec. 2 describes the base model. sec. 3 describes clar. sec. 4 demonstrate its efficacy with extensive empirical evaluation. sec. 5 reviews the existing literature. sec. 6 makes concluding remarks. ## base model the srl task consists of four subtasks: 1) predicate identification (e.g., reach); 2) sense disambiguation of the identified predicate (e.g., reach.01); 3) argument identification for each predicate (e.g., market) and 4) role classification of the identified arguments (e.g., a0). following @xcite and @xcite , we focus on argument labeling and predicate sense disambiguation, both sequence tagging problems. ## the clar algorithm the underlying motivation for polyglot training @xcite is that arguments from different languages often help enhance each other. it is reasonable to assume that if corresponding arguments from source and target languages are located closer in the feature space, their mutual enhancements can be strengthened. the possibility for doing so is based on the following observation. in neural network models that generate labels, the last layer is usually a softmax layer of the form step 1: pair matching: detect the best pairing of the arguments between a pair of languages. step 2: regularization: find a transformation that brings the feature vectors corresponding to the paired argument labels close to each other. these two steps are described in detail below. pair matching: the goal of this step is to identify matching label pairs in the two languages. we start with the simple polyglot training @xcite for the first few epochs without clar and collect the last layer weights for all the target and source language arguments. given the k s vectors u i and k t vectors v j , solve this constraint optimization problem minimize this is an integer linear programming problem, for which many excellent solvers exist. we use glpk solver from cvxopt foot_1 . we observe that the frequency distribution of the argument labels is quite skewed in the training dataset: a few labels (e.g., a0, a1) have much larger number of training examples than other labels. experiments show that low-frequency labels cause noisy pair matching that degrades the output quality. therefore, we consider only labels that have more than 1% of the total number occurrences in the respective language training data. typically, 40 -50% of the total labels in each language match this criterion. the k s and k t in the general algorithm are replaced by ks and kt for the number of arguments satisfying this criterion in the source and the target language, respectively. regularization: the goal of this step is to learn an affine transform to bring the target vectors closest to the corresponding source vectors. this step is performed iteratively during the overall training process. given the k pairs (u i , v i ) detected in the previous step, the objective of the overall optimization objective function is amended as follows ## conclusion we introduces clar, a cross-lingual argument regularizer. it explores linguistic annotation similarity across languages and exploits this obtained information during srl model training to map the target language arguments as the deformation of a space on which source language arguments lie. we confirm the effectiveness of clar for srl on conll 2009 dataset over monolingual and polyglot methods, without prior knowledge of cross-lingual alignments or parallel data. this paper demonstrates the promise of understanding and exploiting linguistic annotation similarity across languages during polyglot training. we plan to explore other ways of identifying and leveraging linguistic annotation similarity across languages. mariona taulé, m antònia martí, and marta recasens. jiateng xie, zhilin yang, graham neubig, noah a smith, and jaime carbonell. 2018. neural crosslingual named entity recognition with minimal resources. in proceedings of the 2018 conference on empirical methods in natural language processing, pages 369-379. wen-tau yih, matthew richardson, chris meek, ming-wei chang, and jina suh. 2016. the value of semantic parse labeling for knowledge base question answering. in proceedings of the 54th annual meeting of the association for computational linguistics (volume 2: short papers), pages 201-206. hai zhao, wenliang chen, jun'ichi kazama, kiyotaka uchimoto, and kentaro torisawa. 2009. multilingual dependency learning: exploiting rich features for tagging syntactic and semantic dependencies. in proceedings of the thirteenth conference on computational natural language learning: shared task, pages 61-66. association for computational linguistics. jie zhou and wei xu. 2015. end-to-end learning of semantic role labeling using recurrent neural networks. in proceedings of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing (volume 1: long papers), pages 1127-1137.