Unifying Cross-Lingual Semantic Role Labeling with Heterogeneous Linguistic Resources (2021)

Abstract: While cross-lingual techniques are finding increasing success in a wide range of Natural Language Processing tasks, their application to Semantic Role Labeling (SRL) has been strongly limited by the fact that each language adopts its own linguistic formalism, from PropBank for English to AnCora for Spanish and PDT-Vallex for Czech, inter alia. In this work, we address this issue and present a unified model to perform cross-lingual SRL over heterogeneous linguistic resources. Our model implicitly learns a high-quality mapping for different formalisms across diverse languages without resorting to word alignment and/or translation techniques. We find that, not only is our cross-lingual system competitive with the current state of the art but that it is also robust to low-data scenarios. Most interestingly, our unified model is able to annotate a sentence in a single forward pass with all the inventories it was trained with, providing a tool for the analysis and comparison of linguistic theories across different languages. We release our code and model at https://github.com/SapienzaNLP/unify-srl .

## introduction semantic role labeling (srl) -a long-standing open problem in natural language processing (nlp) and a key building block of language understanding @xcite -is often defined as the task of automatically addressing the question "who did what to whom, when, where, and how?" @xcite . while the need to manually engineer and fine-tune complex feature templates severely limited early work @xcite , the great success of neural networks in nlp has resulted in impressive progress in srl, thanks especially to the ability of recurrent networks to better capture relations over sequences @xcite . owing to the recent wide availability of robust multilingual representations, such as multilingual word embeddings @xcite and multilingual language models @xcite , researchers have been able to shift their focus to the development of models that work on multiple languages @xcite @xcite . a robust multilingual representation is nevertheless just one piece of the puzzle: a key challenge in multilingual srl is that the task is tightly bound to linguistic formalisms @xcite which may present significant structural differences from language to language @xcite . in the recent literature, it is standard practice to sidestep this issue by training and evaluating a model on each language separately @xcite @xcite @xcite . although this strategy allows a model to adapt itself to the characteristics of a given formalism, it is burdened by the non-negligible need for training and maintaining one model instance for each language, resulting in a set of monolingual systems. instead of dealing with heterogeneous linguistic theories, another line of research consists in actively studying the effect of using a single formalism across multiple languages through annotation projection or other transfer techniques @xcite @xcite @xcite . however, such approaches often rely on word aligners and/or automatic translation tools which may introduce a considerable amount of noise, especially in lowresource languages. more importantly, they rely on the strong assumption that the linguistic formalism of choice, which may have been developed with a specific language in mind, is also suitable for other languages. in this work, we take the best of both worlds and propose a novel approach to cross-lingual srl. our contributions can be summarized as follows: we hope that our unified model will further advance cross-lingual srl and represent a tool for the analysis and comparison of linguistic theories across multiple languages. end-to-end srl. the srl pipeline is usually divided into four steps: predicate identification, predicate sense disambiguation, argument identification, and argument classification. while early research focused its efforts on addressing each step individually @xcite @xcite , recent work has successfully demonstrated that tackling some of these subtasks jointly with multitask learning @xcite is beneficial. in particular, he et al. (2018) and, subsequently, cai et al. (2018), li et al. (2019) and conia et al. (2020), indicate that predicate sense signals aid the identification of predicateargument relations. therefore, we follow this line and propose an end-to-end system for cross-lingual srl. multilingual srl. current work in multilingual srl revolves mainly around the development of novel neural architectures, which fall into two broad categories, syntax-aware and syntax-agnostic ones. on one hand, the quality and diversity of the information encoded by syntax is an enticing prospect that has resulted in a wide range of contributions: marcheggiani and titov (2017) made use of graph convolutional networks (gcns) to better capture relations between neighboring nodes in syntactic dependency trees; strubell et al. (2018) demonstrated the effectiveness of linguisticallyinformed self-attention layers in srl; cai and lapata (2019b) observed that syntactic dependencies often mirror semantic relations and proposed a model that jointly learns to perform syntactic dependency parsing and srl; he et al. (2019) devised syntax-based pruning rules that work for multiple languages. on the other hand, the complexity of syntax and the noisy performance of automatic syntactic parsers have deterred other researchers who, instead, have found methods to improve srl without syntax: cai et al. (2018) took advantage of an attentive biaffine layer (dozat and manning, 2017) to better model predicate-argument relations; chen et al. (2019) and lyu et al. (2019) obtained remarkable results in multiple languages by capturing predicate-argument interactions via capsule networks and iteratively refining the sequence of output labels, respectively; cai and lapata (2019a) proposed a semi-supervised approach that scales across different languages. while we follow the latter trend and develop a syntax-agnostic model, we underline that both the aforementioned syntax-aware and syntax-agnostic approaches suffer from a significant drawback: they require training one model instance for each language of interest. their two main limitations are, therefore, that i) the number of trainable parameters increases linearly with the number of languages, and ii) the information available in one language cannot be exploited to make srl more robust in other languages. in contrast, one of the main objectives of our work is to develop a unified cross-lingual model which can mitigate the paucity of training data in some languages by exploiting the information available in other, resource-richer languages. ## model description in the wake of recent work in srl, our model falls into the broad category of end-to-end systems as it learns to jointly tackle predicate identification, predicate sense disambiguation, argument identification and argument classification. the model architecture can be roughly divided into the following components: unlike previous work, our model does not require any preexisting cross-resource mappings, word alignment techniques, translation tools, other annotation transfer techniques, or parallel data, to perform high-quality cross-lingual srl, as it relies solely on implicit cross-lingual knowledge transfer. ## experiments we evaluate our model in dependency-based multilingual srl. the remainder of this section describes the experimental setup (section 4.1), provides a brief overview of the multilingual dataset we use for training, validation and testing (section 4.2), and shows the results obtained on each language (section 4.3). ## analysis and discussion cross-formalism srl. in contrast to existing multilingual systems, a key benefit of our unified cross-lingual model is its ability to provide annotations for predicate senses and semantic roles in any linguistic formalism. as we can see from figure 1 (left), given the english sentence "the cat threw its ball out of the window", our language-specific decoders produce predicate sense and semantic role labels not only according to the english propbank inventory, but also for all the other resources, as it correctly identifies the agentive and patientive constituents independently of the formalism of interest. and this is not all, our model may potentially work on any of the 100 languages supported by the underlying language model (m-bert or xlm-roberta), e.g., in italian, as shown in figure 1 (right). this is vital for those languages for which a predicate-argument structure inventory has not yet been developed -an endeavor that may take en -english propbank en -verbatlas es -ancora ca -ancora de -german propbank zh -chinese propbank empezar.c1 starten.2 empezar.a2 starten.1 start-v.01 comencar.c1 comenzar.c1 iniciar.b2 iniciar.b2 begin-v.01 开始.01 iniciar.a2 ini�ate-v.01 lementar.a2 lementar.a2 regret-v.01 bedauem.1 a�ack.01 批评.01 攻击.01 atacar.a2 cri�car.a2 cri�cism.01 anklagen.1 suit.01 acusar.a2 case.01 acusar.a2 acusado.b2 vorwerfen.2 acusado.b2 rugen.1 acusat.b2 kri�sieren.1 cri�car.a2 cri�cize-v.01 denunciar.a2 informieren.1 informieren.1 berichten.1 report.01 report-v.01 报道.01 informar.a32 informar.a32 adver�r.a2 warnen.2 alertar.a2 warn-v.01 alertar.a31 提醒.01 警告.01 adver�r.a32 cau�on-v.01 warnen.1 adver�r.a2 mahnen.1 warning.01 warn inform informieren.2 criticize accuse 指责.02 regret_sorry iniciar.a2 comencar.a2 years to come to fruition -and, therefore, manually annotated data are unavailable. thus, as long as a large amount of pretraining data is openly accessible, our system provides a robust cross-lingual tool to compare and analyze different linguistic theories and formalisms across a wide range of languages, on the one hand, and to overcome the issue of performing srl on languages where no inventory is available, on the other. aligning heterogeneous resources. as briefly mentioned previously, the universal encoders in the model architecture force our system to learn cross-lingual features that are important across different formalisms. a crucial consequence of this approach is that the model learns to implicitly align the resources it is trained on, without the aid of word aligners and translation tools, even when these resources may be designed around specific languages and, therefore, present significant differences. in order to bring to light what our model implicitly learns to align in its shared crosslingual space (see sections 3.2 and 3.3), we exploit its language-specific decoders to build a mapping from any source inventory, e.g., ancora, to a target inventory, e.g., the english propbank. in particular, we use our cross-lingual model to label a training set originally tagged with a source inventory to produce silver annotations according to a target inventory, similarly to what is shown in figure 1 . while producing the silver annotations, we keep track of the number of times each predicate sense in the source inventory is associated by the model with a predicate sense of the target inventory. as a result, we produce a weighted directed graph in which the nodes are predicate senses and an edge (a, b) with weight w indicates that our model maps the source predicate sense a to the target predicate sense b at least w times. a portion of this graph is displayed in figure 2 where, for visualization purposes, we show the most frequent alignments for each language, i.e., the top-3 edges with largest weight from the nodes of each inventory to the nodes of the english propbank (figure 2 , left) and to the nodes of verbatlas (figure 2 , right). 7 for example, figure 2 (left) shows that our model learns to map the spanish ancora sense empezar.c1 and the german propbank sense starten.2 to the english propbank sense start.01, but also that, depending on the context, the chinese prop-bank sense 开始.01 can correspond to both start.01 and begin.01. figure 2 (right) also shows that our model learns to map senses from different languages and formalisms to the coarse-grained senses of verbatlas, even though the latter formalism is quite distant from the others as its frames are based on clustering wordnet synsets -sets of synonymous words -that share similar semantic behavior, rather than enumerating and defining all the possible senses of a lexeme as in the english and chinese propbanks. to the best of our knowledge, our unified model is the first transfer-based tool to automatically align diverse linguistic resources across languages without relying on human supervision. ## conclusion and future work on one hand, recent research in multilingual srl has focused mainly on proposing novel model architectures that achieve state-of-the-art results, but require a model instance to be trained on and for each language of interest. on the other hand, the latest developments in cross-lingual srl have revolved around using the english propbank inventory as a universal resource for other languages through annotation transfer techniques. following our hunch that semantic relations may be deeply rooted beyond the surface realizations that distinguish one language from another, we propose a new approach to cross-lingual srl and present a model which learns from heterogeneous linguistic resources in order to obtain a deeper understanding of sentence-level semantics. to achieve this objective, we equip our model architecture with "universal" encoders which share their weights across languages and are, therefore, forced to learn knowledge that spans across varying formalisms. our unified cross-lingual model, evaluated on the gold multilingual benchmark of conll-2009, outperforms previous state-of-the-art multilingual systems over 6 diverse languages, ranging from catalan to czech, from german to chinese, and, at the same time, also considerably reduces the amount of trainable parameters required to support different linguistic formalisms. and this is not all. we find that our approach is robust to low-resource scenarios where the model is able to exploit the complementary knowledge contained in the training set of different languages. most importantly, our model is able to provide predicate sense and semantic role labels according to 7 predicate-argument structure inventories in a single forward pass, facilitating comparisons between different linguistic formalisms and investigations about interlingual phenomena. our analysis shows that, thanks to the prior knowledge encoded in recent pretrained language models and our focus on learning from cross-lingual features, our model can be used on languages that were never seen at training time, opening the door to alignment-free cross-lingual srl on languages where a predicateargument structure inventory is not yet available. finally, we show that our model implicitly learns to align heterogeneous resources, providing useful insights into inter-resource relations. we leave an in-depth qualitative and quantitative analysis of the learnt inter-resource mappings for future work. we hope that our work can set a stepping stone for future developments towards the unification of heterogeneous srl. we release the code to reproduce our experiments and the checkpoints of our best models at https://github. com/sapienzanlp/unify-srl .