Bridging the Gap in Multilingual Semantic Role Labeling: a Language-Agnostic Approach (2020)

Abstract: Recent research indicates that taking advantage of complex syntactic features leads to favorable results in Semantic Role Labeling. Nonetheless, an analysis of the latest state-of-the-art multilingual systems reveals the difficulty of bridging the wide gap in performance between high-resource (e.g., English) and low-resource (e.g., German) settings. To overcome this issue, we propose a fully language-agnostic model that does away with morphological and syntactic features to achieve robustness across languages. Our approach outperforms the state of the art in all the languages of the CoNLL-2009 benchmark dataset, especially whenever a scarce amount of training data is available. Our objective is not to reject approaches that rely on syntax, rather to set a strong and consistent language-independent baseline for future innovations in Semantic Role Labeling. We release our model code and checkpoints at https://github.com/SapienzaNLP/multi-srl .

## introduction semantic role labeling (srl) -the task of automatically addressing "who did what to whom, how, when and where?" @xcite -is a long standing open problem in natural language processing (nlp), and a central task required to complete the puzzle of natural lan-guage understanding @xcite . its roots date back to several decades ago, to when @xcite first theorized the existence of deep semantic relations between a predicate and other sentential constituents. over the years, different linguistic formalisms and their corresponding predicate-argument structure inventories expanded fillmore's seminal intuition @xcite , yet the need to rely on manually designed complex feature templates severely limited early srl models @xcite . fortunately, the recent great success of neural networks in nlp has drawn attention back to srl and has led to considerable performance gains. this has been particularly the case for recurrent neural networks, thanks to their ability to better capture relations over sequences @xcite . the positive results obtained in srl were rapidly extended to other fields where they proved to be beneficial to several downstream tasks, from machine translation @xcite to information extraction @xcite , opinion role labeling @xcite , and question answering @xcite . as researchers constantly explored new approaches to improve srl, the exploitation of syntactic features soon emerged as a natural choice. @xcite suggested that syntax ought to help semantic role labelers since i) a significant portion of the predicate-argument relations in a semantic dependency graph mirrors the edges that appear in a syntactic dependency graph, and ii) there is often a deterministic mapping from syntactic to semantic roles. following this line of thought, numerous papers from major venues reported improvements in srl by explicitly taking advantage of different properties in syntactic dependency trees to various extents @xcite @xcite . however, if we step back to observe the larger picture, a significant gap among languages strikes the eye. indeed, among the state-of-the-art multilingual srl systems that reported their results on all the languages of the conll-2009 benchmark dataset @xcite , the currently best-performing system @xcite manifests a very significant discrepancy in performance between high-and low-resource languages (around 10% in f 1 score); the works of @xcite and @xcite , inter alia, also show the same behavior, with gaps in f 1 scores that fluctuate around 15% and 14%, respectively. these large differences from language to language suggest that recently proposed innovations do not seem to generalize consistently across different languages, especially when syntax plays a central role or the novelty is tested on out-of-domain data. anyhow, and perhaps most importantly, these approaches do not address the ever-present performance gap between high-resource and low-resource languages, such as english and german, respectively, where the disparity in f 1 score still hovers around 10% in the in-domain evaluations of conll-2009, and is even wider in the out-of-domain evaluations. we argue that correctly harnessing contextual intra-sentence information can lead to large gains in performance on single languages and a more robust generalization capability when scaling across languages, especially when these belong to different linguistic families (e.g., english and chinese). more precisely, the main contributions of this paper are as follows: we stress that our objective is not to reject syntax in srl. on the contrary, we strongly believe that clever integration of syntactic features is a promising avenue to advancing research. here, however, our aim is to provide a strong language-agnostic baseline that is robust and consistent across languages, especially low-resource ones. we hope our effort can become a stepping stone for future developments of both syntax-agnostic and syntax-focused srl. end-to-end approaches. regardless of the formalism of choice, srl is traditionally divided into a set of simpler subtasks: predicate identification, predicate sense disambiguation, argument identification and argument classification. early work used different sets of template features and statistical/neural models to tackle each predicate and argument subtask separately @xcite , but the recent success of the multi-task learning paradigm @xcite prompted the development of end-to-end models that jointly address some of the subtasks @xcite @xcite . since the conll-2009 shared task provides pre-identified predicates, systems -end-to-end approaches included -usually process the same sentence n p times, where n p is the number of predicates in the sentence; at the cost of longer training times, this approach lets a system contextualize a sentence with respect to a specific pre-identified predicate. in our work we take another approach: our system contextualizes a sentence to all the predicates it contains in a single forward pass. this results in shorter inference time and a significant reduction in training time, because our model converges in less than 30 epochs compared to the 300 epochs required by the current state-of-the-art multilingual system of @xcite . syntax-agnostic srl. @xcite opened the door to the initial wave of syntaxagnostic models for srl by efficiently employing a bilstm-based encoder to capture longer predicateargument relations within an input sequence, thereby outperforming syntax-aware systems in the conll-2009 english, czech and spanish evaluation sets for the first time. @xcite proposed the first full end-to-end syntax-agnostic srl model to jointly learn to disambiguate predicate senses and recognize their corresponding semantic arguments, and further enhanced the model with an attentive biaffine scorer @xcite to better condition argument predictions on a given predicate in the input sentence. the combined contribution of these innovations realigned the performances of syntax-agnostic systems to the best syntax-aware systems. most recently, @xcite showed that the use of contextualized word embeddings such as elmo @xcite leads to further progress, thus lending support to the hunch that high-quality contextual information is key to enabling high-performing srl systems. we stress that there is a subtle catch in the definition of syntax-agnostic: the foregoing approaches do not make use of sentence-level syntax, but do still consider lexical-level syntactic features, such as part-of-speech tags. as a result, their input is still language-dependent. in contrast, our approach does away with any lexical-or sentence-level syntactic feature and is therefore truly syntax-agnostic. syntax-aware srl. syntactic features have recently gained traction in srl research, mostly due to the diversity of the available representations, the quality of the information they encode, and the wide range of techniques that can be used to take advantage of them. notable work includes the use of graph convolutional networks to capture short-distance relations between neighbors in the syntactic dependency graph @xcite , deriving argument pruning rules from syntactic dependency trees @xcite , clustering dependency relations @xcite , and syntax-based attention mechanisms @xcite . however, most of the work that reported performance improvements in the conll-2009 benchmark dataset only did so in a subset of its in-and out-of-domain evaluations of the 6 available languages, with the noteworthy exception of @xcite . this may fuel the suspicion that either there is a lack of real interest in syntax-aware multilingual srl, or syntaxfocused innovations do not scale immediately across languages. in any case, reporting fragmented results strongly limits full-fledged comparisons and, therefore, hinders progress in multilingual srl. ## model description building on top of recent successes in deep learning, our model learns to tackle predicate sense disambiguation and argument identification/classification jointly. in particular: a predicate-aware word encoder which re-contextualizes the word representations with respect to all the predicates in a sentence in a single forward pass (section 3.2); a predicate-argument encoder which specializes the representation of each argument to a single predicate, for each predicate in the input sentence (section 3.3). ## experiments we evaluate our model in multilingual dependency-based srl (conll-2009) and english span-based srl @xcite . hereafter, we provide a brief overview of the conll-2009 and conll-2012 benchmark datasets (section 4.1), we describe our experimental setup (section 4.2), and report results for each dataset and language (section 4.3). ## analysis and discussion ablation study. we conducted an ablation study of our model architecture to better understand the individual contribution of its main components. as shown in ## conclusion and future work recently, research in semantic role labeling has revolved predominantly around syntax, with several studies showing the benefits of integrating syntactic features into existing models. syntax-based innovations, however, have struggled to transfer improvements from high-to low-resource languages, as they appear to require a substantial amount of high-quality annotations. in this paper, we have gone against the flow and proposed a model that puts both sentence-and lexicallevel syntax aside, in order to avoid relying on noisy language-specific features. our truly syntax-agnostic approach surpasses the previous state of the art among all syntax-agnostic and syntax-aware systems in the in-and out-of-domain evaluations of all 6 languages in the conll-2009 benchmark dataset. most crucially, our model seamlessly scales across languages, finally bridging the long-standing gap between high-and low-resource settings. our analysis delineates the strengths of our approach and highlights its exceptional robustness: our model only needs 50% and 25%, respectively, of the training data to surpass the previous best-performing system in the in-domain and out-of-domain german test sets. to the best of our knowledge, we are the first to evaluate a model on zero-shot cross-lingual srl, where we obtain results that are promising with a view to further developments in transfer learning techniques. finally, we demonstrate that, not only is our model able to achieve state-of-the-art results in dependency-based srl, but it also surpasses current syntax-agnostic and syntax-aware techniques in span-based srl on conll-2012. as previously stated, our objective has not been to dismiss the undeniable importance of syntaxbased innovation in srl, but rather to establish a launch pad from which future syntactic developments can take off. in order to encourage future work on joint syntactic and semantic dependency parsing @xcite , the use of more powerful or cleverly trained language models @xcite , the integration of srl into other cross-lingual semantics-first tasks such as semantic parsing @xcite and word sense disambiguation @xcite , and the exploitation and integration of newly available knowledge from recently released resources, such as verbatlas @xcite and conception @xcite , we make available not only the code for our srl model and experiments, but also our model checkpoints and training/validation logs at https://github.com/ sapienzanlp/multi-srl.