UniteD-SRL : A Unified Dataset for Span- and Dependency-Based Multilingual and Cross-Lingual S emantic R ole L abeling (2021)

Abstract: Multilingual and cross-lingual Semantic Role Labeling (SRL) have recently garnered increasing attention as multilingual text representation techniques have become more effective and widely available. While recent work has attained growing success, results on gold multilingual benchmarks are still not easily comparable across languages, making it difficult to grasp where we stand. For example, in CoNLL-2009, the standard benchmark for multilingual SRL, language-to-language comparisons are affected by the fact that each language has its own dataset which differs from the others in size, domains, sets of labels and annotation guidelines. In this paper, we address this issue and propose UniteD-SRL, a new benchmark for multilingual and cross-lingual, span- and dependency-based SRL. UniteD-SRL provides expert-curated parallel annotations using a common predicate-argument structure inventory, allowing direct comparisons across languages and encouraging studies on cross-lingual transfer in SRL. We release UniteD-SRL v1.0 at https://github.com/SapienzaNLP/united-srl .

## introduction semantic role labeling (srl) -often considered to be a fundamental step towards natural language understanding @xcite -consists in recovering the latent predicate-argument structure of a sentence by identifying the semantic relationship between a predicate and its arguments @xcite . srl can be used to extract information from text and to provide a shallow semantic representation of sentences, finding applications in a wide range of natural language processing (nlp) areas such as machine translation @xcite , question answering @xcite , visual semantic role labeling @xcite , semantic parsing @xcite and story generation @xcite . given the popularity of this task, over the years several competitions have been organized within the conference on computational language learning (conll) to evaluate srl systems @xcite @xcite @xcite . these shared tasks led to the release of several datasets that nowadays represent the de facto standard benchmarks for srl, namely, @xcite @xcite however, despite their widespread use, the conll datasets suffer from a considerable level of heterogeneity, which prevents systems from easily scaling across task formulations and languages: conll-2005 @xcite is devised for span-based srl where systems are required to identify and classify argument spans, whereas @xcite is framed as a dependency-based task, where only the syntactic head of an argument has to be tagged. moreover, when multiple languages are covered, for example in @xcite @xcite , different inventories are used, such as english propositional bank @xcite , chinese propbank (xue and @xcite and @xcite for spanish and catalan, making it difficult to evaluate whether a system is able to generalize across languages and, if so, to what extent. in fact, these multilingual datasets are not aligned, they are considerably different in size and show significant dissimilarities in their domain distribution, strongly limiting language-to-language comparisons. some studies @xcite worked around this issue by electing the english propbank as a universal semantic inventory and employing cross-lingual annotation projection techniques to produce annotations for other languages starting from english annotated data. these approaches, however, tend to ignore the fact that probbank was conceived expressly for english predicates. forcing the semantics of different languages to adapt to english, without considering possible translation divergences in parallel sentences @xcite , can lead to two distinct problems: i) incorrect projections, if divergent sentences are retained in the dataset; ii) elimination from the dataset of all the sources of linguistic divergences, if divergent sentence pairs are discarded. another issue of srl annotation projection techniques regards the use of propbank as verbal resource. this, in fact, limits the informativeness of the annotations, since it does not mark semantic roles with semanticallyconsistent labels, leading to ambiguous or unclear annotations. for example, in john is sleeping and john loves mary, john would be tagged arg0 in both cases, but we argue that agent and experi-encer are clearer and more fitting semantic roles to tag john with. the consequence of all these limitations is that, in order to engage in this task, there is a plethora of features and settings to choose before selecting the proper dataset. to address the above-mentioned issues and encourage future work on cross-lingual approaches for srl, we propose united-srl, a unified srl dataset with the following features: we expect that the release of a parallel multilingual dataset will provide a fair evaluation for multilingual and cross-lingual srl systems, making the results directly comparable from language to language. we release united-srl v1.0 at https: //github.com/sapienzanlp/united-srl. ## related work multilingual srl datasets. due to the complexity of the task, srl annotations are expensive to produce as they require expert annotators who are comfortable with the linguistic theories behind the predicate-argument inventory of choice. this makes the creation of multilingual srl datasets even more difficult. perhaps the largest effort in this direction was made on the occasion of the conll-2009 shared task @xcite . the conll-2009 multilingual dataset for dependencybased srl originally featured 7 languages: english, chinese, czech, german, catalan, spanish and japanese. 1 however, each of these datasets was annotated separately, starting from different corpora and using different predicate-argument structure inventories, e.g., the english propbank @xcite for english, @xcite for czech, ancora @xcite for spanish and catalan. universal propositional bank foot_1 @xcite adds srl annotations on top of the universal dependency corpus @xcite . the limitations of this dataset are that, even if it covers 8 languages, the sentences in it are not parallel and were annotated automatically. we argue that this heterogeneity inhibits, or at least slows down, further advances in multilingual and cross-lingual srl. ## experiments one of the main objectives of united-srl is to allow past and future systems and their results to be directly comparable across diverse languages without having to deal with or take into account heterogeneous linguistic resources, different domains and varying dataset sizes. to this end, we use united-srl to train and evaluate a recently proposed state-of-the-art srl system, showing how our dataset provides interesting insights into the cross-lingual transferability of predicate senses and their argument structures. ## conclusion in this paper, we presented the first version of united-srl, a unified dataset for span-and dependency-based multilingual and cross-lingual srl. our dataset can be used as test bed to investigate different aspects of multilingual and crosslingual semantic role labeling. the same models can be evaluated for both span-and dependencybased srl on different languages and on the same verb inventory. these features allow us to have a realistic view of the transfer learning and language adaptation capabilities of past, current and future srl systems, thereby enabling studies on cross-lingual transfer also in this task. we conducted an extensive evaluation of a stateof-the-art srl model on united-srl, through which we were able to validate different hypotheses on individual languages and across multiple languages. thanks to the shared verbal inventory employed by united-srl we were able to train with examples in different languages and to test the effect that this has on the performances of the model. the results obtained with our evaluation reinforce the idea that cross-lingual annotated data with predicate sense and semantic role labels from a single inventory shared across languages could open the door to the development of more robust cross-lingual srl systems. one of our most important findings was that with just 500 training examples in a language the performances of a model evaluated in a 0-shot setting was raised by more than 10 points in accuracy, encouraging the study of language adaptation techniques and the development of other small parallel datasets not only for other languages but also for other tasks. indeed, as future work, we plan to extend the number of languages covered by united-srl, starting with arabic and russian, which are already part of the un corpus, and then moving on to integrate low-resourced languages from other parallel corpora. manchester, england. coling 2008 organizing committee. mariona taulé, m. antònia martí, and marta recasens. 2008. ancora: multilevel annotated corpora for catalan and spanish. in proceedings of the sixth international conference on language resources and evaluation (lrec'08), marrakech, morocco. lonneke van der plas, paola merlo, and james henderson. 2011. scaling up automatic cross-lingual semantic role annotation. in proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies, pages 299-304, portland, oregon, usa. association for computational linguistics. nianwen xue and martha palmer. 2003. annotating the propositions in the penn chinese treebank. in proceedings of the second workshop on chinese language processing, sighan 2003, sapporo, japan, july 11-12, 2003. michał ziemski, marcin junczys-dowmunt, and bruno pouliquen. 2016. the united nations parallel corpus v1.0. in proceedings of the tenth international conference on language resources and evaluation (lrec'16), pages 3530-3534, portorož, slovenia.