On the Benefit of Syntactic Supervision for Cross-lingual Transfer in Semantic Role Labeling (2021)

Abstract: Although recent developments in neural architectures and pre-trained representations have greatly increased state-of-the-art model performance on fully-supervised semantic role labeling (SRL), the task remains challenging for languages where supervised SRL training data are not abundant. Cross-lingual learning can improve performance in this setting by transferring knowledge from high-resource languages to low-resource ones. Moreover, we hypothesize that annotations of syntactic dependencies can be leveraged to further facilitate cross-lingual transfer. In this work, we perform an empirical exploration of the helpfulness of syntactic supervision for crosslingual SRL within a simple multitask learning scheme. With comprehensive evaluations across ten languages (in addition to English) and three SRL benchmark datasets, including both dependency- and span-based SRL, we show the effectiveness of syntactic supervision in low-resource scenarios.

## introduction the task of semantic role labeling (srl) annotates predicate-argument structures in text and is thus a desirable output of natural language processing (nlp) pipelines designed to extract information from text @xcite . recent developments in neural architectures @xcite and pre-trained contextualized representations @xcite have greatly improved the performance of srl systems @xcite @xcite . however, most previous work focuses on high-resource english srl scenarios, and it remains a challenge to extend these approaches, which require plentiful supervised examples, to other languages where training resources may be limited. a popular approach addressing this challenge is cross-lingual learning: leveraging the shared struc-tures across human languages to transfer knowledge from high-resource languages to low-resource ones. model transfer, where an srl model is directly transferred across languages using shared representations @xcite @xcite , is a particularly promising approach thanks to recent developments in multilingual contextualized representations @xcite , which have proven effective for cross-lingual transfer @xcite . another common strategy for improving srl model performance in both high-and low-resource scenarios is incorporating syntactic information. syntactic analysis was until recently considered a prerequisite for most srl systems @xcite and has been shown to benefit recent neural models as well @xcite @xcite . despite much work exploring cross-lingual learning and incorporating syntactic information into srl systems, most such previous work explores these two avenues separately, though there are numerous reasons that carefully incorporating syntax into a cross-lingual system for srl could provide further benefits: first, whereas srl annotations are limited to only about a dozen languages, much richer resources are available for syntax, thanks to the development of the universal dependencies (ud) framework and accompanying corpora @xcite , which defines syntactic annotations that are consistent across languages, with treebanks in over 100 languages to date. second, ud treebanks in particular have the potential to increase beneficial sharing of information across languages by providing a unified syntactic structure to ground cross-lingual representations. most previous work utilizing syntax for crosslingual srl have incorporated syntactic information only as an input to the model, either as sparse features @xcite or as structures for tree encoders @xcite . these strategies require syntactic pre-processing by an additional model and can suffer from error propagation. in this work, we explore an alternative approach that has yet to be explored in the cross-lingual setting: adopting syntactic annotations as auxiliary supervision and performing multitask learning @xcite together with srl @xcite @xcite . to evaluate the extent to which syntactic supervision can help facilitate cross-lingual transfer in srl, we perform a comprehensive empirical analysis on three srl benchmark datasets, covering ten languages (in addition to english). we evaluate our models in both zero-shot and semi-supervised scenarios, and on both dependency-and span-based srl. highlights of our findings include: our implementation is available at https:// github.com/zzsfornlp/zmsp/ . ## model we adopt the typical encoder-decoder paradigm for multi-task learning to perform syntactic dependency parsing and srl together in one model. a shared encoder gives the hidden representations for the input words and each task has its own decoder that takes those shared representations as inputs and predicts task-specific labels. we hypothesize that syntactic training can provide helpful signals for srl through the shared encoder. ## conclusion in this work, we provide a comprehensive empirical exploration of the helpfulness of syntactic supervision for cross-lingual srl. with extensive evaluations across a variety of datasets and settings, we show that auxiliary syntactic signals are generally beneficial, especially in low-resource srl cases. we hope that this work can shed some light on the relations between syntax and srl in the cross-lingual scenarios.