Transferability of Syntax-Aware Graph Neural Networks in Zero-Shot Cross-Lingual Semantic Role Labeling (2024)

Abstract: Recent models in cross-lingual semantic role labeling (SRL) barely analyze the applicability of their network selection.We believe that network selection is important since it affects the transferability of cross-lingual models, i.e., how the model can extract universal features from source languages to label target languages.Therefore, we comprehensively compare the transferability of different graph neural network (GNN)-based models enriched with universal dependency trees.GNN-based models include transformer-based, graph convolutional network-based, and graph attention network (GAT)-based models.We focus our study on a zero-shot setting by training the models in English and evaluating the models in 23 target languages provided by the Universal Proposition Bank.Based on our experiments, we consistently show that syntax from universal dependency trees is essential for cross-lingual SRL models to achieve better transferability.Dependency-aware self-attention with relative position representations (SAN-RPRs) transfer best across languages, especially in the long-range dependency distance.We also show that dependency-aware two-attention relational GATs transfer better than SAN-RPRs in languages where most arguments lie in a 1-2 dependency distance.

## introduction semantic role labeling (srl) is a task to assign semantic roles to words or phrases in a sentence concerning a specific predicate, as shown in figure 1 . srl supports many natural language processing (nlp) tasks, e.g., information extraction @xcite , abstractive summarization @xcite , and machine translation @xcite . however, srl resource availability is still low, hindering the performance of other nlp tasks in diverse languages. cross-lingual approaches try to solve this problem by enriching the models with knowledge from resource-rich languages (source languages) so that they perform better in resource-poor languages (target languages). existing studies in cross-lingual srl improve existing models by (1) separating languageuniversal and language-specific components @xcite to capture both language-universal and language-specific features from each language and (2) minimizing the dependence on external tools, e.g., word alignment and machine translation tools @xcite . nevertheless, existing studies in cross-lingual srl barely analyze the network selection's applicability, which affects the models' transferability. therefore, in this work, we comprehensively compare diverse graph neural networks (gnns) to investigate their transferability in cross-lingual srl. various gnns have been shown to be useful to encode and transfer linguistic structures in similar tasks cross-lingually. furthermore, we encode universal dependency trees in the networks, for the following reasons: (1) many predicate-argument paths and argument roles in srl co-occur with dependency paths and dependency relations in depen-dency parsing @xcite (figure 1 ). ( 2 ) a universal dependency tree represents a sentence's grammatical structure in a languageuniversal scheme. (3) universal dependency trees help cross-lingual models achieve better transferability @xcite @xcite . we conduct comprehensive experiments on various networks as encoders, including transformerbased, graph convolutional network (gcn)-based, and graph attention network (gat)-based encoders, which we categorize as gnn-based encoders. we also compare the results with a bilstm-based encoder as a baseline. transformer-based models have been proven effective in performing crosslingually in dependency parsing @xcite and event argument role labeling (earl) @xcite . furthermore, we also investigate gcn-based and gat-based models because different nlp tasks, e.g., monolingual srl @xcite , cross-lingual srl @xcite , aspect-based sentiment analysis (absa) @xcite , and relation prediction @xcite , have shown the effectiveness of exploiting the networks to encode dependency trees in their models. following previous work @xcite , we limit our exploration to argument detection and argument labeling in the dependency-based srl. we conduct experiments in a zero-shot setting to find the most transferable network across languages. we train and evaluate the models in 23 languages provided by universal proposition bank (upb) v2. we show that: (1) universal dependency trees are essential for cross-lingual srl models to achieve better transferability. (2) a dependency-aware transformer-based model, i.e., san-rprs, outperforms other models, especially as the dependency distance increases. (3) a dependency-aware gatbased model, i.e., two-attention relational gats (tagats), transfer best in languages where most arguments lie in 1-2 dependency distance. in addition, our findings can also shed some light on selecting the most appropriate network for building cross-lingual models in similar tasks. ## models we apply a standard encoder-decoder architecture to compare transformer-based, gcn-based, gatbased, and bilstm-based cross-lingual srl models. section 3.1 describes a method to convert a dependency tree into a graph for building graph neural networks (gnns). section 3.2, 3.3, and 3.4 describe the input layer, encoder, and decoder of the architecture. the implementation of the model architecture is available on github.com/mynlp/ syntax-aware-cross-lingual-srl. ## conclusions and future work through comprehensive experiments, we consistently show that incorporating syntax from dependency trees can improve the transferability of crosslingual srl models across languages. overall, we show that the transformer-based model, i.e., san-rprs that encode dre without pose in the node representation and rpr in the edge representation, stacked in three layers, is the most transferable among all models, especially as the dependency distance increases. however, tagats that encode sape in the node representation and dr and rpr in the edge representation, stacked in two layers, transfer better than san-rprs in languages where most of the arguments lie in 1-2 dependency distance. in the future, we can extend the model training to a few-shot setting where we include a certain proportion of target sentences in the training set. we can modify the network's objective function and/or parameters to maximize the learning of universal features without ignoring the language-specific features.