Biaffine Dependency and Semantic Graph Parsing for E nhanced U niversal Dependencies (2021)

Abstract: This paper presents the system used in our submission to the IWPT 2021 Shared Task . This year the official evaluation metrics was ELAS, therefore dependency parsing might have been avoided as well as other pipeline stages like POS tagging and lemmatization. We nevertheless chose to deploy a combination of a dependency parser and a graph parser. The dependency parser is a biaffine parser, that uses transformers for representing input sentences, with no other feature. The graph parser is a semantic parser that exploits a similar architecture except for using a sigmoid crossentropy loss function to return multiple values for the predicted arcs. The final output is obtained by merging the output of the two parsers. The dependency parser achieves top or close to top LAS performance with respect to other systems that report results on such metrics, except on low resource languages (Tamil, Estonian, Latvian).

## system overview the shared task 2021 aims specifically at performing enhanced dependency parsing, starting from raw text, in a multi-language setting consisting of seventeen languages @xcite . we concentrate on the syntactic parsing and enhancement stages, by exploiting existing tools for tokenization, sentence splitting. ## syntactic parsing state of the art dependency parsers currently often adopt the graph-based model, based on neural networks for the choice of arcs and labels. in particular the bi-lstm-based deep biaffine neural dependency parser by @xcite has been quite popular and used in three out of five of the top submissions to the conll 2018 shared task: multilingual parsing from raw text to universal dependencies @xcite , in particular in the top non-ensemble submission @xcite . we trained our own models for each language on the shared task treebanks using diaparser, which uses the stanza tokenizer and multi-word splitter. ## pretrained multilingual model after the submission deadline, we experimented building a single model on the concatenation of the training corpora of all languages. the corpora was preprocessed to eliminate empty nodes, which represent implicit nodes, denoted with ids such as 2.1 in the con-llu file format. we used the official script enhanced collapse empty nodes.pl, which collapses graphs reducing such empty nodes into non-empty nodes and introducing new dependency labels. we used the official script to collapse graphs through reducing such empty nodes into non-empty nodes and introducing new dependency labels. in the post-process, we add empty nodes according to the dependency labels. as the official evaluation only score the collapsed graphs, such a process does not impact the system performance. then the enhanced dependency labels in the training corpus were de-lexicalized, stripping lexical information from labels, like in @xcite , replacing them with placeholders (e.g. obl:[case]) indicating where in the dependency graph the lexical information is expected to be found. this process allowed us to reduce the total number of enhanced dependency labels from 6125 to 1282. this also made it possible to fit the model to be trained into the 32gb of memory of our v100 gpu. we run the training in parallel on 4 such gpus: each epoch took about 45 minutes and run for 29 epochs. the model was trained using contextualized word embeddings from roberta @xcite , more precisely xlm-roberta-large from huggingface foot_2 using a scalar mixture of the top 4 hidden layers @xcite . then the model was fine tuned on each language with its specific language corpus. the enhanced dependency labels in the output of the parser are converted back to their lexical notation using a heuristic processing similar to the one outlined in @xcite : furthermore, for languages that have case morphology, like czech, the case is added to the label. the multilingual model does provide significant improvements for languages with smaller corpora, in particular latvian, lithuanian and tamil, as shown in notably lithuanian improves on eulas by 6.75 points. the elas scores do not improve as much, possibly due to the ri-lexicalization algorithms that may need tuning to each language. ## conclusions we experimented using two parsers with the same architecture to perform syntactic and semantic parsing. we first trained parser models on the specific corpus for each language. the final output is obtained by merging the outputs of the two parsers. this simple approach works reasonably well for languages with large enough corpora. to address the difficulty in handling low resource languages, we explored building a single model trained on all corpora and fine tuning it on each specific corpora. since enhanced dependency labels contain lexical parts and the number of such labels is quite large, we adopted a preprocessing step to de-lexicalize the labels. the approach gave promising results on some languages, but the backconvertion algorithm that introduces the lexical parts in the labels after parsing still needs to be improved. given the similarity of the architectures of the syntactic and semantic parsers, the prospect of performing joint training is promising and has been