Not All Linearizations Are Equally Data-Hungry in Sequence Labeling Parsing (2021)

Abstract: Different linearizations have been proposed to cast dependency parsing as sequence labeling and solve the task as: (i) a head selection problem, (ii) finding a representation of the token arcs as bracket strings, or (iii) associating partial transition sequences of a transition-based parser to words. Yet, there is little understanding about how these linearizations behave in low-resource setups. Here, we first study their data efficiency, simulating data-restricted setups from a diverse set of rich-resource treebanks. Second, we test whether such differences manifest in truly low-resource setups. The results show that head selection encodings are more data-efficient and perform better in an ideal (gold) framework, but that such advantage greatly vanishes in favour of bracketing formats when the running setup resembles a real-world low-resource configuration.

## introduction dependency parsing @xcite has achieved clear improvements in recent years, to the point that graph-based @xcite and transition-based @xcite parsers are already very accurate on certain setups, such as english news. in this line, @xcite have pointed out that the performance on these setups is already on par with that expected from experienced human annotators. thus, the efforts have started to focus on related problems such as parsing different domains or multi-lingual scenarios @xcite @xcite , creating faster models @xcite , designing low-resource and cross-lingual parsing techniques @xcite , or infusing syntactic knowledge into models @xcite . this work will lie in the intersection between fast parsing and low-resource languages. recent work has proposed encodings to cast parsing as sequence labeling @xcite @xcite @xcite . this approach computes a linearized tree of a sentence of length n in n tagging actions, providing a good speed/accuracy trade-off. also, it offers a naïve way to infuse syntactic information as an embedding or feature @xcite . such encodings have been evaluated on english and multi-lingual setups, but there is no study about their behaviour on low-resource setups, and what strengths and weaknesses they might exhibit. ## related work low-resource parsing has been explored from perspectives such as unsupervised parsing, data augmentation, cross-lingual learning, or dataefficiency of models. for instance, on unsupervised parsing, @xcite and @xcite have worked on generative models to determine whether to continue or stop attaching de-pendents to a token, while others @xcite have studied how to use self-training for unsupervised parsing. on data augmentation, @xcite used self-training to annotate extra data, while others have focused on linguistically motivated approaches to augment treebanks. this is the case of @xcite or @xcite , who have proposed methods to replace subtrees within a given sentence. on cross-lingual learning, authors such as @xcite trained delexicalized parsers in a source rich-resource treebank, which are then used to parse a low-resource target language. @xcite explored lexicalized versus delexicalized parsers and compared them on low-resource treebanks, depending on factors such as the treebank size and the pos tags performance. @xcite created synthetic treebanks that resemble the target language by permuting constituents of distant treebanks. @xcite and @xcite tackled this same issue, but from the model side, training on rich-resource languages in such way the model learns to detect the aspects of the source languages that are relevant for the target language. recently, @xcite used a lstm to build a polyglot language model, which is then used to train on top of it a parser that shows cross-lingual abilities in zero-shot setups. on data-efficiency, research work has explored the impact of the use of different amounts of data, motivated by the lack of annotated data or by the lack of quality of it. for instance, @xcite showed how a transition-based parser with a dynamic oracle can be used without any modifications to parse partially annotated data. they found that this setup is useful to train low-resource parsers on sentence-aligned texts, from a rich-resource treebank to an automatically translated low-resource language, where only precisely aligned tokens are used for the projection in the target dataset. @xcite studied the effect that pre-processing and post-processing has in annotation projection, and concluded that quality should prevail over quantity. related to training with restricted data, @xcite showed that when distilling a graph-based parser for faster inference time, models with smaller treebanks suffered less. @xcite also distilled models for enhanced universal dependencies (eud) parsing with different amounts of data, observing that less training data usually translated into slightly lower performance, while offering better energy consumption. @xcite showed, in the context of romance languages, that peeking samples from related languages and adapting them to the target language is useful to train a model that performs on par with one trained on fully (but still limited) manually annotated data. restricted to constituent parsing, @xcite analyzed the role of the dev data in unsupervised parsing. they pointed out that many unsupervised parsers use the score on the dev set as a signal for hyperparameter updates, and show that by using a handful of samples from that development set to train a counterpart supervised model, the results outperformed those of the unsupervised setup. finally, there is work describing the impact that the size of the parsing training data has on downstream tasks that use syntactic information as part of the input @xcite . ## preliminaries in what follows, we review the existing families of encodings for parsing as sequence labeling ( §3.1) and the models that we will be using ( §3.2). ## methodology and experiments we design two studies, detailed in §4.1 and 4.2: ## conclusion we have studied sequence labeling encodings for dependency parsing in low-resource setups. first, we explored which encodings are more dataefficient under different conditions that include the use of gold pos tags, predicted pos tags and no pos tags as part of the input. by restricting training data for rich-resource treebanks, we observe that although bracketing encodings are less data-efficient than head-selection ones under ideal conditions, this disadvantage can vanish when the input conditions are not gold and data is limited. second, we studied their performance under the same running configurations, but on truly low-resource languages. these results show more clearly the greatest utility of bracketing encodings over the rest of the ones when training data is limited and the quality of external factors, such as pos tags, is affected by the low-resource nature of the problem.