Enhancing Structure-aware Encoder with Extremely Limited Data for Graph-based Dependency Parsing (2022)

Abstract: Dependency parsing is an important fundamental natural language processing task which analyzes the syntactic structure of an input sentence by illustrating the syntactic relations between words. To improve dependency parsing, leveraging existing dependency parsers and extra data (e.g., through semi-supervised learning) has been demonstrated to be effective, even though the final parsers are trained on inaccurate (but massive) data. In this paper, we propose a frustratingly easy approach to improve graph-based dependency parsing, where a structure-aware encoder is pre-trained on auto-parsed data by predicting the word dependencies and then fine-tuned on gold dependency trees, which differs from the usual pre-training process that aims to predict the context words along dependency paths. Experimental results and analyses demonstrate the effectiveness and robustness of our approach to benefit from the data (even with noise) processed by different parsers, where our approach outperforms strong baselines under different settings with different dependency standards and model architectures used in pre-training and fine-tuning. More importantly, further analyses find that only 2K auto-parsed sentences are required to obtain improvement when pre-training vanilla BERT-large based parser without requiring extra parameters.

## introduction dependency parsing aims to produce the syntactic structure of a sentence by illustrating the syntactic relations between words, where the words with dependency relations are connected by directed and labeled arcs. it is an important fundamental natural language processing (nlp) task that is widely used to enhance downstream nlp tasks @xcite @xcite @xcite @xcite @xcite such as coreference resolution, relation extraction, and sentiment analysis. to produce the dependency structure of a sentence, the contextual information is of great importance to achieve good model performance. thus, most recent studies @xcite @xcite @xcite leverage advanced encoders (e.g., bi-lstm, transformer @xcite ) to model the contextual information of the input and obtain outstanding performance. in addition, because leveraging different models to obtain better results is an important technique for many nlp tasks @xcite @xcite , many previous studies apply this technique to dependency parsing to further improve model performance. under this paradigm, many studies utilize semi-supervised methods (e.g., self-training) to benefit from auto-processed extra data which is used to extract useful features @xcite @xcite @xcite or training data @xcite @xcite . however, since the auto-generated parse tree is not always accurate, semi-supervised methods need to handle the noise with care to achieve better performance @xcite . to address the noise issue, in this paper, we propose to apply pre-training and fine-tuning to enhance dependency parsing with the auto-parsed data generated by existing parsers. although the effectiveness of pre-training and fine-tuning paradigm has been demonstrated to leverage extra data in many nlp tasks, it is still worth studying whether this paradigm works well for dependency figure 1 : our parser is trained in two stages: pre-training with auto-parsed data (figure 1(a) ) and fine-tuning with gold dependency trees (figure 1(b) ). fine-tuning uses the same encoder architecture as in pre-training but further adjusts its weights. in contrast, the decoder for fine-tuning is different from the one in pre-training and its weights are randomly initialized. ## training the dependency parser in this study, we use neural graph-based dependency parsers, because they have achieved state-ofthe-art performance on this task @xcite @xcite . specifically, training our graph-based parser follows a two-stage procedure with pre-training and fine-tuning, where pre-training is performed on auto-parsed data with the same object to train a dependency parser, and the fine-tuning is conducted on the gold dependency trees with the pretrained encoder (other modules in the final parser are freshly learned in fine-tuning). figure 1 shows the model architecture and the two-stage procedure to train our final dependency parser. in the following text, we firstly introduce the neural graph-based dependency parser and then illustrate the process to pre-train the structureaware encoder. ## related work recent studies for dependency parsing use advanced architectures (e.g., bi-lstm, bert) to capture contextual information so as to achieve outstanding performance @xcite @xcite . to further improve dependency parsing, approaches such as bi-affine attentions @xcite @xcite , hpsg parsing @xcite @xcite , treecrf @xcite are further applied. besides, to improve model performance, there are studies that use existing dependency parsers and auto-parsed data through model ensemble @xcite @xcite or semi-supervised approaches @xcite @xcite @xcite . compared to previous studies that use autoparsed data, our approach differs in several ways. first, our encoder is structure-aware as it is pretrained with dependency trees. second, because auto-parsed data is noisy and may use dependency standard different from that of the test data (in the cross-standard setting), it is used in pre-training only. in contrast, training a parser on the union of auto-parsed data and gold data would not work well, especially in the cross-standard setting, as shown in ## conclusion in this study, we propose a simple and effective solution to improve dependency parser through pretraining on auto-parsed data. in doing so, the encoder is able to learn structural information from the auto-parsed data in pre-training. during finetuning, a different decoder is used and its weights initialized randomly, thus reducing the impact of errors in the auto-parsed data. we have run a large number of experiments under different settings (e.g., cross-domain vs. cross-standard, -bf vs. +bf, different parsers used to parse wiki) and shown that our approach outperforms strong baselines and many previous studies under those settings. furthermore, pre-training needs only a small amount of auto-parsed data (e.g., 2k sentences for a bert-large based parser on the ptb test set) to ensure improvement over strong baselines.