Revisiting Tri-training of Dependency Parsers (2021)

Abstract: We compare two orthogonal semi-supervised learning techniques, namely tri-training and pretrained word embeddings, in the task of dependency parsing. We explore language-specific FastText and ELMo embeddings and multilingual BERT embeddings. We focus on a low resource scenario as semi-supervised learning can be expected to have the most impact here. Based on treebank size and available ELMo models, we select Hungarian, Uyghur (a zero-shot language for mBERT) and Vietnamese. Furthermore, we include English in a simulated low-resource setting. We find that pretrained word embeddings make more effective use of unlabelled data than tri-training but that the two approaches can be successfully combined.

## introduction pre-trained neural architectures and contextualised word embeddings are state-of-the-art approaches to combining labelled and unlabelled data in natural language processing tasks taking text as input. a large corpus of unlabelled text is processed once and the resulting model is either fine-tuned for a specific task or its hidden states are used as input for a separate model. in the task of dependency parsing, recent work is no exception to the above. however, earlier, pre-neural work explored many other ways to use unlabelled data to enrich a parsing model. among these, self-, co-and tritraining had most impact (charniak, 1997; steedman et al., 2003; mcclosky et al., 2006a,b; søgaard and rishøj, 2010; sagae, 2010). self-training augments the labelled training data with automatically labelled parse trees predicted by a baseline model in an iterative process: co-training proceeds similarly to self-training but uses two different learners, each teaching the other learner, i. e. output of learner a is added to the training data of learner b and vice versa. tri-training uses three learners and only adds predictions to a learner that the other two learners, the teachers, agree on. as with co-training, the roles of teachers are rotated so that all three learners can receive newly labelled data. we compare tri-training and contextualised word embeddings in the task of dependency parsing, using the same unlabelled data for both approaches. in this comparison, we will try to answer: we focus on low-resource languages as (a) maximising the benefits from semi-supervised learning even at high computational costs is most needed for low-resource languages, e. g. to reducing editing effort in the manual annotation of additional data, and (b) tri-training with high-resource languages comes at much higher computational costs as not only is the manually labelled data much larger but also the automatically labelled data can be expected to need to be at least equally larger to have a relevant effect. we select three low-resource languages, namely hungarian, uyghur and vietnamese, (see section 3.3 for selection criteria) and, english, simulating a low-resource scenario by sampling a subset of the available data. the results of our experiments show that 1) both tri-training and pretrained word embeddings offer an obvious improvement over a fully supervised approach, 2) pretrained word embeddings clearly outperform tri-training (between 2 and 5 las points, depending on the language), and 3) there is some merit in combining the two approaches since the best performing model for each of the four languages is one in which tri-training is applied with models which use pretrained embeddings. ## experimental setup this section describes the technical details of the experimental setup. ## development set results we compare semi-supervised learning with tritraining to semi-supervised learning with a combination of context-independent and contextualised word embeddings, and we compare these two approaches to combining them and to training without unlabelled data, i. e. supervised learning, addressing the three questions posed in section 1. as described in section 3.2, we obtain twelve las scores from tri-training with each parser and language. tri-training with t iterations can choose an ensemble from t + 1 ensembles (appendix e). to give the baselines the same number of models to choose from, we sample from the baseline ensembles described in section 3.6. for each tri-training run with t iterations, we select the best score from t +1 baseline scores. as the choice of the subset of t + 1 scores is random, we repeat the random sampling 250,000 times to get an accurate estimate of the las distribution. in other words, we simulate what would happen if the additional data obtained through tri-training had no effect on the parser. figure 1 compares the parsing performance with and without tri-training for the four development languages and for the three types of parsing models udpf, elmo and mbert. most distributions for each language are clearly separated and the order of methods is the same: both tri-training and external word embeddings yield clear improvements. external word embeddings have a much stronger effect than tri-training. in combination, the two semi-supervised learning methods yield a small additional improvement with an average score difference of over half an las point. ## error analysis in this section, we probe, using the development sets, how the error distribution changes as we add tri-training and/or word elmo embeddings trained on unlabelled data to the basic parser. we compare the following 1. tokens that are out-of-vocabulary relative to the manually labelled training data versus those that are in the training data ("oov"/"iv") 2. different sentence length distributions: up to 9, 10 to 19, 20-39, and 40 or more tokens 3. different dependency labels, e.g is there a marked difference in the effect of tri-training or word embeddings for particular label types, e.g. nsubj? how does tri-training help (with no embeddings)? as expected, tri-training brings a clearly greater improvement for oovs than ivs for all four languages. the role of sentence length is not consistent across languages. for english, tri-training helps most on longer sentences (> 20 words), for hungarian, short sentences (< 10 words), and for uyghur, very long sentences (> 40 words). sentence length does not appear to be a factor for vietnamese. regarding dependency labels, there are no clear pattern across languages. how do word embeddings help (with no tritraining)? analysis of improvement by sentence length and by oov status show similar trends to the tri-training improvements described above. across languages, the use of pretrained embeddings helps to correctly identify the flat relation (which is used in names and dates). ## test set results in this section, we verify to what extent our main observations on development data carry over to test data and include results for a parser using only fast-text as external word embedding. for each of the las distributions using tri-training in figure 1 and the distribution for fasttext not shown, we select the ensemble with highest development las for testing. since we also use model selection based on development las to choose the final model of each tri-training run from its t + 1 iterations, the best model is selected from a set of 50 models given the values of t listed in section 3.2, exceeding the number of baseline models available from iteration 0. for a fair comparison, we therefore leverage the 4096 baseline ensembles described in section 3.6. as a choice of 50 out of 4096 ensembles would introduce noise, we repeatedly draw samples, for each sample find the best model according to development las, obtain test las and report the average las over all samples, i. e. the expectation value. as was the case for development results, we run the linear tree combiner 21 times on the three individual predictions of the tri-training learners and take the average las over all combiner runs as the score of the ensemble. ## conclusion we compared two semi-supervised learning methods in the task of dependency parsing for three low-resource languages and english in a simulated low-resource setting. tri-training was effective but could not come close to the performance gains of contextualised word embeddings. combined, the two learning methods achieved small additional improvements between 0.2 las points for uyghur and 1.3 las points for vietnamese. whether these gains can justify the additional costs of tri-training will depend on the application. we recommend that users of tri-training vary settings and repeat runs to find good models. future work could therefore explore how to best combine the many models or the large amount of automatically labelled data that such experiments produce. to obtain a fast and strong final model, a combination of ensemble search and model distillation or up-training may be the next step. integrating crossview training (clark et al., 2018) into tri-training may also be fruitful similarly to the integration of multi-view learning in co-training (lim et al., 2020). the requirement of tri-training that two teachers must agree changes the sentence length distribution of the data selected and may introduce other biases. future work could try to counter this effect be re-sampling the predictions similarly to how droganova et al. (2018) corrected for such effects in self-training. while our literature review suggests that tritraining performs better than co-and self-training, it would be interesting how these methods compare under a fixed computation budget as the latter methods train fewer parsing models per iteration.