Scalable Cross-lingual Treebank Synthesis for Improved Production Dependency Parsers (2020)

Abstract: We present scalable Universal Dependency (UD) treebank synthesis techniques that exploit advances in language representation modeling which leverage vast amounts of unlabeled general-purpose multilingual text. We introduce a data augmentation technique that uses synthetic treebanks to improve production-grade parsers. The synthetic treebanks are generated using a state-of-the-art biaffine parser adapted with pretrained Transformer models, such as Multilingual BERT (M-BERT). The new parser improves LAS by up to two points on seven languages. The production models’ LAS performance improves as the augmented treebanks scale in size, surpassing performance of production models trained on originally annotated UD treebanks.

## introduction dependency parsers are important components in many nlp systems, such as language understanding, semantic role labeling and relation extraction @xcite . universal dependencies (ud) @xcite are becoming a widely accepted standard among many nlp practitioners for definition of syntactic structures and treebanks. however, production parsers require custom tokenization policies and part of speech (pos) tagging, mostly dictated by supported downstream applications. in addition, parsers in production environments require fine balancing of demands for model accuracy, service performance, response time and constraints on hardware resources, making the design of an industrial-grade parser a challenge. hereby, we introduce data augmentation techniques to improve production parsers without violating their architectural constraints. since their early inception, advances in language representation modeling lead to major improvements in many nlp tasks @xcite . representations trained on various language modeling objectives, ranging from context free embeddings @xcite , to deep context aware representations @xcite @xcite , were trained on massive amounts of unlabeled multilingual text, greatly enabling transfer learning opportunities for nlp tasks. particularly, models such as bert @xcite , albert @xcite , roberta @xcite and xlm @xcite employ a masked language modeling objective @xcite on a bidirectional self-attention encoder @xcite enabling such models to utilize both left and right context for each word representation. pretrained multilingual bert (m-bert) was used for dependency parsing in @xcite aiming to create a single multilingual model. this work, in contrast, shows that parsing performance for a particular language can considerably be improved when adapting the biaffine-attention parser @xcite with a selected set of pretrained transformer models while training on multilingual subsets of selected language family treebanks. we then use this novel parser to project synthetic treebanks, which are used in a teacher-student technique to improve the accuracy of a fast production parser. our approach can generally be described as a form of model compression which was introduced by @xcite , and later reformulated and generalized as neural network knowledge distillation by @xcite . however, instead of using a large number of ensemble for a teacher model, we use a deep neural network parser augmented with a large transformer-based pretrained model creating a new parser that advances the current state of the art. since the pretrained transformer model can be trained on large amounts of unlabeled monolingual and multilingual data of various domains and languages, the teacher model gains improved generalization performance that is facilitated by both cross-lingual and cross-domain transfer learning. our student model is a non-neural net based model that is designed to be fast and efficient in production environments. conventionally, parsers are trained on human annotated treebanks, which can be both costly and limited in quantity. certain languages may not have enough annotation resources, have very small amount of data, or data that carries non-commercial licenses. in other cases, the data may be available in a specific topical domain resulting in models that perform poorly on unrelated domains. in addition, annotation errors can be common in some treebanks. to address these data challenges, we use cross-lingual transfer learning and pretrained deep contextualized representations to create a novel parser that helps generate synthetic data. we describe a production parser trained on these data, whose performance increases as the synthetic data size surpasses that of human annotated data. ## system description the aim of our system is to produce synthetically labeled treebanks in order to significantly improve the accuracy of a production parser. the synthetic data will be generated using a different parser that is higher in quality. we create a new parser using two key components: the deep biaffine-attention parser @xcite and a pretrained transformer model. not only does such a setup improve the parsing accuracy, as shown in section 3, the incorporation of transformer models facilitates greater degree of generalization and domain adaptation. in the sections below, we detail the training data augmentation process as well as the new parser architecture. ## experiments in this section, we show results demonstrating the improved performance of the new tbap architecture on seven languages. we also show the effectiveness of the treebank synthesis technique when used in the augmented training of a production parser. ## conclusion and future work we presented a data augmentation approach for ud parsing that improves fast production parsers accuracy and overcomes critical treebank limitations. a new transformer enhanced biaffine parser is used to generate scalable synthetic data. we showed that utilizing deep contextualized representations pretrained on massive multilingual corpora can be used to considerably improve parsing accuracy. in the future, we plan to extending our method to generate synthetic data for additional languages.