LIMIT - BERT : Linguistics Informed Multi-Task BERT (2020)

Abstract: In this paper, we present Linguistics Informed Multi-Task BERT (LIMIT-BERT) for learning language representations across multiple linguistics tasks by Multi-Task Learning. LIMIT-BERT includes five key linguistics tasks: Part-Of-Speech (POS) tags, constituent and dependency syntactic parsing, span and dependency semantic role labeling (SRL). Different from recent Multi-Task Deep Neural Networks (MT-DNN), our LIMIT-BERT is fully linguistics motivated and thus is capable of adopting an improved masked training objective according to syntactic and semantic constituents. Besides, LIMIT-BERT takes a semi-supervised learning strategy to offer the same large amount of linguistics task data as that for the language model training. As a result, LIMIT-BERT not only improves linguistics tasks performance but also benefits from a regularization effect and linguistics information that leads to more general representations to help adapt to new tasks and domains. LIMIT-BERT outperforms the strong baseline Whole Word Masking BERT on both dependency and constituent syntactic/semantic parsing, GLUE benchmark, and SNLI task. Our practice on the proposed LIMIT-BERT also enables us to release a well pre-trained model for multi-purpose of natural language processing tasks once for all.

## introduction recently, pre-trained language models have shown greatly effective across a range of linguistics inspired natural language processing (nlp) tasks such as syntactic parsing, semantic parsing and so on @xcite @xcite @xcite , when taking the latter as downstream tasks for the former. in the meantime, introducing linguistic clues such as syntax and semantics into the pre-trained language models may furthermore enhance other downstream tasks such as various natural language understanding (nlu) tasks @xcite . however, nearly all existing language models are usually trained on large amounts of unlabeled text data @xcite , without explicitly exploiting linguistic knowledge. such observations motivate us to jointly consider both types of tasks, pre-training language models, and solving linguistics inspired nlp problems. we argue such a treatment may benefit from two-fold. (1) joint learning is a better way to let the former help the latter in a bidirectional mode, rather than in a unidirectional mode, taking the latter as downstream tasks of the former. since universal language representations are learning by leveraging large amounts of unlabeled data which has quite different data volume compared with linguistics tasks dataset such as penn treebank (ptb) 1 @xcite . to alleviate such data unbalance on multi-task learning, we apply semi-supervised learning approach that uses a pre-trained linguistics model foot_1 to annotate large amounts of unlabeled text data and to combine with gold linguistics task dataset as our final training data. for such pre-processing, it is easy to train our limit-bert on large amounts of data with many tasks concurrently by simply summing up all the concerned losses together. moreover, since every sentence has labeled with predicted syntax and semantics, we can furthermore improve the masked training objective by fully exploiting the known syntactic or semantic constituents during the language model training process. unlike the previous work mt-dnn @xcite which only fine-tunes bert on glue tasks, our limit-bert is trained on large amounts of data in a semi-supervised way and firmly supported by explicit linguistic clues. we verify the effectiveness and applicability of limit-bert on propbank semantic parsing foot_2 in both span style @xcite and dependency style, (conll-2009) @xcite and penn treebank (ptb) @xcite for both constituent and dependency syntactic parsing. our empirical results show that semantics and syntax can indeed benefit the language representation model via multitask learning and outperforms the strong baseline whole word masking bert (bert wwm ). ## tasks and datasets limit-bert includes five types of downstream tasks: part-of-speech, constituent and dependency syntactic parsing, span and dependency semantic role labeling (srl). both span (constituent) and dependency are two broadly-adopted annotation styles for either semantics or syntax, which have been well studied and discussed from both linguistic and computational perspectives @xcite . constituency parsing aims to build a constituency-based parse tree from a sentence that represents its syntactic structure according to a phrase structure grammar. while dependency parsing identifies syntactic relations (such as an adjective modifying a noun) between word pairs in a sentence. the constituent structure is better at disclosing phrasal continuity, while the dependency structure is better at indicating dependency relation among words. semantic role labeling (srl) is dedicated to recognizing the predicate-argument structure of a sentence, such as who did what to whom, where and when, etc. for argument annotation, there are two formulizations. one is based on text spans, namely span-based srl. the other is dependency-based srl, which annotates the syntactic head of argument rather than the entire argument span. srl is an important method to obtain semantic information beneficial to a wide range of nlp tasks @xcite . bert is typically trained on quite large unlabeled text datasets, bookscorpus and english wikipedia, which have 13gb plain text, while the datasets for specific linguistics tasks are less than 100mb. thus we employ semi-supervised learning to alleviate such data unbalance on multi-task learning by using a pre-trained linguistics model to label bookscorpus and english wikipedia data. the pre-trained model jointly learns pos tags and the four types of structures on semantics and syntax, in which the latter is from the xlnet version of @xcite , giving state-of-the-art or comparable performance for the concerned four parsing tasks. during training, we set 10% probability to use gold syntactic parsing and srl data: penn treebank (ptb) @xcite , span style srl @xcite and dependency style srl (conll-2009) @xcite . ## conclusions in this work, we present limit-bert which applies multi-task learning with multiple linguistic tasks by semi-supervised learning. we use five key syntax and semantics tasks : part-of-speech (pos) tags, constituent and dependency syntactic parsing, span and dependency semantic role labeling (srl). and further improve the masking strategy of bert training by effectively exploiting the available syntactic and semantic clues for language model training. the experiments show that limit-bert outperforms the strong baseline bert wwm on four benchmark parsing treebanks and two nlu tasks. the results of glue and snli empirically illustrate that incorporating linguistic knowledge into pre-training language bert by multi-task and semi-supervised learning can also enhance downstream tasks. there are many future areas to explore to improve limit-bert, including a deeper understanding of model structure sharing in mtl, a more effective training method that leverages relat-edness among multiple tasks, for both fine-tuning and pre-training, and ways of incorporating the linguistic structure of text in a more explicit and controllable manner.