Retrofitting Structure-aware Transformer Language Model for End Tasks (2020)

Abstract: We consider retrofitting structure-aware Transformer language model for facilitating end tasks by proposing to exploit syntactic distance to encode both the phrasal constituency and dependency connection into the language model. A middle-layer structural learning strategy is leveraged for structure integration, accomplished with main semantic task training under multi-task learning scheme. Experimental results show that the retrofitted structure-aware Transformer language model achieves improved perplexity, meanwhile inducing accurate syntactic phrases. By performing structure-aware fine-tuning, our model achieves significant improvements for both semantic- and syntactic-dependent tasks.

## introduction natural language models (lm) can generate fluent text and encode factual knowledge @xcite @xcite . recently, pre-trained contextualized language models have given remarkable improvements on various nlp tasks @xcite @xcite @xcite . among such methods, the transformer-based @xcite bert has become a most popular encoder for obtaining state-of-the-art nlp task performance. it has been shown @xcite that besides rich semantic information, implicit language structure knowledge can be captured by a deep bert @xcite @xcite . however, such structure features learnt via the vanilla transformer lm are insufficient for those nlp tasks that heavily rely on syntactic or linguistic knowledge @xcite . some effort devote to improved the ability of structure learning in transformer lm by installing novel syntax-attention mechanisms @xcite . nevertheless, several limitations can be observed. first, according to the recent findings by probing tasks @xcite @xcite , the syntactic structure representations are best retained right at the middle layers @xcite . nevertheless, existing tree transformers employ traditional full-scale training over the whole deep transformer architecture (as shown in figure 1(a) ), consequently weakening the upper-layer semantic learning that can be crucial for end tasks. second, these tree transformer methods encode either standalone constituency or dependency structure, while different tasks can depend on varying types of structural knowledge. the constituent and dependency representation for syntactic structure share underlying linguistic characteristics, while the former focuses on disclosing phrasal continuity and the latter aims at indicating dependency relations among elements. for example, semantic parsing tasks are more dependent on the dependency features @xcite , while constituency information is much needed for sentiment classification @xcite . in this paper, we aim to retrofit structure-aware transformer lm for facilitating end tasks. we conduct experiments on language modeling and a wide range of nlp tasks. results show that the structure-aware transformer retrofitted via our proposed middle-layer training strategy achieves better language perplexity, meanwhile inducing high-quality syntactic phrases. besides, the lm after structure-aware fine-tuning can give significantly improved performance for various end tasks, including semantic-dependent and syntacticdependent tasks. we also find that supervised structured pre-training brings more benefits to syntactic-dependent tasks, while the unsupervised lm pre-training brings more benefits to semanticdependent tasks. further experimental results on unsupervised structure induction demonstrate that different nlp tasks rely on varying types of structure knowledge as well as distinct granularity of phrases, and our retrofitting method can help to induce structure phrases that are most adapted to the needs of end tasks. ## related work contextual language modeling. contextual language models pre-trained on a large-scale corpus have witnessed significant advances @xcite @xcite @xcite . in contrast to the traditional static and context-independent word embedding, contextual language models can strengthen word representations by dynamically encoding the contextual sentences for each word during pre-training. by further fine-tuning with end tasks, the contextualized word representation from language models can help to give the most task-related context-sensitive fea-tures @xcite . in this work, we follow the line of transformer-based @xcite lm (e.g., bert), considering its prominence. structure induction. the idea of introducing tree structures into deep models for structure-aware language modeling has long been explored by supervised structure learning, which generally relies on annotated parse trees during training and maximizes the joint likelihood of sentence-tree pairs @xcite @xcite @xcite @xcite @xcite . there has been much attention paid to unsupervised grammar induction task @xcite @xcite @xcite @xcite . for example, prpn @xcite computes the syntactic distance of word pairs. on-lstm @xcite allows hidden neurons to learn long-term or shortterm information by a gate mechanism. urnng @xcite applies amortized variational inference, encouraging the decoder to generate reasonable tree structures. diora @xcite uses inside-outside dynamic programming to compose latent representations from all possible binary trees. pcfg @xcite achieves grammar induction by probabilistic context-free grammar. unlike these recurrent network based structure-aware lm, our work focuses on structure learning for a deep transformer lm. structure-aware transformer language model. some efforts have been paid for the transformerbased pre-trained language models (e.g. bert) by visualizing the attention @xcite @xcite or probing tasks @xcite . they find that the latent language structure knowledge is best retained at the middle-layer in bert @xcite @xcite . @xcite employ a decomposable attention mechanism for recursively learn the tree structure for transformer. wang et al. (2019) integrate tree structures into transformer via constituency-attention. however, these transformer lms suffer from the full-scale structural training and monotonous types of the structure, limiting the performance of structure lms for end ... ... weighted sum l th l-1 th l+1 th + + + + + + phrase generation structure learning module transformer encoders transformer layer syntax distance layer phrase context phrase embedding mlm or end task + + phrasal attention word context tasks. our work is partially inspired by shen et al. (2018a) and luo et al. (2019) on employing syntax distance measurements, while their works focus on the syntax learning by recurrent lms. ## model the proposed structure-aware transformer language model mainly consists of two components: the transformer encoders and structure learning module, which are illustrated in figure 2 . ## analysis we take a further step, evaluating the fine-grained quality on phrasal structure induction after pretraining. instead of checking whether the induced constituent spans are identical to the gold counterparts, we now consider measuring the devia- ## conclusion we presented a retrofitting method for structureaware transformer-based language model. we adopted the syntax distance to encode both the constituency and dependency structure. to relieve the conflict of structure learning and semantic learning in transformer lm, we proposed a middlelayer structure learning strategy under a multitasks scheme. results showed that structure-aware transformer retrofitted via our proposed method achieved better language perplexity, inducing highquality syntactic phrase. furthermore, our lm after structure-aware fine-tuning gave significantly improved performance for both semantic-dependent and syntactic-dependent tasks, also yielding most task-related and interpretable syntactic structures.