D eep S truct: Pretraining of Language Models for Structure Prediction (2022)

Abstract: We introduce a method for improving the structural understanding abilities of language models. Unlike previous approaches that finetune the models with task-specific augmentation, we pretrain language models to generate structures from the text on a collection of task-agnostic corpora. Our structure pretraining enables zero-shot transfer of the learned knowledge that models have about the structure tasks. We study the performance of this approach on 28 datasets, spanning 10 structure prediction tasks including open information extraction, joint entity and relation extraction, named entity recognition, relation classification, semantic role labeling, event extraction, coreference resolution, factual probe, intent detection, and dialogue state tracking. We further enhance the pretraining with the task-specific training sets. We show that a 10B parameter language model transfers non-trivially to most tasks and obtains state-of-the-art performance on 21 of 28 datasets that we evaluate. Our code and datasets will be made publicly available.

## introduction pretrained language models (lms) have revolutionized nlp over the last few years @xcite @xcite , increasingly adept in performing the flexible and task-agnostic downstream transfer. their transfer performance is less studied in structure prediction tasks, however. well-studied tasks mainly focus on understanding one particular aspect of the text, such as predicting the next word that comes after as in language modeling. unlike those downstream tasks, structure prediction requires the structural understanding of the text for further integrating multiple relevant aspects into a structure. for instance, a typical structure prediction task, called open information extraction, seeks the entire structural in- † equal contribution. 1 the code and datasets are available at https:// github.com/cgraywang/deepstruct . formation in a sentence (figure 2 ). different from traditional nlp tasks, structure prediction takes one step further and serves as a natural testbed for the structural understanding competence of lms. it is non-trivial to transfer lms to downstream structure prediction tasks. while the structure prediction requires structural understanding, the lms are pretrained to understand an independent aspect. for example, @xcite is trained to predict the next word, and bert @xcite is trained to recover the masked tokens. recent work has made efforts in bridging the gap in transferring pretrained models to structure prediction tasks with a focus on two directions. as shown in figure 3 , first, task-specific architectures are proposed to model the structures for different structure prediction tasks @xcite . second, task-specific data augmentation @xcite @xcite is introduced, aiming to enrich text format with structure information. these approaches involve custom-designed task augmentations, impeding their usability in general structure prediction tasks. in this paper, we improve the structural understanding capabilities of lms. in contrast to previous approaches relying on task augmentations, we introduce structure pretraining, which systematically teaches lms to better understand structures of text beyond independent aspects in a pretraining phase (figure 1 ). this enables the zero-shot transfer of knowledge that lms learned about structures during our pretraining to downstream structure prediction tasks. for example, our zero-shot 10b parameter lm significantly outperforms the zero-shot gpt-3 (175b) on a structure prediction benchmark dataset (figure 1 ). we accomplish this by reformulating structure prediction as a series of unit taskstriple prediction tasks. we then train lms on a collection of task-agnostic structural corpora to generate triples from text. the design of triple representation is important: it unifies a wide set of standard structure prediction tasks into the same task format. we apply our pretrained model deepstruct to 28 datasets spanning 10 structure prediction tasks, including open information extraction, joint entity and relation extraction, named entity recognition, relation classification, semantic role labeling, event extraction, coreference resolution, factual probe, intent detection, and dialogue state tracking. we further enhance the pretraining with multiple downstream structure prediction training sets and obtain state-of-the-art performance on 21 of 28 datasets. our contributions are as follows: pretrained lm (1) task augmented pretrain-finetune task augmentation finetune on task 0 inference on task 0 pretrained lm structure-pretrain on task-agnostic tasks inference on task 0,1,2,3,... (2) structure pretraining (ours) requires task-specific architectures or data augmentation requires task-specific examples a task-specific model for each task lm learns to generate structural triples from text inference on multiple tasks zero-shot pretrained lm structure-pretrain on task-agnostic tasks and multiple tasks: 0,1,2,3,... inference on task 0,1,2,3,... lm learns to generate structural triples from text with additional task-specific examples ## experiments in this section, we show that deepstruct successfully transfers to the structure prediction tasks considered and obtain state-of-the-art results on 21 of 28 datasets we evaluate. all results are obtained via structure pretraining a pretrained 10b parameter lm, glm @xcite . the details of the experimental setup, datasets, and comparison methods are described in appendix a. ## related work pretrained lms @xcite @xcite are the key ingredients in contemporary nlp. sequence-to-sequence (seq2seq) lms target conditional generation, such as t5 @xcite , bart @xcite and glm @xcite . these models have benefited a wide range of nature language generation tasks such as summarization @xcite ) and text infilling @xcite . recent attempts of generative prediction @xcite @xcite have found that seq2seq models are able to provide a unified solution for modeling a wide set of nlp tasks. while existing approaches focus on text-to-text generation, deep-struct aims to perform text-to-triple generation. multi-task learning @xcite aims to train a model for multiple tasks simultaneously. in deep learning, it is usually categorized into hard weight sharing and soft weight constraints @xcite . in the context of nlp, weight sharing has been adopted in @xcite @xcite . since the emergence of large pretrained lms @xcite @xcite , multi-task training has been shown effective to enhance lms' transferability to downstream tasks @xcite . recent studies @xcite also show that pretrained models finetuned with abundant downstream tasks can conduct effective zero-shot learning. the main difference is that deepstruct trains across multiple structure prediction datasets in structure pretraining with task-agnostic corpora, where we cast all datasets into triple formats. structure prediction is a long-standing challenge that relates to many nlp applications such as open information extraction @xcite , named entity recognition @xcite , and relation classification @xcite @xcite . to handle different structure prediction problems, prior work presents a variety of task-specific models in the form of sequence tagging @xcite , machine reading comprehension @xcite ) and text classification @xcite , which hinders the knowledge transfer across different tasks. tanl @xcite proposes a translation-based approach to unify different structure prediction tasks with task-specific data augmentation. by contrast, our deepstruct unifies more structure prediction tasks via a single model and a uniform data format. ## discussion related models recent studies have provided unified solutions for structural prediction tasks. we focus on the comparison between our deep-struct to the state-of-the-art tanl @xcite and deepex @xcite . tanl @xcite proposes task-specific data augmentation (i.e., augmented natural language) that annotates task information and predictions in the input and output respectively for each structure prediction task. the main difference is that deepstruct decomposes the structure prediction tasks into a collection of triple generation tasks. the triple format serves as the unified representation for all considered structure prediction tasks without the need of introducing new data augmentation as in tanl. while tanl mainly works in the multi-task setting, we additionally enable the zero-shot transfer via the task-agnostic structure pretraining. deepex @xcite explores the attention matrices of pretrained lms via beam search to generate triples for information extraction tasks. following the search, deepex introduces an extra ranking stage to improve the quality of the triples. differently, deepstruct aims to generate the triples for a wide set structure prediction tasks in an end-to-end fashion thanks to the proposed structure pretraining. besides, both tanl and deepex explore relatively small-scale pretrained lms. instead, deep-struct scales up to 10 billion parameters. figure 6 shows that the performance improvements follow the scaling law @xcite @xcite ; liu et al., 2021). based on our results, deepstruct generalizes better to more structure prediction tasks compared to tanl and deepex. zero-shot setup for our zero-shot setup, we follow the zero-shot usage in recent pretrained lm studies @xcite @xcite . it refers to the setting where a pretrained model is used to directly perform downstream tasks without including downstream training sets in its own pretraining data. for deep-struct, our pretraining data is task-agnostic. for each task, we build an offline alignment between the schema of the pretraining data and the task dataset based on co-occurrence information in the pretraining data and downstream data @xcite . we then manually curate the alignment. the resulting schema alignment is part of our release 1 . at test time, we convert each task to one or a combination of the pretraining tasks based on figure 5 : entity, relation, or triple prediction. after producing the triples, we use the pre-built schema alignment to obtain the task predictions. for gpt-3 zero-shot setting, we follow the prompting method introduced by @xcite . in more details, we aim to test the upper bound performance of gpt-3 for structure prediction, in particular the jer task. therefore, instead of using standard prompts in the form of question answering, we design the prompts for "trueor-false" questions based on the ground truth. in this case, gpt-3 only answers with "yes" or "no" to produce a task prediction (figure 7 ), which is apparently an easier task compared to generating the structures from scratch. ## conclusion we improve structural understanding capabilities of language models. we evaluate it on a wide set of structure prediction tasks including 10 tasks and 28 datasets, and successfully transfer pretrained language models to them through the proposed structure pretraining, which teaches language models to output triples from the text. we enable both zero-shot and multi-task transfer learning. deep-struct obtains state-of-the-art results on 21 of 28 datasets. the result shows that pretrained language models can handle higher-level understanding (e.g., structural understanding), which may benefit more nlp tasks. we hope it will foster future research along the language structural understanding direction.