Autoregressive Structured Prediction with Language Models (2022)

Abstract: Recent years have seen a paradigm shift in NLP towards using pretrained language models (PLM) for a wide range of tasks. However, there are many difficult design decisions to represent structures (e.g. tagged text, coreference chains) in a way such that they can be captured by PLMs. Prior work on structured prediction with PLMs typically flattens the structured output into a sequence, which limits the quality of structural information being learned and leads to inferior performance compared to classic discriminative models. In this work, we describe an approach to model structures as sequences of actions in an autoregressive manner with PLMs, allowing in-structure dependencies to be learned without any loss. Our approach achieves the new state-of-the-art on all the structured prediction tasks we looked at, namely, named entity recognition, end-to-end relation extraction, and coreference resolution.

## introduction many common nlp tasks, e.g., named entity recognition, relation extraction, and coreference resolution are naturally taxonomized as structured prediction, the supervised machine-learning task of predicting a structure from a large foot_0 set. to generalize well to held-out data in a structured prediction problem, the received wisdom has been that it is necessary to correctly model complex dependencies between different pieces of the structure. however, a recent trend in structured prediction for language has been to forgo explicitly modeling such dependencies @xcite @xcite , and, instead, to apply an expressive black-box model, e.g., a neural network, with the hope that the model picks up on the dependencies without explicit instruction. framing structured prediction as conditional language modeling is an increasingly common black-box technique for building structured predictors that has led to empirical success @xcite @xcite @xcite . the idea behind the framework is to encode the target structure as a string, flattening out the structure. then, one uses a conditional language model to predict the flattened string encoding the structure. for instance, @xcite flatten parse trees into strings and predict the strings encoding the flattened trees from the sentence with a machine translation architecture. the hope is that the autoregressive nature of the language model allows it to learn to model the intra-structure dependencies and the necessary hard constraints that ensure the model even produces well-formed structures. additionally, many modelers make use of pre-trained language models @xcite to further improve the language models. however, despite their empirical success, simply hoping that a black-box approach correctly models intricate intra-structure dependencies is often insufficient for highly structured tasks @xcite . indeed, the act of flattening a structured object into a string makes properly modeling the intra-structure dependencies harder for many tasks, e.g., those that involve nested spans or long-distance dependencies. for instance, in coreference resolution, a conference link between two mentions can stretch across thousands of words, and a coreference chain can also contain over a hundred mentions @xcite . flattening such a large amount of structured information into a string makes the task more difficult to model. in this paper, we propose a simple framework that augments a conditional language model with explicit modeling of structure. instead of modeling strings that encode a flattened representation of the target structure, we model a constrained set of actions that build the target structure step by step; see fig. 1 for an example of our proposed ## autoregressive structured prediction in this section, we describe our proposed approach, which we refer to as autoregressive structured prediction (asp). unlike previous approaches for structured prediction based on conditional language modeling, we represent structures as sequences of actions, which build pieces of the target structure step by step. for instance, in the task of coreference resolution, the actions build spans as well as the relations between the spans, contiguous sequences of tokens. we give an example in fig. 1 . ## experiments we experiment on three nlp structured prediction tasks: named entity recognition, end-to-end relation extraction, and coreference resolution. we are primarily interested in understanding whether asp provides advantages over two existing formalisms: (i) conditional language models @xcite that flatten the structure into a string (augmented language models), and (ii) the classic discriminative models whose autoregressivity is bounded. we experiment with three pre-trained language models, t5 @xcite , t0 @xcite , and flan-t5 (chung et al., 2022) for the three tasks under consideration. additional experimental details are given in app. a.1 and app. a.2. ## related work most similar to our approach is the model of @xcite , which also predicts structures in an iterative manner using conditional language models. similar approaches exist for constituency parsing @xcite , entity retrieval @xcite , semantic parsing @xcite , slot labeling, and intent classification @xcite ## conclusion in this paper, we propose a novel framework for structured prediction that encodes a structure as a series of structure-building actions that obtains state-of-the-art performance across three tasks. in contrast to past approaches for structured prediction, our approach is compatible with pre-trained large language models. this allows us to reduce structured prediction to the problem of fine-tuning pre-trained language models over an enlarged alphabet. we show empirically that asp outperforms previous structured prediction models by a large margin. indeed, we set the new state of the art on three tasks: named entity recognition, end-to-end relation extraction, and coreference resolution.