Promptly Predicting Structures: The Return of Inference (2024)

Abstract: Prompt-based methods have been used extensively across NLP to build zero- and few-shot label predictors. Many NLP tasks are naturally structured: that is, their outputs consist of multiple labels which constrain each other. Annotating data for such tasks can be cumbersome. Can the promise of the prompt-based paradigm be extended to such structured outputs? In this paper, we present a framework for constructing zero- and few-shot linguistic structure predictors. Our key insight is that we can use structural constraints—and combinatorial inference derived from them—to filter out inconsistent structures predicted by large language models. We instantiated this framework on two structured prediction tasks, and five datasets. Across all cases, our results show that enforcing consistency not only constructs structurally valid outputs, but also improves performance over the unconstrained variants.

## introduction structured prediction requires making multiple inter-dependent structurally constrained decisions @xcite . many nlp tasks are naturally structured. consider, for instance, the task of semantic role labeling (srl) that aims to identify the semantic roles played by sentence constituents for a predicate @xcite . given a sentence 'elrond gave aragorn the sword.' and predicate gave, the task entails extracting its semantic arguments -the giver (agent), the receiver (recipient), and the thing given (theme). these decisions are mutually constrained; e.g., the arguments have to be distinct phrases. existing methods for structured prediction rely on large labeled datasets. however, obtaining structured labels is not straightforward, especially from non-experts. it invariably requires detailed annotation guidelines about the task, the label set, and the interactions between labels. methods like the elrond gave aragorn the sword. who gave someone something ? who was given something? what was given ? ## "promptly" predicting structures problem statement and notation given an input x (e.g., phrase, sentence, etc.), the goal of a conditional model is to characterize the distribution p (y |x) over outputs y . the outputs y in structured prediction tasks consist of a set of decisions y 1 , y 2 , • • • . for instance, consider the srl task from §1. given a predicate, identifying the token spans correponding to its arguments requires multiple decisions; i.e., each y i corresponds to a semantic argument. since the size of the output y is variable and depends on the input x, the standard approach for modeling such problems calls for factorizing the distribution p (y |x) over its components y 1 , y 2 , • • • . the factorization is a design choice; the simplest one decomposes the probability as a product over individual local decisions: p (y |x) = i p (y i |x). with such a factorization, the prediction y * for an input x is given by these local probabilities relate to unary potentials in factor graph notation, and in our work, will correspond to logits from neural models. importantly, the maximization problem above is trivial to solve. each y i can be independently assigned. 2 even the simple factorization above presents two drawbacks: i) it requires an explicit training step using expensive-to-acquire training data to obtain the unary potentials. ii) it does not guarantee structural consistency, leading to invalid structures. in this work, we ask: can we predict valid structured outputs in the zero-to few-shot regime? now, let us address these drawbacks, leading to our proposed framework. unary potentials from prompts. prompt-based methods can help bypass the need for supervised training. for every local decision y i , we construct a query q i to convert the decision into a promptfriendly format. a pretrained model can use these queries in the standard prompt-and-predict fashion to get scored candidate answers. formally, we write p (y |x, q) = i p (y i |x, q i ). here, q is a set of questions corresponding to the components of the structure. in the case of srl, x would be the input sentence and the predicate, and q would contain one question per semantic argument type for that predicate. figure 1 gives an example. the return of inference. merely aggregating outputs generated with prompts does not address the second drawback, i.e., the need for structurally valid outputs. structural validity is a constraint that, when applied to the prediction problem, ensures that only valid output collections are considered as candidate predictions. in other words, instead of the unconstrained prediction of eq. 1, we have the inference problem above requires combinatorial search over valid structures. inference can be realized algorithmically in various forms like beam search, an ilp @xcite or weighted graph search (e.g., @xcite . the framework. we propose the following blueprint for predicting valid structures in a zeroshot manner: i) break the prediction task into its constituent sub-tasks. ii) convert each sub-task into a prompt-friendly format. iii) implement an inference algorithm that takes scores from the llm and produces a valid structured output. this recipe naturally extends to a few-shot setting by prefixing in-context examples with the original prompt. inference constructs outputs that satisfy constraints inherent in the definition of a task. moreover, it can help correct errors of the zero-or fewshot models using other predictions of the same model. without this mechanism (i.e., using the labels from the prompts directly as in equation 1 ), not only will we have partially wrong outputs, they may be invalid from the perspective of the task. finally, as we will see in the case of the coreference task, inference can help make long-range decisions about parts of an input even when the entire input is not presented to an llm. in this sense, it can be seen as a mechanism to "bring back the context" without explicitly having to use a long-document model such as longformer @xcite . ## experiments this section describes the tasks and datasets where we instantiate the framework from section §3. ## results this section presents the impact of inference on zero-shot performance and consistency. we use the 3 billion variants from the t5 @xcite family of encoder-decoder models for our experiments. we discuss our implementation briefly here; appendix d has details like prompt templates. ## analysis experiments next, we show how performance and inconsistency vary with model size and in-context examples. ## conclusions prompt-based methods have impressed when deployed in zero-and few-shot manner. structured prediction problems can use them by breaking the structure down into smaller components and querying for local decisions. this aspect has been explored by some in the literature. however, these works leave out a pivotal part of such structured prediction tasks-the structure. in this work, we proposed a new framework to predict structurally consistent outputs by following local predictions with an inference step. in our experiments, our framework not only guarantees consistency but also provides consistent performance gains.