Conditional set generation using Seq2seq models (2022)

Abstract: Conditional set generation learns a mapping from an input sequence of tokens to a set. Several NLP tasks, such as entity typing and dialogue emotion tagging, are instances of set generation. Seq2Seq models are a popular choice to model set generation but they treat a set as a sequence and do not fully leverage its key properties, namely order-invariance and cardinality. We propose a novel algorithm for effectively sampling informative orders over the combinatorial space of label orders. Further, we jointly model the set cardinality and output by listing the set size as the first element and taking advantage of the autoregressive factorization used by Seq2Seq models. Our method is a model-independent data augmentation approach that endows any Seq2Seq model with the signals of order-invariance and cardinality. Training a Seq2Seq model on this new augmented data (without any additional annotations), gets an average relative improvement of 20% for four benchmarks datasets across models spanning from BART-base, T5-11B, and GPT-3. We will release all code and data upon acceptance.

## introduction conditional set generation is the task of modeling the distribution of an output set given an input sequence of tokens @xcite . several nlp tasks are instances of set generation, including open-entity typing @xcite , fine-grained emotion classification @xcite , and keyphrase generation @xcite @xcite . the recent successes of the pretraining-finetuning paradigm have encouraged a formulation of set generation as a seq2seq generation task @xcite @xcite . in this paper, we posit that modeling set generation as a vanilla seq2seq generation task is suboptimal, because the seq2seq formulations do not explicitly account for two key properties of a set output: order-invariance and cardinality. forgoing order-invariance, vanilla seq2seq generation treats a set as a sequence, assuming an arbitrary order between the elements it outputs. similarly, the cardinality of sets is ignored, as the number of elements to be generated is typically not modeled. prior work has highlighted the importance of these two properties for set output through loss functions that encourage order invariance @xcite , exhaustive search over the label space for finding an optimal order @xcite @xcite , and post-processing the output @xcite . despite the progress, several important gaps remain. first, exhaustive search does not scale with large output spaces typically found in nlp problems, thus stressing the need for an optimal sampling strategy for the labels. second, cardinality is still not explicitly modeled in the seq2seq setting despite being an essential aspect for a set. finally, architectural modifications required for specialized set-generation techniques might not be viable for modern large-language models. we address these challenges with a novel data augmentation strategy. specifically, we take advantage of the auto-regressive factorization used by seq2seq models and (i) impose an informative order over the label space, and (ii) explicitly model cardinality. first, the label sets are converted to sequences using informative orders by grouping labels and leveraging their dependency structure. our method induces a partial order graph over label space where the nodes are the labels, and the edges denote the conditional dependence relations. this graph provides a natural way to obtain informative orders while reinforcing order-invariance. specifically, sequences obtained via topological traversals of this graph allow independent labels to appear at different locations in the sequence, while restricting order for dependent labels. next, we figure 1 : an illustrative task where given an input x, the output is a set of emotions. our method first discovers a partial order graph (middle) in which specific labels (joy) come before more general labels (pride). listing the specific labels first gives the model more clues about the rest of the set. topological samples from this partial order graph are label sequences that can be efficiently generated using seq2seq models. the size of each set is also added as the first element for joint modeling of output with size. figure 1 illustrates the key intuitions behind our method using sample task where given an input x (say a conversation), the output is a set of emotions (y). to see why certain orders might be more meaningful, consider a case where one of the emotions is joy, which leads to a more general emotion of pride. after first generating joy, the model can generate pride with certainty (joy leads to pride in all samples). in contrast, the reverse order (generating pride first) still leaves room for multiple possible emotions (joy and love). the order [joy, pride] is thus more informative than [pride, joy]. the cardinality of a set can also be helpful. in our example, joy contains two sub-emotions, and love contains one. a model that first predicts the number of sub-emotions can be more precise and avoid overgeneration, a significant challenge with language generation models @xcite . we efficiently sample such informative orders from the combinatorial space of all possible orders and jointly model cardinality by leveraging the auto-regressive nature of seq2seq models. ## task we are given a corpus @xmath0, y t )} m @xmath1, y 2 , . . . , y k } is a set. for example, in multilabel fine-grained sentiment classification, x t is a paragraph, and y t is a set of sentiments expressed by the paragraph. we use y i to denote an output symbol, [y i , y j , y k ] to denote an ordered sequence of symbols and {y i , y j , y k } to denote a set. ## method this section expands on two critical components of our system, setaug. section 3.1 presents tsam-ple, a method to create informative orders over sets tractably. section 3.2 presents our method for jointly modeling cardinality and set output. ## experiments setaug comprises: i) tsample, a way to generate informative orders to convert sets to sequences, and ii) card: jointly modeling cardinality and the set output. this section answers two questions: rq1: how well does setaug improve existing models? specifically, how well setaug can take an existing seq2seq model and improve it just using our data augmentation and joint cardinality prediction, without making any changes to the model architecture. we also measure if these performance improvements carry across diverse datasets, model classes, and inference settings. rq2: why does our approach improve performance? we study the contributions of tsample and joint cardinality prediction (card), and analyze where setaug works or fails. ## conclusion and discussion we present setaug, a novel data augmentation method for conditional set generation that incorporates informative orders and adds cardinality information. our key idea is using the most likely order (vs. a randomly selected order) to represent a set as a sequence and conditioning the generation of a set on predicted cardinality. as a computationally efficient and general-purpose plug-in data augmentation algorithm, setaug improves seq2seq models for set generation across a wide spectrum of tasks. for future work, it would be interesting to investigate if the general ideas in this work have applications in settings beyond set generation. examples include generating additional data to improve language modeling in low-resource scenarios and determining the ideal order of in-context examples in a prompt.