Label Representations in Modeling Classification as Text Generation (2020)

Abstract: Several recent state-of-the-art transfer learning methods model classification tasks as text generation, where labels are represented as strings for the model to generate. We investigate the effect that the choice of strings used to represent labels has on how effectively the model learns the task. For four standard text classification tasks, we design a diverse set of possible string representations for labels, ranging from canonical label definitions to random strings. We experiment with T5 on these tasks, varying the label representations as well as the amount of training data. We find that, in the low data setting, label representation impacts task performance on some tasks, with task-related labels being most effective, but fails to have an impact on others. In the full data setting, our results are largely negative: Different label representations do not affect overall task performance.

## introduction state-of-the-art transfer learning methods model classification tasks as text generation, such as gpt2 @xcite and t5 @xcite , and have led to significant improvements across a variety of nlp tasks. in this setting, labels are represented as strings for the model to generate, and the pretrained language model is finetuned to maximize the probability of generating the chosen label representation. for example, for cola @xcite task, a classifier will be trained to output 0 (representing unacceptable) or 1 (representing acceptable), while t5 models it as a generation task, it is trained to generate the string "unacceptable" or "acceptable". the advantage of this approach is that the language model can be applied to the classification task as-is, without the * these two authors contributed equally. however, in this setting, the impact of the particular strings used to represent labels remains unclear on the end task performance. one of the few studies on this question find that the linguistic properties (relatedness, polarity scale, etc.) of the labels do affect task performance @xcite , though their results are limited to document retrieval. we therefore further investigate the impact of string representation when modeling text classification as text generation. we experiment with t5-base and four diverse, standard text classification tasks. for each task, we design a wide range of label representations, including canonical task labels; task-unrelated antonyms; and completely random strings. as previous works by @xcite have noted that the impact of label representation is particularly noticeable in lower data settings, we also vary the amount of training data for each task. our experiments reveal that, in the full data setting, the choice of label representation largely does not affect overall performance, with only one of the four datasets seeing any impact. in the low data setting, label representations sometimes have an impact on the overall performance, with task-related labels being the most effective in these cases. ## related work target word probing experiments @xcite probe the effects of label representation on document retrieval and ranking. they set the baseline mapping as {@xmath0, @xmath1, and also try the reverse mapping, antonyms, related words, unrelated words and subwords. in the low data setting, they find that the baseline mapping yields accuracy significantly higher than other types of mappings do. in the high data setting, related words mapping is the most effective, though the differences between mappings are not as large as in the low-data regime. we extend these experiments with more diverse tasks and label representations. cloze reformulation schick and schütze (2020) introduce pattern exploiting training, where the input is transformed into a cloze-style sentence. for example, the task of identifying whether two sentences a and b contradict or agree with each other is reformulated into "a? [blank], b.", and a pretrained language model is used as-is to generate yes or no to fill in the blank. they claim that this procedure significantly improves the performance on several tasks in the zero-shot setting. similarly, @xcite probe the knowledge presented in state-of-the-art language models without fine-tuning using similar sentences with blanks. they find that these language models contain rich factual knowledge from pretraining, and are effective at recalling knowledge when answering fact-related questions. prompt design @xcite propose several methods to automatically generate efficient prompts for extracting knowledge from pretrained models, rather than manually designing them. they find that different templates, e.g. "x who converted to y", compared to "x is affiliated with y religion", can improve accuracy by as much as 60%. this line of work is orthogonal to ours: we focus on optimal label representation whereas they focus on the best way to format inputs. ## label representations we consider four standard text classification datasets representing different tasks and textual genres: (i) sentence acceptability judgments with cola @xcite ; (ii) sentiment analysis with sst-2 @xcite ; (iii) paraphrase detection with paws @xcite ; and (iv) commonsense reasoning with copa @xcite . for each of these datasets, we test a wide variety of label representations. ## conclusion in this work, we investigate the impact of label representation in modeling classification as a seq2seq task. for four standard text classification datasets and task types, we design a wide range of label representations, ranging from canonical task-related labels to task-unrelated antonyms to random words and strings. we experiment with the t5-base model on these datasets and label representations in a range of regimes with various data sizes. overall, we find that the choice of label representation largely does not affect task performance, though it varies by task and dataset size. in the high data settings, there is generally no differences between choices of label representations, and even random strings can function well as label repre-sentations. in low data settings, the influence of label representations varies significantly between tasks. for paws, we observe no variation, but for copa and cola, we note that task-related labels generally perform best. our experiments represent preliminary negative evidence that label representations have limited impact on task performance, but there are a number of dimensions that our work does not investigate that might affect a model's sensitivity to label representation. first, we take as given that canonical task labels as sufficient for all tasks. however, for some tasks, the canonical label representation might be suboptimal or not sufficiently convey the semantics of the task, e.g. choice1/choice2 for copa. a natural way to mitigate this issue would be to use a much larger set of possible label representations, or even automatically discover label representations. however, we expect this issue will be especially problematic with more complex tasks that are difficult to represent with single-word labels or to succinctly represent in text at all. second, we use a single task input format, but the task formatting may be suboptimal and affect the ability of the model to learn from the label semantics. schick and schütze (2020) use multiple example templates per task and find that performance between templates can vary substantially. while an obvious direction would be to simply use multiple templates, automatically discovering effective templates also seems like a promising direction.