Paper: Gen-Z: Generative Zero-Shot Text Classification with Contextualized Label Descriptions (ICLR-2024)

Reviewer comment (Experiments):
Though this work argues that utilizing label descriptions in a bayesian form benefits the performance, it relies heavily on the quality of the label description, e.g., how clear the context can present the task, domain the label. However, in text classification, sometimes the label is not natural language, but a less literally meaningful symbol. For such labels, it is not likely to encourage the model to generate text $x$ given $y$. Then it requires human beings to provide very good summaries or descriptions for such labels, which is sometimes challenging. Such a problem limits the application of the proposed GEN-Z framework. Instead, conventional in-context learning can provide examples and have the model to learn from simpler label descriptions and demonstrations, which is more flexible.

Author response:
Thank you for pointing this out. It is possible that some tasks may require more descriptive labels than the ones we considered. But, for any text classification task where annotations are involved in creating the training/development/test sets, the prevalent approach is to provide detailed instructions to the annotators including describing what each label is supposed to capture. That is to say,  human labor is already involved in describing labels in most text classification tasks. Such descriptions can easily be reused to work with our setup if available publicly. It is an interesting avenue for future work to study this behavior. We will include this in the limitations section in the updated draft. 

Furthermore, the utility of our method lies in cases where few-shot examples may be not be straightforward to obtain. There are several tasks where it is not possible to access any labeled data to create few-shot examples such as those discussed in our personalization classification experiments. For example, when we’d like to classify text on dialects, but also social variations spoken by intersectional identities (e.g. an older, non-binary, educated speaker of New-Zealand English), finding annotators who can accurately annotate such data is very challenging. Despite requiring potentially long descriptions, it is possible to use Gen-Z in these settings.

Reviewer comment (Reproducibility):
For LMs, the context can influence the language model probability significantly. Combining multiple label descriptions may result in changed language model probability. So how is multiple label descriptions utilized in this work?

Author response:
Multiple label descriptions are simple paraphrases of each other (we provide all the descriptions we used here:  https://pastebin.com/UwH8zRuV). It is indeed true that conditioning on different paraphrases can result in different text probabilities. We propose to simply add the conditional probabilities corresponding to each label and then compare the aggregated probabilities to predict the label which results in improvement in mean performance and reduction in the performance variance. We also ablate with other aggregation methods such as multiplying them or computing their harmonic mean (reported in the Appendix), but overall found simple summation to perform consistently well.

Reviewer comment (Novelty):
The method proposed is conceptually similar to Min et al.'s noisy channel method, with the primary distinction being the use of multiple "label descriptions". This similarity raises questions regarding the novelty of the proposed approach.

Author response:
> conceptually similar to Min et al.'s noisy channel method

This paper serves as a source of inspiration for our work, but we make two simple but significant changes to their method, which lead to substantial improvements over Min et al’s performance. By incorporating multiple label descriptions and adding contextual information to this task, Gen-Z outperforms Min et al’s performance in both zero- and few-shot versions. These changes, albeit simple, are non-trivial and have theoretical grounding. Our method is easily reproducible and usable by others, especially since it is zero-shot. In addition to prompt variations, prior work has shown that selection of in-context examples has a significant impact on task performance. Not requiring any in-context examples at all and is a significant contribution.

Reviewer comment (Theory):
The assumptions outlined in Section 2.3 may be overly simplistic, potentially detracting from the final performance of the proposed method. The paper would gain interest if the authors could suggest reasonable approximations for the omitted probabilities, namely i.e., $p(z|y_i, u,v,\dots)$ and $p(y_i|u,v,\dots)$.

Author response:
> assumptions in section 2.3

Thank you for pointing this out. The assumptions made in this section serve to simplify the inference objective and have sound motivations. We have clarified them in the paper further, we repeat them here.

In the graphical model that we design as shown in Figure 1 (right), each conditional represents the true probability. Give the label $y$ and attributes $u, v,  …$ if there are $N$ ways in which we can describe them in textual form, $p(z|....)$ should really be equal for each possible paraphrase, we are not defining the grammatical plausibility of the paraphrase, just the existence probability, hence using the LM’s probably for $p(z|...)$ is not appropriate. 

For $p(y | u, v, …),$, first, we assume that the probability of the label is independent of contextual factors. For most factors we consider in this work, this assumption is largely true. The true prior probability of any label is unlikely to depend on the domain of the text, or the personal attributes of the user reading, writing, or being described in the text (we agree that this might not hold in datasets on which classifiers are typically trained). Second, we assume that each label is equally likely, this is a simplifying assumption and may not hold in practice but it is not trivial to estimate this using an LM. Our initial exploration in this direction showed decline in classification performance.

Reviewer comment (Experiments):
What potential impacts could arise from the combination of the proposed method with few-shot learning scenarios? This consideration is crucial, especially given that while generative zero-shot classification yields reasonable results, it does not yet match the performance of few-shot-based methods, like ICL (DC) as depicted in Table 2.

Author response:
> Combining Gen-Z with few-shot prompting

This is a great suggestion and avenue for future work. Zero-shot classification is usually considered a weak baseline and has been shown to perform substantially worse than in-context methods. Our goal is in this paper to two fold (1) make a case that pretrained LMs can indeed perform strongly zero-shot and it should not be discarded as weak baseline, and, (2) shine a light on why in-context learning might be performing as well as it does. We do not claim to build a new state of the art prompting method or suggesting that in-context learning is not necessary. In our initial exploration, combining our approach with few-shot examples trivially did not show improvements over either methods, most likely due to the label description getting ignored due to long context between the test example and the description. But alternative methods of incorporating in-context examples such as ensembling might be interesting to explore in future work. 

Further, we would like to point out that the ICL method DC is not strictly comparable to our approach as in addition to few-shot examples, it requires access to a several unlabeled examples in the same domain as the test set to be able to compute calibrated LM probabilities which gives them a boost in performance. We do not assume access to such additional dataset in our work.