Label-Aware Automatic Verbalizer for Few-Shot Text Classification in Mid-To-Low Resource Languages (2024)

Abstract: Prompt-based learning has shown its effectiveness in few-shot text classification. A key factor in its success is a verbalizer, which translates output from a language model into a predicted class. Notably, the simplest and widely acknowledged verbalizer employs manual labels to represent the classes. However, manual selection may not yield the optimal words for a given language model, potentially leading to subpar classification performance, especially in mid-to-low resource languages with weaker language models. Therefore, we propose Label-Aware Automatic Verbalizer (LAAV), effectively augmenting manual labels for improved few-shot classification results. Specifically, we utilize the label name along with the conjunction “and” to induce the model to generate more effective words for the verbalizer. Experimental results on four mid-to-low resource Southeast Asian languages demonstrate that LAAV significantly outperforms existing verbalizers.

## introduction in recent years, we have seen many promising applications of prompt-based learning for text classification @xcite @xcite . while the traditional approach trains or fine-tunes a machine learning model to directly predict a class for an input text, the prompt-based approach fits the input text into a template that has some slots to be filled. next, it asks a language model (lm) foot_0 to fill in the slots and then translates what the model filled to be a predicted class @xcite . to predict sentiment in a movie review like "great movie!" as positive or negative, we may prompt a masked lm with "great movie! it was @xcite ." the model may predict the word "fun" for the [mask] token, and we can apply a function, so-called a verbalizer, to map "fun" to the positive class. certainly, the success of a prompt-based text classifier heavily relies on its verbalizer. @xcite proposed pet, which manually chooses a word to represent each class. during inference, it compares the likelihood of those words at the [mask] token (as predicted by the lm) to find the most probable class. in contrast, @xcite proposed amulap, which represents each class with a set of words, automatically derived from those predicted by the lm for training examples. @xcite proposed npprompt, which represents each class using a set of tokens with the highest embedding similarity to the manual class label. its performance, therefore, relies solely on the lm's embedding space. additionally, there is no guarantee that the chosen words will be relevant to the classes of interest, potentially affecting the classifier's performance. this issue disproportionately impacts mid-to-low resource languages, where the lm may have received less comprehensive training data @xcite . in figure 1 @xcite , to predict whether an object "feather" is light with a prompt "feather is @xcite .", the lm suggests "king", "good", and "strong", which are irrelevant to the task but used by amulap to construct the verbalizer. meanwhile, as shown in figure 1 (middle), npprompt suggests "light", "lights", and "lighter", which are variations related to the class "light" but hardly provide additional information about the class. with the smaller size of lms, particularly for mid-to-low languages, predicting relevant words becomes more challenging. @xcite . in this paper, we propose laav (label-aware automatic verbalizer), integrating pet and amulap by exploiting the class labels to induce the model to generate more relevant words for the verbalizer. as shown in figure 1 (bottom), we could construct a better verbalizer by asking "feather is light and @xcite ." now, the lm suggests "fluffy", "smooth", and "soft", which are closely connected to the light class and can be used to construct an effective verbalizer. the contributions of this paper are as follows. ## label-aware automatic verbalizer as illustrated in figure 1 , the words in s(y i ), selected by amulap, could be unrelated to their corresponding class. so, when constructing s(y i ), our method laav integrates the label name of y i into the template t , using a conjunction. this helps induce m to predict words that are related to y i . our choice for the conjunction is "and" because it serves to connect words or phrases with the same grammatical category and similar meaning. also, "and" is one of the most widely used conjunctions in many languages @xcite . as a result, our laav template for creating s(y i ) is note that we will explore other conjunction options in section 5.2. now, the score of token v for class y i for laav will be since the objective of the laav template t y i is solely for seeking better representative words for each class, we use the original template t without the conjunction during training and inference. ## conclusion our method, laav, constructs a better verbalizer by exploiting class labels to collect more relevant words. as shown in the experiments, laav outperforms other existing verbalizers in few-shot text classification across four languages, even surpassing llm with in-context learning. our comprehensive analysis highlights "and" as a particularly effective conjunction for retrieving words that exhibit high discriminative power crucial for enhancing text classification performance.