ASCM : An Answer Space Clustered Prompting Method without Answer Engineering (2022)

Abstract: Prompt-based learning, which exploits knowledge from pre-trained language models by providing textual prompts and designing appropriate answer-category mapping methods, has achieved impressive successes on few-shot text classification and natural language inference (NLI). Because of the diverse linguistic expression, there exist many answer tokens for the same category. However, both manual answer design and automatic answer search constrain answer space and therefore hardly achieve ideal performance. To address this issue, we propose an answer space clustered prompting model (ASCM) together with a synonym initialization method (SI) which automatically categorizes all answer tokens in a semantic-clustered embedding space. We also propose a stable semi-supervised method named stair learning (SL) that orderly distills knowledge from better models to weaker models. Extensive experiments demonstrate that our ASCM+SL significantly outperforms existing state-of-the-art techniques in few-shot settings.

## introduction pre-trained language models (plms, @xcite @xcite @xcite have shown a great impact on natural language processing (nlp) tasks. by adding task-specific head and fine-tuning on labeled corpora, plms surpass conventional fully supervised learning paradigm and come into being a "pre-train, fine-tune" paradigm @xcite . however, @xcite demonstrate plms can perform downstream tasks without any additional data and modification, which reveals plms have the potential for knowledge exploration. and @xcite find that bert contains relational knowledge, factual knowledge and can be applied to qa tasks without fine-tuning. recently, prompting methods @xcite , which reformulate downstream tasks with taskspecific textual prompts, is proved successful in many tasks such as few-shot text classification and natural language inference. for example, to classify textual news such as "france won the 2018 world cup", prompting methods may add the textual prompt "__ news:" before the news, then plms probably fill in the blank with token "soccer", "sports", "football", "match", "fifa", etc. taking these tokens as the answer space of sports news category, plms may correctly predict the category of news even without fine-tuning. nevertheless, former prompting methods bring in extra prompt engineering and answer engineering, which have significant influences on performance and need to be designed carefully. there exist various prompt engineering methods @xcite @xcite @xcite ), but answer engineering hasn't been researched enough. former answer engineering can be categorized into manual answer design and automatic answer search. both methods select limited answers space for each category and force plms to predict in that answer space. for example, former methods may force plms to fill the blank of "__ news: it was the 12th title for the lakers." with "sports" if the answer space of sports category doesn't include "nba". besides, manual answer design methods need additional expertise and automatic answer search methods may damage the performance @xcite @xcite . we find that answer tokens belonging to the same category get some kind of relationship. for sentiment analysis tasks and natural language inference tasks, same-category answer tokens usually get similar semantics (glad, happy). for topic classification tasks, same-category answer tokens may get relationships such as synonym (soccer, football), hyponym(soccer, football), hypernym, co-hyponym, etc. for convenience, we adopt "synonym" and "semantic" to represent all the relationships above. but the distribution of token embedding in plms isn't specially designed for text classification or nli. it came to us that cluster centers of intra-class answer tokens can be used for classification if all token embedding distributes according to semantics. in that case, any tokens that are relevant to certain categories will get close to the corresponding cluster center and be automatically included in the corresponding answer space, which means no constraint on plms and no answer engineering. following this idea, we propose the ascm, as illustrated in figure 1 , which focuses on text classification and natural language inference. our contributions can be summarized as follows: we conduct extensive experiments which demonstrate the superiority of our method. our ascm+sl outperforms the previous prompt-based learning (manual answer design) by 10.3, 2.6, 2.3, and 2.1 on mnli, yahoo, yelp, and ag's news with 50 labeled examples. ## experiments following prior work, we evaluate our work on four tasks yelp reviews, ag's news, yahoo questions @xcite , and mnli @xcite . for comparison, we adopt roberta large @xcite as the pre-trained language model in all experiments except for to evaluate the few-shot performances, we randomly sample |t | (10, 50, 100, and 1000) examples as the labeled dataset with labels distributing uniformly. and we randomly sample 10000 examples for each label to form the unlabeled dataset d for sl. we choose the adam optimizer with a slanted triangular schedule, an initial learning rate of 1e-5, and a maximum sequence length of 256. the batch size is set to 16 for |t | equals to 50, 100, 1000 and 8 when |t | equals to 10. for each training step, we randomly sample the same number of examples from d to compute auxiliary language modeling loss and the loss weight α is set to 0.5. for supervised training and individual sl, training steps are set to 300. for the final plm classifier, training steps are set to 5000. for sl, we set @xmath0, @xmath1. notably, ipet trains three times for each model and the ensemble of them will improve the performance. training details of synonym initialization can be found in appendix a. ## conclusion in conclusion, we propose an answer space clustered prompting model and a synonym initializa-tion method that doesn't need answer engineering or expertise. our method clusters token embeddings according to semantics and classifies them on unconstrained answer space. experiments show that our method combined with a stable stair learning method outperforms the previous prompt-based learning methods based on manual answer design. clustering multi-tokens words and phrases based on semantics is desirable for future work. in addition, research on adapting the thought of token embedding semantic-clustering to machine translation, text generation, information retrieval, and text summarization might also prove valuable.