CELDA : Leveraging Black-box Language Model as Enhanced Classifier without Labels (2023)

Abstract: Utilizing language models (LMs) without internal access is becoming an attractive paradigm in the field of NLP as many cutting-edge LMs are released through APIs and boast a massive scale. The de-facto method in this type of black-box scenario is known as prompting, which has shown progressive performance enhancements in situations where data labels are scarce or unavailable. Despite their efficacy, they still fall short in comparison to fully supervised counterparts and are generally brittle to slight modifications. In this paper, we propose Clustering-enhanced Linear Discriminative Analysis (CELDA), a novel approach that improves the text classification accuracy with a very weak-supervision signal (i.e., name of the labels).Our framework draws a precise decision boundary without accessing weights or gradients of the LM model or data labels. The core ideas of CELDA are twofold:(1) extracting a refined pseudo-labeled dataset from an unlabeled dataset, and (2) training a lightweight and robust model on the top of LM, which learns an accurate decision boundary from an extracted noisy dataset. Throughout in-depth investigations on various datasets, we demonstrated that CELDA reaches new state-of-the-art in weakly-supervised text classification and narrows the gap with a fully-supervised model. Additionally, our proposed methodology can be applied universally to any LM and has the potential to scale to larger models, making it a more viable option for utilizing large LMs.

## introduction large-scale language models (lms) have been a driving force behind a series of breakthroughs in the machine-learning community. despite their preeminence in wide applications, large lms are often costly or infeasible to fine-tune as many distributed large models, such as gpt-3 @xcite , are provided in a black-box manner, which only allows limited access through commercial apis. to circumvent the explicit adaptation of the models, many recent research leverage prompting, a training-free approach that elicits desired predictions from lms by curating input into a more conceivable form. by doing so, prompting has shown remarkable improvements in data-scarce scenarios (i.e., zero-shot, few-shot), reminiscing the potential of large lms as a universal, off-the-shelf solution for diverse tasks. yet, it is still premature as their performance lags far behind the fine-tuned model and displays fragility to negligible changes @xcite . in this paper, we aim to bridge this gap by utilizing an unlabeled dataset without adapting the model and propose clustering-enhanced linear discriminative analysis (celda) that maximizes the potential of black-box language models further. celda enhances text classification performance by training a lightweight module stacked on the top of lm. the improvement is a result of attaining the following two key objectives: (1) composing a highly reliable pseudo-labeled dataset with black-box lm. (2) training a compact but robust model with the previously composed dataset (pseudo-labels). we accomplish the first goal by sorting out some uncertain data points via clustering the features from lm. we draw inspiration from recent findings @xcite that lm effectively groups semantically similar sentences into clusters. furthermore, we achieve the latter objective by employing linear discriminative analysis (lda) which is efficient in terms of parameter requirements and robust against spurious inputs @xcite . the mentioned characteristics of lda have strong compatibility with our training dataset, considering the presence of noise and its reduced scale. to verify our method, we compare with recently proposed state-of-the-art methods on 8 different text classification datasets, spanning from binary to multi-class tasks (maximum 150 classes). moreover, we also report the performance of baseline lda when the data labels are fully available, serving as the upper bound performance. as illustrated in figure 1 , our method significantly outperforms the precedently proposed prompting-based zeroshot learning (zsl) method @xcite and closes the gap between fully labeled methods. in summary, our contributions are as follows: (1) we propose dubbed celda, a novel weaklysupervised learning framework for black-box language models that consistently outperforms other competing methods and close the gap between fully fine-tuned model. additionally, most recent research in zsl utilizes external knowledge bases or corpus @xcite @xcite . @xcite retrieves semantically similar samples from the additional corpus and labels them randomly. @xcite employs automatically expanded fuzzy verbalizers to converge the mapping between the verbalizer tokens and the class labels. @xcite additionally uses semantically similar sentences from supplementary corpora to compensate for the poorly described labels. weakly-supervised learning approaches, unlike zsl methods, generally require an unlabeled dataset relevant to the target task and re-train the backbone model. generally, most wsl studies @xcite @xcite utilize keywords from metadata (e.g., name of each class) to annotate unlabeled datasets and iteratively re-train models with the pseudo-labeled dataset. namely, x-class @xcite incrementally adds several similar words to each class until in-consistency arises and utilizes the weighted average of contextualized word representations to retrieve the most confident documents from each cluster to train a text classifier. simptc @xcite trains a gaussian mixture model (gmm) on top of the lm's representations, similar to our approach in that it utilizes gaussian distribution to fit the model. while effective, most wsl methods are challenging to utilize in black-box scenarios as they often require direct model tuning or are tailored for a particular language model. black-box tuning is a research field aiming to maximize downstream task performance without accessing weights or gradients of the model, which has diverse potential and benefits. black-box tuned models can process mixed-task input batch with a single model as it circumvents the explicit model retraining phase. furthermore, it can leverage some commercial models available only through apis @xcite or even when models are too large @xcite to optimize directly. in this scenario, the prevailing paradigms are: (1) manipulating the input text or (2) training a lightweight model on top of the final representations from the model. specifically, the most recent work utilizes the language model's ability to learn in-context @xcite and tailors the input through appending templates or few-shot samples (i.e., demonstrations) to the original inputs. additionally, some studies @xcite attempt to find the optimal prompt for the task without explicitly calculating the gradient. ## celda we introduce celda, clustering-enhanced linear discriminative analysis which enhances the text classification ability of black-box lms without labels. celda first (1) consists pseudo labeled dataset by passing an unlabeled dataset to lm(•) and extracts high-quality representation and pseudolabel pair as introduced in § 3.2. then, we (2) refine the pseudo labeled dataset into a small subset dataset dubbed a certain dataset by leveraging clustering and the concept of entropy. (figure 2 illustrates the (1), (2) stage of celda training.) finally, (3) we recursively train a third-party lda module stacked on lm with the certain dataset, which learns a precise decision boundary from the noisy dataset. ## analysis we conduct an in-depth investigation of celda to elucidate its underlying mechanism in conjunction with our intuitions. for all analysis experiments, we utilize t5-large as a backbone model. model accuracy vanilla prompting (pet) 80.96 + lda 84.49 + recursive train 85.45 + multi-verbalizer 88.10 + feature augmentation 88.57 + data cleansing (celda) 90.03 remaining data samples (%) (b) remaining portion of dataset after filtering. figure 3 : filtered dataset has more accurate data samples while the accuracy of dropped samples exhibits poor performance. setting a high strong threshold increases the pseudo label accuracy but the overall performance drops. as a support, we can verify that the accuracy of the dropped dataset also increases with a higher τ , and the total number of the survived sample decreases, as shown in figure 3b . ## conclusion this work presents celda, a practical framework for employing a black-box language model. we have sought room for improvement in three orthogonal directions: (1) utilizing language models with high-quality representations (from the last layer and logit distribution). (2) filtering unreliable data samples from the noisy dataset. (3) recursively trains lda, which is robust to noisy samples and avoids over-fitting by minimizing the overall model parameters. by fusing these elements, we demon-strate the significant performance of celda on sundry classification tasks and its scalability with the language model size. in our follow-up study, we aim to employ sample-wise entropy from pseudolabeling in the data cleansing phase instead of utilizing the entropy of the cluster, which is highly course-grained. we expect that looking at finegrained sample-wise entropy can yield a more precise data filtering effect, reducing meaningful samples from being dropped.