Transductive Learning for Textual Few-Shot Classification in API -based Embedding Models (2023)

Abstract: Proprietary and closed APIs are becoming increasingly common to process natural language, and are impacting the practical applications of natural language processing, including few-shot classification. Few-shot classification involves training a model to perform a new classification task with a handful of labeled data. This paper presents three contributions. First, we introduce a scenario where the embedding of a pre-trained model is served through a gated API with compute-cost and data-privacy constraints. Second, we propose a transductive inference, a learning paradigm that has been overlooked by the NLP community. Transductive inference, unlike traditional inductive learning, leverages the statistics of unlabelled data. We also introduce a new parameter-free transductive regularizer based on the Fisher-Rao loss, which can be used on top of the gated API embeddings. This method fully utilizes unlabelled data, does not share any label with the third-party API provider and could serve as a baseline for future research. Third, we propose an improved experimental setting and compile a benchmark of eight datasets involving multiclass classification in four different languages, with up to 151 classes. We evaluate our methods using eight backbone models, along with an episodic evaluation over 1,000 episodes, which demonstrate the superiority of transductive inference over the standard inductive setting.

## introduction recent advances in natural language processing (nlp) have been largely driven by the scaling paradigm @xcite , where larger models with increased parameters have been shown to achieve state-of-the-art results in various nlp tasks @xcite . this approach has led to the development of foundation models such as chat-gpt @xcite @xcite * these authors contributed equally to this work @xcite , @xcite , gpt-3 @xcite , t5 @xcite , and bert @xcite , which have achieved unprecedented performance in text classification @xcite , language modeling, machine translation @xcite , and coding tasks @xcite . despite the success of the scaling paradigm, significant challenges still exist especially when the many practical constraints of real-world scenarios have to be met: labeled data can be severely limited (i.e., few-shot scenario @xcite ), data privacy is critical for many industries and has become the subject of increasingly many regulatory pieces @xcite , compute costs need to be optimized @xcite . furthermore, these challenges are made even more complex as stronger foundation models are now available only through apis (e.g., ope-nai's gpt-3, gpt-4 or chatgpt, anthropic's claude or google's palm @xcite ) which has led to some of their parameters being concealed, presenting new challenges for model adaptation @xcite . this paper is centered on the fundamental task of fewshot text classification, specifically focusing on cloud-based/api access. specifically, we formulate three requirements for api-based few-shot learning (fsl) (see fig. 1 ): (r1) black-box scenario. we focus on learning from models that are opaquely deployed in production to the end-user, who only has access to the end-point of the encoder, i.e., the resulting text embedding produced by the final layer of the network. (r2) low resources / computation time. ai systems are often required to make rapid predictions at high frequencies in various real-world applications. therefore, any few-shot classifier used in such scenarios should have a low training and inference time, as well as require minimal computational resources. under r1, r2, and r3 requirements, the standard inductive learning @xcite may be quite limiting. to mitigate the labeled data scarcity while retaining api compliance, we revisit transduction @xcite in the context of textual few-shot classification. specifically, in the context of fsl, transductive fsl @xcite advocates leveraging unlabeled test samples of a task as an additional source of information on the underlying task's data distribution in order to better define decision boundaries. such additional source essentially comes for free in many offline applications, including sentiment analysis for customer feedback, legal document classification, or text-based medical diagnosis. our findings corroborate recent findings in computer vision @xcite @xcite @xcite , that substantial gains can be obtained from using transduction over induction, opening new avenue of research for the nlp community. however, the transductive gain comes at the cost of introducing additional hyperparameters, and carefully tuning them. motivated by occam's razor principle, we propose a novel hyperparameter-free transductive regularizer based on fisher-rao distances and demonstrate the strongest predictive performances across various benchmarks and models while keeping hyperparameter tuning minimal. we believe that this parameter-free transductive regu-larizer can serve as a baseline for future research. ## conclusions this paper presents a novel fsl framework that utilizes api models while meeting critical constraints of real-world applications (i.e., r1, r2, r3). this approach is particularly appealing as it shifts the computational requirements (r2), eliminating the need for heavy computations for the user and reducing the cost of embedding. to provide a better understanding, embedding over 400k sequences cost as low as 7 dollars. in this scenario, our research highlights the potential of transductive losses, which have previously been disregarded by the nlp community. a candidate loss is the fisher-rao distance which is parameter-free and could serve as a simple baseline in the future.