Revisiting k- NN for Fine-tuning Pre-trained Language Models (2023)

Abstract: “Pre-trained Language Models (PLMs), as parametric-based eager learners, have become thede-facto choice for current paradigms of Natural Language Processing (NLP). In contrast, k-Nearest-Neighbor (k-NN) classifiers, as the lazy learning paradigm, tend to mitigate over-fittingand isolated noise. In this paper, we revisit k-NN classifiers for augmenting the PLMs-based clas-sifiers. From the methodological level, we propose to adopt k-NN with textual representationsof PLMs in two steps: (1) Utilize k-NN as prior knowledge to calibrate the training process.(2) Linearly interpolate the probability distribution predicted by k-NN with that of the PLMs’classifier. At the heart of our approach is the implementation of k-NN-calibrated training, whichtreats predicted results as indicators for easy versus hard examples during the training process. From the perspective of the diversity of application scenarios, we conduct extensive experimentson fine-tuning, prompt-tuning paradigms and zero-shot, few-shot and fully-supervised settings,respectively, across eight diverse end-tasks. We hope our exploration will encourage the commu-nity to revisit the power of classical methods for efficient NLP1.”

## methodology the overall framework is presented in figure 2 . we regard the plm as the feature extractor that transforms the input textual sequence x into an instance representation x with dimensions d. we revisit k-nn in §3.1 and then introduce our method to integrate k-nn with tuning paradigms in §3.2. ## limitations we only explore leveraging the training data for k-nn search, while various external domain data are also suitable for k-nearest neighbor retrieval. moreover, incorporating k-nn also faces the following limitations: (1) the requirement of a large memory for retrieval; (2) hyper-parameters (such as λ and α) used for retrieval have an impact on the performance of model training; ## conclusion and future work in this paper, we propose a novel method to enhance plm-based classifiers using k-nn. specifically, we introduce a calibration process and linear interpolation of inference phrases to effectively integrate k-nn into the training pipeline. to evaluate the effectiveness of our approach, we conduct a comprehensive and in-depth analysis of the role of k-nn in various nlu tasks and tuning paradigms. our results demonstrate that the integration of k-nn is flexible and can significantly enhance the performance of large models. future work should explore the combination of k-nn and llms such as (1) inject external knowledge into the llms with k-nn. specifically, k-nn can be used to retrieve relevant knowledge from an external database during the reasoning process, which can help correct errors and reduce the prevalence of gibberish output and factual errors that are common in llms.