Unsupervised Calibration through Prior Adaptation for Text Classification using Large Language Models (2023)

Abstract: A wide variety of natural language tasks are currently being addressed with large-scale language models (LLMs). These models are usually trained with a very large amount of unsupervised text data and adapted to perform a downstream natural language task using methods like fine-tuning, calibration or in-context learning. In this work, we propose an approach to adapt the prior class distribution to perform text classification tasks without the need for labelled samples and only a few in-domain sample queries. The proposed approach treats the LLM as a black box, adding a stage where the model posteriors are calibrated to the task. Results show that these methods outperform the un-adapted model for different number of training shots in the prompt and a previous approach where calibration is performed without using any adaptation data.

## introduction in the last years, large language models (llms) like @xcite , @xcite , instructgpt @xcite have proven to be useful for a large variety of complex natural language understanding tasks, showing outstanding performance on many benchmarks related to reading comprehension, summarization, information retrieval, and generative question-answering, among others @xcite @xcite . llms are pre-trained on a large amount of unsupervised text data following a cost function that is usually self-supervised (autoregressive, denoising, etc.) @xcite @xcite . notably, llms achieve competitive results in a zero-shot scenarios, i.e., without being adapted to the downstream task of interest @xcite @xcite . nevertheless, when data is available for adaptation, significant gains can be achieved over the zero-shot scenario. in these cases, the adaptation is done, for example, through (full or selective) fine-tuning @xcite , in-context learning @xcite , or post-processing of the model's outputs @xcite , depending on the size of adaptation dataset, whether this data is labelled or not, and the amount of computational resources available. in this work, we propose an approach to adapt llms to text classification tasks using unlabelled in-domain data. that is, we assume we have examples of the type of text that needs to be classified, but we do not have the actual class of these examples. we propose a light-weight method inspired by the theory of calibration which, for the datasets we experimented with, required only a few dozen of in-domain samples to achieve optimal performance. we call this method ucpa (unsupervised calibration through prior adaptation). we compare our proposed approach with a previously proposed approach which does not rely on any indomain data @xcite and show that the additional information provides significant performance improvements. further, we compare our method, theoretically and empirically, with supervised calibration of the posteriors using logistic regression. we show that our approach performs similarly to supervised calibration, without the need for labelled data. finally, another version of the method is presented where we assume that, even though no labelled data is available, the class priors can be estimated from knowledge of the task. we call this variant sucpa (semi-unsupervised calibration through prior adaptation) since some information about the task is needed to estimate the priors. fine-tuning pre-trained llms can be adapted to a specific task of interest using finetuning techniques like parameter efficient finetuning (peft) @xcite , soft-prompt @xcite and reinforcement learning from human feedback (rlhf) @xcite . despite the attempt of some methods to reduce the number of trainable parameters without sacrificing performance, finetuning is generally an expensive way of adapting a llm to a certain task since it requires significant amounts of in-domain data, as well as computational resources to load and train the llm. in-context learning given the large computational and data requirements of the fine-tuning approach, alternative approaches to adapt llms to a certain task of interest have been proposed. in-context learning refers to the practice of prepending instructions about the task of interest before the text to be classified, summarized or continued in some other way. besides these instructions, examples (usually called "shots") on how to perform the task can be added to the prompt. the gpt-3 paper @xcite showed that close to the state-of-the-art performance in many tasks could be obtained by providing instructions in the prompt without any further adaptation to the task. this work led to the study of good prompt design practices @xcite . calibration there is a large body of literature regarding calibration of classifiers' outputs @xcite , with some recent applications to natural language processing tasks @xcite . recently, a work from @xcite used the concept of "content-free" input to perform an ad-hoc unsupervised form of calibration for different tasks carried out by a llm. our work can be seen as a generalization and formalization of this work, where we derive the approach as unsupervised calibration with an affine expression where the parameters are obtained through the minimization of the crossentropy. ## ucpa: unsupervised calibration through posterior adaptation an llm produces posterior probabilities p (n|h, q, e) for the next token, n, given the history of previously generated words or tokens, h, and the query, q, and preface, e, which together form the prompt (e, q). the bolded variables indicate sequences of one or more tokens, while n is a single token. in our case, the preface contains instructions on the classification task to be solved @xcite and, optionally, a set of training examples for in-context learning @xcite . when using a llm to do classification, we need to use the model to obtain p (y|q, e) from a prompt (e, q), where @xmath0, . . . , y k } is the set of possible classes. to do this, we first define an ad-hoc label name w k for every label y @xmath1. then, we prompt the llm, which we will call θ, with the word sequence given by e followed by q to get a score for the class y k . finally, the probability distribution p (y|q, e) is computed by normalizing this score to get a probability distribution over the classes: note that there may be cases in which the label name is represented with more than one token (see using the definition of conditional probability, the posterior p (y|q, e) can be written as: the factor p (q | y, e)/p (q | e) is the ratio between the likelihood of the query given the preface and the class name, and the likelihood given only the preface. this likelihood ratio (lr) reflects the increase in the likelihood of the query obtained by adding the class name to the response. the quantity p (y|e) can be understood as a prior probability in the sense that it is not conditioned on the query: it is the probability of the class given only the preface. this prior depends strongly on the task of interest. while the llm might have a tendency to predict a certain class given the preface, this may not be the most likely class for the task of interest. as we mentioned in section 1, adapting the model to the application of interest is key for obtaining relevant responses from the llm. while the preface e is, in fact, a way to adapt the llms outputs to the task of interest, it may not be sufficient to fully adapt the posteriors since not all the information about the task can be represented in a short text explanation. in this work we propose to improve the posteriors computed from the llm's scores by explicitly adjusting the priors to the task of interest. this is done by assuming the following expression for the in-domain posterior: the prior p (y|e) is the prior we expect for our task of interest. we may or may not know this distribution. in this work we compare results under two assumptions: 1) that we do not know anything about the prior distribution in which case we simply assume a uniform distribution p (y k |e) = 1/k for all k, and 2) that, even though we do not have labelled data, we do have a good estimate of the frequencies of the classes we expect to see in practice. in our experiments, for this second scenario we compute p (y k |e) = n k /n , where n k is the number of training samples of class k. in practice, though, these priors could be estimated from knowledge of the task rather than from in-domain labelled data. arguably, this second scenario is no longer unsupervised, so we will call this method semi-unsupervised calibration through prior adaptation (sucpa), while the method that uses uniform priors will be called unsupervised calibration through prior adaptation (ucpa). the prior p (y | e) is the prior for y that is implicit in our llm. it is the distribution of classes that the model would output for this task, across all possible relevant queries we may provide. hence, we estimate this prior by simply running the llm on (unlabelled) training data q @xmath2, . . . , q (n ) } and averaging the re-sulting posteriors: the method above is a heuristic that relies on an assumption (equation ( 5 )) that may or may not hold, as well as on the approximation of the prior above. when using this expression to obtain the prior we call the method "ucpa/sucpa-naive". as we will see, this heuristic works quite well in our experiments. further, as we explain in section 4, we can also obtain the prior in a more principled way using an expression derived from linear logistic regression. ## supervised affine calibration and semi-ucpa (sucpa) a standard way to adapt the posterior probabilities from a classifier to a certain domain of interest is to calibrate them using in-domain labelled data. calibration refers to the process of transforming the scores of a system to optimize the quality of the scores as posteriors. this is usually done by choosing a certain parameterized form for the transform and training those parameters to minimize a proper scoring rule like the cross-entropy @xcite . one instance of this approach is linear logistic regression. linear logistic regression assumes that the logarithm of the calibrated posteriors are given by: the α k and β k parameters are estimated by minimizing the cross-entropy on a training set c @xmath3, y (1) ), . . . , (q (n ) , y (n ) )} where q (i) and y (i) are the query and the class of sample i: in this work we take α k to be a scalar, independent of the class. this is what is usually done for calibration @xcite @xcite . in particular, temperature scaling @xcite , one of the most widely used calibration methods, corresponds to taking β @xmath4. if we restrict the calibration transformation to have α @xmath5, we can derive the following expression for β k : note that this is not a closed-form expression for β k but rather a system of equations since the righthand side contains all the β k within the γ (i) . we can now compare equation (5) and equation (7). taking the logarithm of equation ( 5 ) for one specific class k we get: the form of this expression is identical to that of equation ( 7 ) when taking α @xmath6. both γ and γ ′ are determined so that the posterior on the left-hand side adds to one. hence, if β @xmath7, the two posteriors would be identical. comparing the expressions for β k and β ′ k we can see that they coincide if we take p (y k |e) = n k /n as we assume in our experiments for the sucpa approach, and if we take γ (i) = 0, since in that case the second term in β k coincides with equation (6). of course, γ (i) is not necessarily zero. yet, as we will see in the experiments, making this assumption has little effect on the results. nevertheless, we can also estimate the β k that satisfies equation (10) exactly using an iterative approach where we first set γ (i) = 0 and compute β k for all k, plug those values into equation ( 8 ) to get a new value for γ (i) and plug that back into equation ( 10 ), repeating these steps until convergence. we find that this algorithm leads to identical results as running linear logistic regression with α @xmath8. in the following, we will refer to the ucpa (and sucpa) approach described in section 3 as "upca-naive" (and "sucpa-naive"), whereas the iterative version of this method will be called simply ucpa (and sucpa). figure 1 summarizes both approaches for both the ucpa and sucpa variants. ## experimental set up we evaluate the proposed approach on the n-shot text classification task following a similar procedure as @xcite . we use four datasets for which the task is classification of a single text into known categories: binary sentiment analysis using sst-2 @xcite , 6-way question classification using trec @xcite , 4-way news classification using agnews @xcite , and the 14-way ontology classification using dbpedia @xcite . we used the standard train and test partitions for all datasets. for agnews and dbpedia we selected 1000 random samples from the test set since their test splits were too large for computation in our infrastructure. this approach was similar to the one used by @xcite where they selected 300 test samples (see their github repository foot_0 ). also following this work, we used gpt-2 xl which has 1.5b parameters and consists of a decoder-only transformer architecture @xcite . the checkpoint was downloaded from the huggingface website foot_1 , and the code used to run experiments is available on github foot_2 . for each dataset, a set of n shots were selected by random sampling the train split and added to the preface (see we show results in terms of error rate (1accuracy), equivalently to @xcite . further, in the final results, we also include the crossentropy performance. cross-entropy is a proper scoring rule which means that it evaluates the performance of the provided scores as posterior probabilities @xcite . in the figures we show normalized cross-entropy, where the crossentropy is divided by the cross-entropy of a naive system that always outputs the prior distribution, ignoring the input sample. a normalized crossentropy larger than 1.0 indicates that the system is so badly calibrated that its performance is worse than that of a naive system @xcite . ## effect of the training set size figure 2 shows the error rate for all datasets as a function of the number of training samples used to perform domain adaptation for the 0-shot scenario (no examples added to the preface). this set is used either to train the calibration model (in which case the class labels are used), to compute equation (6) for ucpa/sucpa-naive, and to compute equation (10) for ucpa/sucpa. for sucpa, the training labels are used to compute n k /n which is used to obtain p (y|e) and in equation (10). for ucpa, the training labels are never used and n k /n is assumed uniform over the classes. the figure also shows the results for the baseline system which takes the posteriors from equation (2) without any prior adaptation. finally, we show the performance of a naive baseline that always chooses the most likely class in the training data. we first note that, as explained in section 4, sucpa and linear logistic calibration with α = 1 gives identical performance. we found the same results for the case of 1, 4 and 8-shot learning, indicating that our iterative algorithm for estimating the β k that satisfies equation ( 10 the instruction text for each case is taken from @xcite . ## effect of the number of shots figure 3 shows the effect of the number of examples (shots) added to the preface for each dataset in terms of error rate and cross-entropy when the number of training samples is 600. we compare our proposed methods with four systems: 1) the nonadapted posteriors (solid black line), 2) the affine calibration method (solid green line) in which pa-rameters α and β are trained using the labelled training data, and 3) and 4) the two content-free calibration methods explained in section 3.1 with the two sets of content-free inputs that were used in the authors' code (see footnote above), namely, {'idk'} and {'[mask]', 'n/a', '' }. we can see that, for 600 training samples, our proposed methods consistently improve upon the content-free baseline, as well as over the non-adapted posteriors. in most cases, the non-adapted model presented a mean cross-entropy close to or higher than 1 for all number of shots, which implies that the model is useless for this task and cannot learn from the prompt shots. they also tend to have small standard deviation and a more stable tendency across the number of shots. of course, the content-free approaches have the advantage that they do not require training data at all. yet, these results show that, if unlabelled training data is available, we can obtain significant gains from our proposed ucpa approach. further, if an estimate of the class priors is available, the sucpa approach can lead to additional gains in datasets with imbalanced priors, like trec. figure 3 also shows that the affine calibration system performs consistently better than our proposed approaches, particularly in terms of crossentropy which better highlights issues of miscalibration compared to accuracy. this is expected since the calibrator is taking advantage of the labelled training data. note that affine calibration is one specific case of a downstream classifier, one with very few parameters which can be trained with a small number of samples. a more complex downstream classifier may give further improvements, but would require larger labelled training datasets. with some exceptions (like in sst-2), increasing the number of examples added to the preface leads to gains in all methods that do some kind of adaptation. the original posteriors, on the other hand, have erratic behavior as a function of the number of shots with a very large standard deviation resulting from the specific selection of examples. additional experiments were performed when the number of training samples is set to 40 showing similar trends to those in figure 3 with the exception that sucpa works worse than ucpa in most cases due to a bad estimate of the class priors. in practice, the class priors would be estimated from knowledge of the task rather than from the training dataset, so that the sucpa performance would in fact depend on how good that estimate is. overall, we can see that the proposed method leads to a large gain with respect to the non-adapted posteriors, even in a scenario where a relatively small amount of unlabelled data is available for training. in terms of computational requirements in comparison with the baseline method, the proposed approach requires the computation of the probabil-ities p (y|e). the time needed to compute these probabilities depends linearly on the number of training samples. during evaluation, the proposed approach has a negligible overhead with respect to the baseline system since the only difference is that it needs to compute equation (5) using the pre-computed p (y|e) and target priors p (y|e). ## conclusion and future work in this work we proposed a method for calibrating the posteriors generated by an llm for a certain text-classification task. we assume that only a relatively small number of unlabelled in-domain samples are available for adaptation. we propose a simple method for calibrating the posteriors generated by the llm by adapting the prior class distribution to the task of interest in an unsupervised manner. optionally, the method allows the new priors to be set to the ones we expect to see during deployment, when known. when such priors are unknown, they can be assumed uniform. we show that, as long as the test priors can be estimated reasonably well, or that the uniform assumption is not too far off from the test distribution, the proposed approach works significantly better than the un-adapted posteriors even with a small amount of available adaptation samples. further, we show that it works better than a previous approach where calibration is performed without using any adaptation data. in our experiments, the proposed method was shown to work well for classification tasks where the number of classes was relatively small in number compared to the amount of training data. we hypothesize that the requirement on training data would increase linearly with the number of classes. future work will include a comparison with finetuning techniques, as well as experiments with llms larger than gpt-2 xl. further, we will explore the generalization of the proposed method to other nlp tasks like question-answering and summarization where the required output is not necessarily restricted to just a few tokens and to tasks outside of the nlp domain where prior mismatch may also be a common scenario.