Measuring Pointwise \mathcal{V} -Usable Information In-Context-ly (2023)

Abstract: In-context learning (ICL) is a new learning paradigm that has gained popularity along with the development of large language models. In this work, we adapt a recently proposed hardness metric, pointwise \mathcal{V} -usable information (PVI), to an in-context version (in-context PVI). Compared to the original PVI, in-context PVI is more efficient in that it requires only a few exemplars and does not require fine-tuning. We conducted a comprehensive empirical analysis to evaluate the reliability of in-context PVI. Our findings indicate that in-context PVI estimates exhibit similar characteristics to the original PVI. Specific to the in-context setting, we show that in-context PVI estimates remain consistent across different exemplar selections and numbers of shots. The variance of in-context PVI estimates across different exemplar selections is insignificant, which suggests that in-context PVI estimates are stable. Furthermore, we demonstrate how in-context PVI can be employed to identify challenging instances. Our work highlights the potential of in-context PVI and provides new insights into the capabilities of ICL.

## introduction understanding the hardness of a dataset or an instance is crucial for understanding the progress in machine learning, since a dataset is designed as a proxy for real-world tasks @xcite . the significance of hardness has been acknowledged in the field of natural language processing (nlp) @xcite @xcite . extended from the predictive v-information framework @xcite , pointwise v-usable information (pvi) is a recently proposed metric for measuring the hardness of individual instances @xcite . pvi is estimated through supervised learning, which involves fine-tuning two models: one model that is fine-tuned on input-target pairs, and another model fine-tuned on only the target labels. pvi measures the amount of usable information in an input for a given model, which reflects the ease with which a model can predict a certain label given an input. though it is a recently proposed method, the effectiveness of pvi has been demonstrated in various nlp tasks @xcite @xcite . recent years have seen remarkable progress in the development of large language models (llms), and it is expected that they will continue to be an essential topic in nlp @xcite @xcite . in the era of llms, pvi is useful in many aspects as a measure of hardness, such as the development of high-quality new benchmarks that can better evaluate the capabilities of llms and the selection of in-context exemplars that enhance model performance. however, given the scales of the state-of-the-art llms, the calculation of pvi can be challenging due to the need for fine-tuning. motivated by the need to leverage pvi for llms and the recent discoveries that in-context learning (icl) is similar to fine-tuning @xcite @xcite , we implement pvi in an in-context manner (i.e., incontext pvi). rather than fine-tuning, we prompt a model using two few-shot prompts, an input-target prompt and a target-only prompt (see a.2 for an example). figure 1 shows the difference between the calculation of pvi and in-context pvi. this work aims to investigate the feasibility of obtaining reliable pvi estimates in an in-context setting. we conducted experiments with various datasets and a range of llms. we showed that in-context pvi behaves similarly to the original pvi: in-context pvi estimates are consistent across different models, and the threshold at which predictions become incorrect is similar across datasets. 1 15739 specific to the in-context setting, we found that incontext pvi estimates are consistent across different selections of exemplars and numbers of exemplars in the prompt. we also found that the variance of in-context pvi estimates across different exemplar selections is insignificant, which suggests that incontext pvi estimates are quite stable. we assessed the correlation between in-context pvi estimates and inter-annotator agreement, and our findings indicate that in-context pvi estimates made by larger models are more reliable. we conducted a qualitative analysis to show how in-context pvi assists in identifying challenging instances. additionally, we presented a preliminary analysis on the potential of in-context pvi to improve model performance through exemplar selection. our contributions are summarized as follows: ## method algorithm 1 shows the calculation of pvi, which involves fine-tuning a model g on two different datasets. in-context pointwise v-usable information (in-context pvi) is adapted from the original pvi. instead of fine-tuning a model g, two few-shot prompts, i.e., an input-target prompt p ′ = (x 1 , y 1 , x 2 , y 2 , ..., x n , y n , x) and a null-target prompt @xmath0, y 1 , ∅, y 2 , ..., ∅, y n , ∅), are used to prompt g. in-context pvi, denoted as c(x, y), is calculated as equation ( 1 ): given the observation that icl resembles finetuning @xcite , log 2 g[p](y) and log 2 g[p ′ ](y) are the icl approximations of log 2 g[∅](y) and log 2 g ′ [x](y) in algorithm 1. the calculation of the original pvi is based on sequence classification, i.e., the dimension of the output space of g ′ and g is the number of unique labels in a task. different to the original pvi, the calculation of in-context pvi is based on generation, where the output is a sequence of tokens such as ["un", "acceptable"]. instead of asking the model to produce a label prediction such as "unacceptable", we matched a numerical index to a label as the target in p ′ and p (see a.2 for an example), so that the expected output is a single token, which makes the calculation of log 2 g[p](y) and log 2 g[p ′ ](y) easier. ## experiment settings dataset we conducted experiments on 7 datasets, including boolq @xcite , cola @xcite , multinli @xcite , @xcite , rte @xcite , and two domain specific datasets-health advice @xcite , and causal language @xcite . the selected datasets cover question answering, natural language inference and reasoning in both general and domain specific settings. we included the three tasks that are used in @xcite , i.e., cola, multinli, and snli. boolq and rte are two commonly used datasets from superglue @xcite . health advice and causal language involve fundamental tasks in the biomedical domains, which require domain specific knowledge. the details of the datasets are shown in tables 9, 10, and 11 in a.1. model we tested models with sizes varying from 125m to 175b: gpt2-125m @xcite , gpt-neo-1.3b, gpt-neo-2.7b @xcite , gpt-j-6b, gpt-jt-6b @xcite , alpaca-7b, alpaca-13b @xcite , and openai text-davinci-003 (gpt3-175b). we limited the use of gpt3-175b to experiments on cola, rte, and the first 500 test instances in multinli and snli due to cost considerations. ## results figure 2 shows the difference in prediction accuracy between instances with the top 20th/50th percentile and bottom 20th/50th percentile of incontext pvi estimates. we observed an increase in accuracy gain when model scales with one exception (gpt2-125m). ## conclusion this paper introduces in-context pvi, an alternative approach to access the hardness of an instance. compared to the original pvi, our proposed method significantly reduces computational cost while behaving similarly to it. we showed that in-context pvi estimates are consistent across different models, selections of exemplars, and numbers of shots. we discovered that larger models tend to produce more reliable in-context pvi estimates, suggesting that in-context pvi is not a strict replacement for the original pvi, and that for smaller models (especially for those that are readily "fine-tunable"), the original pvi is a better choice. we demonstrated that in-context pvi helps identify challenging instances. we also presented a preliminary analysis on the potential of in-context pvi to enhance model performance through exemplar selection. the utility of in-context pvi for discerning instance difficulty may be effectively harnessed for dataset selection in joint learning scenarios, facilitating the construction of balanced and optimized training sets. concurrently, future work may utilize in-context pvi as a tool to design an efficient curriculum learning protocol for llms @xcite . developing an adaptive curriculum for icl, progressing from easier to more difficult instances, as gauged by their in-context pvi estimates, represents a significant direction. a further avenue of exploration lies in developing an efficient training method that strategically prioritizes challenging instances, as identified by in-context pvi, thereby creating focused mini-batches for training llms. this approach could potentially provide both computational savings and enhance learning efficacy. achieving these objectives, however, necessitates addressing a series of research questions, including optimizing joint learning dataset composition and creating algorithms that effectively prioritize difficult instances in the training process. these questions constitute the main focus of our future research. 8 limitations a theoretical analysis is still needed to enhance our understanding and validation of in-context pvi. in order to manage costs, we limited our experiments with openai text-davinci-003 (gpt3-175b) to cola, rte, and the initial 500 test instances in multinli and snli. due to the closed source nature of gpt3-175b, the reproducibility of the results related to it may be a concern as well.