Logit Separability-Driven Samples and Multiple Class-Related Words Selection for Advancing In-Context Learning (2025)

Abstract: Effective organization of in-context learning (ICL) demonstrations is key to improving the quality of large language model (LLM) responses. To create better sample-label pairs that instruct LLM understanding, we introduce logit separability, a criterion to assess the clarity of both samples and class-related words at the logit level. This facilitates the optimization of sample and label selection, enhancing the precision of information provided in ICL demonstrations. Additionally, we find that incorporating multiple class-related words for each sample, rather than relying on a single class name, improves performance by offering a broader range of label information. Building on these insights, we propose LICL, a logit separability-based method that jointly organizes samples and integrates multiple class-related words into each sample-label pair. Evaluations across seven classification datasets show that this approach significantly improves ICL performance by providing clearer instructions and richer label information.

## introduction in-context learning (icl) enables large language models (llms) to perform new tasks using samplelabel pairs as demonstrations, without the need for retraining or fine-tuning @xcite . however, the organization of these sample-label pairs is critical, as it can significantly affect icl performance 1 @xcite . how do we assess the suitability of samples and labels for demonstration? in traditional machine learning, effective features are those that exhibit high discriminative power, enabling clear distinction between their respective classes @xcite . analogously, we introduce logit separability, a criterion to assess how well in-demonstration samples and class-related words differentiate between classes at the logit level. logit separability captures two aspects: (1) how significantly a sample is predicted to its true label, where the correct label has substantially higher logit than others, and (2) how consistently a class-related word yields high logit values across samples of the same label. this measure provides insight into both the separability of samples through class-related words and the ability of these words to distinguish between samples of different classes. building on this concept, we observe that not all samples or class-related words exhibit the same degree of logit separability. as shown in fig. 1(a) , some samples produce strong separation in the logits of class-related words, assigning high logits to words matching the true label and low logits to others, while others show marginal differences between them. additionally, certain class-related words consistently display higher logit in samples of their corresponding label compared to samples of other labels. as shown in fig. 1(b) , "bad" shows a clearer differentiation between positive and negative samples than the class name "negative". given the importance of providing clear labelinstructed information in demonstrations, leveraging logit separability for selecting samples and labels might enhance icl performance. as shown in fig. 1 (a)-accuracy, samples with strong logit separability provide more precise signals by aligning class-related words with the model's internal representations. besides, class-related words with distinct logit separability can beat predefined class names in guiding the model toward accurate predictions (appx.a.2.1). therefore, selecting both samples and class-related words via logit separability ensures clearer, more effective demonstrations, ultimately improving the performance of icl. besides, the variability observed in logit values of class-related words across different samples shows that relying on a single class-related word may not adequately capture the full semantics associated with a class name. unlike expanding candidates via a verbalizer in prediction mapping, directly inserting class-related words into demonstrations offers explicit instructions that guide llm responses to potential candidates. consequently, transitioning from single to multiple class-related words could enhance linguistic precision and deepen the semantic representation of class names, potentially leading to enhanced model performance. fig. 1(c ) supports that using multiple class-related words in sample-label pairs indeed strengthens icl performance. given samples and labels' critical roles in demonstration effectiveness for icl, in this work, we propose a logit separability-based demonstration organization method. we organize demonstration samples based on their zero-shot logit separability across a pool of class-related words. in demonstration label selection, we propose a novel method that utilizes multiple class-related words instead of a single one. the selection, ordering, and quantity of class-related words are determined through sequential forward search, guided by logit feedback from the selected samples and their validation performance. moreover, while most research investigates either the sample selection @xcite @xcite or label selection @xcite @xcite ) individually, we comprehensively address both aspects to enhance icl. we summarize our contributions as follows: ## related work demonstration organization in icl to improve icl performance with better demonstration organization, some studies use pre-trained models like s-bert @xcite or bm25 @xcite to select and rank demonstrations. while these unsupervised methods have advantages, they may cause inconsistencies in knowledge transfer. other approaches organize demonstration based on the lm's output distribution. some methods take part of the training set as validation to enable supervised learning methods for demonstration organization @xcite @xcite . however, this will shrink the pool of candidates, risking sub-optimal selection. additionally, some methods use the lm's output, like label confidence, to organize demonstrations under the full training set @xcite @xcite . in the above works, the class names are preassumed to be the label word when creating the demonstration sample-label pairs. label matters in icl the significance of demonstration labels in icl has been debated. @xcite demonstrates a positive relationship between icl performance and accurate sample-label mapping. @xcite reveals that using the same sample but different class-related words as labels in the demonstrations can result in very different icl performance. @xcite suggests that labels derive semantic representations from demonstrations for use in deep layers to make final predictions. yu and ananiadou (2024) further shows that these demonstration features are integrated into corresponding labels, with each incontext head extracting features specific to these labels. @xcite uses different class-related words in augmented samples to enhance icl performance, indicating the potential for enhanced demonstration effectiveness when multiple class-related words are used. however, this approach significantly increases the required token length and running time in augmentation settings. ## method in this section, we introduce our proposed method licl, which comprises logit separability-based demonstrated sample selection and ordering, and corresponding multiple class-related words insertion, as shown in fig. 2 . ## experiments in this section, we examine the capacities of our method in icl from the following perspectives: ## conclusion in this paper, we introduce logit separability, a criterion for measuring the instruction clarity of samples and labels based on their logit value distributions, assessing whether a sample is easily predictable by its true label and whether a label performs well across multiple samples of the same class. to improve icl with clearer class-related guidance, we propose licl, a logit separabilitybased method for selecting and ordering demonstrated samples and labels. besides, to enrich icl with broader label information, we present a novel label demonstration method by forming samplelabel pairs through the insertion of multiple classrelated words, guided by sequential forward search on the logits from selected samples and validation performance. experimental results confirm that this combined approach significantly enhances icl classification performance, yielding superior outcomes.