Linear Layer Extrapolation for Fine-Grained Emotion Classification (2024)

Abstract: Certain abilities of Transformer-based language models consistently emerge in their later layers. Previous research has leveraged this phenomenon to improve factual accuracy through self-contrast, penalizing early-exit predictions based on the premise that later-layer updates are more factually reliable than earlier-layer associations. We observe a similar pattern for fine-grained emotion classification in text, demonstrating that self-contrast can enhance encoder-based text classifiers. Additionally, we reinterpret self-contrast as a form of linear extrapolation, which motivates a refined approach that dynamically adjusts the contrastive strength based on the selected intermediate layer. Experiments across multiple models and emotion classification datasets show that our method outperforms standard classification techniques in fine-grained emotion classification tasks.

## introduction despite the success of large language models on a variety of natural language processing (nlp) tasks @xcite , they still struggle with commonsense reasoning and factual recall @xcite , often hallucinating incorrect information @xcite . recently, contrastive methods, which maximize differences between a desirable "expert" and undesirable "amateur" model, have been proposed to address these issues @xcite . in particular, decoding by contrasting layers, or dola, improves factuality by contrasting model outputs against early-exit predictions from intermediate layers of the same model. @xcite dola operates under the premise that later layers encode factuality, and thus late-emerging changes to predictions likely update towards more factual predictions. recent work has demonstrated that intermediate layer features can also effectively quantify emotions in text @xcite . identifying emotions in text is crucial in nlp for applications ranging from detecting harmful behavior to enhancing conversational agents @xcite . while many systems often focus on mutually exclusive emotions like joy or sadness, fine-grained emotions like grief and remorse are more nuanced and distinct. many language-model systems still struggle to classify fine-grained emotions and opinions @xcite . given this, we explore dola-style layer contrast to improve finegrained emotion classification. the main contributions of the paper are: early exiting: early-exiting predictions are obtained by applying the classification head of a model to the residual stream earlier in the network. these have been used to accelerate inference and dynamically allocate compute on a per-input basis @xcite @xcite . contrastive steering: contrastive methods optimize the difference in predictions between a favorable "expert" and an unfavorable "amateur," to steer text decoding in language models. @xcite gedi @xcite contrasts between classspecific control codes to improve text-conditioned factuality and emotion control. coherence boosting @xcite provides the language model with only the final k tokens of the prompt to obtain amateur scores, encouraging longer-term coherence over locality. contrastive decoding @xcite improves long-form generation and reasoning ability by contrasting between large and small models of the same family. other works use cd-like methods to reduce model toxicity, surface biases, and increase faithfulness to a provided context. @xcite @xcite 3 method here we define the main components of contrastive decoding (cd) and dola, alongside our proposed method for dynamically selecting contrastive strength. following earlier work, we use early-exit distributions to choose an amateur layer, contrasting its predictions against the final layer (the expert). we apply mask candidate classes based on a plausibility constraint to filter out lowprobability labels. we experiment with two methods to determine the contrastive strength: static β and dynamic β. the details of each component are discussed next. ## results we report performance across models, datasets and inference algorithms in traditional vs contrastive classification: we observe that contrastive classification improves performance significantly in both recall and f1. this trend holds for all models used in our experiments. empatheticdialogue: for this dataset, we only see a slight increase in performance using the deberta-xl model. analyzing the probability distributions between layers, we did not observe a significant change in the probability distribution for different emotions across layers. the probability was distributed over a single label, increasing gradually across layers. this led to minimal contribution, positive or negative, from layer contrast. effect of amateur layer selection: we use a bucket of layers for amateur layer selection defined by hyperparameter k. figure 2c shows the trend of k against f1 using the deberta-xl for tweetemotions dataset. we observe that the performance generally increases up to a layer where the benefit of contrastive action is maximum, followed by a drop in performance. upon evaluating earlyexiting on intermediate layers, we observed that some layers are more adept at identifying specific classes than others, providing a variety of skills to contrast against for improved performance. ## analysis we find that contrastive inference tends to flip neutral predictions to underrepresented classes. we also report the most frequent samples corrected for the tweetemotion dataset in ## conclusion we propose a linear extrapolation approach for dynamically determining contrastive strength in layerwise contrastive decoding. applied to finegrained emotion classification tasks, this method enhances classifier performance by effectively identifying under-represented classes.this strengthens the promise of layer-contrast methods in domains other than text generation, and provides a technical contribution that reduces the variance of the method with respect to a core hyperparameter k, encouraging further research into how best to exploit the layerwise emergence of textual understanding to improve performance on a wide range of tasks.