CURE : Controlled Unlearning for Robust Embeddings — Mitigating Conceptual Shortcuts in Pre-Trained Language Models (2025)

Abstract: Pre-trained language models have achieved remarkable success across diverse applications but remain susceptible to spurious, concept-driven correlations that impair robustness and fairness. In this work, we introduce CURE, a novel and lightweight framework that systematically disentangles and suppresses conceptual shortcuts while preserving essential content information. Our method first extracts concept-irrelevant representations via a dedicated content extractor reinforced by a reversal network, ensuring minimal loss of task-relevant information. A subsequent controllable debiasing module employs contrastive learning to finely adjust the influence of residual conceptual cues, enabling the model to either diminish harmful biases or harness beneficial correlations as appropriate for the target task. Evaluated on the IMDB and Yelp datasets using three pre-trained architectures, CURE achieves an absolute improvement of +10 points in F1 score on IMDB and +2 points on Yelp, while introducing minimal computational overhead. Our approach establishes a flexible, unsupervised blueprint for combating conceptual biases, paving the way for more reliable and fair language understanding systems.

## introduction with the rapid advancement of artificial intelligence, pre-trained language models (plms) have been widely adopted across various domains, including education, healthcare and e-commerce @xcite @xcite . a predominant strategy for applying these models is fine-tuning, where a plm is further adapted to task-specific data, aiming to enhance its performance or better align with human intent @xcite . however, training -the wood-fired pizza had the perfect balance of crispy crust, tangy tomato sauce, and gooey cheese-absolutely delicious! (f θ : positive) -i wasn't expecting much, but the homemade lasagna was rich, and bursting with flavor! (f θ : positive) testing -the breading was soggy, and the meat was disappointingly dry. (f θ : positive) figure 1 : example of shortcut learning in sentiment classification, where a classification model f θ wrongly associate reviews about food to a positive sentiment. fine-tuning often exposes models to dataset biases, leading to shortcuts-spurious correlations between features and labels @xcite . for instance, @xcite demonstrated that on the yelp dataset @xcite , a llama2based @xcite sentiment classifier mistakenly associated the concept of "food" with a "positive" label. these fragile dependencies not only limit the robustness of plms but also pose significant risks. in medical diagnosis, a biased detector might incorrectly associate certain biological attributes with diseases, leading to inaccurate predictions @xcite . similarly, in automated recruitment systems, a shortcut may result in a favor to applicants with certain demographic attributes, exacerbating fairness problem. in figure 1 , we present an example where the classifier incorrectly associates the concept of food with positive sentiment. contemporary debiasing research primarily focus on two strategies: (1) modifying shortcutinducing terms in training data @xcite , and (2) generating counterfactual samples @xcite via large language models (llms). however, both approaches suffer from notable limitations. lexical modification requires prior knowledge of shortcutinducing terms, which is often challenging to ob-tain @xcite . moreover, its effectiveness is restricted to lexical shortcuts rather than conceptual biases. on the other hand, llmbased counterfactual generation is computationally expensive and increases training costs significantly. while llm-free counterfactual generation still relies on prior knowledge @xcite , making it similarly constrained. as an unsupervised and lightweight solution, we propose cure-controlled unlearning for robust embeddings. cure remaps the semantic space to disentangle conceptual and content-related information without human annotation, offering finegrained control over shortcut effects. it first trains a content extractor using a concept classifier and back-translation to produce concept-irrelevant representations. a contrastive learning-based debiasing module then refines sample representations, adjusting conceptual features as needed. finally, the module is jointly trained with a classification head to enhance model robustness. unlike traditional approaches, cure offers three key advantages: prior knowledge independence -cure uses unsupervised learning, eliminating the need for manual annotations of shortcuts. resource efficiency -cure eliminates the need for llm-driven data augmentation, reducing the training time to approximately one-tenth of the original. controllability -cure can quantify the impact of conceptual bias on classification results. this facilitates both the mitigation of conceptual biases to enhance performance on out-of-distribution (ood) data and the exploitation of shortcuts to improve performance on independent and identically distributed (i.i.d.) data. such adaptability enables users to align training objectives with their generalization requirement, while also providing a quantifiable framework for future debiasing research. our contributions are as follows: ## related work addressing spurious correlations in plms has become a critical research focus, as these correlations can lead to biased and unreliable predictions, limiting model robustness and fairness. traditional works have explored various strategies to mitigate these issues, including causal inference techniques @xcite , adversarial training @xcite , and data augmentation methods designed to reduce model reliance on spurious features @xcite . additionally, approaches leveraging counterfactual reasoning @xcite have shown promise in improving fairness and robustness in llms. these advancements collectively contribute to a growing body of research aimed at developing more reliable and ethically sound language models. @xcite addresses the challenge of models learning spurious topical shortcuts instead of relevant features in tasks like native language identification. they introduce an adversarial model to demote these latent topical confounds using logodds ratios, guiding the model to focus on stylistic rather than topic-based features. ## conclusion in this work, we introduced cure, a novel and lightweight framework for mitigating conceptual shortcuts in pre-trained language models. cure enables fine-grained control over conceptual bias retention by systematically disentangling conceptrelevant and content-relevant representations. it balances robustness and accuracy based on task requirements. our experiments on imdb and yelp datasets demonstrate that cure significantly improves out-of-distribution robustness, achieving up to 5-point accuracy gains and 10-points f1 gains over baselines while maintaining minimal computational overhead. notably, cure reduces training time by an order of magnitude compared to llm-driven debiasing approaches, making it a scalable and efficient solution. these results highlight cure, which reveals the potential of unsupervised conceptual debiasing in enhancing the reliability of language models while preserving critical task-relevant features.