SALAD : Improving Robustness and Generalization through Contrastive Learning with Structure-Aware and LLM -Driven Augmented Data (2025)

Abstract: In various natural language processing (NLP) tasks, fine-tuning Pre-trained Language Models (PLMs) often leads to the issue of spurious correlations, which negatively impacts performance, particularly when dealing with out-of-distribution data.To address this problem, we propose **SALAD** (**S**tructure **A**ware and **L**LM-driven **A**ugmented **D**ata), a novel approach designed to enhance model robustness and generalization by generating structure-aware and counterfactually augmented data for contrastive learning.Our method leverages a tagging-based approach to generate structure-aware positive samples and utilizes large language models (LLMs) to generate counterfactual negative samples with diverse sentence patterns. By applying contrastive learning, *SALAD* enables the model to focus on learning the structural relationships between key sentence components while minimizing reliance on spurious correlations.We validate our approach through experiments on three tasks: Sentiment Classification, Sexism Detection, and Natural Language Inference. The results demonstrate that *SALAD* not only improves model robustness and performance across different environments but also enhances generalization to out-of-distribution datasets and cross-domain scenarios.

## introduction in many natural language processing (nlp) tasks, machine learning models often suffer from the issue of spurious correlations (a.k.a shortcuts) between input text tokens and output labels @xcite . these shortcuts allow models to rely on irrelevant patterns in the data, leading to biased predictions. for example, a model trained mainly on positive reviews of spielberg movies may incorrectly associate the word spielberg with favor-able sentiment, regardless of the actual content of the review @xcite . this reliance on superficial patterns results in poor performance, especially when handling outof-distribution data. many research has addressed this problem by exploring methods for generating counterfactually augmented data (cad) @xcite , intended to disrupt these false correlations. the cad method involves altering input data to flip its label, providing models with examples that help overcome shortcut-based learning and improve generalization across diverse datasets. however, previous approaches have focused mainly on manual generation methods and statistical techniques to automate cad creation. although manual methods can produce high-quality counterfactual data, they are both costly and time-consuming. to reduce these costs and improve scalability, automated techniques have been developed, such as using sentiment dictionaries @xcite , statistical matching, or predefined antonyms @xcite . however, these methods are dependent on fixed rules that restrict the quality of the generated data. for example, sentiment dictionaries and statistical techniques struggle to fully capture the complex context within the data, limiting their ability to produce diverse patterns that can enhance model performance. recently, there has been a shift towards generating cad using pre-trained language models (plms) @xcite @xcite @xcite @xcite . these models offer the advantage of automatically producing high-quality data by reflecting the context of the input data, thereby reducing the reliance on manual methods. however, since plms are trained based on the distribution of the training data, they can easily become biased toward frequently occurring contextual patterns. this causes plms to rely more on common patterns, rather than capturing complex sentence structures or uncovering hidden causal relationships. moreover, most current approaches focus solely on data augmentation, neglecting the need to enhance model robustness during training. to improve model robustness while reducing shortcuts, it is necessary not only to focus on contextual patterns but also to learn structural patterns where shortcuts occur. models that learn these structural patterns can capture the fundamental meaning of sentences and the relationships between key components, rather than relying on word frequency or specific terms. by doing so, the model effectively avoids shortcuts and achieves stronger generalization. structural patterns, particularly those involving the roles of subjects, verbs, and objects, remain consistent across different contexts, allowing models to perform more reliably across diverse datasets. therefore, the model should learn the causal relationships within sentences to effectively learn structural patterns, rather than just word frequency or simple textual features. this allows the model to focus on important words @xcite to grasp the core meaning of the sentence while reducing its reliance on less significant words @xcite . consequently, models that learn structural patterns can provide consistent performance across a variety of sentence structures. in this paper, we propose an effective method, salad, for enhancing model robustness while addressing spurious correlations, called structure aware and llm-driven augmented data for contrastive learning. this method combines structureaware augmented data (positive) generated by a tagging-based approach that considers structural patterns and counterfactually augmented data (negative) generated by large language models (llms) to create complex and diverse sentence patterns. then, contrastive learning is applied to enable the model not only to classify patterns but also to capture relationships between samples. using the original sentence as the anchor, the positive sample is trained to remain close to it while maintaining the structural patterns where shortcuts occur, whereas the negative sample, generated by the llm, is trained to diverge from the anchor by incorporating diverse patterns. our salad helps the model effectively learn key patterns in simple sentence structures, reducing its dependence on shortcuts. simultaneously, the model learns negative patterns in various contexts, enhancing its generalization performance. we conducted experiments on three tasks: sentiment classification, sexism, and natural language inference, to verify the model's robustness across various environments. furthermore, we demonstrated our model's generalization performance in out-of-distribution datasets and crossdomain settings. ## related work text data augmentation data augmentation has traditionally been used to improve model performance and increase data diversity. eda @xcite applied simple heuristic transformations such as synonym replacement, word insertion, deletion, and swapping. while it was easy to implement and low-cost, it has the limitation of not considering context. plm-based methods, such as ssmba @xcite , used pre-trained models like bert @xcite to corrupt and reconstruct data, improving performance, particularly on out-of-domain datasets. however, this approach requires significant computational resources and could introduce noise. llm-based methods, like auggpt @xcite , leveraged gpt-3 to rephrase sentences and enhance performance in few-shot learning settings, but they relied heavily on large language models, which were resourceintensive. while these methods contributed to increased data diversity, they had the drawback of lacking fine-grained control over the quality and relevance of the generated data. ## proposed method figure 1 illustrates an overview of salad, consisting of the following three processes: structureaware data generation, llm-driven counterfactual data generation, and contrastive learning with triplet loss. ## results in this section, we demonstrate the outperformance of salad for the robustness and generalization abilities of the model in three tasks. we also conduct comprehensive ablation studies to demonstrate its superiority. ## conclusion in this paper, we introduced salad, a method that enhances model robustness and generalization through the use of structure-aware and counterfactually augmented data for contrastive learning. by combining a tagging-based method for generating structure-preserving positive samples and llmgenerated counterfactual negative samples, salad enables models to learn meaningful structural relationships while reducing their reliance on spurious correlations. our experiments, conducted on three tasks demonstrate that salad significantly improves model performance across diverse environments, particularly in out-of-distribution and cross-domain settings. these results highlight the potential of our approach to address the challenges of spurious correlations in natural language processing, providing a more robust and generalizable solution.