Weed Out, Then Harvest: Dual Low-Rank Adaptation is an Effective Noisy Label Detector for Noise-Robust Learning (2025)

Abstract: Parameter-efficient fine-tuning (PEFT) large language models (LLMs) have shown impressive performance in various downstream tasks. However, in many real-world scenarios, the collected training data inevitably contains noisy labels. To learn from noisy labels, most solutions select samples with small losses for model training. However, the selected samples, in turn, impact the loss computation in the next iteration. An inaccurate initial selection can create a vicious cycle, leading to suboptimal performance. To break this cycle, we propose Delora, a novel framework that decouples the sample selection from model training. For sample selection, Delora establishes a noisy label detector by introducing clean and noisy LoRA. Benefiting from the memory effect, the clean LoRA is encouraged to memorize clean data, while the noisy LoRA is constrained to memorize mislabeled data, which serves as a learnable threshold for selecting clean and noisy samples. For model training, Delora can use carefully selected samples to fine-tune language models seamlessly. Experimental results on synthetic and real-world noisy datasets demonstrate the effectiveness of Delora in noisy label detection and text classification.

## introduction llms are extremely powerful, yet they are expensive to train. by balancing performance with practicality, peft has become a popular technique for adapting the llm for downstream applications. notable peft methods include low-rank adaptation (lora) @xcite , adaptors @xcite , and prompts @xcite . instead of fine-tuning all weights, peft fixes the backbone model parameters while adding a few learnable parameters for adaptation. while promising, peft techniques rely on perfectly labeled datasets, which * corresponding author our method decouples sample selection (stage 1) from model training (stage 2) by training a noisy label detector and classifier model separately. to tackle this issue, a recent work @xcite explores peft methods on imperfectly labeled datasets (i.e., datasets with noisy labels) to learn with noisy labels (lnl). it uses training losses to select clean data and suggests a routingbased peft method that preferentially trains peft modules on clean data. similar to previous lnl methods @xcite @xcite based on sample selection, they both adopt the "small-loss" mechanism to select clean data because the model tends to fit clean samples earlier than noisy samples during training. however, these methods are inherently affected by the label noise, as losses used for sample selections are extracted from the model that is being trained. specifically, the sample selection process affects subsequent training, and training loss in turn influences the sample selection. if the initial sample selection is poor, it leads to an inescapable vicious cycle. we contend that in such a strategic feedback loop, a good sample selection can itself fall into a new paradox, akin to: it's a catch-22 situation: a good sample selection requires generalizability, while generalizability requires sample selection. given this, let us leave the existing framework and revisit the lnl task. the main difficulty here is that we need to select clean samples to train a strong model while avoiding the effects of noisy samples. can we, perhaps, decouple sample selection from the model training, making them independent of each other? with this question in mind, we propose a new framework that firstly leverages peft modules to construct a noisy label detector for sample selection and then trains the model using selected samples, as shown in figure 1 . in the first stage, we introduce two distinct peft modules (i.e., clean lora and noisy lora) to construct a noisy label detector. the parameters in clean lora are termed the ideal parameters used to memorize clean data, while parameters in noisy lora are called noise parameters used to memorize mislabeled data. our noisy label detector can be derived from the cross-entropy between the predictions of clean/noisy lora on the training text and its corresponding labels. if the text exhibits a higher cross-entropy value with the noisy lora than the clean lora, it is considered a correctly labeled sample, i.e., a clean sample. this design draws inspiration from a recent study @xcite that decomposes an image into its constituent subject and style, represented as two distinct loras. this insight has inspired us to explore different loras to memorize clean and noisy samples separately in lnl. however, as we cannot directly obtain clean and noisy samples from datasets, controlling loras to accomplish our notion is a challenge. note that the memorizing effect @xcite demonstrates that deep networks would first memorize clean samples and then noisy samples. based on this, we introduce dynamic regularization to adjust the parameter updates of the two loras over time. specifically, we constrain noise parameter updates of noisy lora in the early training stage and make ideal parameters of clean lora completely memorize clean data. as training progresses, the restrictions on noisy lora are gradually lessened while the constraints on clean lora are reinforced, resulting in noisy samples being mostly memorized by noisy lora. in the second stage, we train the classifier model using samples carefully selected through the noisy label detector. in this step, we first utilize selected clean samples as contextual references for reliable relabeling of noisy samples, and then merge the clean samples and relabeled noisy samples to finetune the classifier model. in this way, we can obtain a denoised fine-tuning dataset while maximizing data utility to improve the generalization ability of our framework in lnl tasks. it is worth noting that the sample selection is independent of the in-training classifier model, which can effectively avoid the issue of vicious cycles. overall, our main contributions can be summarized as follows: ## method in this section, we present our proposed denoising learning framework delora in detail. the main idea is to decouple sample selection and model training during the fine-tuning of llms on downstream tasks, avoiding the issue of vicious cycles. overall, delora comprises two pivotal stages. in the first stage, delora introduces dual loras to construct a noisy label detector, selecting clean samples and noisy samples. in the second stage, delora leverages the curated clean samples and relabeled noisy samples to train the classifier model, further boosting the performance of our framework on text classification in the context of noisy environments. figure 2 shows our proposed framework. ## conclusion in this work, we mitigate the issues of the vicious cycle in current mainstream sample selection methods and further explore the peft methods in the era of llms to solve the lnl tasks. specifically, we decouple this task into sample selection and classifier model training. for sample selection, we introduce the dual lora to construct a new noisy label detection approach, which restricts the clean lora to fit clean data and the noisy lora to absorb the side effects of mislabeled data. then we can train the robust classifier model by leveraging the carefully selected samples. extensive experiments have convincingly demonstrated the superior performance of delora in both noisy label detection and text classification. moreover, our in-depth analysis has demonstrated that delora can generalize to other language models.