S ea PO : Strategic Error Amplification for Robust Preference Optimization of Large Language Models (2025)

Abstract: Existing alignment methods for preference optimization of large language models (LLMs) aim to enhance model performance by utilizing pairs of positive and negative samples. However, due to the limited capacity of models in scoring or generating responses, the quality of positive and negative samples may become similar during training, which complicates optimization for preference learning. To address this issue, we introduce SeaPO, a Strategic Error Amplification method that leverages three error types commonly occurring in LLMs to introduce specific error patterns into the model Preference Optimization. This strategy ensures that negative samples are more erroneous than positive samples and preference-based training is employed to mitigate the occurrence of these errors, thereby enhancing model performance. Evaluations across five capability dimensions and different model scales (1.5B to 14B) demonstrate that the generated data significantly improved overall model performance, particularly in terms of truthfulness, with improvements of 5–10 percentage points observed. Further analysis reveals that task performance varies depending on the error types introduced. Injecting the most common error types improves performance in related tasks, while a mix of error types leads to a broader performance enhancement: most tasks show stable improvements, while a few tasks exhibit significant gains.

## introduction since the advent of gpt4 @xcite , data have become the major affect factor in large language models (llms) of capability @xcite @xcite . extensive research has explored various aspects of data, including knowledge distillation @xcite @xcite figure 1 : due to the limitations of the reward model and the sampling, the rejection may be as good as the chosen one, making preference-based optimization more difficult. in the vanilla approach, positive and negative samples are obtained through multiple sampling and a scoring model, which are required to identify positive and negative samples. in contrast, our method only requires introducing specific error types to the original responses without the need for rejection sampling. however, due to the inherent limitations of a model's capabilities @xcite @xcite , achieving human-like precision in specific tasks @xcite , such as evaluating sentence quality or generating accurate responses @xcite to complex questions, remains challenging. this results in scoring models potentially misjudging @xcite positive and negative samples, leading to minimal differences between them and, consequently, difficulties in preferencebased training, which can result in potential logical or hallucination generation. to address this challenge, we propose seapo, which utilizes error-injected negative samples that have been largely overlooked in prior studies. specifically, we define three strategic errors commonly occurring in llms and introduce a straightforward error-injected negative sample generation method to produce these target error types. after that, seapo employs an error-focused preference optimization objective, aiming to reduce the probability of negative samples and, in turn, minimize the likelihood of the introduced strategic errors, thereby enhancing overall model performance. seapo reduces the dependence on powerful models, requiring only minimal error injection ability and minimizes computational overhead by eliminating the scoring process, as shown in figure 1 . through extensive experimental validation, we demonstrate that seapo effectively generates more erroneous samples compared to the original negative samples, thereby enhancing both overall and specific model capabilities, such as mathematical reasoning, code solving, and truthfulness, through negative sample augmentation with these strategic errors. the results show improvements of up to 18.8% in llama3-8b-instruct and 12.3% in qwen2.5-7b-instruct on truthfulness, surpassing existing methods such as @xcite . additionally, ablation experiments highlight the significance of both error definition and error injection. our main contributions are as follows: ## conclusion due to model limitations and no explicit quality control in preference data construction, positive and negative samples may have similar quality; in some cases, negative samples may even be better. to address this issue, we propose seapo that explicitly introduces strategically defined errors. applying preference optimization reduces error probabilities and enhances overall performance. evaluations across five capability dimensions and multiple models (1.5b-14b) demonstrate the robustness of our approach. these strategic errors vary depending on the task, and targeting the most error-prone types can significantly boost task performance.