Sentimatic: Sentiment-guided Automatic Generation of Preference Datasets for Customer Support Dialogue System (2025)

Abstract: Supervised Fine-tuning (SFT) and preference optimization (PO) are key methods for enhancing language models and aligning them with human preferences. However, scaling preference datasets for PO training is challenging, leading AI customer support systems to rely on SFT. To address this, we propose the Sentiment-guided Automatic Generation of Preference Datasets (Sentimatic) methodology to automatically generate customer preference datasets without human intervention using a publicly available dataset constructed for SFT. Our approach classifies responses by sentiment, fine-tunes models on them, and applies advanced sampling and evaluation techniques to ensure diversity and quality. Ultimately, we generated 1,174 customer preference datasets based on 357 test datasets, and through experiments, we confirmed that the AI customer support system trained on these datasets is capable of carefully considering customer emotions and generating professional and appropriate responses.

## introduction previous studies have used the sft approach primarily to train ai models for customer service @xcite @xcite . however, sft focuses solely on the accuracy of individual tokens generated by the model, failing to adequately reflect the overall quality of conversations. this limitation can lead to inefficiencies in performance evaluation and optimization. in contrast, po addresses these issues by evaluating the quality of the entire response generated by the model @xcite . however, the preference datasets required for po training are created through response comparisons, which require the involvement of human annotators. this dependency significantly increases the time and cost of large-scale data collection, posing challenges to the widespread adoption of po. to address these challenges, ai-based feedback approaches that utilize large language models (llms) have been proposed to minimize human intervention @xcite . however, these approaches still rely on human-authored evaluation criteria for practical application. in the customer service domain, where providing responses that align with customer preferences and mitigate negative emotions is critical, the ambiguity of the evaluation criteria further highlights the limitations of existing methods. to overcome these challenges, this study proposes a novel methodology for generating customer preference datasets without human intervention. this methodology provides a foundation for the efficient construction and scalability of po datasets, enabling a wider adoption of po in ai customer support systems. the proposed methodology consists of the following three key steps: