Learning LLM Preference over Intra-Dialogue Pairs: A Framework for Utterance-level Understandings (2025)

Abstract: Large language models (LLMs) have demonstrated remarkable capabilities in handling complex dialogue tasks without requiring use case-specific fine-tuning. However, analyzing live dialogues in real-time necessitates low-latency processing systems, making it impractical to deploy models with billions of parameters due to latency constraints. As a result, practitioners often prefer smaller models with millions of parameters, trained on high-quality, human-annotated datasets. Yet, curating such datasets is both time-consuming and costly. Consequently, there is a growing need to combine the scalability of LLM-generated labels with the precision of human annotations, enabling fine-tuned smaller models to achieve both higher speed and accuracy comparable to larger models. In this paper, we introduce a simple yet effective framework to address this challenge. Our approach is specifically designed for per-utterance classification problems, which encompass tasks such as intent detection, dialogue state tracking, and more. To mitigate the impact of labeling errors from LLMs – the primary source of inaccuracies in student models – we propose a noise-reduced preference learning loss. Experimental results demonstrate that our method significantly improves accuracy across utterance-level dialogue tasks, including sentiment detection (over 2%), dialogue act classification (over 1.5%), etc.

## introduction maintaining high annotation quality, scaling the size of labeled datasets, and managing annotation budgets are three critical yet often conflicting objectives in deploying real-world ml applications. a widely adopted paradigm involves a two-stage process: unsupervised pretraining followed by supervised fine-tuning (e.g., @xcite @xcite . this approach effectively reduces the size of the labeled dataset required because, during the pretraining phase, models learn to generate universal embeddings across various modalities. consequently, such pretrained models are often straightforward to adapt to downstream tasks. in dialogue understanding, moving beyond bert-like models is essential, as dialogues possess unique characteristics compared to the bert pretraining corpus (which primarily consists of books and web pages). these differences arise from several factors: first, dialogues involve spoken language exchanges between two or more individuals and are often structured differently, with one line per speaker. this format reduces the effectiveness of tasks such as masked token prediction and next-sentence prediction. second, the vocabulary in daily dialogues tends to be informal. finally, dialogues are frequently transcribed from voice recordings, introducing asr errors and background noise. these distinctive properties have inspired research into developing specialized unsupervised pretraining algorithms for dialogue data @xcite @xcite . benchmark evaluations on common dialogue tasks -such as intent detection, next-utterance prediction, summarization, dialogue act classification, and dialogue state tracking -demonstrate the advantages of dialogue-optimized models. these models generally adhere to the classical bert framework, pretraining on largescale unsupervised dialogue datasets with dialoguespecific loss functions, including random mask filling, utterance swapping, and contrastive learning. however, it remains unclear whether such pretrained embedding models generalize effectively to specific downstream tasks. to address this challenge, we require direct supervision signals that are closely aligned with downstream tasks. this motivates the use of in-struction fine-tuned llms as phase-2 supervision signals, while retaining traditional unsupervised pretraining as phase-1. however, simply employing llms as data labelers and fine-tuning a student model using traditional cross-entropy loss proves suboptimal. the accuracy of llm-generated labels can be unpredictable, influenced by factors such as the quality of the llm, the prompting strategy, and the inherent difficulty of the dialogue task. consequently, the knowledge transferred from the llm to the student model often deviates from the intended objective. this paper proposes an alternative approach based on preference learning, where pairs of chunks sampled from the same dialogue session (intra-session pairs) are labeled by ensembled llms. under reasonable assumption on llm labeling errors, our method outperforms traditional training algorithms in both data efficiency and generalizability. ## experiments datasets. we benchmark our method on three important tasks in task-oriented dialogues (tod): intent/sentiment-detection, dialogue act classification, and dialogue state tracking. we benchmark intent/sentiment detection on meld @xcite and silicone @xcite ; benchmark dialogue act classification on daily-dialog @xcite , mrda @xcite , bt-oasis @xcite and dyda_da @xcite ; benchmark dialogue state tracking on sgd @xcite and multiwoz-2.2 @xcite . we put statistics and other details of datasets in appendix a. baselines. we want to see how the accuracy change after plugging our workflow into some strong models. we select following baselines accordingly: • claude3-sonnet: we pick this model as a strong baseline for measuring llm annotator performance.