Improving Conversational Abilities of Quantized Large Language Models via Direct Preference Alignment (2024)

Abstract: The rapid advancement of large language models (LLMs) has facilitated their transformation into conversational chatbots that can grasp contextual nuances and generate pertinent sentences, closely mirroring human values through advanced techniques such as instruction tuning and reinforcement learning from human feedback (RLHF). However, the computational efficiency required for LLMs, achieved through techniques like post-training quantization (PTQ), presents challenges such as token-flipping that can impair chatbot performance. In response, we propose a novel preference alignment approach, quantization-aware direct preference optimization (QDPO), that aligns quantized LLMs with their full-precision counterparts, improving conversational abilities. Evaluated on two instruction-tuned LLMs in various languages, QDPO demonstrated superior performance in improving conversational abilities compared to established PTQ and knowledge-distillation fine-tuning techniques, marking a significant step forward in the development of efficient and effective conversational LLMs.

## introduction as large language models (llms) advance in understanding the context of language and generating relevant sentences, llms are evolving into conversational chatbots that can naturally respond to a wide array of user requests @xcite @xcite . particularly noteworthy is the remarkable ability of llms to follow user instructions and align with human values, such as providing helpful and engaging responses through techniques like instruction tuning and reinforcement learning from human feedback (rlhf) @xcite @xcite ; mukherjee et @xcite . these advancements have greatly enhanced the capability to finetune pre-trained llms for various tasks and user preferences. for the effective implementation of llm-based chatbots, addressing llms' computational complexity is essential. weight load overhead, a critical bottleneck in llm deployment, has led to the development of weight quantization techniques like post-training quantization (ptq). ptq reduces storage requirements by applying quantization to the weights of trained llms, thereby decreasing the necessary bit count for weight data storage @xcite . techniques such as awq @xcite address quantizationinduced accuracy loss through methods like scaling data distribution and weight updates aimed at preserving accuracy. the effectiveness of these quantization strategies has been measured by taskdependent benchmarks to evaluate model accuracy instead of multifaceted conversational qualities. evaluating the conversational abilities of llmbased chat assistants, especially for open-ended tasks requiring alignment with human preferences, challenges traditional score-based benchmarks due to the assistants' varied capabilities. to address this, new methods have been introduced for a more objective assessment of llm chatbot performance @xcite . the "llm as a judge" approach @xcite employs advanced llms like @xcite to evaluate responsiveness in multi-turn conversations across eight conversational categories, focusing on conversational continuity and adherence to instructions. furthermore, flask @xcite offers fine-grained evaluation criteria that dissect conversational skills linguistically. yet, these methods mainly target full-precision chatbots, leaving the performance of cost-efficient quantized llm chatbots less explored. to assess quantization's effect on llm-based chatbots' conversational abilities, we qualitatively compared the responses of quantized llms with a 16-bit baseline. fig. 1 reveals that quantized models often fail to maintain engaging dialogues with repetitive phrases. we identify "tokenflipping" -a phenomenon where quantization errors skew token distribution, causing incorrect token selection -as a crucial factor for this quality degradation. traditional task-dependent evaluation metrics, such as common sense question answering (csqa) @xcite and massive multitask language understanding (mmlu) @xcite , may not fully detect these nuances. for example, as shown in fig. @xcite and (b), 16-bit and w4a16 inference exhibit similar task accuracy, but w4a16 inference produces responses that are not helpful to the user. this observation underscores the need for a new quantization approach that preserves userperceived effectiveness beyond the task-dependent benchmarks. to address the issue of token-flipping in quantized llms, we propose a novel preference alignment method that aligns quantized llms with full-precision counterparts. drawing inspiration from direct preference optimization (dpo) strategies @xcite , our approach generates preference datasets directly from the quantized llm and its full-precision counterpart to implement quantization-aware optimization for preference-reflective weight adjustments. our quantization-aware direct preference optimization (qdpo) method improves the dis-parity between the top-1 and top-2 logits of token distribution, reducing token-flipping, and fostering more relevant and consistent text output. we rigorously tested qdpo on two instruction-tuned llms, vicuna @xcite and mi:dm (kt-ai, 2023), assessing their conversational performance in both english and korean. the results, as illustrated in fig. 1(c ), demonstrate that qdpo markedly enhances conversational abilities beyond those achieved with established quantization techniques. ## qdpo: quantization-aware direct preference optimization as described in section 3, quantization significantly degrades the conversational ability of llms. to address this issue, we introduce an algorithm named quantization-aware direct preference optimization (qdpo), which aims to align the conversational abilities of quantized llms with those of llms prior to quantization. qdpo has two main contributions: 1) providing an efficient method for generating the dataset d qdpo without costly human annotations. 2) offering a theoretical foundation that ensures the automatic distinction of preferences during dataset generation. ## conclusion in this work, we address the conversational abilities of quantized llm-based chatbots. after identifying token-flipping as a crucial factor for degraded text generation quality, we propose a novel quantization-aware direct preference optimization (qdpo) method that effectively aligns quantized and full-precision llms, enhancing conversational performance. tested across multiple languages and models, qdpo outperforms traditional fine-tuning techniques, setting a new benchmark for conversational chatbot development.