Knowledgeable Preference Alignment for LLM s in Domain-specific Question Answering (2024)

Abstract: Deploying large language models (LLMs) to real scenarios for domain-specific question answering (QA) is a key thrust for LLM applications, which poses numerous challenges, especially in ensuring that responses are both accommodating to user requirements and appropriately leveraging domain-specific knowledge bases. They are the two major difficulties for LLM application as vanilla fine-tuning falls short of addressing. Combining these requirements, we conceive of them as the requirement for the model’s preference to be harmoniously aligned with humans’. Thus, we introduce Knowledgeable Preference AlignmenT (KnowPAT), which constructs two kinds of preference sets to tackle the two issues. Besides, we design a new alignment objective to align the LLM preference with different human preferences uniformly, aiming to optimize LLM performance in real-world, domain-specific QA settings. Adequate experiments and comprehensive comparisons with 15 baseline methods illustrate that our KnowPAT is a superior pipeline for real-scenario domain-specific QA with LLMs.

## introduction in contemporary digital commerce platforms, the deployment of automated and intelligent questionanswering (qa) services is a pivotal task to augment service quality. these services are designed to furnish answers to domain-specific customer queries. building such a domain-specific qa system, while highly sought after, remains a daunting challenge in practical scenarios. domain-specific qa necessitates a comprehensive understanding of a specific domain to answer specialized questions. however, traditional deep learning models (devlin et al., 2019; raffel et al., * equal contribution. ## problem setting in this section, we will first introduce our problem scenario and basic notations. our overall target is to fine-tune a llm m with our qa datasets @xmath0, a i ) | @xmath1, 2, . . . , n } where q i , a i represent a question and answer pair. the questions in the dataset are all about common usage issues with our cloud products while the questions and answers are manually collected and labeled, which are gloden answers with decent and knowledgeable responses. for vanilla fine-tuning (vft), we first wrap the qa pair with a prompt template i and the model m is autogressively @xcite optimized as: however, such a vft approach can not achieve pretty good results for the domain-specific qa. on the one hand, applications in real scenarios should be user-friendly, otherwise, they will not bring commercial value. thus, the text style of the generated response should be more acceptable for users. on the other hand, the knowledge retrieval process is unsupervised and the effectiveness of the retrieved knowledge is hard to guarantee, which means that the model m needs to acquire the ability to judge and selectively utilize the knowledge triples. therefore, we should improve the basic vft to solve these two problems. actually, both of these problems can be summarised as model preference. the llm m has its style preference to generate texts and its knowledge preference to selectively utilize the retrieved knowledge. for the model to be practically applicable, the model preference should align with human preference, aiming to generate high-quality answers that humans prefer. preference alignment (pa) is an important topic for llms. to apply pa during llm fine-tuning, we sample a preference set @xmath2, b 2 , . . . , b l } with l different answers for each qa pair (q, a). we denote r i as the preference score of each answer b i where higher r i represents that humans prefer this answer. during training, we will define another objective l align to align the model m with the preference set p, aiming to increase the probability of a human preferred answer appearing and simultaneously decrease the probability of an unpreferred answer. the human preference of each answer is the preference score r. the overall training objective then becomes @xmath3.with such a multi-task objective, the llm is fine-tuned to fit the golden answers while avoiding unpreferred results. the next question is how to generate a preference set to reflect both the style and knowledge preference. ## our knowpat pipeline in this section, we will present our pipeline of knowledgeable preference alignment (knowpat), which consists of three key parts: unsupervised knowledge retrieve, knowledgeable preference set construction, fine-tuning, and training. figure 2 demonstrates an intuitive view of the three parts in our pipeline design. ## experiments and analysis in this section, we present the detailed experimental settings and analyze the experiment results to investigate the following four research questions: ## conlusion in this paper, we introduce a novel framework, knowledgeable preference alignment (knowpat), for domain-specific qa tasks in cloud product services, leveraging llms and kgs in a practical application setting. our approach constructs a knowledgeable preference set by retrieving and utilizing knowledge triples to generate answers with different quantities. a new alignment objective is designed to unleash the power of the preference set. comprehensive experiments demonstrate that our method surpasses existing solutions for this realworld challenge. looking ahead, we aim to apply knowpat to more real scenarios such as enterpriseclass services and further investigate the potential of kg-enhanced llm application in the future. infrastructure of internet industry to provide exact factual knowledge, which is widely leveraged in e-commerce @xcite , telecom fault analysis @xcite , health care @xcite and so on. it is a popular topic to utilize kgs in real industry applications. in our scenario, we construct a domain-specific kg for cloud service products to benefit our question answering (qa) task. qa stands as a cornerstone in nlp, aiming at equipping machines with the capability to autonomously respond to human queries @xcite . qa tasks can take on various forms. some require the selection from multiple choices, as seen in certain knowledge base qa (kbqa) @xcite @xcite and visual question answering (vqa) @xcite @xcite . conversely, tasks like open-domain qa often challenge systems to directly produce textual responses without a set answer pool @xcite . in the last few years, fine-tuning pre-trained language models has been a leading approach for qa tasks. models like bert @xcite and t5 @xcite have previously achieved notable performance when adapted with question-answer pairs. we hold that qa doesn't just remain an academic pursuit; it acts as a bridge, facilitating the adoption of ai technologies in real-world applications. numerous industrial efforts have been directed toward developing domain-specific qa systems to meet the needs of their users @xcite . such systems often rely on domainspecific knowledge bases, like knowledge graphs (kgs), to provide relevant information for the posed questions. our current investigation aligns with this trend, focusing on a domain-specific qa scenario for cloud service products. moreover, our approach diverges from these recent kg-based qa systems @xcite @xcite that utilize prompts for dialog with (large) language models to facilitate path reasoning and refine the scope of kg retrieval. we propose an innovative knowledgeable preference alignment framework that enhances kgaware qa with the knowledge preference.