Empower Large Language Model to Perform Better on Industrial Domain-Specific Question Answering (2023)

Abstract: Large Language Model (LLM) has gained popularity and achieved remarkable results in open-domain tasks, but its performance in real industrial domain-specific scenarios is average due to its lack of specific domain knowledge. This issue has attracted widespread attention, but there are few relevant benchmarks available. In this paper, we provide a benchmark Question Answering (QA) dataset named MSQA, centered around Microsoft products and IT technical problems encountered by customers. This dataset contains industry cloud-specific QA knowledge, an area not extensively covered in general LLMs, making it well-suited for evaluating methods aiming to enhance LLMs’ domain-specific capabilities. In addition, we propose a new model interaction paradigm that can empower LLM to achieve better performance on domain-specific tasks where it is not proficient. Extensive experiments demonstrate that the approach following our method outperforms the commonly used LLM with retrieval methods. We make our source code and sample data available at: https://aka.ms/Microsoft_QA.

## introduction recent advancements in large language models (llms), including openai's gpt-3.5 @xcite , gpt-4 (openai, 2023), google's palm @xcite , and other benchmark models @xcite @xcite , have demonstrated impressive performance across various natural language processing (nlp) tasks. these models are pretrained on extensive data, which imbues them with remarkable language understanding and generation capabilities @xcite . however, when it comes to domain-specific problems, llms exhibit limited performance due to their insufficient pretraining on domain knowledge, where the overwhelming * work done during the internship at microsoft. presence of domain-general data causes them to prioritize common knowledge, leading to a potential oversight of crucial domain-specific information @xcite @xcite . fine-tuning and maintaining llms to incorporate domain-specific knowledge can be expensive for most companies and researchers. moreover, the availability of domain-specific data is often restricted and confidential, introducing risks of potential data leakage during fine-tuning of llms @xcite . the existing works primarily focus on enhancing the performance of llms in specific domains by employing retrieval-based methods @xcite @xcite or external modules @xcite to extract domain-specific knowledge. however, these approaches suffer from certain limitations. firstly, retrieval-based methods face challenges in handling complex queries as they may not retrieve all the necessary domain-specific context, leading to incomplete information. additionally, retrieved chunks can suffer from the issue of "quote out of context" where the retrieved information may not fully capture the intended meaning @xcite . as the example shown in figure 1 , retrieved chunks contain keywords or spans of the question, but they do not understand the question, resulting in generating inaccurate answers. secondly, due to the length limitation of prompts, it becomes challenging to incorporate all retrieved data for in-context learning. this poses a constraint on these methods in leveraging domain-specific knowledge. humans integrate domain-specific knowledge with domain-general knowledge through interactions @xcite @xcite . for example, penner et al. @xcite conducted an experiment where children inferred factors affecting the sinking rates of objects. initially, children believed weight alone determined sinking, but the experiment helped ## msqa dataset creation current public q&a forums, such as quora, reddit, stack overflow, contain responses to a variety of open-ended questions. however, there are limited q&a forums dedicated to specific domains that have a large number of active users. in light of this, we chose to focus on the publicly available microsoft q&a forum 3 for our dataset creation, primarily due to its extensive collection of available questions and corresponding answers. these domain-specific qas cover a wide range of microsoft technologies and products, such as azure and microsoft 365. additionally, microsoft offers publicly available and well-documented documentation, which serves as a valuable external resource for extracting domain-specific knowledge. we make our msqa dataset openly accessible to the nlp community. we hope this resource could facilitate the exploration of llm's capabilities in handling industrial domain-specific questions. ## methodology the model interaction paradigm (shown in figure 2 ) involves two key steps: (a) obtaining a domain-specific model that incorporates aligned knowledge, (b) providing the generated domainspecific knowledge to llms, enabling them to generate answers enriched with domain knowledge. in the first step, we pre-train small language models foot_2 using the publicly available azure documentation foot_3 . this documentation serves as a comprehensive knowledge base for microsoft azure, containing cloud-domain knowledge of microsoft's cloud products. note that microsoft maintains extensive documentation covering various product offerings. however, we specifically focus on azure documentation as it aligns with the prevalent tags related to azure found in the q&a forum, which captures the most frequently asked questions (shown in after completing the pre-training phase, we imbue the small language models with domainspecific knowledge from the azure knowledge base. we then adapt the model to the lfqa task through instruction tuning @xcite , allowing it to specialize and become more accurate in the qa task. to facilitate instruction tuning, we construct instructions from the training set of the msqa dataset. each instruction consists of a threeelement tuple, including an instruction prompt, an input query or statement, and a corresponding response. the instruction template is shown in instruction: please answer the following questions concerning {tags}. ## results as suggested in @xcite and our experiments, the lexical-overlap-based metrics are not an informative way to evaluate the quality of llm-generated answers due to their poor correlation with grounded human-written answers. as shown in llm+dpr performs better than llm+bm25, as indicated by the previous analysis. hence, we select llm+dpr as the representative data-retrieval method for both llm-based metric evaluation and human evaluation to optimize resources and reduce human efforts. in the llm-based metric evalua- the results align with the llm-based metric, highlighting the significant performance of our method and the potential of using the llm-based metric as an automation evaluation. we present case studies in appendix i to give a comprehensive comparison of different methods. the retrieved-based methods tend to provide scattered and often "quote out of context" chunks. in contrast, the domain knowledge from our method offers more concise and relevant information, with a significantly shorter length compared with the retrieved chunks. ## conclusion in this paper, we deal with the challenge of empowering llms with domain-specific knowledge, enabling them to accurately answer questions in industrial scenarios. due to the limited availability of relevant benchmarks, we introduce the msqa dataset, tailored for cloud domain qa. our novel model interaction paradigm effectively equips llms with domain-specific knowledge, bridging the gap between general models and industry demands. experiments demonstrate and highlight the effectiveness of our proposed paradigm in standard and newly proposed metrics.