RAG 4 ITO ps: A Supervised Fine-Tunable and Comprehensive RAG Framework for IT Operations and Maintenance (2024)

Abstract: With the ever-increasing demands on Question Answering (QA) systems for IT operations and maintenance, an efficient and supervised fine-tunable framework is necessary to ensure the data security, private deployment and continuous upgrading. Although Large Language Models (LLMs) have notably improved the open-domain QA’s performance, how to efficiently handle enterprise-exclusive corpora and build domain-specific QA systems are still less-studied for industrial applications. In this paper, we propose a general and comprehensive framework based on Retrieval Augmented Generation (RAG) and facilitate the whole business process of establishing QA systems for IT operations and maintenance. In accordance with the prevailing RAG method, our proposed framework, named with RAG4ITOps, composes of two major stages: (1) Models Fine-tuning & Data Vectorization, and (2) Online QA System Process. At the Stage 1, we leverage a contrastive learning method with two negative sampling strategies to fine-tune the embedding model, and design the instruction templates to fine-tune the LLM with a Retrieval Augmented Fine-Tuning method. At the Stage 2, an efficient process of QA system is built for serving. We collect enterprise-exclusive corpora from the domain of cloud computing, and the extensive experiments show that our method achieves superior results than counterparts on two kinds of QA tasks. Our experiment also provide a case for applying the RAG4ITOps to real-world enterprise-level applications.

## introduction in recent years, the field of it operations and maintenance has become increasingly significant due to the rapid expansion of massive data and complex it systems, such as in cloud computing and telecommunications @xcite . efficient it figure 1 : two examples of typical and important qa scenarios for it operations and maintenance. the words with underlines are domain-specific terminologies, and the [ * ] represents enterprise-exclusive terms, e.g. status codes or service names. traditionally, to operate and maintain those systems, it highly depends on it operators' personal experience, while often leading to difficulties in incident management, problem resolution, and maintaining service quality @xcite . later with the advancements of qa techniques, some qa systems are developed, and it operators can leverage them to retrieve useful information and make a plan on troubleshooting efficiently in a natural-language human-machine interacting manner @xcite @xcite . as shown in figure 1 , the two typical and important qa tasks are knowledge acquisition and troubleshooting @xcite . the former is usually for junior it operators to promote their experience, while the latter is for senior ones to obtain guidance on resolving difficult software and hardware faults during their daily work. therefore, qa systems have become greatly important in contemporary it operations and maintenance. to build the qa systems for it operations and maintenance, we observed numerous examples such as those in figure 1 . some characteristics and challenges can be summarized as follows: first, the qa utterances contain many technical terminologies (e.g., status codes, service names and other underlined words/[ * ] as illustrated in figure 1 ), and their semantics are exclusive to a specific domain or even an enterprise. therefore, the enterpriseexclusive semantics should be thoroughly modeled. second, in terms of data forms, a vast amount of enterprise-exclusive documents, guides and manuals should be processed and modeled into a uniformed format to support the continual upgrading of qa systems. third, the difficulties of various qa tasks are distinct. for example, qa for knowledge acquisition requires a system only to answer the question with straightforward information, while the qa for troubleshooting demands a much longer answer that involves referring to multiple resources. to this end, all the above challenges lead to a complex problem of how to build an efficient framework that addresses exclusive data and specific qa tasks. intuitively, some open-domain qa systems that are trained on massive public corpora can be leveraged to further fine-tune on domain-specific corpora and tasks, especially with the recent breakthroughs of llms @xcite @xcite such as bert, llama-3 @xcite , qwen, and chatglm3 @xcite . however, those llms are still too general to be adaptive for distinct qa tasks, or efficiently support the continuous data or/and system upgrading in real-world applications. to address the above-mentioned problems, in this paper, we leverage the idea of retrieval augmented generation (rag) which can strengthen llms @xcite , and propose a comprehensive rag framework specific for the domain of it operations and maintenance, named with rag4itops. in accordance with the prevailing rag methodology, our framework composes of two stages: (1) models fine-tuning & data vectorization, and (2) online qa system process. the framework features include a data pipeline for efficiently processing multi-source and multi-form enterprise-exclusive corpora, a domain knowledge augmented embedding model for modeling exclusive semantics, and a supervised fine-tuned llm which can support grounded qa tasks. more specifically, to build a qa system by using the proposed rag4itops, firstly the enterpriseexclusive corpora should be collected and preprocessed in advance. after several automatic steps of data cleaning, chunking and distillation, we can obtain a high-quality set of text chunks and two datasets with annotations for fine-tuning the embedding model and the llm respectively. to better distinguish the qa tasks and model enterpriseexclusive semantics, we fine-tune the embedding model by adopting the contrastive learning @xcite with homogeneous in-batch negative sampling (his) @xcite and auxiliary hard negative sampling (ahns) strategies. then the set of text chunks are embedded into vectors by using the fine-tuned embedding model, and stored in the vector database. as to the llm, we also fine-tune it with qa pairs by adopting a retrieval augmented fine-tuning method. the details can be found in the methodology section. in summary, the proposed rag4itops is supervised fine-tunable on both exclusive data and grounded qa tasks. note that due to the nature of rag mechanism, the vector database can be easily updated by inserting new data, instead of frequently refine-tuning the llm. and the llm can dynamically incorporate retrieved top-k contents from the database, which are always latest and most relevant. in this way, the requirement of continuous data or/and system upgrading is fulfilled with a low cost. we collect enterprise-exclusive corpora from the domain of cloud computing, and the experiment results show that our framework can achieve superior performance than counterparts on both qa tasks. our experiment also provides a case of how to apply the rag4itops into real-world enterprise-level applications. the contributions of this paper include: large language models. recent llms have demonstrated significant advancements in opendomain qa tasks @xcite . as to those closed-source models, like gpt-4, claude and gemini, they cannot answer domain-specific or even enterprise-exclusive questions well since they do not trained on any private documents. the other thing is that those open-source models, like llama-3 @xcite , qwen, and chatglm3 @xcite , can be directly fine-tuned on specific corpora and then provide qa services. however there are two major concerns. firstly, llms often tend to generate hallucinated information @xcite , which is unbearable in industrial area. secondly, faced with the everincreasing massive data, the qa systems based on llms have to be refine-tuned frequently, leading to a much high expense. therefore, intuitively, rag frameworks can remove the concerns and strengthen the llms-based qa systems. a recent effort to develop domain-specific llms, such as owl @xcite , have shown promise. but it still struggles to be adaptive for grounded qa scenarios in real-world industrial it operations. ## methodology to facilitate the business process of data modeling and model fine-tuning of qa systems for it operations and maintenance, we present the rag4itops framework and introduce its details in this section. as shown in figure 2 , the framework includes two stages. one is for offline model fine-tuning and data vectorization, and the other is about the online qa system process based on rag mechanism. ## conclusion in this paper, we introduce rag4itops, a comprehensive framework for qa systems tailored for it operations and maintenance. initially, we developed a dataset construction pipeline, incorporating data cleaning, chunking, and distillation of enterprise-exclusive corpora. additionally, we finetuned an embedding model and enhanced its retrieval performance using homogeneous in-batch negative sampling and auxiliary hard negative sampling strategies. furthermore, we leveraged and fine-tuned a llm enhancing its capabilities for domain-specific qa tasks with continue pre-training and retrieval augmented fine-tuning. we evaluated our framework through a series of experiments, designed to assess its performance on distinct qa tasks with different difficulties, demonstrating the effectiveness of our approach in the domain of it operations and maintenance. lianmin zheng, wei-lin chiang, ying sheng, siyuan zhuang, zhanghao wu, yonghao zhuang, zi lin, zhuohan li, dacheng li, eric xing, et al. 2024. judging llm-as-a-judge with mt-bench and chatbot arena. advances in neural information processing systems, 36.