K a PQA : Knowledge-Augmented Product Question-Answering (2024)

Abstract: Question-answering for domain-specific applications has recently attracted much interest due to the latest advancements in large language models (LLMs). However, accurately assessing the performance of these applications remains a challenge, mainly due to the lack of suitable benchmarks that effectively simulate real-world scenarios. To address this challenge, we introduce two product question-answering (QA) datasets focused on Adobe Acrobat and Photoshop products to help evaluate the performance of existing models on domain-specific product QA tasks. Additionally, we propose a novel knowledge-driven RAG-QA framework to enhance the performance of the models in the product QA task. Our experiments demonstrated that inducing domain knowledge through query reformulation allowed for increased retrieval and generative performance when compared to standard RAG-QA methods. This improvement, however, is slight, and thus illustrates the challenge posed by the datasets introduced.

## introduction the advancements in large language models (llms) led to exponential growth in domainspecific applications. question answering has emerged as one of the prominent domain-specific applications. as the demand for accurate and reliable qa systems increases, generic rag-qa approaches often struggle to deliver satisfactory results within the specialized domains. this challenge has spurred active exploration in this area, with researchers employing various novel methodologies @xcite @xcite to improve qa systems. additionally, training and evaluating these systems rigorously remains crucial. this trend underscores the critical need for domain-specific qa datasets to facilitate the training and evaluation of such systems. notably, while efforts have been directed towards releasing datasets across prominent and expansive domains such as medicine @xcite , finance @xcite , and legal @xcite , there remains an apparent scarcity of such datasets in the area of software products. to address this gap, our work investigates such industry-specific qa datasets, namely the adobe helpx datasets, and releases them for others to benchmark against and further improve their qa systems. these datasets comprise of user queries and their corresponding answers pertaining to adobe products, specifically acrobat and photoshop. by providing these benchmark datasets, we aim to offer valuable resources for assessing the performance of domain-specific rag-qa systems. the datasets will be released after obtaining relevant permissions from adobe. furthermore, we introduce a novel llm-based knowledge-driven rag-qa framework designed to seamlessly accommodate domain knowledge into rag-qa systems. this framework leverages comprehensive knowledge bases for query expansion, thereby enhancing both retrieval and generation in domain-specific qa tasks. through extensive experimentation, we've determined that performing accurate retrieval over these datasets poses a unique challenge. even introducing this concept of query augmentation using knowledge directly from the corpora only helped improve the model so much. this illustrates how these datasets are challenging ones -as even complex frameworks such as the one we've proposed, could only result in so much improvement. by contributing these datasets and proposing an innovative framework, we aim to advance llm technology and its application in domain-specific qa tasks, ultimately improving user experiences and operational efficiency across various industries. ## data analysis and statistics the adobe acrobat and the photoshop datasets contain questions, answers, and corresponding source web page urls. all questions in this dataset are how-to type asking steps to perform operations such as changing text font, editing text in a pdf, and creating certificate-based signatures. the gold answers to these questions provide procedural steps to accomplish these operations and the urls of the web pages allow for independent verification of the answer. nificant challenges due to the critical nature of every step involved. it's essential to emphasize that for an answer to the "how to" query to be entirely accurate, each step must be precise, and all steps must be in the correct sequence. even a minor mistake in the explanation of a step or its order can invalidate the entire utility of the answer. moreover, the average number of steps per response for a query in the adobe acrobat dataset is 4.71, indicating that most queries requires multi-step solutions. below is an example highlighting the necessity for accurate and detailed instructions within every step of the response. sample q&a from the dataset q: how to insert images into a pdf? a: 1. open the pdf in acrobat and go to the edit menu. 2. select image from the add content submenu. 3. in the dialog box, choose the image file you want to insert. 4. select the location where you want to insert the image or use the drag option to resize it as you insert. 5. a copy of the image file will appear on the page with the same resolution as the original file. adding to the complexity, a substantial portion of the acrobat dataset, over 50%, is comprised of implicit questions, defined by their brevity and conversational tone, lacking clear indications of user intent, as shown in to assess the generalizability of our proposed qa framework, we curated an additional synthetic dataset focused around adobe photoshop product, closely resembling adobe acrobat in terms of question type, question and answer lengths, as shown in moreover, both datasets serve as evaluation benchmarks, representing real-world user queries (both implicit and ambiguous) in adobe acrobat and controlled questions in the synthetic adobe photoshop dataset. they offer question-answer pairs for diverse scenarios, making them valuable resources for research in the software product domain. ## methodology as summarized by @xcite @xcite , in a standard rag-qa process, upon receiving an input query, the retriever identifies and retrieves pertinent data sources, which are subsequently utilized by the response generator to enrich the overall generation process. to enable adobe domain-specific qa, we add an initial query reformulation stage which enhances the user query using knowledge base triples. query reformulation or rewriting @xcite encompasses a set of techniques to transform a user's original query into one that's better aligned with the user's intent, thereby enhancing the retrieval outcomes. our proposed query reformulation pipeline consists of multiple steps as shown in fig. 1 . it starts with the generation of knowledge base triples using the triples generator. next, all matching triples to the user query are retrieved using the triple retriever, classified based on their relevance to the original query using the relevance classifier and finally reformulated using the query enhancer. refer to appendix e for the llm prompts used at various stages. given below is a detailed outline of the different components in our pipeline: ## experiments we conducted a variety of experiments (as shown in tables 3 and 4 ) on the adobe datasets, where the main datasets formed the corpus of texts to be retrieved from, a selected few of which would be passed in as part of the prompt to the llm; and the gold set was used for measuring performance. they ranged from using different retrievers to incorporating multiple components and techniques into the rag-qa pipeline to achieve a performance gain. throughout the processes we used gpt-3.5 0301 and gpt-4 0314 from azure openai. ## conclusion in this paper, we introduce two qa datasets focused on adobe acrobat and photoshop products, that serve as benchmarks to evaluate an rag-qa framework tailored for domain-specific procedural long-form qa datasets. we also introduce a novel and detailed pipeline with components directed to improve the retrieval and generation metrics in such qa datasets. it equips llms with domain-specific knowledge through the use of knowledge base triples to bridge the gap between general rag-qa methods and industry demands. through various experiments we showcase the effectiveness and limitations of our proposed pipeline in standard metrics.