Technical Question Answering across Tasks and Domains (2021)

Abstract: Building automatic technical support system is an important yet challenge task. Conceptually, to answer a user question on a technical forum, a human expert has to first retrieve relevant documents, and then read them carefully to identify the answer snippet. Despite huge success the researchers have achieved in coping with general domain question answering (QA), much less attentions have been paid for investigating technical QA. Specifically, existing methods suffer from several unique challenges (i) the question and answer rarely overlaps substantially and (ii) very limited data size. In this paper, we propose a novel framework of deep transfer learning to effectively address technical QA across tasks and domains. To this end, we present an adjustable joint learning approach for document retrieval and reading comprehension tasks. Our experiments on the TechQA demonstrates superior performance compared with state-of-the-art methods.

## introduction recent years have seen a surge of interests in building automatic technical support system, partially due to high cost of training and maintaining human experts and significant difficulty in providing timely responses during the peak season. huge successes have been achieved in coping with opendomain qa tasks @xcite , especially with advancement of large pre-training language models @xcite . among them, twostage retrieve-then-read framework is the mainstream way to solve open-domain qa tasks, pioneered by @xcite : a retriever component finding a document that might contain an answer from a large collection of documents, followed by a reader component finding the answer snippet in a given paragraph or a document. recently, various pre-training language models (e.g., bert) have dominated the encoder design for solving different open-domain qa tasks (karpukhin et al., 2020 @xcite . despite the tremendous successes achieved in general qa domain, technical qa have not yet been well investigated due to several unique challenges. first, technical qas are non-factoid. the question and answer can hardly overlap substantially, because the answer typically fills in missing information and actionable solutions to the question such as steps for installing a software package and configuring an application. different from factoid questions that are typically aligned with a span of text in document @xcite , semantic similarities between such non-factoid qa pairs could have a large gap as shown in fig. 1 . therefore, the retrieval module in retrieve-thenread framework might find documents that do not contain correct answers due to the semantic gap in non-factoid qas @xcite @xcite . second, compared to squad (with more than 100,000 qa pairs), technical domain datasets typically have a much smaller number of labelled qa pairs (e.g., about 1,400 in techqa), partially due to the prohibitive cost of creating labelled data. in addition, there are limited real user questions and technical support documents, especially for some new tech products and communities. since the pre-trained language models are mainly trained on general domain corpora, directly fine-tuning pre-trained language models may lead to unsatisfying performance due to the large discrepancy between source tasks (general domains) and target tasks (technical domains) @xcite . to address the aforementioned challenges, we propose a novel deep transfer learning framework that explores knowledge transfer across tasks and domains (transtd). transtd consists of two components: transt (knowledge transfer across tasks) and transd (knowledge transfer across domains). transtd jointly learns snippet prediction (reading comprehension) task and matching prediction (document retrieval) task simultaneously, applying it on both general domain qa and target domain qa. to address the first challenge of non-factoid qas, transt leverages a joint learning model that directly ranks all predicted snippets by reading each pair of query and candidate document. it optimizes matching prediction and snippet prediction in parallel. compared to two-stage retrieve-then-read methods that only read most semantically related documents, transt considers potential snippets in every candidate document. when jointly training these two tasks, snippet prediction pays attention to local correspondence and matching prediction helps understand the semantic relationship from a global perspective, allowing the multi-head attentions in bert-based encoders to jointly attend to information from different representation subspaces at different positions. besides, the weights of two training objectives can be dynamically learned to pay more attention on the more difficult task when training different data samples. to address the second challenge of learning with limited data, transd leverages a deep transfer learning model to transfer knowledge from general domain qas to technical domain qas. general domain qa dataset like squad has a much larger data size and a similar task setting (i.e., snippet prediction). though knowledge is different between two domains, by learning the ability to answer questions in general domains, the model can quickly adapt and learn efficiently when changing into a new domain, reflected in faster convergence and better performance. transfer learning helps avoid overfitting on technical qas with limited size of data. specifically, our model first applies the multi-task joint learning in general domain qas (squad), then transfers model parameters to initialize the training in the target domain qas (techqa), making knowledge transfer across domains to address data limitation. we conducted extensive experiments on the techqa dataset and utilized bert as basic models. experiments show that transtd can provide superior performance than models with no knowledge transfer and other state-of-the-art methods. transfer learning transfer learning studies how to transfer knowledge from auxiliary domains to a target domain @xcite @xcite . recent advances of deep learning technologies with transfer learning has achieved great success in a variety of nlp tasks @xcite . several research work in this domain greatly enrich the application and technology of transfer learning on question answering from different perspectives @xcite @xcite . although transfer learning has been successfully applied to various qa applications, its applicability to technical qa has yet to be investigated. in this work, we focus on leveraging transfer learning to enhance qa in tech domain. ## research problem in the technical support domain, suppose we have a set of questions q and a large collection of documents d. for each question @xmath0, we aim at finding a relevant document @xmath1, d end ) in the document d. note that the answer may not exist, and so, the relevant document may not exist, either. all predicted snippets are ranked by a specific span score calculation method, and (usually) the top-1 foot_0 answer span is chosen to answer the given question. ## proposed framework in this section, we present our proposed framework for technical qa. given a query, we first obtain 50 technotes by issuing the query to the search engine elasticsearch foot_1 . instead of using a document retriever based on semantic similarity between the query and each document, our proposed transtd jointly optimizes snippet prediction and matching prediction in a parallel style. figure 2 illustrates the design of the framework. it has a multi-task learning method to transfer knowledge across the snippet prediction (reading comprehension) and matching prediction (document retrieval) tasks. this method is further applied to pre-train the model on auxiliary domain qas foot_2 . furthermore, the weights of two training objectives are dynamically adjusted by calculating the difference between real answer snippet and predicted snippet. so, the model can focus on optimizing the more difficult task when training different data samples. lastly, our model has a novel snippet ranking function that uses snippet prediction to obtain an alignment score and linearly combines it with the matching prediction score. ## conclusion in this paper, we studied qa in the technical domain, which was not well investigated. technical qa faces two unique challenges: (i) the question and answer rarely overlaps substantially (onfactoid questions) and (ii) very limited data size. to address the challenges, we propose a novel framework of deep transfer learning to effectively address techqa across tasks and domains. to this end, we present an adjustable joint learning approach for document retrieval and reading comprehension tasks. our experiments on the techqa dataset demonstrates superior performance compared with non-transfer learning state-of-the-art methods.