A Technical Question Answering System with Transfer Learning (2020)

Abstract: In recent years, the need for community technical question-answering sites has increased significantly. However, it is often expensive for human experts to provide timely and helpful responses on those forums. We develop TransTQA, which is a novel system that offers automatic responses by retrieving proper answers based on correctly answered similar questions in the past. TransTQA is built upon a siamese ALBERT network, which enables it to respond quickly and accurately. Furthermore, TransTQA adopts a standard deep transfer learning strategy to improve its capability of supporting multiple technical domains.

## introduction technical community (e.g., stackoverflow) is the most active and compelling open forum community for question answering (cqa) that has played an important role for technicians sharing and spreading professional knowledge @xcite . recently, many technology companies have been developing and actively maintaining their own technical support forums (e.g., ibm developer-works) where users/consumers can post technical issues to seek advice and solutions from peers and experts. technology companies are investing in training their employees with professional communication skills and technical knowledge to effectively respond to users' questions. however, it is often expensive to provide timely and helpful responses on these forums. statistics show that at least 30% questions do not hold an accepted answer on stackoverflow and askubuntu. first, the users/peers often have limited domain-specific knowledge to describe their issues or accurately answer complex questions in appropriate technical terminology. second, it is common for such forum threads to become multi-turn asynchronous interactions, such that the users/peers give additional information, forming a series of discussions, which is time-consuming for experts to track such ongoing interactions. therefore, building an intelligent system to automatically respond to new questions with useful information or a solution is in high demand. in scenarios where the automatic response does not address a user's technical issue, human experts can further intervene such discussions. a straightforward approach is to properly match potential candidate answers to the new question, since many questions have recurred, allowing for new questions to be answered using the historically accepted answers. the information retrieval (ir) and natural language processing (nlp) communities have witnessed the growing dominance of retrieval approaches based on neural networks over the last decade @xcite . recent advances of pre-training language models (e.g., bert) achieved superior performance on the retrieval question answering (qa) task @xcite . several bert based qa systems in the general domain have been deployed for real-world applications @xcite . in these approaches, bert first concatenates a qa pair as [cls, q, sep, a, sep], then passes the concatenation to the transformer network @xcite . this setup is time-consuming and resource intensive for retrieval task because of too many possible combinations. finding the pair of the highest similarity in a collection of 10,000 sentences requires 49,995,000 inference computations in bert. on a modern v100 gpu, it requires around 60 hours. furthermore, while different technical communities have different focuses, there is still substantial overlap in the abilities of intelligent systems required to answer questions across multiple technical domains. to the best of our knowledge, there is no existing system that uses transfer learning to address technical qa tasks. we develop a novel system called transtqa (in short for "transfer learning for technical question answering"). our system leverages accepted questions and answers (tagged by human users) for answering similar technical questions. the system consists of two modes: evaluation mode and usage mode. the evaluation mode demonstrates retrieved answers by four different transfer learning strategies. the usage mode displays three highest ranked answers retrieved from the database of answer candidates. we employ the latest algorithm albert @xcite with a siamese network, to make our system efficient and accurate. first, an albert has 18x fewer parameters than bertlarge with the same configuration. a siamese network has two separate albert encoders to generate question and answer embeddings, and applies a similarity measure at the top layer. the similarity measuring process can perform efficiently on modern hardware, allowing our system to be used for real-time usage. second, our system demonstrates superior accuracy compared to traditional lstm based qa systems @xcite . third, we adopt a standard deep transfer learning technique, so our system is able to work on different technical domains. experiments show our proposed system is both time efficient and memory efficient. we have provide the following links for readers to easily access our (1) demo video and (2) source code (3) website for research purpose. retrieval question answering retrieval question answering (a.k.a answer retrieval) finds the most similar answer between multiple candidate answers for a given question @xcite . recently, several researchers have proposed different deep neural models in text-based qa that compare two segments of texts and produces a similarity score. both documentlevel @xcite @xcite and sentence-level @xcite retrieval has been studied on many public datasets such as squad @xcite and nq @xcite . transfer learning for qa transfer learning studies how to transfer knowledge from a source domain to a target domain @xcite . recent advances of deep transfer learning technologies have achieved great success in various nlp tasks @xcite . several research work in this domain greatly enrich the application and technology of transfer learning on question answering from different perspectives @xcite @xcite @xcite . although transfer learning has been successfully applied to various qa applications, its applicability to technical qa has yet to be investigated. ## front end and user interface our system consists of two different modes: evaluation mode and usage mode. first, the usage mode allows users to select an existing question from the drop-down box or ask a question themselves. then, the system returns three retrieved responses with highest probabilities from the database to the user. second, the evaluation mode retrieves responses through different transfer learning settings, which can be used to compare different model performance. overall, the qa front end service was implemented in python with flask web framework. it is fully configurable and allows multiple candidate ranking services to be used at the same time. ## conclusions we developed transtqa, which is a novel system that offers automatic response by retrieving proper answers based on correctly answered similar questions. transtqa was built upon a siamese albert network, which enables it to respond to questions quickly and accurately. furthermore, transtqa adopted transfer learning to improve its performance on multiple tech domain qa.