Improving Knowledge Production Efficiency With Question Answering on Conversation (2023)

Abstract: Through an online customer service application, we have collected many conversations between customer service agents and customers. Building a knowledge production system can help reduce the labor cost of maintaining the FAQ database for the customer service chatbot, whose core module is question answering (QA) on these conversations. However, most existing researches focus on document-based QA tasks, and there is a lack of researches on conversation-based QA and related datasets, especially in Chinese language. The challenges of conversation-based QA include: 1) answers may be scattered among multiple dialogue turns; 2) understanding complex dialogue contexts is more complicated than documents. To address these challenges, we propose a multi-span extraction model on this task and introduce continual pre-training and multi-task learning schemes to further improve model performance. To validate our approach, we construct two Chinese datasets using dialogues as the knowledge source, namely cs-qaconv and kd-qaconv, respectively. Experimental results demonstrate that the proposed model outperforms the baseline on both datasets. The online application also verifies the effectiveness of our method. The dataset kd-qaconv will be released publicly for research purposes.

## introduction with the rapid advance of natural language processing (nlp), customer service chatbots have been widely applied in industries, as they can significantly reduce the cost of human customer service. retrieval-based question answering model often plays an essential role in a chatbot system. however, building and maintaining a high-quality frequently asked question (faq) database is laborintensive, which relies on human experts to produce the answers. in alipay's online customer ser-vice system, there are many dialogues between customers and service agents collected daily, which contain customer questions and corresponding answers. to utilize these data, we design a knowledge production system, as depicted in figure 1 , to improve the efficiency of building the faq database. our system extracts appropriate answers from the retrieved conversations for user questions that the chatbot cannot answer due to lacking relevant qa pairs in the faq database. after updating the faq database with produced qa pairs, the chatbot can answer the user questions correctly. the core of our system is the qa on conversations module, which extracts the answer from the candidate dialogues for a given question. ## dataset the data collection pipeline (shown in figure 2 ) includes three stages as follows: ## conclusion we design a knowledge production system including qa on conversations to help mine answers from human customer service dialogues. based on the span-based extraction model, we add a multi-span extraction module trained with multi-task learning and continual pre-training schemes to extract incontiguous answers from conversational contexts. experimental results show our approach outperforms the baseline models on both ant-qaconv and kd-qaconv datasets, the latter of which will be publicly released. finally, the proposed method has been deployed to support the alipay's customer service chatbot system, which significantly saves the time cost of human operators' producing new qa pairs.