Question Answering over Electronic Devices: A New Benchmark Dataset and a Multi-Task Learning based QA Framework (2021)

Abstract: Answering questions asked from instructional corpora such as E-manuals, recipe books, etc., has been far less studied than open-domain factoid context-based question answering. This can be primarily attributed to the absence of standard benchmark datasets. In this paper, we meticulously create a large amount of data connected with E-manuals and develop a suitable algorithm to exploit it. We collect E-Manual Corpus, a huge corpus of 307,957 E-manuals, and pretrain RoBERTa on this large corpus. We create various benchmark QA datasets which include question answer pairs curated by experts based upon two E-manuals, real user questions from Community Question Answering Forum pertaining to E-manuals etc. We introduce EMQAP (E-Manual Question Answering Pipeline) that answers questions pertaining to electronics devices. Built upon the pretrained RoBERTa, it harbors a supervised multi-task learning framework which efficiently performs the dual tasks of identifying the section in the E-manual where the answer can be found and the exact answer span within that section. For E-Manual annotated question-answer pairs, we show an improvement of about 40% in ROUGE-L F1 scores over most competitive baseline. We perform a detailed ablation study and establish the versatility of EMQAP across different circumstances. The code and datasets are shared at https://github.com/abhi1nandy2/EMNLP-2021-Findings , and the corresponding project website is https://sites.google.com/view/emanualqa/home .

## introduction an e-manual, or electronic manual, is a document that provides technical support to the consumers of a product by giving instructions and procedures to operate the device along with know-how of its specifications. it is often difficult to find the relevant instructions from an e-manual; hence, an automated question answering support to use the information present in the e-manual effectively would be of great help. e-manuals typically provide lengthy instructions structured in a sequential fashion explaining various uses of a device. this often poses a challenge in building a question answering system because the answer to a question may come from multiple disjointed portions within a section of the e-manual. due to the instructional nature of e-manuals, we also find that often adjacent instructions are not related to each other but may be related to a parental instruction leading to long-range dependencies in context. this, therefore, deems a domain-specific natural language understanding which may, in turn, suffer from lack of domainspecific labeled data @xcite and presence of formal syntax in the corpus @xcite . these challenges have led recent works to pre-train the state-of-the-art transformer models on unlabelled domain-specific corpora @xcite @xcite . inspired by such works, we painstakingly collect e-manual corpus: a huge corpus of 307,957 e-manuals foot_0 and pre-train the transformer-based language model, roberta_base foot_1 on the corpus (section 3.1). a question answering system needs to select the relevant section of the e-manual, which contains the answer to the given question (section retrieval (sr)) and subsequently, extract the answer from that relevant section (answer retrieval (ar)). there are currently four main types of approaches in state-of-the-art literature that utilize the sr and ar systems (1). @xcite uses a two-stage training pipeline where the sr model consists of an unsupervised information retrieval (ir) method like tf-idf or bm25, followed by an extractive ar model; (2) an end-to-end learning setup of sr cascaded by ar @xcite ; (3) single-span @xcite or multi-span @xcite answers given questions and corresponding candidate contexts as inputs and (4) a multi-task learning (mtl) framework, where sr and ar are the two underlying tasks @xcite ; @xcite performs mtl using separate sr and ar pipelines sharing feature extraction layers. the simultaneous training of sr and ar using mtl helps the model build a combined and hierarchical understanding of question answering at a global (section) and a local (sentence/token) level. however, these methods apply a span-based selection approach for extracting answers, whereas the answers to questions on e-manuals are usually non-contiguous; hence while we principally use this multi-task learning (mtl) framework, we make some customization to accommodate the peculiarity of the data. summing up, the paper makes the following contributions: (1) since no data is available for the e-manual domain, we create a huge corpus for pretraining containing 307,957 e-manuals known as the e-manual corpus. (2) since no qa dataset is available for this domain, we apply multi-pronged strategy to create a large enough corpus of question answering (qa) datasets: two datasets manually annotated by experts containing 904 and 950 questions respectively, and another collected from amazon question answering forum containing 1,028 questions and a set of 10 questionanswer pairs for 40 different devices each (section 2). (3) emqap (e-manual question answering pipeline) develops on two basic pillars -a domain-specific pre-trained roberta architecture and a multi-task learning framework. in the next section we discuss in detail the different types of data rigorously created. the system design is discussed in detail in section 3, followed by the experimental results in section 4. the experimental results emphatically establish that the performance of emqap is way superior to its nearest baseline. ## corpus and datasets in this section, we elaborate the corpus of e-manuals and the benchmark datasets we create. these datasets are used for pre-training and to test the performance of the qa algorithms. ## methodology in this section, we describe each step from the pipeline of emqap. the pipeline consists of two major steps (a). pre-training the e-manual and (b). multi-task learning framework to select the answer. however, before employing multi-task learning, the first step is to reduce the pipeline's search space and provide it with only a few candidate sections for a question. we use an unsupervised ir method that accepts a question and all sections of the e-manual as input and provides similarity scores for each question-section as output (details in suppl.) the flow of the entire emqap is depicted in fig. 2 . the steps are also presented as algorithm in suppl.. ## experiments and results to assess the efficiency of emqap, we first evaluate the performance of the unsupervised retrieval algorithm followed by the mtl framework on the datasets specifically curated in sections 2.2 -2.4. the experimental results of unsupervised algorithm is detailed in suppl. we found that the proposed algorithm tf-idf + t5 performs the best. ## conclusion in this paper, we worked on a far less studied problem of question answering from e-manuals. in order to work the subject, a pre-condition was to create benchmark datasets which we painstakingly developed. we created a large corpus from e-manuals which was used in pre-training a roberta architecture. this in turn helped in developing a domainspecific natural language understanding; the fruits of which can be observed in the huge improvement in performance over competing baselines. we believe that the e-manuals specific qa dataset is extensive and well-rounded and will help the community in various ways.