The T ech QA Dataset (2020)

Abstract: We introduce TECHQA, a domain-adaptation question answering dataset for the technical support domain. The TECHQA corpus highlights two real-world issues from the automated customer support domain. First, it contains actual questions posed by users on a technical forum, rather than questions generated specifically for a competition or a task. Second, it has a real-world size – 600 training, 310 dev, and 490 evaluation question/answer pairs – thus reflecting the cost of creating large labeled datasets with actual data. Hence, TECHQA is meant to stimulate research in domain adaptation rather than as a resource to build QA systems from scratch. TECHQA was obtained by crawling the IBMDeveloper and DeveloperWorks forums for questions with accepted answers provided in an IBM Technote—a technical document that addresses a specific technical issue. We also release a collection of the 801,998 Technotes available on the web as of April 4, 2019 as a companion resource that can be used to learn representations of the IT domain language.

## introduction there is a tension between the development of novel capabilities in the early phases of the technology lifecycle, using unlimited data and compute power, and the later development of practical solutions as that technology matures. the challenges of creating practical solutions are twofold: developing robust, efficient algorithms and curating appropriate training data. here we describe the curation and public release of a dataset intended to further those algorithmic advances. the application domain is it support, a notable component of the trillion-dollar it services industry foot_0 . we created a dataset using publicly available data: questions from technical forums and answers from technical documents, all in english. we manually selected question-answer pairs that are appropriate for machine reading comprehension techniques, and reserved questions where the answer is distributed across multiple separate spans or documents, and those that require reasoning or substantial real world knowledge for future datasets. we release 600 questions for training purposes, of which 150 are not answerable from the provided documents, as well as 160 answerable and 150 non-answerable questions as development set. the blind test set contains 490 questions with similar answerable/nonanswerable statistics to the development set. the purpose of the techqa dataset is to stimulate transfer learning research from popular question-answering scenarios-driven by largescale open-domain datasets with short questions and answers-to a use case with involved questions and often long answers. we expect that simple approaches based on tuning models trained on generic datasets will perform poorly on techqa, and that systems that are successful at the task embody algorithmic advances and novel approaches. we are hosting a leaderboard for the techqa dataset at ibm.biz/tech qa where the datatraining and development sets, as well as a collection of more than 800, 000 technotes published on the internet-is available subject to registration. to maintain the integrity of the test set, the site provides the tools for authors evaluate their system on cloud infrastructure. the rest of the paper is organized as follows. question: title: netcool/impact 7.1.0: the statechange value being used by the omnibuseventreader is too high the value being used is a date and time in the future and as such is preventing the eventreader from capturing the current events. the simplest solution is to manually reset the eventreader statechange value via the gui. stop the eventreader, open it for edit, click the "clear state" button, exit the editor and restart the eventreader. ## related work recent notable datasets for machine reading comprehension (henceforth, mrc) include squad 1.1 @xcite , squad 2.0 @xcite , narra-tiveqa @xcite and hotpotqa. a common problem of the earlier mrc datasets is observation bias: annotators first read a paragraph and then wrote appropriate questions and answers, which, as a result, have substantial lexical overlap with the paragraph. also, systems trained on squad 1.1 could be easily fooled by the insertion of distractor sentences that should not change the answer, as shown in @xcite . based on these considerations, squad 2.0 added "unanswerable" questions. however, large pretrained language models @xcite were able to achieve super-human performance in less than a year on squad 2.0 as well; this suggests that the evidence needed to correctly identify unanswerable questions also are present as specific patterns in the paragraphs. recently, the nq dataset has been introduced which overcomes the above problems and constitutes a much harder and realistic benchmark. the questions came from a commercial search engine and were asked by humans who had actual information needs. the answers were manually extracted from a wikipedia page which the user may have selected among the search results. hotpotqa is a recent multi-hop questionanswering dataset (i.e., based on multi-step inference) where questions require reasoning over text from multiple wikipedia pages. systems must both produce answers and extract passages that contain supporting evidence. all of the above datasets are said to be "opendomain", as the corpus is wikipedia. there are also datasets for specialized domains. the biomedical qa dataset @xcite contained 29 development questions (arguably too few for training an automated system) and 282 test questions, divided into four categories-'yes/no', factoid, list, and summary. insuranceqa (ins), a dataset for the insurance industry, is a corpus for intent detection, rather than for mrc. our dataset, techqa, consists of questions posed in a technical forum by technical users who questions count total retrieved 276,968 with accepted answers 57,990 with link to technote in accepted answer 15,918 ## techqa dataset collection the questions for the techqa dataset were posed by real users on public forums maintained and hosted by ibm at the developer.ibm.com answers 2 and ibm developerworks 3 sites. the questions are related to products running in environments supported by ibm and mostly fall into three categories: i) generic requests for information; ii) requests for information on how to perform specific operations; iii) questions about causes of and solutions to observed problems. the questions are very specific: when describing an issue, the writer typically provides the versions of the affected software products, a description of the operations that yield the error, information about the error including portions of stack traces, and recent changes to the computing environment, such as upgrades, that might have bearing on the problem. questions have a title and a body. the title is often an integral part of the question and therefore we include both title and body of the question in techqa. as shown in about 6% of the accepted answers contain links to one or more technotes, documents written and maintained by ibm support personnel that contain information about common questions asked by customers, product upgrade information, and official solutions to well-scoped problems. technotes follow templates: for example, a troubleshooting technote has an informative title, a description of the problem, an explanation of the cause, the products, versions, and configurations affected, steps to diagnose the problem, steps to solve the problem, and, if appropriate, temporary workarounds. metadata in an infobox also describes the components, software version/editions, operating systems, and environments to which the technote applies, as needed. ## techqa dataset characteristics the techqa dataset consists of a training set, a development set, a test set, and a small validation set. the training set contains 450 answerable questions and 150 non-answerable questions, the development set consists of 160 answerable and 150 non-answerable questions, and the evaluation set consists of 490 questions with similar answerable vs. non-answerable ratio as the development set. the ratios of non-answerable to answerable questions in the splits are similar to those of squad 2.0 @xcite . the validation set consists of the first 20 entries of the development set and is used in the leaderboard described in section 5. we also provide the full collection of the unique 801, 998 technotes that were available on the web as of april 4, 2019. the dataset is designed for mrc, rather than for open-domain qa. specifically, instead of requiring users to search the technote collection to find one containing the answer, we provide for each question a candidate list of 50 technote ids. systems should analyze only the 50 technotes associated with the question. a question is answerable if the annotators found an answer in one of these 50 technotes, and is unanswerable otherwise. systems can access the entire technote collection but only answers from the 50 technotes associated with each questions will be scored. the 50 technotes were obtained by issuing a query to an instance of elasticsearch foot_2 that indexes the 801, 998 technotes. this query consisted of the concatenation of the question title and question text; thus, the retrieved technotes are expected to contain at least some of the low-frequency terms in the question. if the answer is in a technote not retrieved by the search engine, we randomly removed one of the 50 technotes and substituted it with the one containing the answer. we did not include the search engine scores of the technotes and we randomized their order to obfuscate their search engine ranking. techqa questions and answers are substantially longer than those found in common datasets. the questions and answers contain numerous technical terms. we estimated the number of mentions of technical support entities with a model built with the mention detection data produced by our annotators during their training period (see section 3.2). on average the training set questions contain 1.67 detected mentions of errors, error codes, error messages or log messages (we do not further extract mentions from error messages or log messages, hence the subsequent counts are from other parts of the question), 3.8 mentions of hardware or software products or components, 2.0 mentions of parameters, settings, or configurations, and 2.23 mentions of operations or specific commands issued by the person asking the question, among others. many of these terms are likely not part of the vocabulary of most generalpurpose contextual language models. hence, one of the reasons for including the whole technotes corpus is to provide data for enhancing the language models by appropriately enlarging the vocabulary to include technical support terms. ## leaderboard task the dataset is available by registering to the leaderboard at ibm.biz/tech qa. registered users have access to the data and to means for submitting systems for evaluation against the blind test set. as with other leaderboards, this approach will help maintain the integrity of the blind set. a submitted system must be packaged as a docker image, containing all the needed compo-nents. the container will run in isolation from the network: systems will not be allowed to download anything-including models or other resourceswhile running in the evaluation environment. the systems will read the evaluation data from a readonly input directory and will write results to an output directory. detailed instructions on how to package the system are available from the leaderboard site. we ask that systems submitted to the leaderboard do not use information from the developer.ibm.com answers foot_3 and ibm developerworks foot_4 web sites except for the data provided with the dataset. submitted systems will run on a machine with 128 gb of memory and two 16g v100 gpus, with 64 gb local disk space available for temporary files or logs. upon submission, the system will run against the 20-question validation set. the results of the validation run are made available on the user's personal dashboard. a user satisfied with the validation run can submit the system to be run against the 490 evaluation questions. runs will be limited to 24 hours, after which they will be terminated and the submission will be in an error state in the dashboard. successful runs are added to the dashboard. the user can monitor the progress of each submission from the dashboard, and cancel the submission at any point previous to completion of the evaluation run. the results of successful evaluation runs are automatically posted on the leaderboard. a user is prevented from submitting a new system for a week starting from the date of the most recent submission, as it appears on the public leaderboard. the user dashboard provides means for anonymizing and de-anonymizing a successful submission (for example, for paper review purposes). an anonymized submission retains the name of the system provided by the user, but hides the user's affiliation as well as the optional link to a paper. to indicate that no answer exists in the technote; however, they also must indicate the best span extracted from the document: this is used to compute the two ancillary metrics described below. the evaluation score computed for the leaderboard is a zero-one value for a question/document pair with score below the threshold, and characteroverlap f1 for a question/document pair with score greater than or equal to the threshold. the main metric, called f1 on the leaderboard, is the macro average of the evaluation scores computed on the first of the five answers provided by the system in response to each question. the leaderboard displays three ancillary metrics. ha f1@1 is the macro average of the evaluation scores computed on the first of the five answers and averaged over the answerable evaluation questions. this metric should be compared to the inter-annotator agreement of 76.3 reported in section 3. ha f1@5 consists of computing the evaluation score for each of the 5 answers, selecting the maximum, and computing the macro average over all answerable questions. best f1 is the value of the f1 metric corresponding to the optimal choice of the threshold. the time required for the run will also be made available. ## baseline results the tap system consists of a document ranker module followed by an answer span selector, both based on pretrained bert small. if the largest score produced by the ranker exceeds a threshold, the question is declared answerable and the answer span selection is invoked on the documents. ## discussion and future work we have introduced techqa, a questionanswering dataset for the it technical support domain. the overall size of the released data (600 training questions) is in line with real-world scenarios, where the high cost of domain expert time limits the amount of quality data that can reasonably be collected. thus, the dataset is meant to stimulate research in domain adaptation, in addition to developing algorithms for longer questions and answers than the current leaderboards. we have created a leaderboard to evaluate systems against a blind dataset of 490 questions with a ratio of answerable to unanswerable questions similar to that of the development set. the leaderboard ranks submissions according to a metric consisting of the character overlap f1 measure for answerable questions and the zero-one metric for non-answerable questions. the leaderboard also reports the f1 at the top result and at the top 5 results averaged over the answerable questions. techqa is a challenging dataset for models developed for existing open-domain mrc systems. their out-of-the box performance is very low, especially considering that a system that declares every question as unanswerable achieves @xmath0.4% on the development set. the obvious approach of fine-tuning these models using the techqa training set yields systems that barely beat the baseline. the initial version of the dataset was created by selecting questions and answers that are relevant to the it technical support domain but at the same time do not diverge excessively from the spirit of other existing mrc datasets. we consider techqa to be a stepping stone on which to build future data collections and leaderboards. we plan on releasing questions with answers in a broader and more diverse collection that will include documents with a less formulaic structure than the technotes. we will also relax the length limitations to include questions rich in details, and