On the portability of extractive Question-Answering systems on scientific papers to real-life application scenarios (2022)

Abstract: There are still hurdles standing in the way of faster and more efficient knowledge consumption in industrial environments seeking to foster innovation. In this work, we address the portability of extractive Question Answering systems from academic spheres to industries basing their decisions on thorough scientific papers analysis. Keeping in mind that such industrial contexts often lack high-quality data to develop their own QA systems, we illustrate the misalignment between application requirements and cost sensitivity of such industries and some widespread practices tackling the domain-adaptation problem in the academic world. Through a series of extractive QA experiments on QASPER, we adopt the pipeline-based retriever-ranker-reader architecture for answering a question on a scientific paper and show the impact of modeling choices in different stages on the quality of answer prediction. We thus provide a characterization of practical aspects of real-life application scenarios and notice that appropriate trade-offs can be efficient and add value in those industrial environments.

## introduction it is widely recognized today that the most advanced countries have moved to the so-called knowledge-based economy. in the industrial field, including service providers, this new paradigm has particular consequences for most players in r&d and innovation activities where decisions are based on the analysis of huge corpora of documents @xcite . the thorough exploitation of this pre-existing knowledge by highly-skilled workers is costly and timeconsuming, but such costs can be significantly reduced by nlp technologies that make exploitation and consumption of textual content faster and more efficient. for instance, information-seeking question-answering is of particular interest to industrial environments conducting scientific monitoring, but there still remain significant hurdles to efficiently adopt such systems in those environments, predominantly the complexity and accessibility of the data landscape. as a matter of fact, extracting information from scientific publications is a cognitively complex process and requires domain expertise, but obtaining and ensuring such high-quality annotations could become unreasonably expensive and unreliable. the scarcity of in-house annotation efforts, frequent domain shifts, and lack of deep understanding of data-model interaction and evaluation make these technologies inaccessible especially for industrial environments lacking computational resources. one direction would be to entirely rely on models' transfer learning capabilities and make use of the knowledge they learn on academic benchmarks that meet the size requirement. however, zero and fewshot settings successes, i.e, when few to no annotations are available, seem to be largely dominated by large-scale autoregressive models @xcite , which are accessible only to a handful of researchers and practitioners with enormous compute power. in this paper, we take on extractive informationseeking qa on scientific papers from an industrial point of view. we identify the hurdles standing in the way of adopting such systems and show through a simulation of such context that some modeling and evaluation practices might not align with a suitable return on investment sought by such industries. our contributions can be summarized as follows: first, we explore the portability challenges of qa models toward scientific content-consuming industrial environments and split them into three major long-standing issues. second, we simulate through a series of experiments on qasper @xcite the context where information is sought in research papers and thus illustrate the identified portability issues. third, we discuss based on the results the relevance of modeling and evaluation choices when compared to the goal of adequately solving the task in a cost-effective way. ## the portability challenge in industrial environments for small and medium-sized enterprises (smes) interested in information-seeking qa on scientific publications, the question of work to be done compared to the benefit of it is very important as it informs the way resources are allocated. when bringing advancements like qa systems into realworld applications suffering data scarcity issues, choosing a benchmark representative of contexts, questions, and answers one would expect in their application remains the most widely adopted practice for maximizing accuracy. unfortunately, due to the fact that meeting an information need is a hard concept to quantify, adopting such technologies can fall short of quantitatively measuring the impact and the business value created. we discuss hereafter three major inter-connected long-standing issues that restrain from successful portability: issue 1: modeling real-world problems is challenging. question answering aims at meeting an information need and providing a user with relevant answers to their questions. however, in domains with high levels of expertise, assisting professionals in such complex processes requires, depending on the nature of the query, cognitive abilities that ai systems have not yet matured to @xcite . the ai community has factually been benchmarking intelligence by comparing the defined skill exhibited by ai and humans at specific tasks, and building special-purpose systems capable of handling narrow, well-described tasks, more and more above human-level performance. this created a plethora of qa benchmarks/tasks measuring very specific skills @xcite as opposed to the complex processes one would long for in intelligent systems. further, annotating the required amount of quality data to build such systems can be unaffordable for many industries and organizations. the question that arises here is whether to favor quantity in task format adequacy and thus potentially model performance, or limited content representativeness with complexity that guarantees quality and better alignment with real-world applications. issue 2: there is a real need for transparency and confidence not only in predictions but also in the whole predictive process in a way that allows users to assess how well-informed their decisions would be. however, there still remains insufficient understanding of the capabilities and limitations of models and the way they interact with data during the different stages of their training @xcite . up until recently, there has been little guidance on the suitability of which models for which cases in question answering @xcite . tremendous work continues to be done on modeling and exploring new model architectures and training schemes, however interpretation and explanation of models' behaviors that inform modeling choices in adopting such technologies, have not developed at the same pace. this makes it challenging for adopters to select their models for real-world settings, whether the intended use is at early stages or later in production. the obvious issue here is to identify what makes a certain model a trustworthy fit for the project motivation rather than another. issue 3: a good performance metric is not synonymous with how well application requirements are met. while current evaluation schemes contribute to overly specializing solutions for performance benchmarks, adopters and end-users are not only more sensitive to the plus-value models provide, but also the costs of developing and deploying such systems. extractive qa systems are mainly evaluated using the f-measure, but a token-overlap metric is not informative on how well the system is assisting the user and providing relevant answers. for this reason, misaligning what is measured and what is intended and desired might lead in certain cases to misallocating resources, and although progress has been made towards user-centered evaluation @xcite , real-world applications still have more complexity and demands whereas models' evaluation is lagging behind. these issues impact different phases of the development cycle of qa systems in real-world expert applications. for instance, issue 1 impacts problem definition and adequate data collection, which are the backbone of the whole cycle. issue 2 introduces hurdles to experiment design and model training, while issue 3 directly impacts evaluation and complicates the path to successful model deployment. in the rest of the paper, we simulate a scenario of seeking information in research papers and consider our end-user to be an expert in decision support based on scientific publications analysis. we particularly focus on the extractive qa setting where the goal is to provide the user with answers to a particular question on a given paper. this translates to the following formulation of the issues mentioned above: 1. what kind of task and data should we use, given the complexity and the level of expertise present both in the questions and the context? 2. what models would solve this task and would interact well with such data, peculiarly since we need as much transparency as possible in the process of identifying the answers? 3. how does the performance of the chosen model on the chosen task and data reflect the return on investment for deploying such systems? ## results we present in this section the results of the different stages of the pipeline when adding components or using different training strategies. ## discussion we discuss hereafter the sources of improvements and their alignment with the portability challenges. ## conclusion information-seeking qa on scientific content is gaining popularity in a world of knowledge-based economies. in this paper, we identified the hurdles that stand in the way of efficient portability of such systems into industrial environments suffering data scarcity. we revealed through a series of experiments on extractive qa anchored in research papers, that bridging the gap between academic benchmarks along with their models' performance, and concrete user needs that are most often hindered by resource allocation constraints in business can be done with appropriate trade-offs and that caution needs be taken when investing in widespread but costly practices.