Talk to Papers: Bringing Neural Question Answering to Academic Search (2020)

Abstract: We introduce Talk to Papers, which exploits the recent open-domain question answering (QA) techniques to improve the current experience of academic search. It’s designed to enable researchers to use natural language queries to find precise answers and extract insights from a massive amount of academic papers. We present a large improvement over classic search engine baseline on several standard QA datasets and provide the community a collaborative data collection tool to curate the first natural language processing research QA dataset via a community effort.

## introduction natural language processing (nlp) is one of the fastest growing field in computational linguistics and artificial intelligence, e.g. acl has experienced a 140% growth from 2017 (1419 submissions) to 2020 (3429 submissions). plus, there are more than 4000 pre-prints published at arxiv in 2019. as a result, it has become increasingly stressful for researchers to keep up with the evolution of new methods. today, the common way for researchers to find relevant papers is via searching keywords in google scholar foot_1 or semantic scholar 3 . although these search engines are great at curating all the papers, they are limited in the following ways: (1) they are based on classic information retrieval methods, and do not handle natural language queries well, e.g. what effects can we get from label smoothing? (2) they are designed to find relevant documents (title and abstract) instead of direct answers to users' questions. often researchers are looking for answers on specific research questions, e.g, how to prevent posterior collapse in vae? or how much is it to label sentences via crowdsourcing? with current search engine, it requires one to read several papers to find these answer. therefore, it is necessary to create better tools for researchers to find answers from the scientific publications in a more efficient manner. meanwhile, machine reading comprehension (mrc), aka question answering (qa) has advanced significantly. pretrained and then fine-tuned transformer models @xcite have surpassed human performance on a number of datasets, e.g. squad @xcite . further, @xcite extended single document mrc to machine reading at scale (mrs), combining the challenges of document retrieval with reading comprehension. their open-domain qa system is able to find precise answers from millions of unstructured documents using natural language queries and has successfully been applied to the entire wikipedia which contains more than 5 million articles. the goal of talk to papers is to create a new way of finding answers from scientific publications and advance qa research. concretely, we first adapted mrs techniques to create a conversational search portal that enable users to ask natural language questions to find precise answers and extract insights from the last 3 year papers published in top-tier nlp conferences, including acl, naacl, emnlp and etc. second, an initial corpus on these papers is collected and will be released as a publicly available dataset for qa research. we also developed a collaborative annotation toolkit that enable any researcher to contribute to this dataset so that more potential answers from these papers can be annotated. the annotation results will be fed back to the qa corpus after manual validation. 2 related work developing question answering system with text knowledge base has been studied for decades @xcite . many of the classic system as well as recent mrc-based opendomain qa systems have relied a pipeline approach @xcite @xcite : (1) a information retrieval-based retriever module first finds relevant passages from all the documents and then (2) a reader component (computationally more expensive) extracts precise answer spans from the retrieved passages. @xcite has shown that using paragraphs as the unit of passage outperform sentences or documents. @xcite proposes a trainable first-stage retriever that improves the recall performance. pipeline-based system often suffer from error propagation @xcite . thus another line of research has been finding an end-toend approach that enable precise-answer extraction from the entire dataset instead of only the output from the first-stage retriever. @xcite introduced the phrase level representation model that index every potential answer span as vector representation and exploited approximate nearest neighbour (ann) methods to retrieve the final answer span directly from a large vector index @xcite . @xcite argued that phrase-level answer may not always be required or preferred. instead they proposed to find the right "sentence" as an answer from large body of text, and used universal sentence encoder @xcite to retrieved the correct sentence given a question. our approach follows the sentence-level qa system from @xcite for two reasons: (1) answers to many research questions cannot be cov-ered in a short phrase-level span, and a sentence answer can provide more context to deliver relevant solutions. (2) our preliminary study found that it is important to have a trainable retriever that goes beyond tf-idf keyword matching to ensure enough recall in the paper domain. nonetheless, we keep a machine reader as optional post-process to extract phrase-level span from the sentences. ## the proposed qa toolkits: soco we first introduce soco (search oriented conversation), which we used to build our talk to paper. soco foot_3 is an answer-engine platform that enables developers to easily build universal question answering systems with unstructured documents as its knowledge base. figure 1 shows the overall architecture of soco engine. it's designed to enable users to use natural language queries to find precise answers and extract insights from massive amount of text data. the typical workflow is as following: ## experiments and results in this section, we first present quantitative preliminary evaluation results the effectiveness of the proposed soco-qa framework on a number of standard qa datasets. then we show results on the data collected from our initial user study. ## conclusion we present talk to paper, a qa system for nlp papers powered by soco-qa. experiments confirm the effectiveness of our proposed approach and show superior search experience compared to traditional search engine. we welcome contributions from the research community to curate useful resources together for the further research. future work include (1) expanding the database to more papers (2) improving the qa model using the collected data to better handle question answering in the context of research domain.