A sk M e: A LAPPS G rid-based NLP Query and Retrieval System for Covid-19 Literature (2020)

Abstract: In a recent project, the Language Application Grid was augmented to support the mining of scientific publications. The results of that ef- fort have now been repurposed to focus on Covid-19 literature, including modification of the LAPPS Grid “AskMe” query and retrieval engine. We describe the AskMe system and discuss its functionality as compared to other query engines available to search covid-related publications.

## introduction the onset of the coronavirus pandemic has prompted concerted efforts within the field of natural language processing (nlp) to enable researchers and practitioners to easily access and mine the growing body of literature concerned with the virus. between february and may 2020, the number of scientific papers published on covid-19 research increased from 29,000 to more than 138,000; this number is expected to exceed one million by the end of 2020. in a recent project, the language applications (lapps) grid foot_0 @xcite was augmented to support the mining of scientific publications foot_1 @xcite . the results of that effort have now been repurposed to focus on covid-19 literature, including modification of the lapps grid "askme" query and retrieval engine to access nightly updates of the cord-19 dataset 3 (wang et al., 2020) available from the allen institute for ai. in this paper, we describe the askme system and discuss its functionality as compared to other query engines available to search covid-related publications. because the askme engine is deployed primarily as a front end to the lapps grid, one of its most salient features is the ability to further process query results with the large array of nlp tools available in the grid. the askme system also provides numerous options for displaying results as well as means to fine-tune parameters of the search heuristics to suit particular needs and interests. ## related work several literature search engines have been developed or adapted in the wake of the covid-19 pandemic in order to expedite the progress of research focusing on the virus. among these are facilities developed by major corporations, most notably ibm (ibm inc., 2020), amazon web services (aws, 2020), and google (google inc., 2020), which generally do not access daily updates of covid-19 literature and may include licensed databases. non-commercially developed search engines for covid-19 data include vespa, litcovid and isearch. vespa (vespa, 2020) provides an open source search engine for version 2020-05-19 of the cord-19 dataset. litcovid @xcite is a curated open-resource literature hub developed by the national institutes of health (nih) and the national library of medicine (nlm), and is arguably the most comprehensive freely-available resource for covid-19 literature discovery. its database is limited to articles in pubmed specific to covid-19, vs. cord-19, which includes other viruses (e.g., sars, mers) and covers a time period well before the current outbreak. litcovid divides the articles into categories (e.g. mechanism, transmission, diagnosis, and treatment), and, based on automatically generated annotations, reports the chemicals, journals, and countries contained in the results. results can be sorted by relevance or recency. nih's isearch covid-19 portfolio (national institutes of health, 2020), also updated daily, includes both publications and preprints (from medrxiv, ssrn, arxiv, biorxiv, research square and chemrxiv) and provides interactive visualizations that allow users to select topics within their search results for download. the user can view results as a list in either chronological or reverse chronological order, or in "facet view" which shows results grouped by category (date, journal, author, etc.). the system provides advanced filters for more exact timeframes, publications types, source dataset, and inclusion or exclusion of icite articles. ## lapps grid the lapps grid provides a large collection of nlp tools from a wide variety of sources exposed as web services, together with multiple commonly used resources (e.g., gold standard corpora). the services and resources are made available via the galaxy web-based workflow development engine @xcite , and may also be accessed directly via soap calls or programmatically through java and python interfaces. all tools and resources in the lapps grid are rendered mutually interop-erable via transduction to the lapps grid interchange format (lif) @xcite and the web service exchange vocabulary (wsev) @xcite , both designed to capture fundamental properties of existing annotation models in order to serve as a common pivot among them. the lapps grid also provides interoperable, two-way access to the pubannotation annotation repository in response to a natural language query, askme applies several scoring algorithms to hone the results of solr's tf-idf similarity processing (based on the vector space model). currently, it scores only a document's title and abstract; askme will be expanded to handle full text in the near future. the score for each of these sections is computed from the following: a section's score is the sum of the component algorithm scores; the overall document score is computed by adding the scores for the individual sections. the main askme page (figure 1 ) includes an option to adjust the weights for inividual scoring algorithms to suit specific document types or interests, or simply fine-tune results. results can be viewed as a list including only score and title (figure 2 ) or coupled with document scores for each of the component algorithms. in the latter view, overall score for each section is highlighted. clicking on any column in either view sorts the results according to the values for that column. the full text for any document in the results can be viewed by clicking on the document id. once results are generated, the user may directly import them into the lapps grid for processing by any of a wide array of nlp tools from popular sources (stanford, opennlp, lingpipe, and many others), for example to identify entities, relations, dependencies, etc. annotations can be manually edited by accessing the editor in either pubannotation or inception and brought back into the grid for further processing, e.g., to train any of the various machine learning algorithms available in the lapps grid. foot_3 ## comparison with other systems we limit our comparison to litcovid and isearch, the most popular covid literature search engines. in terms of functionality, litcovid and isearch currently provide more advanced filtering by date, source, journal, etc. than askme, although there are plans to add such options (see section 6). all provide one-click access to the original publications. the most salient differences between askme and other systems include: comparison of askme results with those of other query engines is not straightforward, as none of the systems described in section 2 works exclusively with the most recently (daily) updated version of cord-19. therefore, we first focus on comparing askme and litcovid, which is the most closely related and relevant system, and include only documents included in both the litcovid and cord-19 corpora (as of 28 june 30 2020, 16,929 documents) in the results. to evaluate the results of askme, we asked an expert to compare results from askme for several queries against results from litcovid, vespa, and cord19.aws. the assessment shows that papers that the top 20 results from all of these search engines are very similar, with a few aberrations. for example, in one case where askme, vespa, and cord-19.aws rated the same documents at the top of their lists, litcovid did not return that document among its top 20; and for one question askme's top few results were deemed highly relevant, while none of the other engines found these documents. obviously, this is a preliminary assessment, but it gives us confidence that askme performs at the same level as other publication search enginesincluding engines such as cord19.aws that use sophisticated machine learning techniques to filter results. we will in the near future provide results of an evaluation using the trec-covid (cord-19-based) benchmark ( https://ir.nist.gov/covidsubmit/index.html ) to test how askme performs on this labeled data, when compared to other search engines. ## future work and conclusion our evaluation so far indicates that askme produces results on a par with litcovid and several other publication search engines. as a front end to the lapps grid coupled with the grid's integration with annotation generators and editors, askme provides a unique, one-stop platform for data discovery, annotation, and text mining. we are currently working on a number of performance and usability enhancements to the askme query engine, e.g., expanding scoring to other sections in full text articles and implementing a lapps grid service to generate named entity annotations over cord-19 documents for referral during the retrieval process. we are also adding nightly-generated embeddings from the cord-19 data and integrating tensorboard for their visualization. our eventual goal is to migrate askme to elasticsearch in order to enable integration with semviz foot_5 , thus providing an end-to-end system for query, search, and discovery over the cord-19 data.