Interactive Extractive Search over Biomedical Corpora (2020)

Abstract: We present a system that allows life-science researchers to search a linguistically annotated corpus of scientific texts using patterns over dependency graphs, as well as using patterns over token sequences and a powerful variant of boolean keyword queries. In contrast to previous attempts to dependency-based search, we introduce a light-weight query language that does not require the user to know the details of the underlying linguistic representations, and instead to query the corpus by providing an example sentence coupled with simple markup. Search is performed at an interactive speed due to efficient linguistic graph-indexing and retrieval engine. This allows for rapid exploration, development and refinement of user queries. We demonstrate the system using example workflows over two corpora: the PubMed corpus including 14,446,243 PubMed abstracts and the CORD-19 dataset, a collection of over 45,000 research papers focused on COVID-19 research. The system is publicly available at https://allenai.github.io/spike

## introduction recent years have seen a surge in the amount of accessible life sciences data. search engines like google scholar, microsoft academic search or semantic scholar allow researchers to search for published papers based on keywords or concepts, but search results often include thousands of papers and extracting the relevant information from the papers is a problem not addressed by the search engines. this paradigm works well when the information need can be answered by reviewing a number of papers from the top of the search results. however, when the information need requires extraction of information nuggets from many papers 1 https://pages.semanticscholar.org/coronavirus-research (e.g. all chemical-protein interactions or all risk factors for a disease) the task becomes challenging and researchers will typically resort to curated knowledge bases or designated survey papers in case ones are available. we present a search system that works in a paradigm which we call extractive search, and which allows rapid information seeking queries that are aimed at extracting facts, rather than documents. our system combines three query modes: boolean, sequential and syntactic, targeting different stages of the analysis process, and different extraction scenarios. boolean queries ( §4.1) are the most standard, and look for the existence of search terms, or groups of search terms, in a sentence, regardless of their order. these are very powerful for finding relevant sentences, and for co-occurrence searches. sequential queries ( §4.2) focus on the order and distance between terms. they are intuitive to specify and are very effective where the text includes "anchor-words" near the entity of interest. lastly, syntactic queries ( §4.4) focus on the linguistic constructions that connect the query words to each other. syntactic queries are very powerful, and can work also where the concept to be extracted does not have clear linear anchors. however, they are also traditionally hard to specify and require strong linguistic background to use. our systems lowers their barrier of entry with a specification-by-example interface. our proposed system is based on the following components. ## existing information discovery approaches the primary paradigm for navigating large scientific collections such as medline/pubmed foot_1 is document-level search. the most immediate document-level searching technique is boolean search ("keyword search"). however, these methods suffer from an inability to capture the concepts aimed for by the user, as biomedical terms may have different names in different sub-fields and as the user may not always know exactly what they are looking for. to overcome this issue several databases offer semantic searching by exploiting mesh terms that indicate related concepts. while in some cases mesh terms can be assigned automatically, e.g @xcite , in others obtaining related concepts require a manual assignment which is laborious to obtain. beyond the methods incorporated in the literature databases themselves, there are numerous external tools for biomedical document searching. thalia @xcite is a system for semantic searching over pubmed. it can recognize different types of concepts occurring in biomedical abstracts, and additionally enables search based on abstract metadata; livivo @xcite takes the task of vertically integrating information from divergent research areas in the life sciences; swift-review foot_2 offers iterative screening by reranking the results based on the user's inputs. all of these solutions are focused on the document level, which can be limiting: they often surface hundreds of papers or more, requiring careful reading, assessing and filtering by the user, in order to locate the relevant facts they are looking for. to complement document searching, some systems facilitate automatic extraction of biomedical concepts, or patterns, from documents. such systems are often equipped with analysis capabilities of the extracted information. for example, nactem has created systems that extract biomedical entities, relations and events. foot_3 ; exact and robotreviewer @xcite take a rct report and retrieve sentences that match certain study characteristics. to improve the development of automatic document selection and information extraction the bionlp community organized a series of shared tasks @xcite @xcite @xcite @xcite . the tasks address a diverse set of biomed topics addressed by a range of nlp-based techniques. while effective, such systems require annotated training data and substantial expertise to produce. as such, they are restricted to several "head" information extraction needs, those that enjoy a wide community interest and support. the long tail of information needs of "casual" researchers remain mostly un-addressed. ## interactive ie approach existing approaches to information extraction from bio-medical data suffer from significant practical limitations. techniques based on supervised training require extensive data collection and annotation @xcite @xcite @xcite , or a high degree of technical savviness in producing high quality data sets from distant supervision @xcite @xcite . on the other hand, rule based engines are generally too complex to be used directly by domain experts and require a linguist or an nlp specialist to operate. furthermore, both rule based and supervised systems typically operate in a pipeline approach where an ner engine identifies the relevant entities and subsequent extraction models identify the relations between them. this approach is often problematic in real world biomedical ie scenarios, where relevant entities often cannot be extracted by stock ner models. to address these limitations we present a system allowing domain experts to interactively query linguistically annotated datasets of scientific re-search papers, using a novel multifaceted query language which we designed, and which supports boolean search, sequential patterns search, and byexample syntactic search @xcite , as well as specification of search terms whose matches should be captured or expanded. the queries can be further restricted by contextual information. we demonstrate the system on two datasets: a comprehensive dataset of pubmed abstracts and a dataset of full text papers focused on covid-19 research. comparison to existing systems. in contrast to document level search solutions, the results returned by our system are sentences which include highlighted spans that directly answer the user's information need. in contrast to supervised ie solutions, our solution does not require a lengthy process of data collection and labeling or a precise definition of the problem settings. compared to rule based systems our system differentiates itself in a number of ways: (i) our query engine automatically translates lightly tagged natural language sentences to syntactic queries (queryby-example) thus allowing domain experts to benefit from the advantages of syntactic patterns without a deep understanding of syntax; (ii) our queries run against indexed data, allowing our translated syntactic queries to run at interactive speed; and (iii) our system does not rely on relation schemas and does not make assumptions about the number of arguments involved or their types. in many respects, our system is similar to the propminer system @xcite for exploratory relation extraction @xcite . both propminer and our system support byexample queries in interactive speed. however, the query languages we describe in section 4 are significantly more expressive than propminer's language, which supports only binary relations. furthermore, compared to propminer, our annotation pipeline was optimized specifically for the biomedical domain and our system is freely available online. technical details. the datasets were annotated for biomedical entities and syntax using a custom scispacy pipeline @xcite 6 , and the syntactic trees were enriched to bart format using pybart @xcite . the annotated data is indexed using the odinson engine @xcite . the simplest boolean query is a list of terms, where each term is a word, i.e: 'infection asymptomatic fatal' the semantics is that all the terms must appear in the query. a term can be made optional by prefixing it with a '?' symbol ('infection asymptomatic ?fatal' ). each term can also specify a list of alternatives: 'fatal|deadly|lethal'. ## example workflow: risk-factors we describe a workflow which is based on using our extractive search system over a corpus of all pubmed abstracts. while the described researcher is hypothetical, the results we discuss are real. consider a medical researcher who is trying to compile an up to date list of the risk factors for stroke. a pubmed search for "risk factors for stroke" yields 3317 results, and reading through all results is impractical. a google query for the same phrase brings out an info box from nhlbi 9 listing 16 common risk factors including high blood pressure, diabetes, heart disease, etc. having a curated list which clearly outlines the risk factors is helpful, but curated lists or survey papers will often not include rare or recent research findings. the researcher thus turns to extractive search and tries an exploratory boolean query: 'risk factor stroke' . the figure shows the top results for the query and the majority of sentences retrieved indeed specify specific risk factors for stroke. this is an improvement over the pubmed results as the researcher can quickly identify the risk factors discussed without going through the different papers. furthermore, the top results contain risk factors like migrane or c804a polymorphism not listed in the nhlbi knowledge base. however, the full result list is lengthy and extracting all the risk factors from it manually would be tedious. instead, the researcher notes that many of the top results are variations on the "x is a risk factor for stroke" structure. she thus continues by issuing the following syntactic query, where a capture labeled r is used to directly capture the risk factors: 'r:diabetes is a @xmath0factor for @xmath1risk @xmath2stroke'. the full names are now captured as expected. now that that researcher has verified that the query yields relevant results, she clicks the download button to download the full result set. the resulting tab separated file has 1212 rows. each row includes a result sentence, the captured elements in it (in this case, just the risk factor), and their offsets. using a spreadsheet to group the rows by risk factor and order the results by frequency, the researcher obtains a list of 640 unique risk factors, 114 of them appearing more than once in the data. figure 2a lists the top results. reviewing the list, the researcher decides that she's not interested in general risk factors, but rather in diseases only. she modifies the query by adding an entity restriction to the 'r' capture: as seen in the query graph, even though the researcher didn't specify the exact entity type, the query parser correctly resolved it to disease. the results now include diseases like sleep apnoea and hypertension but do not include smoking, age and alcohol (see figure 2b ). analyzing the results, the researcher now wants to compare the risk factors in the general population to ones listed in research papers dealing with children and infants. luckily, such papers are indexed with corresponding mesh terms and the researcher can utilize this fact by appending '#d mesh:child mesh:infant -mesh:adult' to her query. in cases where a desired mesh term does not exist, an alternative approach is filtering the results based on words in the abstract or title. for example, appending '#d abstract:child abstract:children' to a query will ensure that the result sentences come from abstracts which contain the word child or the word children. happy with the results of the initial query, the researcher can further augment her list by querying for other structures which identify risk factors (e.g. "'r:diabetes @xmath3stroke'", "'@xmath4factors for @xmath5include r:diabetes'", etc.). importantly, once the researcher has identified one or more effective queries to extract the risk factors for stroke, the queries can easily be modified in useful ways. for example, with a small modification to our original query we can extract: risk factors for cancer: ## example workflow: cord-19 the covid-19 open research dataset @xcite is a collection of 45,000 research papers, including over 33,000 with full text, about covid-19 and the coronavirus family. the corpus was released by the allen institute for ai and associated partners in an attempt to encourage researchers to apply recent advances in nlp to the data to generate insights. identifying covid-19 aliases since the cord-19 corpus includes papers about the entire coronavirus family of viruses, it's useful to identify papers and sentences dealing specifically with covid-19. before converging on the acronym covid-19 researchers have referred to the virus by many names: ncov-19, sars-cov-ii, novel coronavirus, etc. luckily, it's fairly easy to identify many of these aliases using a sequential pattern: the pattern looks for the words "novel coronavirus" followed by an open parenthesis, one-or-two words which are to be captured under the 'alias' variable, and a closing parenthesis. the query retrieves 52 unique candidate aliases for covid-19, though some of them refer to older coronaviruses such as "mers", or non-relevant terms such as "fig2". after ranking by frequency and validating the results, we can reuse the pattern on newly retrieved aliases to extend the list. through this iterative process we quickly compile a list of 47 aliases. we marked all occurrences of these terms in the underlying corpus as a new entity type, covid-19, and re-indexed the dataset with this entity information. exploring drugs and treatments. to explore drugs and treatments for covid-19 we search the corpus for chemicals co-occuring with the covid-19 entity using a boolean query: to guide the search toward therapeutic substances in different stages of maturity we can add indicative terms to the query. for example, the following query can be used to detect substances at the stage of clinical trials: while the queries are very basic and include only a few terms for each category, the difference is clearly noticeable: while the malaria drug chloroquine tops both lists, the antiviral drug remdesivir which is currently tested for covid-19 is second on the list of trial related drugs but does not appear at all as a top result for ideation related drugs. importantly, entity co-mention queries like the ones above rely on the availability and accuracy of underlying ner models. as we've seen in section 5, in cases where the relevant types are not extracted by ner, syntactic queries can be used instead. for example the following query captures sentences including chemicals being used on patients (the abstract or paragraph are required to include covid-19 related terms). the top results by frequency are shown in ## more examples while the workflows discussed above pertain mainly to the medical domain, the system is optimized for the broader life science domain. here are a sample of additional queries, showing different potential use-cases. which genes regulate a cell process: ## conclusion we presented a search system that targets extracting facts from a biomed corpus and demonstrated its utility in a research and a clinical context over cord-19 and pubmed. the system works in an extractive search paradigm which allows rapid information seeking practices in 3 modes: boolean, sequential and syntactic. the interactive and flexible nature of the system makes it suitable for users in different levels of sophistication.