Neural Extractive Search (2021)

Abstract: Domain experts often need to extract structured information from large corpora. We advocate for a search paradigm called “extractive search”, in which a search query is enriched with capture-slots, to allow for such rapid extraction. Such an extractive search system can be built around syntactic structures, resulting in high-precision, low-recall results. We show how the recall can be improved using neural retrieval and alignment. The goals of this paper are to concisely introduce the extractive-search paradigm; and to demonstrate a prototype neural retrieval system for extractive search and its benefits and potential. Our prototype is available at https://spike.neural-sim.apps.allenai.org/ and a video demonstration is available at https://vimeo.com/559586687 .

## introduction in this paper we demonstrate how to extend a search paradigm we call "extractive search" with neural similarity techniques. the increasing availability of large datasets calls for search tools which support different types of information needs. search engines like google search or microsoft bing are optimized for surfacing documents addressing information needs that can be satisfied by reviewing a handful of top results. academic search engines (semantic scholar, google scholar, pubmed search, etc) address also information needs targeting more than a handful of documents, yet still require the user to read through the returned documents. however, some information needs require extracting and aggregating sub-sentence information (words, phrases, or entities) from multiple documents (e.g. a list of all the risk factors for a specific disease and their number of mentions, or a comprehensive table of startups and ceos). these typically fall outside the scope of search engines and instead are classified as information extraction (ie), entailing a research project and a dedicated team per use-case, putting them well beyond the abilities of the typical information seeker. in contrast, we advocate for a complementary search paradigm: extractive search, which combines document selection with information extraction. the query is extended with capture slots: these are search terms that act as variables, whose values should be extracted ("captured"). @xcite the user is then presented with the matched documents, each annotated with the corresponding captured spans, as well as aggregate information over the captured spans (e.g., a count-ranked list of the values that were captured in the different slots). the extractive search paradigm is currently implemented in our spike system. @xcite aspects of its earlier versions are presented in shlain et al. (2020); taub-tabib et al. (2020). one way of specifying which slots to capture is by their roles with respect to some predicate, semantic-frame, or a sentence. in particular, the spike system features syntax-based symbolic extractive search-described further in section 2-where the capture slots correspond to specific positions in a syntactic-configuration (i.e., "capture the subject of the predicate founded in the first capture slot, and the object of the predicate in the second capture slot"). these are specified using a "by-example" syntax (shlain et al., 2020), in which the user marks the predicate and capture slots on a provided example sentence, and the syntactic configuration is inferred. while such parse-based matching can be very effective, it also suffers from the known limitations of symbolic systems: it excels in precision and control, but often lacks in recall. in this work, we demonstrate how the symbolic system can be combined with the flexibility of neural semantic similarity as induced by large pre-trained language models. figure 1 presents an overview of the system, containing a query with capture slots, the derived syntactic query, the returned (neural) results with marked spans, and an aggregate summary of the extracted pairs. by allowing fuzzy matches based on neural similarity search, we substantially improve recall, at the expense of some of the precision and control. the incorporation of neural similarity search requires two stages: retrieval of relevant sentences, and locating the roles corresponding to the capturespans on each sentence. we use standard dense passage retrieval methods for the first part (section 3), and present a neural alignment model for the second part (section 4). the alignment model is generic: it is designed to be pre-trained once, and then applied to every query in real time. this allows to provide an interactive search system which returns an initial response in near real-time, and continues to stream additional responses. the purpose of this paper then is twofold: first, it serves as a concise introduction of the extractivesearch paradigm. second, and more importantly, it demonstrates an incorporation of neural similarity techniques into this paradigm. ## symbolic extractive search we introduce the extractive search paradigm through usage examples. boolean extractive search. consider a researcher who would like to compile a list of treatments to bacteremia (bloodstream infection). searching google for "bacteremia treatment" might lead to a healthline article discussing a handful of treatments. foot_2 , which is not a great outcome. a similar query in pubmed search leads to over 30,000 matching papers, not all are relevant and each including only nuggets of relevant information. compare this with the extractive boolean query: bacteremia treatment :@xmath0., 2020), a search system over pubmed abstracts. "en-@xmath1, and the preceding colon (":") designate this term as a capture. the query retrieves 1822 sentences which include the word bactermia, the word treatment (added to establish a therapeutic context) and a chemi-cal entity. the user interface also displays the ranked list of 406 different chemicals captured by the query variable. the researcher can click each one to inspect evidence for its association with bacteremia, quickly arriving at a clean list of the common therapeutic compounds. syntactic extractive search ("by example"). in the previous example, the capture slot was based on pre-annotated span level information ("named entities"). while very effective, it requires the entity type of interest to be pre-annotated, which will likely not be the case for most entity types. additionally, the search is rather loose: it identifies any chemical in the same sentence of the terms "bactermia" and "treatment", but without establishing a semantic connection between them. what can we do when the entity type is not preannotated, or when we want to be more specific in our extraction target? one option is to define the capture slots using their syntactic sentential context. for example, consider a researcher interested in risk factors of stroke. an example of this relation is given in the syntactic configuration: we can search for sentences that match this pattern, foot_3 and extract the information which aligns with the capture node. @xcite however, such syntactic patterns require expertise to specify and are challenging to master. to counter this, shlain et al. (2020) introduced to spike the notion of query by example: the user enters a sentence which demonstrates the configuration: "something is a risk factor of stroke", marks which words are essential and should match exactly (risk, factor, stroke), and which correspond to capture slots (something), resulting in the query: foot_5 something arg is a risk factor for stroke the system then derives the corresponding syntactic query (see (shlain et al., 2020) for the details), returning results like: "these cases illustrate that pxe is a rare but significant risk factor for small vessel disease and stroke in patients of all age groups.", with the top aggregate terms being hypertension, artial fibrillation, af, diabetes, obesity while less frequent terms include vzv reactivation and palmitic acid. by modifying the query such that stroke is also marked as a capture slot: something arg 1 is a risk factor for stroke arg 2 one could easily obtain a table of risk factors for various conditions. ## neural extractive search the syntactic search by example lowers the barriers for ie: it easy to specify, accurate and effective. however, it is also limited in its recall: it considers only a specific configuration (both in terms of syntax and lexical items), and will not allow for alternations unless these are explicitly expressed by the user. neural models, and in particular large pre-trained language models (devlin et al., 2019; beltagy et al., 2019), excel at this kind of fuzzier, less-rigid similarity matching. we show how to incorporate them in the extractive search paradigm. this requires two stages: first, we need to match relevant sentences for a given query. second, we need to identify the relevant capture spans in the returned sentences. crucially, this needs to be done in a reasonable time: we do not have the luxury of re-training a model for each query, nor can we afford to run a large neural model on the entire corpus for every query. we can afford to run a pre-trained model on the query sentence(s), as well as over each of the sentences in the result set (similar to neural-reranking retrieval models (guo et al., 2020)). we operate under these constraints. the final system enables the user to search for specified information with minimal technical expertise. we demonstrate this approach on the cord corpus (wang et al., 2020), a collection of research papers concerning the covid-19 pandemic. ## argument-identification via alignment the dense neural retrieval over the averaged query vector results in topically-related sentences. to obtain the full benefit of extractive search, we need to provide span annotations over the sentences. this is achieved via a span alignment model which is trained to align semantically corresponding spans across sentences. at query time, we apply this model to align the marked spans over the first syntactic-query result, to spans over the neurallyretrieved sentences. the alignment model is pre-trained over a diverse set of relation, with the intent of making it a general-purpose alignment model. we describe the model architecture, training data, and training procedure. the argument-alignment task. the user marked in the query q a two spans, denoted as arg 1 and arg 2 . given a sentence (a dense retrieval result) with n tokens @xmath2, ..., w n , we seek a consecutive sequence of tokens w i:j corresponding to arg 1 , and another consecutive sequence of tokens w k:ℓ corresponding to arg 2 . for example, consider the query: in which the span arg 1 corresponds to a kind of infection, and arg 2 corresponds to the outcome of the infection. the syntactic query may return a result such as: the infection of sars-cov-2 arg 1 causes fever arg 2 . while a neural result might be: it has been noted that headaches are one side effect of flu infection. we would like to align flu to arg 1 (sars-cov-2) and headaches to arg 2 (fever). ## example search we demonstrate the system via an example where one aims to find sentences containing information on compounds and their origin (e.g. plant-derived, isolated from soil, etc.). we start with the query: the syntactic yields only few results, all of them are relevant. among the results: -colchicine is a drug extracted from colchicum autumnale. several additional results are presented below: -allicin is the major biologically active component of garlic. note that the last two examples demonstrate failure modes: in the the fourth example, the model failed to identify dimethyl cardamonin as an argument; and in the last sentence there is no clear capture corresponding to the second argument. finally, we perform an aggregation over the predicted captures (fig 1 , right-pane), allowing the user to quickly get a high-level overview of the interactions. from our experience, users are mostly interested in this table, and turn to the text as support for validating interesting findings. ## limitations of the neural approach while we find the neural approach to be very effective, we would also like to discuss some of its limitations. first, speed and scalability are still lagging behind that of symbolic search systems: dense retrieval systems do not yet scale as well as symbolic ones, and running the (sci)bert-base argument-aligner for each candidate sentence is significantly slower than performing the corresponding similarity search. while the user can see the first results almost immediately, getting extractions from thousands of sentences may take several minutes. we hope to improve this speed in future work. in terms of accuracy, we find the system to be hit-or-miss. for many symbolic queries we get fantastic resutls, while for others we observe failures of the dense retrieval model, or frequent failures of the alignment model, or both. for effective incorporation in a user-facing system, we should-beyond improvements in retrieval and alignment accuracybe able to predict which queries are likely to yield poor results, and not extend them with fuzzy neural matches. ## conclusions we presented a system for neural extractive search. while we found our system to be useful for scientific search, it also has clear limitations and areas for improvement, both in terms of accuracy (only 72.2% of the returned results are relevant, both the alignment and similarity models generalize well to some relations but not to others), and in terms of scale. we see this paper as a beginning rather than an end: we hope that this demonstration will inspire others to consider the usefulness of the neural extractive search paradigm, and develop it further.