I nfo F orager: Leveraging Semantic Search with AMR for COVID -19 Research (2020)

Abstract: This paper examines how Abstract Meaning Representation (AMR) can be utilized for finding answers to research questions in medical scientific documents, in particular, to advance the study of UV (ultraviolet) inactivation of the novel coronavirus that causes the disease COVID-19. We describe the development of a proof-of-concept prototype tool, InfoForager, which uses AMR to conduct a semantic search, targeting the meaning of the user question, and matching this to sentences in medical documents that may contain information to answer that question. This work was conducted as a sprint over a period of six weeks, and reveals both promising results and challenges in reducing the user search time relating to COVID-19 research, and in general, domain adaption of AMR for this task.

## introduction uv light can inactivate viruses by making them unable to infect cells, thereby reducing the transmission of viral diseases. while a wealth of literature pertaining to uv inactivation of viruses exists, searching this literature for trustworthy and relevant information can be difficult and inefficient. this difficulty can be especially evident when focusing on diseases such as covid-19 (caused by the novel coronavirus, sars-cov-2 1 ), which can be transmitted via aerosols or droplets in a complex process spanning disciplines ranging from physiology to optics to fluid mechanics. there are many unknowns, poorly quantified parameters, and even confusions resulting from differing terminology used. nonetheless, efficiently finding needed information may be critical in discovering improved methods, such as the use of germicidal uv, 2 to reduce the transmission of covid-19 and other diseases. this paper identifies an opportunity for nlp tools to aid in automatically sifting through this mass of documents to find relevant answers to specific and targeted questions from subject matter experts working in the space of uv inactivation of viruses. we introduce infoforager, a proof-of-concept prototype tool that utilizes semantic understanding and search, going beyond the words in a user question to focus on its meaning. by employing a semantic search, we hypothesize that the user will more easily search through medical documents because they do not need to rephrase their questions (for example, into keywords) to conform to the system's search limitations and capabilities. infoforager first parses a user research question into abstract meaning representation (amr) @xcite , then compares the resulting amr query to a collection of medical research papers already parsed into amr. all amr query-sentence pairs in each paper are scored for their semantic similarity, and infoforager returns the highest-ranking answer sentence and the source document. given the urgent nature of this research, we allotted six weeks during which four nlp researchers worked with two uv inactivation researchers to perform a shallow pass through the semantic searching problem space. additionally, we worked with test users to obtain an understanding of the system requirements and developed a prototype framework to address those needs. section 2 describes the uv inactivation problem space. our semantic search approach is described in section 3, and section 4 details a wizardof-oz prototyping user study of the framework and the initial development of the prototype system. an evaluation of the prototype and the results are discussed in section 5. ## background: uv inactivation research virus particles (virions) can remain infectious on surfaces (e.g., walls, doorknobs, and masks), and may be contained within sneezed or coughed droplets of airway (e.g., mouth, nose, throat) fluids. such virions can be inactivated by treating them and particles containing them with sufficient uv light @xcite ; however, it remains unclear whether, and to what extent, the virions within dried droplets, or particles, are shielded from uv. researchers need to know how the optical properties of a particle of such respiratory fluids affect the intensity of uv in a virion within that particle. if the particle of dried respiratory fluids actually protects virions within it from uv, then how much more intense does the uv source need to be to achieve adequate inactivation of the virions? furthermore, what techniques could be used to increase the effectiveness of uv light for inactivation of sars-cov-2 in such particles? given the tremendous variation in sizes, shapes, and compositions of dried respiratory particles, answering these questions by experimentation alone is far too expensive. having a computational model to accurately represent intensities within uv-illuminated respiratory particles would be beneficial for testing multiple use-cases, and may be critical in discovering improved methods for using germicidal uv to reduce the transmission of covid-19 and other diseases. such modeling requires knowledge of the optical properties of airway fluids, which can be estimated from the concentrations and optical properties of the materials in these fluids. however, to date and to our knowledge, there are no medical papers or reports that include all the optically relevant materials in any airway fluids to create such a model. we designed infoforager with the aforementioned challenges in mind. the uv researchers and nlp researchers iteratively refined the general topic of uv inactivation of coronavirus into two main research questions of interest, which were used in developing and testing infoforager: q 1 which (bodily) fluids have higher concentrations of sars-cov-2 particles? q 2 what is the range of sizes of respiratory droplets, specifically from coughing and sneezing? for the purposes of training participants in our wizard-of-oz user study, we introduced a basic question: q t what types of uv light are used in coronavirus research? the uv researchers made clear that they were not seeking a system that returned a single answer to their research questions, as there are often multiple, even conflicting answers in cutting-edge medical research. instead, they desired a system that returns all relevant results and that also clearly points to where the relevant content is in the document, allowing them to quickly assess where the information they are searching for is discussed in the approach, background, or results section of a paper. this approach stands in contrast to search systems that the uv researchers have become accustomed to and use primarily for building their initial pool of medical documents, or abstracts when the documents are behind paywalls. for example, pubmed is as an extensive, publicly available, online, searchable database of medical and life sciences research abstracts. 3 in this study, we rely on it as a benchmark resource, given its broad and updating coverage of research papers, and make use of its keyword search, wherein terms from the user query are matched against subject designations for articles, the title, abstract, and author names. if no matches are made, the search terms are broken apart and the search process is repeated with the individual terms. this, indirectly, has led researchers away from posing natural language questions. when backing off to break such questions into keywords, pubmed may return either too many results for very general keywords or no results for more targeted combinations. for example, q 1 yields zero results, and subsequent alterations of the question, e.g., which fluids higher sars-cov-2 particles and which fluids higher concentration particles, yield 1 and 1,157 individual research papers respectively. this approach places the burden on the user to determine which terms in the query are the most important for the system to function. ## infoforager approach: semantic parse and search infoforager was designed with the uv researchers' goals, criteria, and the current tool limitations in mind. given that they expect to conduct an iterative search process (to "forage"), where they may find partial information, including different and possibly discrepant values (especially as new findings emerge), we start with the following working hypothesis: an amr-based approach can assist in identifying such partial information based on where their questions share semantics with sentences within the documents in their collection. thus, we aim to create a system that accepts natural language questions, interprets their underlying meaning as amr, detects a range of answers, and finally points to where that information was found in research papers. our semantic search interaction is presented in figure 1 . first, a user provides a question, which is semantically parsed into amr (left panel of figure 1 ). second, a semantic search is conducted that matches the parsed query against a collection of medical research papers that had also been semantically parsed (center panel). finally, the results are scored and ranked, and the top responses along with the source papers are returned (right panel). the remainder of this section describes the parsing, and search and rank processes. (3) matches are ranked and highest ranking sentence is returned with its source document. ## user studies and proof-of-concept development having developed the initial framework for semantic search to address the needs of uv researchers, we conducted a wizard-of-oz user study (section 4.1) to explore how non-expert users interacted as research assistants with the system, while we developed the proof-of-concept prototype (section 4.2). ## results and discussion to make an initial assessment of the validity of our approach, we evaluated the prototype system on the three research questions, for which we had manually identified documents and sentences that best answer the questions. using these answers and documents as a ground truth, we compared the ranked list of sentences output by our system, thus determining the extent to which our methodology would highly rank and return the same answers as a human researcher. to focus on the potential for this semantic search approach without introducing too much noise from automatic parsing of the question at this stage, we conducted the evaluation using the manually annotated amr gold-queries, as opposed to the amr parsed-queries. note, however, that the target answer documents are all automatically parsed. in the subsequent analysis, we compute both 1) gold-query vs. parsed-answer, and 2) gold-query vs. gold-answer, to determine how effective the smatch metric is in capturing the shared semantic content between query and answer with and without amr parsing error. q 1 which fluids have higher concentrations of sars-cov-2 particles? as of june 2020, this research question was judged most urgent by the uv researchers and they found very little supporting literature on the subject-only a single document with several informative sentences in the results section, and one sentence summarizing the information sought: a 1 overall, we found higher sars-cov-2 titers from saliva than nasopharyngeal swabs from hospital inpatients @xcite . we were encouraged to find that our system ranked this sentence as an answer third by smatch fmeasure (23%) and fifth by recall (17%). notably, this sentence has similar phrasing to the question, and as a result, their amr structures are more similar. the parsed-answer does have a variety of errors, most arising from problems with the have-degree-91 argument assignment (this is the roleset used to express comparatives in amr, including "higher"). as noted above, to test for the noise introduced by automatic parsing errors, we also scored the gold-query vs. gold-answer, shown in figure 4 . this gold pair actually receives a slightly lower smatch f-measure, at 20% (compared to 23% on gold-query vs. parsed-answer). these results made clear that valid query-answer amr pairs may vary widely in smatch scores, ranging from 20% to just under 80%, as found for the simple pair in figure 1 . q1. which fluids had higher concentrations of virus particles ? (h2 / have-degree-91 :arg1 (a / amr-unknown :domain (f / fluid)) :arg2 (h / high-02 :arg1 f :arg2 (v2 / virus :consist-of (p / particle :mod (v / virus) :arg1-of (c / concentrate-02)))) :arg3 (m2 / more)) a1. overall , we found higher sars-cov-2 titers from saliva than nasopharyngeal swabs from hospital inpatients. (f / find-01 :arg0 (w / we) :arg1 (h / have-degree-91 :arg1 (s3 / saliva :source (i3 / inpatient :location (h6 / hospital))) :arg2 (h5 / high-02 :arg1 s3 :arg2 (v / virus :name (n2 / name :op1 "sars-cov-2") :quant (v2 / concentration-quantity :unit (t2 / titer)))) :arg3 (m / more) :arg4 (s5 / swab-01 :arg2 (n3 / nasopharynx :part-of i3))) :mod (o / overall)) figure 4 : gold-query q 1 compared with gold-answer a 1 q 2 what is the range of sizes of respiratory droplets, specifically from coughing and sneezing? this question had two target documents with several answers that provided information on different aspects of this question. as discussed earlier, the expected infoforager output need not be a single answer, but rather might include various types of relevant information, which the experts would examine further for source and approach used to obtain that data point. answers to this question depend on, for example, the methods used to detect the sizes and numbers of droplets, and the location of measurement relative to the droplet source. because definitions of "droplet" or "aerosol" can depend upon the user, it was expected infoforager may not find all relevant answers. answers here include: a 2a and the geometric mean of droplet size of all the sneezes is 360.1 µm for unimodal distribution and 74.4 µm for bimodal distribution... @xcite . a 2b ...while sneezing produces a greater number of particles than coughing, particles from both activities are of a similar size (a sneeze produces 40,000 -4,600 particles with 80% of these particles being smaller than 100 µm compared with coughing which produced up to few hundred particles sized between 20 @xmath0. although both answers mention "size," they are more complex and distinctly phrased than the question. as a result, our system rankings for this question were weak. for the first target document, two answer sentences ranked in 55th and 128th positions (by smatch f-measure) out of 1,056 possible matches. for the second target document, one answer sentence ranked 156th of 850 possible matches, while the other received a 0 smatch score, placing it in the lowest rank where all sentences received a 0 smatch score. given that these low smatch scores could reflect poor automatic parsing preventing a match, or that the smatch algorithm does not capture the overlap seen in a q-a pair, we again examine the gold pair (goldquery vs. gold-answer) for the two answers above (to compare against the gold-query vs parsed-answer). when scored against the gold-query, the gold-answer a 2a receives an smatch f-score of 16%, whereas the parsed-answer a 1 scored significantly higher at 25.8%. for the far longer sentence a 2b , its goldanswer receives an smatch score of 10%, due in large part to its length and complex structure relative to the question. curiously, the parsed-answer a 2b fared far worse, receiving a score of 0. thus, these a 2a and a 2b scores for the gold pairs are, in a rough sense, about as challenging to interpret as the a 2a and a 2b scores comparing the gold-query to their parsed-answers. significantly more data will be needed to ascertain all the factors in the scoring, but we can see that noise alone from the automatic parse is not the primary cause of low smatch scores. q t what types of uv light are used in coronavirus research? for this user-study training question, one target answer document and the following answer sentence were identified: a t in conclusion, air disinfection using 254 nm uv-c may be an effective tool for inactivating viral aerosols @xcite . as with q 2 above, the phrasing of q t leaves open a variety of different answer types, with very different phrasing from the question itself. nonetheless, our system performed better with this q-a pair, ranking this parsed answer at the 29th position (smatch f-measure 24%) out of 406 possible matches, compared to the corresponding gold pair scoring an smatch of 18%, indicating again, that noise alone from the automatic parse is not bringing down the smatch score. to recap, we have seen that this approach using amr and smatch can be effective when an answer sentence shares structured semantics with the question. however, when the question is more open-ended, searching for "ranges" and "types," the smatch approach is much less likely to perform adequately. while these words may on occasion be present in document collections with answers that the uv researchers seek, we realize that infoforager itself must be augmented to recognize such higher-order questions and, more like a dialogue system with an intelligent agent, potentially engage the user in reformulating their question. ## related work previous research has leveraged amr within the biomedical domain for applications that, like ours, entail natural language understanding, such as information extraction. @xcite use amr in a graph kernel learning framework to extract biomolecular interactions and report that the use of amr significantly improves accuracy on this task over surface and syntax-based features, with the best performance achieved by combining amr and syntax-based features. notably, this research also furnished the bio-amr dataset of 400 manually annotated and 3k automatically annotated amrs from pubmed scientific articles, which were incorporated into retraining the amr parser used in infoforager. @xcite use amr to identify molecular events/interactions in biomedical text. the authors first define biomedical events of interest as subgraphs within amr graphs, and develop a neural network-based model that identifies such an event subgraph given an amr. while the method shows promising results, the authors find that improvements in amr parsing are needed for further improvement on the task. @xcite were the first to make use of amr embeddings, in this case along with word and dependency embeddings, to mine for otherwise hidden reports of drug-drug interactions (ddi) in the textual biomedical literature. the best performance the authors report was obtained by combining these three types of embeddings. they also noted that amr embeddings alone, leveraging the jamr parser @xcite , did not perform adequately for this ddi task, and, like others, attributed this to poor parser performance due to limited medical terms and documents in its training dataset. other research that leverages amr in tasks similar to ours has focused on question-answering. @xcite use amr as an intermediate representation for question-answering tasks designed to test an agent's understanding. the authors find that the addition of a formal reasoning layer significantly increases the reasoning capability of an agent, and that amr serves as an effective pivot from natural language to the answer set programming language used for reasoning and inductive logic. amr is leveraged for a machine reading comprehension (mrc) task for question-answering in @xcite . here, the authors first convert a q-a pair into a single amr "hypothesis" graph. additionally, the authors create an amr for an entire passage, as opposed to the standard single-sentence amr, by combining at coreference points. with these two amr graphs, the authors transform the mrc task to a graph containment problem. the authors use a max-margin approach for subgraph matching, reasoning that the hypothesis graph aligns to the passage graph where the question is best answered, not unlike the intuition in infoforager of applying smatch to find the best alignment of triples from the query amr dag to triples from sentence amr dags in a searched document. the authors obtain competitive results and point out that this approach uniquely allows them to combine evidence from multiple sentences. although they do not use the amr formalism per se, du and cardie (2020) achieve state-ofthe-art performance on the automatic content extraction (ace) 2005 benchmark event extraction task by making use of template-based questions within a bert-based question-answering model, suggesting to us that an extension to amr-based question templates might further improve their approach. the authors experimented with several template strategies for forming the questions targeting the event and its participants. most relevant to our work, the authors found that the success of their approach was strongly influenced by different questioning strategies, ranging from posing a single keyword as the question to a multi-turn progression of structured questions, and conclude that more natural language questions lead to significantly better performance. ## conclusions & future work our six-week sprint brought together researchers of different backgrounds (uv inactivation and nlp) to identify research requirements, and design and test a prototype framework with software available from other projects. we also identified facts about covid-19 now more widely known: testing saliva is more effective than nasopharyngeal swabs, given the concentration of the virus is higher in saliva. we maintain our working hypothesis that the infoforager framework is promising for facilitating a search system that allows for natural language questions, and in finding answers matching not only on keywords, but the semantic relations between those keywords. however, this process demonstrated that using smatch for graph matching does not allow us to take full advantage of the semantic structure that amr offers in this task-namely pinpointing the concept sought in an answer and its direct semantic relations to other concepts. one path we are exploring in the short-term is to have users input their hypotheses in lieu of queries. for example, for q 1 , a user might input a hypothesis like saliva has a higher concentration of coronavirus than nasopharyngeal fluid. prompting an expert user for a hypothesis could enable us to leverage smatch more fruitfully to find other graphs more similar overall to the candidate answer, following insights from @xcite . although this approach may prove useful, it runs counter to our users' stated preference for posing natural language questions, and would bias the presentation of system results towards the hypothesis graph. this approach would also be inherently limiting for non-expert users with insufficient background to formulate initial hypotheses. the longerterm solution we are exploring follows the related work mentioned above more closely, searching for subgraphs within larger passage graphs that match the query graph (as opposed to scoring the triples of a full query graph against those of a full sentence graph), with supervision from ontological and lexical resources to determine the categories of words in general that could fill certain semantic slots, thereby augmenting what constitutes subgraph matches. for example, for q 1 , we would search for matching subgraph structures where the concepts in the graph could be filled with any phrase for bodily fluids in the relationship of comparison of virus concentrations. thus, the query would be expanded to a variety of paraphrase alternates, similar to the use of hyter networks in past amr research @xcite , supporting the full range of matching answer subgraphs. these avenues for future work, in combination with our own findings, encourage us to continue this exploration of semantic search for information foraging, and engage with users to rapidly adapt capabilities for such urgent and dynamically changing situations as the coronavirus pandemic.