Developing a Curated Topic Model for COVID -19 Medical Research Literature (2020)

Abstract: Topic models can facilitate search, navigation, and knowledge discovery in large document collections. However, automatic generation of topic models can produce results that fail to meet the needs of users. We advocate for a set of user-focused desiderata in topic modeling for the COVID-19 literature, and describe an effort in progress to develop a curated topic model for COVID-19 articles informed by subject matter expertise and the way medical researchers engage with medical literature.

## introduction the language technology community has been responding in force to the coronavirus pandemic with sustained energy and creativity, and the medical research literature, facilitated by the cord-19 dataset, is a major hub of activity @xcite . a crucial goal of all this technological effort is to support non-technologist users, namely the medical researchers, clinicians, and policy makers who are involved with the response to covid-19. however, to date there has been little explicit discussion in the technology-development community about what functionalities will actually help these users in the trenches. @xcite sum things up nicely when they write,"we don't actually know how our systems . . . can concretely contribute to efforts to tackle the ongoing pandemic until we receive guidance from real users who are engaged in those efforts . . . @xcite how to build improved fire-fighting capabilities for tomorrow without bothering those who are trying to fight the fires that already raging in front of us". in this paper we describe a cross-disciplinary collaborative effort that is intended to help close that gap by developing a curated topic model for covid-19 medical research literature, informed both by subject matter expertise about the do-main and by the way that medical researchers typically engage with medical literature. our efforts make use of a human-in-the-loop platform for topic model development informed by the research literature on interactive topic modeling as well as by practical experience developing topic taxonomies for large scale document collections. the intent is not a new algorithmic contribution; rather, at its heart this is a position paper advocating for a needs-driven rather than technology-driven approach in nlp when tackling crucial real-world problems. we articulate and discuss what that looks like specifically for topic modeling of the covid literature, and we offer for discussion our work in progress toward public release of a curated model that can be used downstream by the community as a resource and as a starting point for further work. ## desiderata the core idea in topic modeling is to take in a collection of documents, discover a latent set of topics characteristic of the collection, and represent each document as a mixture of those topics. typically each latent topic is itself represented as a distribution over the collection's vocabulary. latent dirichlet allocation @xcite is the ancestor of a whole variety of topic model variants, adding hierarchical topics, supervision, temporal structure, discrete covariates, and more, as well as more recent neural variations. in most applications of topic models, the topic-word distribution -i.e. the representation of each topic by its most probable n words, for small n -provides human consumers of the topic model with intuitions about what each topic is about, as a way of understanding the contents of the collection. however, automatically discovered topic models are often subject to noise and poor quality @xcite . in addition, the usual products of topic modeling can be a poor fit for real-world use cases involving non-technical subject matter experts (smes) within a particular domain. from our own experience and discussions with medical researchers, we have identified a set of desiderata for topic modeling in the context of the covid-19 literature. we find that these observations are largely in line with the principles for automated content analysis articulated by @xcite for the social sciences. in particular, they emphasize the role of automated analysis in expanding human capabilities, that models should be evaluated not intrinsically but in terms of their ability to support scientific goals, and they advocate strongly against the use of automated models without some form of validation. these observations are also consistent with blei's (2014) discussion of "the craft of latent variable modeling" and his adaptation of "box's loop" @xcite as an iterative picture of model development, although his discussion is focused much more strongly on the role of modelers than of subject matter experts. we envision many uses for a curated topic model, but our driving use case involves the way many if not most medical researchers engage with the literature. a typical researcher is intimately familiar with boolean searches in pubmed, including use of wildcards (e.g. bacteri * ), quoted multiword phrases (e.g. ''white blood cell''), fielded searches (e.g. aromatherapy[tiab] specifying to look in the articles' title, abstract, or keywords), and mesh taxonomy headings and subheadings (e.g. hypertension[mh] and toxicity @xcite ). when reviewing search results, users will typically look at the article list and then, for a specific article, it is extremely common to drill down by looking at lists of articles surfaced by pubmed as similar to and/or citing the current article being looked at. clicking through to other articles can be interleaved with new or refined boolean searches. the overall picture here is a very structured kind of search. in their day-to-day experience navigating the medical literature, it would be a fairly drastic shift from this way of doing things to typing in full-sentence questions in the style of questionanswering systems or queries in the style of web search engines. when thinking about topic models, this motivates our thinking less in terms of, for example, improvements to ranking or visualizing topic-based clusters, and more in terms of discrete topic labels and how they could be introduced into the user's familiar, discrete experience, more anal-ogous to mesh terms. we would anticipate that one particularly valuable use of curated, labeled topical categories will be in helping researchers to overcome information overload when navigating results. for example, augmenting documents in the collection with discrete topic category labels would make it possible to organize related-article and citing-article lists into subcategories, and to include topical categories in follow-up searches. ## human-in-the-loop process we are in the early stages of using an interactive, human-in-the-loop topic modeling platform to produce a curated topic model. the process we will be following is informed by prior experience using this platform to develop curated taxonomies for large document collections. 1 we begin with preprocessing, including conventional steps such as tokenization (including identification of relevant multi-word expressions), lowercasing, removal of stopwords, and down-selection of the vocabulary to high-value words based on frequency and other statistical properties. foot_1 this is followed by creation of multiple initial topic models of differing granularities. we do not optimize the number k of topics automatically, since doing so typically relies on automatic approximations to human judgment such as normalized pointwise mutual information (npmi, @xcite . rather, we will use human judgments directly by constructing models across a range of k and assessing how promising each model is as a starting point, via a combination of qualitative assessment and by comparing human quality ratings for a random sample of topics as assigned by two independent smes. having selected an initial starting point, the human-in-the-loop process includes drilling down to better understand the model (including, for example, identifying documents that are highly representative of a given topic, or visualizing topic similarity), interleaved with human-feedback operations of the kind investigated by @xcite and @xcite . for example, they discuss within-topic feedback such as removing or adding words as good signals for a topic, as well as modellevel feedback such as merging equivalent topics or removing topics that are too incoherent to be worth refining. topics are also assigned labels manually, and labels can be revised at any time as the sme's understanding of a topic evolves. this feedback is provided in batch, and then the model is recomputed using the feedback to provide inductive bias. after the recalculation, the sme inspects the resulting model, and can either continue another iteration of refinement or designate the model as final. typically two to four iterations of refinement using this process are sufficient to produce a high quality model. once a final model has been produced, we will evaluate its quality using multiple smes in epidemiology or clinical infectious diseases. we plan to use both subjective topic coherence ratings on a likert scale @xcite and word intrusion @xcite , and to look not only at summary measures of agreement (e.g. correlation) but at specific sources of disagreement. ## preliminary analysis we conducted a preliminary analysis of topic modeling using the titles and abstracts in the may 1, 2020 release of the cord-19 dataset (metadata.csv). 3 this exploratory modeling used spacy (spacy.io) for tokenization and identified phrase chunks in preprocessing as meaningful semantic units @xcite . although spacy's phrase chunking performed very well, mimno observes that "for text with lots of technical terms, a carefully curated list of multi-word terms can make a huge difference", and in subsequent exploration of initial models we will integrate covid named entity resources to identify biomedical multi-word expressions, e.g. entities in @xcite . consistent with prior experience, we have found that adding multi-word tokens to a document's representation in preprocessing, rather than replacing unigrams, tends to yield topics that are more useful for smes, even though this does violence to the independence assumptions inherent in the typical topic model's generative process. for example, in a document containing the phrase chronic headache, we would include all of chronic headache, chronic, and headache as tokens. this can introduce some redundancy, but it also allows the model itself to determine, for a given topic, whether (for example) chronic headache or headache is the appropriate level of abstraction for the topic in the context of other terms. in our preliminary analysis, we constructed and inspected initial models with @xmath0, 100, 150 topics. on drilling down into the models, it became apparent that the collection contains a high proportion of articles that had been retrieved by the cord-19 dataset search terms but were not directly relevant to covid-19. on further inspection, we have come to the conclusion that from an epidemiological and public health perspective, a cleaner and more valuable set of curated topics for the covid-19 research literature is likely to be created if the inclusion criteria for the document collection are stricter, limiting to articles sufficiently "about" covid-19 by creating a filter based on the inclusion search terms used in creating the cord-19 dataset @xcite , further tuned to focus on novel (2019) coronavirus, not all previous coronavirus research in general. 4 in addition, as is common in english-dominated but multi-language collections, we have found in our preliminary analyses that modeling tends to produce topics aggregating words in non-english languages. as a simple way to address this issue, we have determined that a simple heuristic filter for english documents works well: requiring that the title or abstract contain at least one of the most common words in english. 5 of the roughly 3000 items filtered using this heuristic (about 5,500 in the june 7 release), only a tiny number include abstracts in english. in addition to non-english content, the heuristic picks out some articles where a title is provided but the abstract is empty; technically speaking these are false positives for the 4 the reasoning here is that, within a given "budget" k, it is important to provide sufficient granularity for covid-19-specific topics. now that we have progressed beyond the earliest days of the epidemic, well-developed subareas are emerging in the covid-19 medical research literature, but casting the net broadly enough to include all prior coronavirus discussion appears to drown out themes related to the novel coronavirus more than it helps capture themes that are relevant to it; for example, most of those studies are in vitro with at best a tangential relationship to our human coronavirus. retaining the sars-and mers-related search terms ensures that relevant pre-2019 literature related to human coronaviruses is still included. our initial impressions of topic granularity suggest that @xmath1. this is consistent with our prior experience with human-inthe-loop modeling for collections with tens of thousands of documents, where we have found that 100-150 topics was a good initial starting point. for the interactive refinement process, it makes sense to err in the direction of fine-grained topics in the initial model, since our refinement process makes it easy to whittle down the set by identifying, sharpening, and merging topics that cover the same conceptual territory and by removing manifestly incoherent topics. to provide an impression of the topic model obtained in our initial process we show three randomly chosen topics from the 100-topic model for the may 1 release, noting that initial model results are expected to change once we implement the stricter inclusion criteria and medical entity processing (and of course reiterating our own prior cautions about looking only at highly ranked topic words): ## conclusions we have argued for the importance and utility of a curated topic model for covid-19 medical research literature, proposed criteria that such modeling should satisfy, and we have provided a preliminary sketch of how we are planning to construct a curated model meeting those criteria. this is work in progress, but preliminary analysis suggests that initial topic models are high quality, we have identified several steps for improving initial automatic modeling, and our process of human-in-the-loop refinement is designed to further refine and curate a model whose content will be driven both bottom-up by the data and top-down by subject matter expertise. one important take-away from our preliminary investigation is the importance of careful sample selection as guided by the end goals. this consideration is entirely de rigeur in content analysis as practiced in other disciplines @xcite , but within nlp there is often a tendency to work with "found data", using it in its entirety. we find that for the goals we are pursuing here, more data is not necessarily better data. at the end of our curated model development process, we will make publicly available topic-word distributions with corresponding labels, document-topic distributions for the input documents, the vocabulary, and a script that enables preprocessing consistent with ours for new documents. this will provide the community with materials needed to visualize the topics and documents, to manually organize topics into a hierarchy, to compute topic posteriors for new documents, to treat a document's most-prevalent topic(s) as discrete labels, and to integrate topic-labeling of documents into users' experience in search engines. more generally, one of our key aims is for this curated model to serve as a useful starting point for further model development by the community, e.g. using our resource to construct informative priors in development of further models. this may help others in bootstrapping work on this dataset using generative models, but it also represents one reasonable answer to the question of how to deal with a literature that is rapidly and continuously evolving, namely starting with the curated model as m 0 and then periodically performing inference on an updated document collection at time t using m t-1 to define a prior. related work along these lines includes, among others, lexical "seeding" of topics @xcite , treating the literature as a streaming document collection @xcite , and dynamic topic models @xcite . the best way to introduce additional curation over time remains an important question for future work. the seriousness and urgency of covid-19 re-quires the creation of new knowledge on an enormous scale. paradoxically, though, the deluge of work on this subject, producing more and more knowledge, is itself an obstacle to progress. clinicians, medical researchers, and policy makers cannot read everything -they need better ways of making sense of what's out there, organizing it, and navigating in the directions that will help them. technology that is focused on their needs can help.