Cydex: Neural Search Infrastructure for the Scholarly Literature (2020)

Abstract: Cydex is a platform that provides neural search infrastructure for domain-specific scholarly literature. The platform represents an abstraction of Covidex, our recently developed full-stack open-source search engine for the COVID-19 Open Research Dataset (CORD-19) from AI2. While Covidex takes advantage of the latest best practices for keyword search using the popular Lucene search library as well as state-of-the-art neural ranking models using T5, parts of the system were hard coded to only work with CORD-19. This paper describes our efforts to generalize Covidex into Cydex, which can be applied to scholarly literature in different domains. By decoupling corpus-specific configurations from the frontend implementation, we are able to demonstrate the generality of Cydex on two very different corpora: the ACL Anthology and a collection of hydrology abstracts. Our platform is entirely open source and available at cydex.ai.

## introduction the ongoing worldwide covid-19 pandemic has brought about a resurgence of interest in natural language analysis applied to the scientific, particularly biomedical, literature. this has been catalyzed by the availability of corpora, such as the covid-19 open research dataset (cord-19) curated by the allen institute for ai @xcite , as well as substantial efforts in the creation of evaluation resources, such as the trec-covid challenge @xcite , which has constructed a test collection on information needs related to covid-19. our project builds on covidex, a full-stack opensource neural search engine for cord-19 that includes three main capabilities: basic keyword search, neural ranking models, and a faceted search and browsing interface. an early description of our project can be found in @xcite and an updated paper appears in this workshop @xcite . despite the successes of covidex, parts of the system remain hard-coded to work only with cord-19. we saw an opportunity to develop our code base into general neural search infrastructure that can be deployed on different corpora of scholarly articles. the contribution of this work is the execution of this vision: we introduce cydex, which abstracts covidex to provide neural search capabilities to scholarly literature in different domains. throughout this paper, we use the terms "domain" and "corpus" interchangeably, as the most precise way to define a particular domain is by a collection of texts that capture the domain. our platform is demonstrated on the acl anthology and a collection of abstracts in the hydrology domain. all of the components described in this paper are open source. ## from covidex to cydex this section begins with a description of covidex, and then describes how we have adapted and generalized each layer in its stack to create cydex. ## case studies to demonstrate the effectiveness of abstractions within cydex, we have built and deployed custom instances on two corpora: although cydex has been verified to work with both corpora, here we focus on the acl anthology. a screenshot is shown in figure 1 . ## evaluation and discussion the focus of our efforts, and the contribution of this work, is search infrastructure to support information access to scholarly literature. however, it is certainly fair to inquire about the quality of the search results produced by covidex (and cydex by extension, since there have been no changes to the t5 ranking models in pygaggle). we have yet to conduct formal domain-specific evaluations, either in the acl or hydrology contexts, and can only point to covidex results based on participation in the multi-round trec-covid challenge @xcite . as detailed in @xcite , the covidex team submitted the best automatic runs in the final two rounds using a combination of techniques that included ranking with t5. note that the covidex ranking model operated in a zeroshot setting, and there is independent evidence that transformer-based ranking models have powerful cross-domain relevance transfer capabilities @xcite . thus, we hope that the ranking models would generalize to the nlp and hydrology domains as well. despite the lack of domain-specific evaluation data, we performed our own informal evaluation of cydex on the acl anthology. our evaluation can be characterized as informal "hallway usability testing", primarily as a sanity check. we asked four colleagues (who were not the co-authors) to compare the top five results of five different natural language questions between cydex and the acl anthology's current site-specific google search. all of our colleagues are familiar with the nlp literature. we came up with the test set of five natural language questions based on our own interests. using a standard evaluation methodology, the human assessors were presented side-by-side results (from the two systems) and asked which one (the left system or the right system) they liked better. the identity of the two systems were blinded and randomized, so the assessors had no way to determine the source of the results. all four asses-sors preferred the results of cydex for at least three out of the five test questions. needless to say, there are not enough assessors or questions in this simple evaluation to draw any meaningful conclusions, although these results suggest that our system does not appear to be obviously worse than google. we consider this to be an encouraging outcome, given that google presents a high bar in terms of search quality, and we have only begun to build cydex. more broadly, however, there are obvious questions worth addressing about the premise of our endeavor: why do we even need cydex? why do we need another search engine to the scholarly literature? isn't google scholar sufficient? while undoubtedly valuable and certainly the most widely used search engine for the scholarly literature, it would be far-fetched to think that any research project can displace google scholar. however, a failure by academic researchers to also engage in the space would be implicitly ceding this important intellectual territory to a commercial search engine that is, at its core, not transparent and controlled by a single entity that may not necessarily act in the best interest of scholars. our project does not aspire to be a comprehensive guide to the literature like google scholar or ai2's semantic scholar. rather, our niche is domain-specific "verticals" that are manageable from the scale perspective, yet sufficiently large to support interesting analyses. our two illustrative case studies of the acl anthology and hydrology abstracts fit this bill exactly: our software stack can run with only modest resources, and both these corpora are sufficiently rich and self-contained to answer interesting questions-see, for example, recent analyses of the acl anthology by mohammad (2020) and the work by @xcite on the collection of hydrology abstracts. ## ongoing work and conclusions there are two main directions we are currently pursuing as part of ongoing work on cydex. the first is expansion of the platform to new domains, in particular examining the scalability of our underlying infrastructure. both the acl anthology and the corpus of hydrology abstracts are tiny by modern standards; even cord-19, with around 300k articles (as of october 2020), is small compared to many information retrieval test collections. a worthwhile target would be the recently released kaggle arxiv dataset, foot_3 which contains over 1.7m preprints, in areas ranging from physics to the many subdisciplines of computer science. the effectiveness of our ranking algorithms is naturally another area of interest, although the development of neural ranking models is orthogonal to the infrastructure that we present here. our group's latest work on a sequence-tosequence ranking formulation using t5 is discussed in @xcite . while we seek to further improve these core models, in the context of cydex we are more interested in the end-to-end user experience, where ranking is just one (albeit important) aspect. beyond features such as faceted browsing and the highlighting of relevant content (already implemented), the platform could benefit from the integration of additional capabilities such as citation recommendation @xcite and related article browsing @xcite . these features all contribute to users' perception of system quality and must be evaluated holistically, for example, via user studies. cydex represents the starting point of a platform for building information access capabilities for domain-specific scholarly literature, powering our own explorations into scientific literature anal-