A Natural Language Processing System for National COVID-19 Surveillance in the US Department of Veterans Affairs (2020)

Abstract: Timely and accurate accounting of positive cases has been an important part of the response to the COVID-19 pandemic. While most positive cases within Veterans Affairs (VA) are identified through structured laboratory results, some patients are tested or diagnosed outside VA so their clinical status is documented only in free-text narratives. We developed a Natural Language Processing pipeline for identifying positively diagnosed COVID19 patients and deployed this system to accelerate chart review. As part of the VA national response to COVID-19, this process identified 6,360 positive cases which did not have corresponding laboratory data. These cases accounted for 36.1% of total confirmed positive cases in VA to date. With available data, performance of the system is estimated as 82.4% precision and 94.2% recall. A public-facing implementation is released as open source and available to the community.

## introduction a robust pandemic response is contingent on timely and accurate information @xcite . during the covid-19 pandemic, public health institutions have established surveillance systems to monitor and track case counts over time. covid-19 is typically diagnosed using laboratory tests. the test results are frequently used as a source for surveillance systems. however, such systems typically only capture laboratory results from the same healthcare system. patients may also be diagnosed with covid-19 in the community, such as in external hospital networks or drive-through testing. these patients may potentially be missed by laboratory-based surveillance methods, leading to these patients not being represented in overall case counts. patient health information needed for biosurveillance is often recorded in free-text narratives in the electronic health record (ehr) @xcite , offering an alternative source of covid-19 status when structured lab evidence is absent. in this work we developed a natural language processing (nlp) system to extract potential positive covid-19 cases from clinical text within the department of veterans affairs (va). following review by a clinical expert, positively identified patients are included in official va surveillance counts. since the va ehr includes data from hospitals and clinics across the united states, this system enables a unique capability for collecting data for national surveillance purposes. ## background manual information gathering draws effort away from patient care priorities and can impede timely and effective responses to public health threats. automated approaches for processing clinical notes have been applied for public health purposes when data is needed as quickly as possible. @xcite developed an automated syndromic surveillance system using clinical text to identify anomalies in symptoms as rapidly as possible. several examples in the literature have utilized clinical text including chief complaints to perform early detection of infectious disease @xcite @xcite @xcite . typical data sources for covid-19 surveillance include government announcements, scientific publications, and news articles @xcite ) most literature to date for nlp related to covid-19 has involved public data sources such as research publications @xcite . others have examined social media sources including twitter to examine sentiment or misinformation related to the virus @xcite . in this work, the objective was to identify the diagnosis of covid-19 in clinical documents to report complete case counts of the disease for public health surveillance in va. ## discussion in this work we described the development and application of a natural language processing system for covid-19 surveillance in a national healthcare system in the united states. we demonstrated that nlp combined with clinical review can be leveraged to improve surveillance for covid-19. within the va surveillance system, over one third of total known cases were identified by nlp and clinical review, with the remainder being identified through structured laboratory data. this capability validated that nlp can provide significant value to such a surveillance system, which requires a timely and sensitive case count. our system achieved high recall while still maintaining acceptable precision. leveraging a rule-based system allowed defining narrow and specific criteria for what is extracted. rules were iteratively developed to filter out irrelevant documents while still identifying positive cases. additionally, the flexibility of a rule-based system allowed us to add new examples and adapt to new concepts as they emerged. this was critical in the covid-19 response, as the pandemic remains a dynamic and evolving situation. for example, the terms covid-19 and sars-cov-2 were not announced until weeks after the surveillance system had been deployed, but requirements dictated immediate addition to our system. similarly, changes in the clinical documentation such as new clinical concerns and semi-structured template texts required quick response and modification. due to the continuously changing nature of covid-19, we required a system which permitted rapid and flexible development. while other mature clinical nlp systems exist, such as ctakes and clamp @xcite , we elected to develop this system using the features and flexibility of the spacy framework. rapid iteration permitted reviewing documents for errors, directly making changes to rules, and then evaluating them without compiling or reloading. visualizations such as figure 1 were useful to troubleshoot rule development and understand the linguistic patterns. one limitation of this work is the evaluation of system performance. our primary objective in this effort was to serve veterans and provide complete public health reporting. the goal of chart review was to identify all positive patients rather than to create a reference set. precision and recall metrics presented here are estimates using sampling and available structured data. in future work, we plan to evaluate machine learning methods to improve identification of positive cases. a machine learning classifier could potentially improve our current system by improving document classification accuracy and identifying high-probability cases for review. this was not feasible in early stages of the response since there were very few known cases and no existing reference set. we have now identified thousands of possible cases which could be included in a training set for a supervised classifier. however, as stated previously, our clinical review did not equate to creating a reference set. specifically, clinical reviewers did not always assign negative labels to reviewed cases which would be needed for training a supervised model. however, we believe that with additional validation and review, a machine learning classifier has the potential to augment our system's performance. ## conclusion we have developed a text processing pipeline and utilized it to perform accelerated review of covid-19 status in clinical documents. this approach was dynamic and allowed us to adapt to an evolving situation where vocabulary and clinical understanding continued to emerge with high data volume. rapid implementation and iteration permitted reaction to shifting clinical documentation and evidence. this pipeline accelerated review of patient charts such that 36.1% of confirmed positive cases in a va surveillance system were identified using this capability.