The elusive goal of a learning healthcare system requires accurate clinical information. Structured 48 healthcare data, although increasingly accessible, suffers from biases ubiquitous in observational 49 data, including misclassification bias. In particular, diagnostic codes, widely used for billing and 50 research, typically have low sensitivity and specificity 1,2 . When used to establish an association 51 of exposures to a clinical outcome, such limitations can lead to misclassified outcomes and 52 interfere with adjustment procedures for confounding and selection bias 3 . 53 161 history shows robustness of results 162 The past medical history of heart failure used for propensity-score matching was derived from an 163 aggregate use of ICD-10 codes from problem lists, encounter diagnoses, medical history, and 164 admission diagnoses. Given concern for inaccuracy of structured data, our finding that RSV 165 infection was associated with higher rate of heart failure hospitalization may have been affected 166 by the definition of HF history (which has large impact on this outcome). To overcome this concern, 167 we first sought to evaluate the accuracy of the ICD-10 code-based approach to define HF history. 168
To this end, we performed a systematic effort of labeling 863 patients and found that the ICD-10 169 approach nonetheless had a good accuracy with sensitivity 0.80 and specificity 0.97 ( Figure 4A) . 170
Although the performance was good, the use of structured data typically leaves no flexibility to 171 explore robustness of the result to different thresholds. We were particularly concerned about the 172 impact of a reduced sensitivity. Therefore, we developed a neural network-based approach for 173 learning past medical history at a given point in time using a neural network-based strategy. We 174 trained three separate models: a model for note-informativeness, a model for heart failure history 175 classification, and finally a time series model that used an aggregate of note scores across time. 176
The final model had an excellent discriminative ability with ROC-AUC of 0.97 ( Figure 4A) . 177
Although the accuracy at the mandatory point of comparison was not superior to the ICD-10 code-178 based definition, the model enabled evaluating robustness of the results to the selected cutoff 179 point. We assessed conclusions using cutoffs with high sensitivity, high specificity and a balanced 180 sensitivity/specificity ratio. Different cutoffs gave different absolute rates of heart failure history 181 (Table S26) . Nonetheless, propensity score matching using model-based heart failure history was 182 successful with SMD<0.1 for all baseline values (Table S27- CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) preprint
The copyright holder for this this version posted November 16, 2020. ; https://doi.org/10.1101 https://doi.org/10. /2020 Discussion 188 Here we outline two primary contributions; 1) machine learning models for automated event 190 adjudication for cardiovascular events and past medical history and 2) application of these models 191 to estimate the risk of delayed heart failure hospitalizations in individuals with respiratory virus 192 infections. In this context, we observed an estimated 23% higher risk of delayed heart failure 193 hospitalizations in individuals infected with RSV compared to those who tested negative but no 194 change in the risk for such events following influenza infection. 195
Our work labeling a large collection of notes across over 17 institutions within an integrated 196 delivery network adds further support to the limitations of diagnostic codes for describing 197 cardiovascular events 2 . The accuracy of diagnostic codes varied strongly with the type of event: 198 at one extreme, ischemic stroke was very well captured whereas there were no clear diagnostic 199 codes for coronary revascularization, and, at least within our database, the use of CPT codes had 200 very low sensitivity. Reimbursement incentives are known to shape the completeness of 201 diagnostic codes 16 and there is also further variation from center to center based on provider and 202 institutional coding practices. Nonetheless, a systematic labeling approach can provide local 203 estimates of the adequacy of diagnostic code for any specific application, and further simulation-204 based modeling can be used to understand the impact of misclassification bias on any 205 downstream uses of the data. 206
Although we found that a historical compilation encompassing encounter diagnoses, admission 207 diagnoses, problem lists, and medical history is highly accurate, at least for heart failure, the 208 sensitivity was relatively low compared to specificity, which is in agreement with previous studies 209
showing that ICD-10 codes for past medical history suffers primarily from problems of sensitivity 17 . 210
The ICD-10 code-based approach forces a single sensitivity/specificity and does not allow 211 adjustment or optimization. In comparison, our model-based definition can provide various 212 sensitivity/specificity ratio by adjusting the threshold. This feature enabled us to perform a 213 sensitivity analysis showing that our results are robust to the sensitivity/specificity of the past 214 medical history detection. Furthermore, we anticipate that our approach should generalize well to 215 capturing conditions for which there are no diagnostic codes (such as inadequate response to a 216 medication). 217
In terms of prior work, most of the focus on EHR-based automated diagnoses has been on the 218 use of heuristic combinations of diagnostic codes, medications, labs, and regular expression 219 matching within unstructured data 2 . Recent advances in convolutional neural network-based text 220 processing have motivated their application to the problem of note classification 18,19 . Our own 221 experience with this process emphasized the need for both domain (i.e., cardiovascular disease) 222 and data science expertise, with an iterative examination of failure cases, augmented labeling, 223 and retraining of models. 224
Our models enabled scalable analysis of millions of discharge summaries and progress notes. 225
We see multiple applications of such an approach, whether it be for pharmacovigilance, quality 226 control, or patient selection for a clinical trial according to criteria that may be poorly reflected in 227 structured data. For our specific application, we were able to conclude that influenza virus 228 infection does not appear to contribute to any delayed increase in heart failure hospitalization, 229 while RSV infection contributes a 23% increase in risk that is apparent across an entire four-year 230 period. 231
Prior work focusing on respiratory viruses have emphasized the acute cardiovascular 232 manifestations of viral infections in a limited subset of patients 5,20 . In the case of RSV, it is well 233 recognized that patients with RSV infections have a higher burden of cardiovascular disease, 234 although this may reflect selection bias, given that cardiovascular consequences of viral infection 235 may add to pre-existing disease to drive presentation to healthcare systems. However, our results 236 are striking in that even controlling for heart failure diagnoses prior to and up to 90 days after the 237 incident event, we still see an increased risk of future events. While previous reports show 238 increased risk of cardiovascular events in patients with influenza infections, they focus entirely on 239 acute events in the context of the infection 6-8 . Our analysis excludes acute events and thus, does 240 not contradict these findings. It is entirely plausible that influenza virus increases heart failure 241 hospitalization in the acute phase, either through direct myocardial involvement 21 or indirectly 242 through a systemic inflammatory response as with many other infections 22 , but patients may not 243 have prolonged damage to the heart. 244
The primary limitation of our work is the unknown contribution of unmeasured confounders with 245 significant impact on both heart failure hospitalization outcomes and probability of positive test for 246 viral infection. Since our analysis focused only on those who had been tested for the virus, these 247 population likely had symptoms suggestive of a cardiopulmonary illness. Thus, the negative 248 results for the virus results may be consistent with other reasons for having respiratory symptoms. 249
In the case of influenza virus, we observed a classic example of collider bias requiring propensity 250 score matching approaches 23 . Nonetheless, such approaches inevitably can suffer from model 251 misspecification. However, it should be noted that possible approaches to address these 252 . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) preprint
The copyright holder for this this version posted November 16, 2020. ; https://doi.org/10.1101 https://doi.org/10. /2020 limitations such as randomizing patients to be infected by a virus or performing a cohort study 253 testing every individual regardless of the symptoms are not feasible. 254
An open question is whether our work has relevance to other respiratory viruses, such as SARS-255 COV2. Although SARS-COV2, like influenza 21 and RSV, can cause myocarditis [9] [10] [11] 24 , there is no 256 clarity on the prevalence of this effect given the selection bias in published reports. Nonetheless, 257
it is reasonable to assume the possibility of long-term cardiac consequences for SARS-COV2, 258 that will only be addressed with multi-year surveillance and methodologic approaches such as 259 our own. 260 261 262 263 264 . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) preprint
The copyright holder for this this version posted November 16, 2020. ; https://doi.org/10.1101 https://doi.org/10. /2020 The test set, consisting of 1,003 discharge summaries, was selected randomly from patients 276 above who had undergone electrocardiograms within the centers participating in the EDW. 277
Following previously published guideline 13 , one of 4 cardiologists labeled each discharge 278 summary for 4 possible diagnoses: acute coronary syndrome, heart failure as the primary cause 279 of hospitalization, percutaneous coronary intervention or coronary artery bypass surgery 280 (PCI/CABG), or ischemic stroke. 281 282 We used SpaCy v2.2.4 25 to tokenize medical notes. We extended the English language model 283 (en-core-web-large) tokenization rules by adding a comprehensive set of abbreviations that are 284 commonly used in medical records. We used fasttext v0.9.2 26 to train 100-dimensional bigram 285 word embeddings with the parameters `fasttext skipgram -wordNgrams 2`. Word embeddings 286 were trained on a compendium of more than 10 million clinical notes ranging from 2002 to 2020, 287 based on more than 330,000 subjects, with up to 50 notes per subject. 288 289 The event adjudication model was constructed with the combination of 1D-CNN and long-short-290 term memory (LSTM). It consisted of a word embedding layer, which converts words into word 291 vectors, followed by a 1D-CNN and LSTM combination network (schematic shown in Figure S1 ). 292
The model was trained using data from derivation dataset. Each discharge summary was labeled 293 as positive or negative for 4 outcomes of interest (heart failure hospitalization, PCI/CABG, ACS 294 and ischemic stroke) and 4 separate models were trained for each outcome label. The model was 295 trained to minimize the binary cross entropy between model prediction and the label using 296
RMSprop optimizer with initial learning rate of 0. 0001 309 To evaluate the influence of viral infection on 3 cardiovascular events (heart failure hospitalization, 310 urgent revascularization and ischemic stroke), we performed a survival analysis for these 311 outcomes. Due to limited numbers or follow-up length for other viruses, we focused our analysis 312 on influenza and RSV. To avoid possible bias caused by the COVID-19 outbreak in Boston area, 313 all the follow up were censored at February 2020. All viral test results between February 2016 314 and February 2020 were identified. Positive patients were identified as those having at least one 315 positive result for the virus of interest within the study period and the date of first positive result 316 was used as the index date. Patients who had the test for the virus of interest but did not have 317 any positive results were identified as a control population within that cohort. Index dates were 318 identified as the date of first viral test for control patients. All patients who had other viral infections 319 before the index date were excluded. 320
Outcome events were defined bases on the event text model predictions for heart failure 321 hospitalization and urgent revascularization and ICD-10 codes for ischemic stroke. The cutoffs 322
for the events were selected as the value that gave the best sensitivity at a specificity over 90%. 323
Urgent revascularization was defined as positive results for PCI/CABG and ACS for the same 324 discharge summary. All 3 outcome events were analyzed independently. Patients were censored 325 at the first event, death, last encounter in the system or the end of study period, whichever 326 occurred first. To account for confounders, we used a propensity-score matching method 15 . A 327 . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) preprint
The copyright holder for this this version posted November 16, 2020. ; https://doi.org/10. 1101 /2020 propensity score for a positive test was calculated with age, sex, race, smoking status, history of 328 heart failure, history of coronary artery disease, history of atrial fibrillation, history of type 2 329 diabetes mellitus, history of COPD, history of hypertension, history of dyslipidemia, use of loop 330 diuretic, use of aspirin, use of P2Y12 inhibitors and the method for viral test using logistic 331 regression. In this analysis, the past histories were identified using ICD-10 codes. The between 332 group difference in baseline characteristics was assessed by the standardized mean difference 333 (SMD) where a value under 0.1 is considered negligible. Patients with positive viral results were 334 matched to controls (without replacement) using a nearest neighbor method, as implemented in 335 the MatchIt package (version 3.0.2) in R (version 3.5.1) 27 . The matching ratio was set to the 336 highest number that did not introduce substantial difference in any baseline characteristics 337 (SMD<0.1). 338
Since past medical history is defined using a cumulative record and not on a single note, we took 340 a 2-step approach for training the history model. Our first step focused on classifying the note. 341
Since some notes are totally unrelated to the history of disease (such as nutrition notes), we 342 trained a model to classify if the note is informative for a specific history or not and a second 343 model to actually classify if the history is positive or negative. To this end, each note was initially 344 labeled with 3 labels (positive/negative and uninformative). During labeling, we also noticed that 345 there were some notes that stated an ambiguous condition (e.g. extremely low ejection fraction 346 without noting if the patient have symptoms or not for heart failure). We labeled these by a 4th 347 label (ambiguous) and excluded these from the training data. Two sets of notes were used for 348 model training and testing from the MGB-EDW database where the training set were partially 349 enriched with the use of ICD10 codes heart failure and the test set selected randomly. The final 350 numbers of notes were 819 for training set and 365 for the test set. The note-level models resulted 351 in note-level accuracy of AUC 0.93 for informativeness and 0.93 for the presence of heart failure 352 history ( Figure S2A ). 353
The second step focused on combining all the note-level output to identify the past medical history 354 status for each patient for a specific date. To this end, we built a time-series neural network-model 355 that takes note-level prediction for informativeness and past medical history status along with the 356 dates of the note relative to the index date as input. The model produces a single binary output 357 reflecting the probability of the patient having the disease history at the index date ( Figure S2B) . 358
For this model, we labeled a separate patient dataset (with no overlap with the note level history 359 model patients) consisting of 561 patient-date pairs for training and 305 patient-date pairs for 360 . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) preprint
The copyright holder for this this version posted November 16, 2020. ; https://doi.org/10.1101/2020.11.12.20230706 doi: medRxiv preprint testing. The patients in the training datasets were partially enriched for past medical history using 361 the event adjudication model to select patients who had an event before the index date. In contrast, 362 the patients in test set were randomly selected from a group of patients 60 years and above who 363 had received an electrocardiogram at MGB. The labeling was done by reading all the notes 364 associated with that patient to identify if the patient had a positive history of the disease at the 365 index date. The input data was constructed as time-series data of the note-level output for all the 366 notes within ± 2500 days of index date. Each output value was inserted into a vector of length 367 5000 (each element corresponding to the date relative to index date). The neural network was 368 trained to take this vector as input and a binary 1/0 value as output. 369 370 To evaluate if the results of our survival analysis was sensitive to the sensitivity/specificity of the 371 past medical history detection method, repeated the survival analysis for heart failure 372 hospitalization using the heart failure history defined based on the time-series history model with 373 various cutoff points. We used 3 cutoff points for this analysis, each of which giving (1) highest 374 specificity with sensitivity ≥ 0.95, (2) sensitivity and specificity both over > 0.90, (3) highest 375 sensitivity with specificity ≥ 0.95. The cutoffs were 0.24 (sensitivity=0.96, specificity=0.83), 0.41 376 (sensitivity=0.90, specificity=0.93) and 0.43 (sensitivity=0.68, specificity=0.98). The propensity 377 score matching was re-done using the model-based heart failure history instead of ICD-10 code-378 based heart failure history. 379 380 381 . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) preprint
The copyright holder for this this version posted November 16, 2020. ; https://doi.org/10.1101/2020.11.12.20230706 doi: medRxiv preprint 
. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) preprint 
. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) preprint CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) preprint
The copyright holder for this this version posted November 16, 2020. ; https://doi.org/10.1101/2020.11.12.20230706 doi: medRxiv preprint
A large compendium of cardiologist-labeled discharge summaries and progress notes were used to train four cardiovascular event adjudication models and a heart failure history model using neural networks. The models were deployed on 17 million notes for 267,596 patients to evaluate the impact of respiratory virus infections on delayed cardiovascular events. Receiver operating characteristics curves (left) and precision-recall curves (right) for event adjudication models for PCI/CABG, ACS, heart failure hospitalization and ischemic stroke in the test dataset. Light blue area indicates 95% confidence interval and the pink diamond indicates the sensitivity and specificities of the ICD-10 code-based diagnosis. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) preprint
The copyright holder for this this version posted November 16, 2020. ; https://doi.org/10.1101/2020.11.12.20230706 doi: medRxiv preprint is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) preprint
The copyright holder for this this version posted November 16, 2020. ; https://doi.org/10.1101/2020.11.12.20230706 doi: medRxiv preprint . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) preprint
The copyright holder for this this version posted November 16, 2020. ; https://doi.org/10.1101/2020.11.12.20230706 doi: medRxiv preprint is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.
The copyright holder for this this version posted November 16, 2020. ; https://doi.org/10.1101/2020.11.12.20230706 doi: medRxiv preprint Labeled 1,003 discharge summaries for 1,000 patient for model testing.
The notes were randomly selected from patients undergoing ECG without any enrichment with ICD-10 codes. (54, 54, 122, 59 notes were positive for PCI/CABG, ACS, HF and Ischemic stroke events respectively)
All the notes were allocated to Test dataset
. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.
The copyright holder for this this version posted November 16, 2020. ;
February 1, 2020. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.
The copyright holder for this this version posted November 16, 2020. ; https://doi.org/10.1101/2020.11.12.20230706 doi: medRxiv preprint . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.
The copyright holder for this this version posted November 16, 2020. ; https://doi.org/10.1101/2020.11.12.20230706 doi: medRxiv preprint . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.
The copyright holder for this this version posted November 16, 2020. ; Figure S7 . Graphical model illustrating potential selection bias in patients who underwent testing for a respiratory virus. Selecting patients based on having undergone a viral test induces an inverse association between viral infection and other causes of respiratory symptoms (collider bias). The latter may also influence the probability of delayed cardiovascular events, and thus were handled by adjustment. 
. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.
The copyright holder for this this version posted November 16, 2020. ; https://doi.org/10.1101 https://doi.org/10. /2020 . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.
The copyright holder for this this version posted November 16, 2020. ; https://doi.org/10.1101 https://doi.org/10. /2020 . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.
The copyright holder for this this version posted November 16, 2020. ; https://doi.org/10.1101 https://doi.org/10. /2020 . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.
The copyright holder for this this version posted November 16, 2020. ; https://doi.org/10.1101 https://doi.org/10. /2020 Heart failure . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) preprint
The copyright holder for this this version posted November 16, 2020. ; https://doi.org/10.1101/2020.11.12.20230706 doi: medRxiv preprint Figure S12 . Flow diagram of patient selection for survival analysis on heart failure hospitalization in patients undergoing influenza viral tests.
Identified 81 . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) preprint
The copyright holder for this this version posted November 16, 2020. ; https://doi.org/10.1101/2020.11.12.20230706 doi: medRxiv preprint Figure S13 . Flow diagram of patient selection for survival analysis on urgent revascularization in patients undergoing influenza viral tests.
Identified 81 . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) preprint
The copyright holder for this this version posted November 16, 2020. ; https://doi.org/10.1101/2020.11.12.20230706 doi: medRxiv preprint Figure S14 . Flow diagram of patient selection for survival analysis on ischemic stroke in patients undergoing influenza viral tests. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) preprint
The copyright holder for this this version posted November 16, 2020. ; https://doi.org/10.1101/2020.11.12.20230706 doi: medRxiv preprint . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) preprint
The copyright holder for this this version posted November 16, 2020. ; https://doi.org/10.1101 https://doi.org/10. /2020 . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) preprint
The copyright holder for this this version posted November 16, 2020. ; https://doi.org/10.1101 https://doi.org/10. /2020 . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) preprint
The copyright holder for this this version posted November 16, 2020. ; https://doi.org/10.1101 https://doi.org/10. /2020 Supplementary Tables ( 