121 Fig 1. Graphical outline of the methods. Purple rectangle shapes correspond to methods subsections, 122 meaning they represent steps in the processing workflow, orange parallelogram shapes represent data, 123 blue diamond shapes represent binary decisions on data, gray rectangle shapes represent excluded data, 124 and green isosceles trapezoid shapes correspond to subcomponents of the algorithm. 125
126 Data Capture and Cleaning
Hospital admission radiology reports from non-contrast head CT scans were extracted from 128 EMRs for subjects participating in the CLASSIFY-TBI study (details in S1 Protocol). Each radiology 129 report was converted to a spaCy [26] container for assessing linguistic annotations and partitioned into 130 sentences. Sentences before "Findings" and after "Impressions" sections were removed. Then, the 131 sentences were concatenated with newline characters replaced with a space, symbols removed, and 132 whitespace stripped. Radiology reports that did not contain "Findings" or "Impressions" sections were 133 removed along with radiology reports containing multiple scan types.
Using scikit-learn [27] TfidfVectorizer, the corpus was converted into a matrix of TF-IDF 137 (term-frequency times inverse document-frequency) features using n-grams with n-range from one to 138 ten. Cosine similarities were calculated between each pair of radiology reports by multiplying the TF-139 IDF matrix by its transpose. Using the cosine similarity for each pair of radiology reports, one 140 radiology report was randomly selected and all radiology reports with at least 0.70 cosine similarity to 141 that radiology report were collected in a set. From this set, one radiology report was randomly selected 142 to keep for further analysis and the remainder were removed. This was applied recursively for each set 143 until each radiology report was retained for further analysis or marked for removal. The purpose of this 144 removal was to reduce the data requiring human annotation. Details in S2 Appendix.
146 Dataset Partitioning
A random deck of three numbers the same size as the number of radiology reports retained for 148 analysis was created. The three numbers represented the proportion of radiology reports to be assigned 149 to each of the datasets: 10% initialization, 40% training, and 50% validation. From the set of radiology 150 reports retained for analysis, one radiology report was randomly selected along with up to three most 151 similar radiology reports, based on cosine similarity. From this subset, each radiology report was 152 assigned the next number in the shuffled deck. This was applied recursively until each radiology report 153 was assigned to one dataset.
The initialization dataset was solely used for training annotators and was not used by the 
As a lexicon-based method, pyConTextNLP inputs tab-separated files for lexical targets 187 (indexed events) and lexical modifiers (contextual features). It then converts these into itemData, which 188 contains a literal, category, regular expression, and rule (the latter two are optional). The literal, 189 belonging to a category (e.g., ABSENT), is the lexical phrase (e.g., is negative) in the text. The regular 190 expression allows for variant text phrases (e.g., was negative) giving rise to the same literal and is 191 generated from the literal if not provided. Further, the rule provides context to the span of the literal 192 (e.g., backward).
For text data, pyConTextNLP marks the text with lexical modifiers and lexical targets 194 according to their representative itemData. The pyConTextNLP algorithm outputs a directional graph 195 via NetworkX[29] which represents these markups. Nodes in the graph represent the concepts (i.e.,
196 lexical modifiers and lexical targets) in the text and edges in the graph represent the relationship 197 between the concepts.
The following three subsections will describe the details used for extending pyConTextNLP. 
ABSENT, NORMAL, and ABNORMAL. Henceforth, the term "annotation" will be used when 209 referencing the category to maintain consistency between annotators and algorithm vocabulary. 
Lexical targets were adapted from the common data elements in radiologic imaging of TBI [3] .
219 In deriving the lexical targets, the literal represents a clinical condition relevant to TBI on a non-220 contrast head CT scan (e.g., microhemorrhage) and the category, in this study, is the same (e.g., 221 MICROHEMORRHAGE). The regular expression for each literal (e.g., microhemorrhage(s)?)
222 was added and updated during the training stage.
Two examples (Fig 2 and Fig 3) are provided for detailed explanation of the application of 224 lexical modifiers and lexical targets during the algorithm process. 
At this stage of processing, each sentence in the radiology report will be marked with lexical 245 targets and linked lexical modifiers. There will be one lexical modifier assigned to one lexical target. 
There were 438 radiology reports extracted: 1 was removed because it did not have both 298 "Findings" and "Impressions" sections, 20 were removed because they contained more than one scan 299 type, and 106 were removed for high cosine similarity. The remaining 311 reports were split into 300 initialization, training, and validation datasets (Table 1) . 
In the training dataset, annotators took an average of 2.84 minutes per radiology report.
310 Between 15% and 16% of annotations across radiology reports were selected from default (Table 2) . 
In the validation dataset, annotators took an average of 1.67 minutes per radiology report.
318 Similar to the training dataset, 16% of annotations across radiology reports were selected from default 319 (Table 2) . For the validation dataset, there was high equivalence in annotations between the annotators 320 (N = 4072), with an additional 598 similar annotations, and only 87 divergent annotations. Overall, the 321 two annotators were in high agreement (κ = 0.913). 332 equivalent was considered the gold standard (Fig 4 dashed line) . The evaluation revealed high 333 performance across all metrics (Table 3) . Six false positives were produced for intracranial pathology and four for hemorrhage (NOS), 366 meaning tbiExtractor identified these lexical targets as PRESENT, while the annotators marked these 367 as ABSENT. This is due to the derivation of these lexical targets in relation to other lexical targets (i.e., 368 if extraaxial fluid collection is PRESENT, then by decision rules, so is intracranial pathology). The 369 remaining lexical targets produced less minimal false positives. Overall, the errors are minimal as 370 measured by the high F1 scores for the majority of lexical targets.
Further examination of divergent cases (i.e., annotators annotated ABSENT and tbiExtractor 372 annotated PRESENT, or vice versa) revealed the most common diverged lexical targets to be 373 intracranial pathology, facial fracture, intraparenchymal hemorrhage, hemorrhage (NOS), and 374 herniation. The remaining lexical targets exhibited less than four diverged responses. The most