Improving Precancerous Case Characterization via Transformer-based Ensemble Learning (2022)

Abstract: The application of natural language processing (NLP) to cancer pathology reports has been focused on detecting cancer cases, largely ignoring precancerous cases. Improving the characterization of precancerous adenomas assists in developing diagnostic tests for early cancer detection and prevention, especially for colorectal cancer (CRC). Here we developed transformer-based deep neural network NLP models to perform the CRC phenotyping, with the goal of extracting precancerous lesion attributes and distinguishing cancer and precancerous cases. We achieved 0.914 macro-F1 scores for classifying patients into negative, non-advanced adenoma, advanced adenoma and CRC. We further improved the performance to 0.923 using an ensemble of classifiers for cancer status classification and lesion size named-entity recognition (NER). Our results demonstrated the potential of using NLP to leverage real-world health record data to facilitate the development of diagnostic tests for early cancer prevention.

## introduction cancer has been the second leading cause of death with more than 1,900k new cases and 600k cancer deaths in the united states in 2022 @xcite . among those, colorectal cancer (crc) is the third most common cancer and the third leading cause of cancer death @xcite . detecting crc at its early stage can dramatically improve clinical outcomes. the 5-year survival rate is 90% when colorectal cancer is identified at the localized stage compared to 73% and 17% survival rates at the regional or distant stage, respectively 1 . crc progresses from asymptomatic nonadvanced adenoma (naa) to advanced adenoma (aa) and then to invasive carcinoma @xcite . aas are adenomas characterized by villous or tubulovillous histology, adenomas or sessile serrated lesions ⩾ 10mm, or high-grade dysplasia @xcite . aa indicates an intermediate or high risk for crc @xcite and requires crc screening every three years @xcite . recent economic studies suggest a test with increasing adenoma sensitivity in a blood-based crc screening test can reduce crc incidence and reduce mortality @xcite . there is great interest in developing noninvasive diagnostic tests with high sensitivity and specificity for advanced adenoma and crc screening @xcite . this development process requires biomarker discovery and clinical validation based on samples collected from large numbers of individuals whose colorectal cancer statuses are confirmed by colonoscopy @xcite . correctly classifying the colorectal cancer statuses, namely negative (neg), naa, aa and crc, requires expertise in distilling and interpreting tumor stage and histology information and size of precancerous adenoma from colonoscopy and pathology reports. such nuanced annotations are typically not documented and collected in structured sections of electronic health records or standardized via international classification of diseases (icd) codes @xcite . therefore, extracting this information from colonoscopy and pathology reports and generating reliable crc status classification has heavily relied on manual review by trained gastrointestinal pathologists. such review is time-consuming, costly and difficult to scale. to reduce the burden of manually annotating thousands to hundreds of thousands of pathology reports, and to facilitate the development of noninvasive diagnostic tools for colorectal cancer prevention, we investigated classical and advanced natural language processing (nlp) methods to automatically extract precancerous lesion information and determine crc status (figure 1 ). we developed transformer models to extract both categorical and numerical attributes from colonoscopy and pathology reports. compared to bag-of-word (bow) and convolutional neural network (cnn) models (see data and methods in section 3), we achieved the best performance by fine-tuning the biobert model. since lesion size is an important factor to distinguish between the aa and naa classes @xcite ), we developed an entity recognition model for lesion size extraction and improved its performance through transfer learning from a non-biomedical domain. we further improved the cancer status classification model performance by explicitly adding extracted lesion size through an ensemble model. our study also addressed two challenges for nlp practice that are specific to the biomedical industry setting: annotation at the sentence level for numerical variable extraction is limited; and most clinical trial studies that enroll patients from various sites still receive health records in the scanned pdf format @xcite , creating challenges for precisely locating the diagnosis section in health records. our research demonstrated that, along with domain knowledge-informed feature learning, fine-tuned advanced deep learning methods are able to achieve high accuracy in highly complex and nuanced disease phenotyping tasks, even with only several thousands of documents for model training. the extraction of numerical cancer attributes is challenging because the semantic context for numerical variables is mostly at the sentence level instead of document/patient level @xcite . this creates a discrepancy between training objectives (sentence level) and output evaluation (patient level). to overcome this limitation, @xcite first identified the sentences that contain the numerical values and extracted them through regular expression matching. @xcite treated tnm staging extraction as a sequence labeling task with pattern matching and conditional random field techniques. @xcite encoded tokens and their context words as bag-of-n-gram features and classified the token sequence for tnm staging and tumor volume extraction. in our work, we treated numerical lesion size extraction as a named entity recognition (ner) task and addressed the challenge of limited annotated sample size by transfer learning from models pre-trained in a nonbiomedical domain. previous attempts to employ nlp methods to parse colonoscopy reports and linked pathology reports have aimed to characterize adenomas due to their importance in estimating colorectal cancer risk @xcite @xcite . the limitations of current studies are two fold: first, most studies still rely on rule-based systems such as linguamatics @xcite and ctakes @xcite for extracting adenomas through pattern-matching and dictionary look-up. deriving rules can be timeconsuming, require extensive domain knowledge, and likely results in overfitting to the development dataset and limited portability; second, nlp studies for colorectal cancer do not perform end-to-end crc phenotyping to classify patients into neg, naa, aa and crc, which are of great interest to characterize crc risk and prioritize patients for cancer screening and early cancer detection. here we performed end-to-end precancerous and cancer status characterization with deep learning methods which promise to be more generalizable and efficient than rule-based approaches. ## conclusion determining cancer status and characterizing precancerous lesions are critical and time-consuming steps for the development and evaluation of diagnostic tests for colorectal cancer screening. here we achieved a 0.914 macro-f1 score for cancer status classification with transformer models finetuned using biobert. informed by the domain knowledge and error analyses, we identified lesion size as a critical factor for differentiating between aas and naas, but one that was not efficiently captured in biobert context-dependent embeddings. using an ensemble model combining a fine-tuned biobert model and a lesion size named entity recognition model, we further improved classification performance to a macro-f1 score of 0.923. the lesion size extraction model was developed through transfer learning, using a transformer model trained in a non-biomedical domain. we showed that directly fine-tuning of transformer models was inadequate for domain-specific nlp tasks, and that precise feature engineering and use of ensemble models was needed to improve classification performance. overall, we provided an accurate algorithm for characterizing precancerous cases that may help to improve early colorectal cancer detection and prevention, and a model training framework that leverages advanced nlp techniques to address complex disease phenotyping tasks in biomedical domain.