Extracting Relations between Radiotherapy Treatment Details (2020)

Abstract: We present work on extraction of radiotherapy treatment information from the clinical narrative in the electronic medical records. Radiotherapy is a central component of the treatment of most solid cancers. Its details are described in non-standardized fashions using jargon not found in other medical specialties, complicating the already difficult task of manual data extraction. We examine the performance of several state-of-the-art neural methods for relation extraction of radiotherapy treatment details, with a goal of automating detailed information extraction. The neural systems perform at 0.82-0.88 macro-average F1, which approximates or in some cases exceeds the inter-annotator agreement. To the best of our knowledge, this is the first effort to develop models for radiotherapy relation extraction and one of the few efforts for relation extraction to describe cancer treatment in general.

## introduction radiotherapy is the use of ionizing radiation, which is radiation with enough energy to remove electrons from atoms and molecules, to treat disease @xcite . the predominant indication for radiotherapy is the treatment of cancer, where it exerts its antineoplastic effect via dna damage, which preferentially kills cancer cells over health tissue cells @xcite . radiotherapy plays a central role in the curative and palliative treatment of many cancers. it is estimated that up to 30% of cancer patients receive radiotherapy as a part of their first-line treatment, and approximately 50% of all cancer patients receive radiotherapy during the course of their cancer care @xcite . despite its importance in cancer treatment, radiotherapy is included in cancer registries in only high-level, often cursory detail, if at all. for example, radiotherapy details are only available by custom request in the publicly available surveillance, epidemiology, and end results program (seer) cancer registry, acknowledging incompleteness and errors in this manually extracted data @xcite . the reason for this is multifold. first, radiotherapy is a highly technical field not extensively taught in medical school, and uses its own jargon not found in other medical texts. additionally, radiation treatment details are frequently not entered into the electronic medical records (emr) as structured data. instead, radiotherapy is described in clinical free text using descriptive and very non-standardized language. radiotherapy treatment descriptions are more similar to the documentation of operative procedures than to documentation of medication regimens. because radiation is personalized to each patient's disease and anatomy, it cannot be described with standard reporting of the type of radiation, dose, and frequency. additionally, radiotherapy is often delivered in multiple phases, each treating a different anatomical site to different doses and with different types of radiation, yielding complex descriptions of treatment courses. these features, in concert with a lack of widely used standardized nomenclatures @xcite @xcite , limit manual data extraction, hindering the potential of big data to improve cancer research and clinical care. while algorithms for named entity recognition have previously been reported for radiotherapy details @xcite and other cancer therapies @xcite @xcite , relation extraction remains a relatively underexplored task in clinical nlp @xcite . there are few examples of relation extraction models for cancer characteristics in general @xcite @xcite , and to the best of our knowledge none for cancer treatment, including radiotherapy. identifying treatment entities in isolation and not linking them to a specific treatment instance is insufficient to coherently describe cancer therapies, especially because concurrent and serial treatments are often described together in the same note. for example, extracting frequency without linking it to a specific treatment is not informative by itself. relation extraction is a critical component of information extraction for radiotherapy as this treatment is often given in multiple sequential or nested phases. linking relevant treatment entities with the same phase is necessary to summarize how and why a treatment was delivered, necessary components to understanding treatment outcomes and quality. however, this is a very challenging task even for expert human annotators, as demonstrated by the challenges in accurate extraction of such data for seer. therefore, there is a need for more reliable relation extraction methods for radiotherapy. relations can be modeled in various ways in neural networks, including inserting special tokens around the arguments of interest and using this augmented text as input into the model @xcite , and using token position embeddings to encode the relative distance of words to the arguments @xcite @xcite . using the former approach, we aimed to investigate several approaches for relation extraction from clinical texts describing radiotherapy, with a goal of augmenting reporting of cancer treatment details for research and clinical purposes. the contributions of the work described in the paper are (1) the definition of the task of radiotherapy information extraction from the emr clinical narrative, (2) the creation of resources for the task (annotation guidelines and corpus), (3) the exploration of state-of-the-art neural methods to this highly impactful clinical task, and (4) establishing a baseline for the task. ## data data for this work consisted of texts describing radiotherapy from three complementary sources. first, we included 270 clinical descriptions of radiotherapy regimens from hemonc.org, which is a publicly available wiki of cancer and blood disorder treatment regimens and interventions. second, we included 73 radiotherapy descriptions from a state cancer registry. these are abstractions from patients' emr, often copied and pasted from clinician notes, describing details of cancer treatment for use by cancer registrars within a patient-level xml. we used the entire text of the xml categories that contained radiotherapy details as model input. third, 79 completed breast and colorectal cancer clinician notes that contained radiotherapy details from the thyme corpus from clinical tempeval @xcite and an internal corpus with breast cancer notes. annotation guidelines for radiotherapy properties and treatment instances were developed @xcite . 1 if an overall radiotherapy treatment course was delivered in more than one dose-treatment site relationship is represented as: …she presented after a screening mammogram showed a nodule in the left breast upper outer quadrant. after lumpectomy, she was treated with radiation to a dose of 50 gy in 25 fractions to the left breast, followed by a boost of rt_dosagestart 10 gy rtdosage_end in 5 fractions to the txsitestart tumor bed txsiteend… phase, as described above, each phase was considered a separate radiotherapy instance (figure 1 ). gold annotations for relations between the following key properties were created: dose, fraction number, fraction frequency, treatment site, and boost. radiotherapy is most often delivered in many small doses, or fractions, over a given period of time. dose is any description of radiation dosage in the text, either the total dose or fractional doses, generally described using the unit gray (gy). fraction number is any mention of the number of fractions delivered, and fraction frequency is the frequency of fraction delivery. treatment site is the actual or relative anatomical site that is targeted with radiotherapy. boost is a mention that conveys the treatment instance is a second phase of radiotherapy that brings a smaller treatment site to a higher dose. properties that described the same treatment instance were linked together as a relation. two expert human annotators completed gold annotations for 47 radiotherapy instances containing 310 relation instances to calculate interannotator agreement, after which a single human annotator completed the gold annotations. as most radiotherapy instances included dose, we chose to classify the relations between each dose mention and every other property mentioned in the radiation instance. thus, dose mentions served as the anchor for the relations within a radiotherapy instance and were labeled as dose-mention. of note, dose-dose refers to a relation of two different dose mentions in the same radiotherapy instance. in our dataset, there were on average 1.4 radiation instances per document. documents were split into train, development, and test sets. the gold annotated hemonc.org corpus will be made publicly available for research purposes. ## methods we explored two state-of-the-art neural network methods for this task. first, we used flair @xcite , which is a pre-trained character language model trained on one billion words of text @xcite to train a multi-layer long short-term memory (lstm) @xcite to generate contextual embeddings. text is then passed into this model, put into an lstm to obtain a text representation, and this is passed into a final linear layer for classification. we were motivated to explore this approach as we hypothesized that the contextual character embedding may be better at handling the rare and misspelled words in cancer texts, as well as numbers and short abbreviations common in radiation descriptions. models had a hidden state of 128 and a dropout of 0.15-0.24, and were trained for 100 epochs using a mini-batch size of 8 and learning rate starting at 0.2 with an anneal factor of 0.5, using an sgd optimizer @xcite @xcite . second, we assessed the performance of bidirectional encoder representations from transformers (bert) base uncased model, finetuned on this relation task using a recurrent neural network to predict the class label @xcite . we chose to explore this method given the excellent relation classification performance of attention-based models in biomedical texts @xcite @xcite . these models had a hidden size of 128 and dropout of 0.5, and were trained for 30 epochs using a minibatch size of 8 and learning rate starting at 0.00003 with an anneal factor of 0.5, using an adam optimizer @xcite . for all, the model that performed best on the development set was evaluated on the held-out test set. rule-based methods were considered, but as there are often more than one radiotherapy instance mentioned in a clinical text, frequently in close proximity and described in nested fashion, we did not feel there was a straightforward approach (figure 1 ). to generate candidate relations, we extracted text windows encompassing two different token lengths on each side of the gold annotated dose mention anchor: 46 tokens and 90 tokens (95 th and 99 th percentile of token span lengths between dose-mentions in the same gold radiotherapy instances in the train and development sets, respectively). every dose-mention pair in the text window was considered a candidate relation. for each relation candidate, start and stop tokens were inserted around the dose and candidate property (figure 1 ). the text segment was labeled with a ## evaluation qualitative error analysis of each model revealed five main categories of failure: 1) human errors in gold labeling, 2) false positives due to two mentions being in close proximity despite relating to different treatment instances, 3) false negatives ## discussion and conclusion the neural models had very good performance on the radiotherapy relations explored in these experiments, often approaching or exceeding iaa. the performance of the bert and flair models were overall comparable, with the best bert model outperforming the best flair model. interestingly, the size of the text windows used for fine-tuning appeared to have a larger impact on performance than the type of model itself, with shorter texts yielding better results. this may be due to the frequent repetition of similar but unrelated entities and treatments in clinical texts, and optimizing this parameter should be explored when developing models for clinical relation extraction. these findings suggest that neural methods may be a good avenue for clinical relation extraction for complex, highly specialized treatments such as radiotherapy. future work will develop models to extract relations between dose and additional relevant entities, and will investigate end-to-end entity and relation extraction systems for robust information extraction pipelines.