An end-to-end neural model based on cliques and scopes for frame extraction in long breast radiology reports (2023)

Abstract: We consider the task of automatically extracting various overlapping frames, i.e, structured entities composed of multiple labels and mentions, from long clinical breast radiology documents. While many methods exist for related topics such as event extraction, slot filling, or discontinuous entity recognition, a challenge in our study resides in the fact that clinical reports typically contain overlapping frames that span multiple sentences or paragraphs. We propose a new method that addresses these difficulties and evaluate it on a new annotated corpus. Despite the small number of documents, we show that the hybridization between knowledge injection and a learning-based system allows us to quickly obtain proper results. We will also introduce the concept of scope relations and show that it both improves the performance of our system, and provides a visual explanation of the predictions.

## introduction in this study, we will address the task of structuring breast radiology reports. 1 this end to end task consists in extracting different structured entities (frames) of various types, each one composed of multiple labels and multiple mentions organized as a list of fields. the creation of a frame is triggered by a "trigger" mention, and some of its fields may be justified by "attribute" mentions. an example of structured lesion frames extracted from a fictitious document is illustrated in figure 1 and therefore, modelling the relations between different mentions is necessary to distinguish between multiple overlapping frames, such as frames 1 and 3 in the example. our contributions foot_2 are the following: the extraction of structured information from clinical reports has been the subject of many studies. most of these works are not specific to breast imaging reports and the objectives vary greatly, in terms of their scope, granularity and form. interested readers can refer to existing surveys on the state of nlp in radiology reports @xcite . several works are only concerned with the extraction of a few report-level attributes, and therefore view the task as a classification or term extraction task for items such as acr scores, histological grade or primary site of frame 1 frame 2 frame 3 field value mention(s) value mention(s) value mention(s) trigger [cysts], [nodules] multiple [cysts] [cysts], [nodules] organ breast [breast] breast breast [breast] laterality left [left]: left on the [left] left [left]: time current current current quadrant size [millimetric] [millimetric] distance 30mm [3 cm] 20mm [2 cm] angle 8 [8 o'clock radius] 6 [6 o'clock radius] other features have also been the subject of specialized systems such as locations @xcite ). an extensive survey of the different systems proposed for different features was conducted by @xcite . other works have sought to produce a more detailed and global extraction, and to detect several types of entities at the same time. the earliest work was the one of @xcite , who proposed a frame based representation and method for annotating abnormal findings, anatomy, and medical procedures frames in radiology reports. @xcite used a rule-based system and terminologies to extract abnormal findings and acr scores. the deepphe system was proposed by @xcite as a fully integrated software built on ctakes @xcite to extracts document and patient level cancer summaries (akin to frames) in clinical reports. @xcite proposed a fact-based scheme, in which each fact is structured around an anchor and may contain modifiers. however, their model makes the assumption that all the mentions that characterize an entity are adjacent inside the fact span. several methods decompose the problem into a first ner step followed by a relation detection step that allows arguments to be non adjacent. @xcite proposed a frame based scheme for annotating cancer information in clinical reports and a method to perform the prediction @xcite . their method first extracts triggers and modifiers with a ner system, and predicts their relations to form frames, but makes the assumption that there is no overlap between the different entities. recently, a more complex scheme has been proposed by @xcite to annotate nested relationships between different entities. however, this work does not specifically address the case of complex or distant relations between entities. the closest task to ours is the one of event extraction, in which models extract one event per trigger mention, and look for related mentions that might be part of the same event. however, trigger mentions (e.g., the first [cysts] in figure 1 ) may belong to different overlapping frames (the 3cm 8 o'clock lesion frame and the 2cm 6 o'clock lesion frame in figure 1 ) that can only be distinguished by considering relations between their attribute mentions ( ). to address this issue, an approach consists in listing all the possible combinations of mentions, then filtering them with a classifier @xcite @xcite @xcite @xcite . however, this solution becomes computationally unsatisfactory when the number of mentions that compose a frame grows. ## method we now detail a neural network based end-to-end method to automatically extract frames from clinical reports. we encode each document as word embeddings and share these with the downstream decoding components. like most relation and event extraction models, our model operates as a pipeline. as illustrated by figure 2 , the first two mentionlevel decoders extract the named entities, or mentions (step 1 ), that are likely to be used in the composition of structured entities, and normalize them (step 2 ) to obtain the value of the field they apply to. the next two decoders focus on frame-level extractions. the frame extraction decoder (step 3 ) extracts groups of mentions to form frames. for each frame, the last frame classification decoder (step 4 ) predicts the values of the fields for which no explicit mention was found, such as a past temporality, which could be indicated by a verb tense. ## experiments we evaluate our proposed approach on the test set of a new annotated dataset described in appendix a, and perform several ablation experiments to investigate the design choices of our model. the dataset is composed of 120 french breast imaging clinical reports annotated with frames. there are five types of objects: acr (cancer risk) scores, breast density scores, diagnostic procedures, therapeutic procedures and lesions. the document-level statistics are detailed in ## acknowledgment we thank the clinical data warehouse (entrepôt de données de santé, eds) of the greater paris university hospitals for its support and the realization of data management and data curation tasks. ## conclusion in this work, we presented a system for extracting structured entities from clinical breast radiology reports. we have shown that the addition of synthetic sentences can improve the performance in the context of a small amount of data. this information is valuable for the annotation and development of new information retrieval systems in other domains, where key words or phrases are known in advance. the method we described introduces the notion of frame extraction in the form of mention cliques, and we have shown that a formulation of the relation extraction task via scopes improves the performance of our system. future work will evaluate this approach on other structured entity extraction tasks such as event extraction.