Information Extraction from P olish Radiology Reports Using Language Models (2023)

Abstract: Radiology reports are vital elements of directing patient care. They are usually delivered in free text form, which makes them prone to errors, such as omission in reporting radiological findings and using difficult-to-comprehend mental shortcuts. Although structured reporting is the recommended method, its adoption continues to be limited. Radiologists find structured reports too limiting and burdensome. In this paper, we propose the model, which is meant to preserve the benefits of free text, while moving towards a structured report. The model automatically parametrizes Polish radiology reports based on language models. The models were trained on a large dataset of 1200 chest computed tomography (CT) reports annotated by multiple medical experts reports with 44 observation tags. Experimental analysis shows that models based on language models are able to achieve satisfactory results despite being pre-trained on general domain corpora. Overall, the model achieves an F1 score of 81% and is able to successfully parametrize the most common radiological observations, allowing for potential adaptation in clinical practice. Our model is publically available.

## introduction a radiology report is the most important product radiologists generate to help direct patient care. they are vital to the referring physicians that depend upon them while making a decision about further treatment of a patient. it represents the highest level of radiologists' synthesis and insight into a patient's condition. however, radiology reports are almost always formulated in natural language. natural language is flexible and enables the writer to express the same idea in a variety of different ways with varied complexity. as a result, the style, 1 github.com/aleksanderobuchowski/plradie length, and level of detail vary among the radiologists, even among those coming from the same institution. moreover, the reports often contain misspellings and mental shortcuts. such properties make them difficult to analyze for referring physicians and incomprehensible to patients. the well-known initiative of the american college of radiology -imaging 3.0 introduced a roadmap to transition radiological practice from volumed-based care to value-based care. the critical element of the roadmap was the adoption of structured reporting. a structured report (sr) is a report generated from a predefined, standardized format. the sr is considered a better strategy in terms of reduction in diagnostic error, comprehensiveness, adherence to consensus guidelines, and reduction in the omission of findings and other preventable errors. the negative effects of medical errors were publicized by the report of the institute of medicine "to err is human" @xcite . the report highlighted the importance of limiting preventable medical errors, such as omission in reporting radiological findings. the adoption of sr was defined as a critical step to provide the best quality of service to referring physicians and patients by both the european society of radiology (esr) and radiological society of north america (rsna) (european society of radiology (esr), 2018). the sr is believed to improve the quality of reports by providing a checklist to ensure that all relevant points were addressed. moreover, the sr is easier to integrate with tools helping radiologists express relevant information, e.g., co-rads classification @xcite . lastly, they could facilitate the adoption of valuebased healthcare -a new healthcare delivery model in which healthcare providers are paid based on patient outcomes, not the number of performed procedures. although structured reports have many benefits, their acceptance among radiologists is still limited @xcite . they require radiologists to change their habits which they often practiced for many years. the radiologists may be reluctant to change for many reasons, including the limited scope of expression resulting in the downgrade of quality, the feeling that there is no clinical necessity to change, and even because they perceive it as an attack on the art of medicine @xcite . with sr, the structure of a report would also have to be manually updated with the changes in classification ontology, possibly resulting in discrepancies between the latest state of knowledge and clinical practice. moreover, while the proposed structured reports schema could be introduced in clinical practice, it does not solve the problem of already generated reports, where the clinical observations may need to be rewritten to follow the parameterized structure, therefore resulting in additional labor. although those older reports might not be used in further clinical practice due to being outdated, their parametrization could still be beneficial for data analysis and training of machine learning models. to bring the most out of both structured reporting and free-texts, in this paper we propose a model for the automatic parametrization of polish radiology reports based on language models. the model's role is to assign one of 44 labels to each radiological observation. example texts with extracted radiological observations are shown in figure 1 . formally, our task falls under the information extraction category, as the goal of the model is to detect spans corresponding to specific radiological findings rather than detect a broader set of entities. this was motivated by the fact, that as shown in @xcite systems that strictly perform named entity recognition-level tasks are insufficient for answering clinical queries. for example, in the sentence "no lesion observed," a ner-only system could (correctly) identify "lesion" as an entity, but cannot correctly answer the intended question. moreover, we decided to model this task as sequence labeling rather than multiclass sequence text classification, as not only more informative to the end user by also previous work has shown that token-level labeling can result in improved accuracy @xcite . to the best of our knowledge, this is the first model for information extraction from radiology reports in the polish language. radlex @xcite ) is a radiology lexicon produced by the radiological society of north america. it contains an ontology of radiology terms for use in radiology reporting, decision support, data mining, data registries, education, and research. it defines standard names and codes for radiology findings. the idea of unifying terminology and linking the reports to the images was combined in the annotation and image markup (aim) project @xcite of the national institutes of health cancer biomedical informatics grid. aim was created to develop a uniform machine-readable format for storing both the image and a radiology report. it enables the description of an image using common data elements and controlled terminologies, such as radlex. the usage of ontology enables easy queries and retrieval of information. the annotations and measurements made with aim can be serialized as xml or dicom sr. another approach was the rsna's radreport.org reporting templates. the templates for various clinical scenarios provide a standardized radiology lexicon with the terms defined in web ontology language @xcite . although there have been some important attempts to make sr feasible, it is still at the early stage of adoption. @xcite proposed an information extraction model for spanish radiology reports using a multilingual bert @xcite model. the model's role was to parametrize ultrasonography reports. the corpus was annotated using ten different labels: abbreviation, anatomical entity, conditional temporal, degree, finding, location, measure, negation, type of measure, and uncertainty, and was split into a training set (175 reports), development set (92 reports), test set (207 reports). similar to our work the authors have also used bio annotation schema, however, in our work, we focus solely on radiological findings but use much more detailed annotations with 44 different possible findings. ## experiments and results the results for different models are presented in after performing additional analysis of the best model shown in ## discussion in this work, we presented a tool for the parametrization of radiological reports for narrative reports written in natural language. in the interest of standardization and to help further research in this area, we introduced a general anno-tation scheme that was developed together with clinical experts based on common radiological observations. the results show that general domain language models can successfully be used in the radiology domain, although there is still room for improvements that can possibly be filled with domainspecific models. the detailed analysis of the results shows that the model is able to better capture the entities with fewer variations and higher representation in the training set. it can also be seen that the model rarely confuses different entities, but has some trouble with capturing the spans accurately. however, the model still achieved satisfactory results and with proper verification could successfully be used in clinical practice. information extraction is especially challenging with medical terminology since there is some interchangeability between the terms and the structure of a phrase may influence the meaning. for instance, "przepuklina przełykowa" or "przepuklina przełyku" ("hiatal hernia" or "hiatus hernia") can also be phrased as "przepuklina wślizgowa przełyku" ("sliding hiatus hernia"). the literal translation of (parenchymal) pulmonary/lung consolidations is: "zgęszczenia (mi ąższowe) płuc/płucne" but in reports it usually comes in a phrase "zgęszczenia (mi ąższowe) w płucu prawym" ("consolidations in the left lung"). extracting information from a report is a difficult task for the model but it is also non-trivial for a referring physician. from a clinical perspective, the automatic generation of structured reports from free texts combines the benefits of both structured reporting and free text, while limiting the drawbacks of a rigidly structured format. ## future work the results generated by general domain language models are satisfactory, but far from perfect. this is likely motivated by the fact that the word distribution in the general domain and medical corpora is vastly different, which can result in an array of problems in the nlp of clinical texts. in the future, we are planning to train domain-specific language models using a larger corpus of unlabeled reports using methods such as masked language modeling. such an approach would most definitely improve the model's results.