Paragraph Retrieval for Enhanced Question Answering in Clinical Documents (2024)

Abstract: Healthcare professionals often manually extract information from large clinical documents to address patient-related questions. The use of Natural Language Processing (NLP) techniques, particularly Question Answering (QA) models, is a promising direction for improving the efficiency of this process. However, document-level QA from large documents is often impractical or even infeasible (for model training and inference). In this work, we solve the document-level QA from clinical reports in a two-step approach: first, the entire report is split into segments and for a given question the most relevant segment is predicted by a NLP model; second, a QA model is applied to the question and the retrieved segment as context. We investigate the effectiveness of heading-based and naive paragraph segmentation approaches for various paragraph lengths on two subsets of the emrQA dataset. Our experiments reveal that an average paragraph length used as a parameter for the segmentation has no significant effect on performance during the whole document-level QA process. That means experiments focusing on segmentation into shorter paragraphs perform similarly to those focusing on entire unsegmented reports. Surprisingly, naive uniform segmentation is sufficient even though it is not based on prior knowledge of the clinical document’s characteristics.

## introduction healthcare professionals spend a lot of time going through extensive clinical documents, such as discharge summaries, to find specific answers to questions about their patients @xcite . this process could be aided by question answering (qa) models, that search for substrings in the text of a clinical document to provide an evidence for a given question @xcite . currently, encoder-based language models demonstrate strong performance in solving qa tasks @xcite , even when we are looking for substrings in a multiparagraph clinical context @xcite . however, the training process and inference of large language models (llms) on document-level qa require significant computational resources that are not always available. in addition, encoder-based and decoder-based models face difficulties in understanding and processing longer documents @xcite . a possible solution might be working with segments (paragraphs) of the document rather than full text. to achieve this, we must first segment the document into paragraphs, then identify the relevant paragraph for a given question, and then apply an qa model only to the selected paragraph as the context instead of the full document text. this alone significantly facilitates healthcare professionals' work in finding answers in clinical texts, which is another reason why it is worth addressing the paragraph retrieval issue. clinical texts often lack structure @xcite and contain information that is not expressed in natural language @xcite . moreover, each clinical text, authored by distinct doctors from various hospitals and even different countries, is arranged uniquely. therefore, the task of segmenting a document into natural language paragraphs is inherently non-trivial. however, the question arises: is it necessary to segment clinical text into such structured paragraphs? will a naive uniform segmentation without knowledge of the text itself have a similar performance? our work addresses qa on differently-sized paragraphs of clinical documents. first, given clinical document paragraphs and a given question, retrieve the most relevant paragraph. in the second step, we perform qa on that paragraph. in addi-tion, we investigate the potential performance of the model on the qa task if the relevant paragraph is always predicted correctly. we work with two subsets of the emrqa dataset: medication and relations @xcite . we propose a heading-based segmentation into sections regarding different average paragraph lengths over all training clinical documents. we analyze the optimal average paragraph length to achieve the best performance and ensure that we preserve the context while keeping the paragraph as concise as possible. we then compare the performance of the encoding-based models on these segmentations with their performance on a naive segmentation approach. finally, we demonstrate how llms perform under the same training conditions as encoderbased models. our main contributions are the following: ## related work the problem of question answering encompasses several different subtypes. one of them is to return an answer for a given question without any context @xcite . another subtype involves text comprehension. for a given question and some context (such as a document or a paragraph), the task is to answer the question based on the content of the context, but the actual formulation of an answer is not restricted @xcite . in our work, however, we focus on finding substrings in a given context that serve as both evidence and answer to a given question. a significant resource in this field is the squad dataset @xcite , containing questions, context paragraphs based on wikipedia articles, and answer substrings. the dataset has been used to train and compare various neural methods, including encoder-based and decoder-based architectures @xcite @xcite . this dataset was later extended into squad v.2 @xcite , which also includes questions and corresponding paragraphs that do not contain an answer for a given question. as an alternative to this dataset in the clinical domain, the emrqa dataset @xcite was published. the emrqa dataset contains synthetically generated questions and substring answers for clinical reports from the n2c2 dataset (previously called i2b2). the emrqa consists of 5 subsets: medication, relations, heart disease, obesity, and smoking, each focusing on different aspects and different complexity. from the emrqa dataset, the emrqa-msquad dataset (eladio and wu, 2024) was derived by summarizing clinical reports into single paragraphs as contexts and providing new manual annotated substring answers. however, this process removes the naturalness of clinical notes written by healthcare professionals. there is also the qa reading comprehension dataset in the medical scientific domain, which is bioasq @xcite . the dataset includes instances consisting of a question, ideal answer, pubmed medical article abstracts containing the answer, and the substring answers of all such related article abstracts. in our work, we exploit the emrqa dataset @xcite . @xcite analyzed the two largest subsets from the emrqa dataset in detail: medication and relations. they preprocessed and filtered these two subsets and trained encoderbased models, such as bert-base @xcite , biobert @xcite , and clinical-bert @xcite , and then compared their performance. however, developments in the field have introduced other medically pre-trained encoder-based models, such as medcpt @xcite , designed specifically for biomedical information retrieval, or biolord @xcite . although the emrqa dataset authors perceive the analysis provided by @xcite as misleading due to the use of only 2 out of 5 subsets, for the purposes of our work, these two subsets with the same preprocessing and filtering are equally suitable. therefore, our work indirectly follows up on the analysis conducted by @xcite . another type of qa task involves multiplechoice questions. in the field of medicine, there is a pubmedqa dataset @xcite , which contains questions related to pubmed article abstracts. furthermore, exam-like multiple-choice question datasets such as medqa @xcite , medm-cqa @xcite , mmlu @xcite were published. these datasets have been used as benchmarks for llms, such as meditron @xcite and biomistral-7b @xcite , which are open-source llms pretrained on medical data. in addition to medical scientific and exam-like multiple-choice question datasets, richter-pechanski et al. ( 2024 ) focused on multiple-choice questions on german doctors' letters. ## setup we solve the task of document-level qa on the emrqa dataset by a two-step method combining paragraph retrieval and paragraph-level qa. we analyze performance of the two tasks in combination and also separately. we follow the work of @xcite focusing on the medication and relations subsets only and applying the same data preprocessing. yue et al. ( 2020 ) concluded that it is sufficient to use only 20% and 5% of training data to train models of the medication and relations subsets, respectively. since a larger amount of training data has no effect, we use the same ratio of data samples for training. their data instances consisted of triples of document+question+answer where the answer was guaranteed to be present in the document. our data instances were generated as triples paragraph+question+answer where the answer was also guaranteed to be present in the paragraph and pairs paragraph+question where the corresponding answer was not present in the paragraph. for each question in a sampled training subset of a given report, we randomly select a paragraph from the same report that does not contain an answer. thus, we have a balanced dataset where the number of paragraphs containing an answer matches the number of paragraphs without them. in our experiments, we train the clinicalbert @xcite and bert-base @xcite ) models, just as yue et al. (2020) did. additionally, we measure the performance of the medcpt article encoder @xcite model. since we are working with a balanced dataset, it is necessary to specify how to handle cases where the answer is missing in the context. if there is no response in the dataset sample, the model is trained to predict the cls token as a response prediction. during the inference, we apply the softmax function to the output logits and use its negative value as confidence that the paragraph contains the answer. then, for a given question and segmented report into paragraphs, the model solves the qa problem for all paragraphs (we evaluate the oracle-qa task using the ground truth relevant paragraph), ranks past surgical history : notable for the above , as well as debridements ... discharge medications : vancomycin 1250 mg iv q d , ofloxacin 200 ... laboratory data : white count 12.6 , hematocrit 28.9 ... physical examination : on admission vital signs were ... all results by confidence, and selects the substring of the paragraph with the highest confidence as the final answer prediction (and then the pr and pr-qa tasks are evaluated as well). ## document segmentation our goal is to design a method for segmenting reports into natural language paragraphs that each contain all necessary context while minimizing unnecessary information. as @xcite pointed out, the segmentation of clinical reports into sentences is not straightforward. these complications arise from factors such as the frequent use of dots in acronyms, list items, and values and the irregular alternation of uppercase and lowercase letters. because our goal is to create concise paragraphs without losing context, we must ensure that paragraph boundaries do not disrupt sentence cohesion or, worse, do not appear in the middle of a word. therefore, our initial step is to split each report into groups of complete sentences, ensuring that no sentence is fragmented across groups and that no substring of a response is split into two paragraphs. to achieve this, we leverage the structure of the official emrqa dataset @xcite . in this dataset, each report text is stored as a list of lines, with the answer evidence (in our case, the answer substring) being one of the report lines. therefore, we set the condition that none of the sentence groups starts or ends in the middle of any line, ensuring that no answer substring is split into two paragraphs. we then split the medication subset into groups of sentences if the following pattern for the end of a line is satisfied: a dot at the end of a line, preceded by five characters that are neither dots nor uppercase letters, and the next line starting with an uppercase letter (eventually this second line can also be an item of a numbered list, which means it could start with a number instead of an uppercase letter). clinical notes in the subset of relations are structured more clearly. dots marking the end of a sentence are surrounded by spaces, while dots forming part of abbreviations are not. thus, such space-surrounded dots at the end of a line indicate a sentence group boundary in the context of the relations subset. another pattern we utilize as a criterion for segmentation contains a sequence of characters ending with a colon, indicating headings followed by corresponding content, as shown in figure 1 . using the end of the previous line of such heading lines as a sentence group separator makes sense. to decrease the risk of detecting not-heading lines, we only consider uppercase titles for medication when determining sentence group boundaries. in the case of the relations subset, only lines ending with a colon preceded by space are considered, similar to the situation with dots. finally, we need to determine how we will group sentence groups together to create a final paragraph segmentation. by using the following regex pattern (^([0 -9]+[\ s ]*[\.\)][\ s ]*)?[ a -z ][ a -za -z \ s @xmath0]*:) we identify all potential headings at the beginning of all sentence groups. subsequently, we calculate how often these headings appear in the training data. we assume that frequently used titles signify sections generally discussed in clinical notes by healthcare professionals that do not need any additional context. therefore, the question arises: what is the minimum number of occurrences of headings in the training data that we want to use for paragraph separations? we call such segmentation as heading-based segmentation. as the range of possible headings serving as paragraph boundaries increases, the average length of paragraphs decreases. as shown in the challenges involved in distinguishing relevant paragraphs from finding exact substring answers. despite the structured nature of the emrqa dataset, the rules for splitting the medication and relations subsets into paragraphs can be generalized to other clinical datasets with caution. although different countries, hospitals, and doctors may structure their reports differently, there are often similar paragraphs and even common headings across various discharge summaries. this observation allows us to take the list of headings collected from the segmentation process of emrqa and use it when segmenting other discharge summaries. however, some level of preprocessing and postprocessing will always be necessary, as this method is not a one-size-fits-all solution for all clinical reports. ## is segmentation into paragraphs necessary? we have shown that heading-based paragraph segmentation has no significant effect on the pr-qa task except for improving the exact match score of the relations when segmenting into shorter paragraphs. however, most clinical texts are unstructured and use unique text formatting; sometimes, finding a segmentation into coherent sections in the sense of meaning as well as syntax is not easy or even possible. to determine its necessity, we conduct experiments with naive segmentation. we choose a target average segment length t to create the naive segmentation. then, we calculate each report's length n and determine the rounded number of segments in the report as @xmath1. subsequently, we compute the actual average segment length of the report for the value of p as @xmath2. finally, the report is divided into segments of r characters. postprocessing is then applied across all segments and all answers in the report. in cases where an answer substring is part of two separate segments, we adjust the segment boundaries so that the entire answer is in one segment only. the target average segment length t for naive segmentation is chosen to match the average segment lengths of the headings-based segmentation experiments. specifically, for each measured segmentation level of the headings-based approach, we also measure the naive segmentation using a target average segment length equal to the average segment length of the given headings-based segmentation. in figures 2 and 6 , we visualize the comparison between clinicalbert @xcite using heading-based and naive approaches. except for segmentation with the shortest paragraphs, the choice of segmentation method has no noticeable effect on the pr-qa task. in the case of the shortest paragraphs of the relations subset, the naive approach begins to decline in pr-qa performance, while the heading-based approach becomes more accurate. the reason for that is worse performance on the pr task as well as the exact match on the oracle-qa. the performance of naive segmentation on the pr task is significantly worse. on the other hand, the oracle-qa naive segmentation experiments show better results. the most confident segment contains less relevant content compared to heading-based segmentation, making it easier to find the correct substring as an answer (fewer relevant and potential words in the segment) if the segment itself is predicted correctly. overall, the pr-qa performance of both heading-based and naive approaches is similar. ## paragraphs and llms considering the impact of segmentation into shorter paragraphs on the scores, it is noteworthy that it does not significantly affect them and may even enhance them. this observation suggests the potential for leveraging llms without the necessity for unlimited computational resources in future applications. in this study, we evaluate the performance of biomistral-7b @xcite in the oracle-qa task and compare it with medcpt @xcite , bert-base @xcite , and clinicalbert @xcite models. biomistral-7b @xcite is trained on question+paragraph+answer triplets where each paragraph contains an answer. negative examples are omitted to focus solely on the oracle-qa task. the model prompt is shown in figure 10 . for evaluation, the model's response is parsed into a json object, and the value of the "answer" field is extracted. ## conclusions our study explores the efficiency of language models in addressing clinical document-level qa. we described an approach to perform headingbased segmentation and extract clinical report headings and found that segmenting documents into shorter sections through heading-based or naive approaches does not decline the performance of clinicalbert @xcite , bert-base @xcite , or medcpt @xcite models. paragraph length has no significant impact on the qa task. furthermore, knowledge of clinical document characteristics is unnecessary since naive segmentation performs similarly to headingbased segmentation. the main difference is that naive segmentation is more challenging for paragraph retrieval but easier for question answering. in both cases, however, we observe that the correct segment containing the answer is almost always found within the three most confident paragraph retrieval predictions. leveraging llms like biomistral-7b @xcite shows potential for document-level clinical qa tasks even when computational resources are limited. however, there is still room for improvement and it is necessary to explore other pre-trained llms with different training approaches. it remains an open question how the m-f1 m-em r-f1 r-em medcpt 70.7 28.3 96.7 91.5 bert-base 73.0 31.9 97.5 94.0 clinicalbert 74.4 32.5 97.9 94.4 biomistral 66.6 29.8 94.4 89.0