TLDR at S em E val-2024 Task 2: T5-generated clinical-Language summaries for D e BERT a Report Analysis (2024)

Abstract: This paper introduces novel methodologies for the Natural Language Inference for Clinical Trials (NLI4CT) task. We present TLDR (T5-generated clinical-Language summaries for DeBERTa Report Analysis) which incorporates T5-model generated premise summaries for improved entailment and contradiction analysis in clinical NLI tasks. This approach overcomes the challenges posed by small context windows and lengthy premises, leading to a substantial improvement in Macro F1 scores: a 0.184 increase over truncated premises. Our comprehensive experimental evaluation, including detailed error analysis and ablations, confirms the superiority of TLDR in achieving consistency and faithfulness in predictions against semantically altered inputs.

## introduction the multi-evidence natural language inference for clinical trial data (nli4ct) task focuses on developing systems that can interpret clinical trial reports (ctrs) and make inferences about them @xcite . the task provides a collection of breast cancer ctrs from clinicaltrials.gov along with hypothesis statements and labels annotated by clinical experts. the nli4ct 2024 task is to classify whether a given ctr entails or contradicts the hypothesis statement. this is challenging as it requires aggregating heterogeneous evidence from different sections of the ctrs like interventions, results, and adverse events. the dataset contains 999 breast cancer ctrs and 2400 annotated hypothesis statements split into training, development, and test sets. the ctrs are summarized into sections aligning with patient intervention comparison outcome framework. the 2024 semeval task has the same training dataset as the semeval 2023 task @xcite but the development and test sets for the 2024 task include interventions of either preserving or inversing the entailment relations for some data points. in this paper, we introduce tldr (t5-generated clinical-language summaries for deberta report analysis), a novel framework developed for the semeval task 2024. our key contribution is the integration of a t5-based summarization approach to preprocess and condense the premises of clinical reports, which are then analyzed alongside the corresponding statements using deberta, an encoderonly transformer to perform natural language inference (nli). our t5-generated summaries address the limitations of small context windows and lengthy premises for this task. this methodology demonstrates a significant improvement in performance on the held-out test set, with our best model achieving an increase of 0.184 in the macro f1 score compared to using truncated premises and 0.046 over extractive summarized premises. to underscore the efficacy of our approach in this task, we have conducted extensive experiments and ablations, complemented by a thorough error analysis. we also demonstrate the efficacy of our model's performance with regards to consistency and faithfulness against semantically altered inputs. these efforts collectively highlight the robustness and effectiveness of the tldr framework in addressing the complexities and nuances of nli task within the domain of clinical report trials. ## background the nli4ct task requires developing systems capable of nli from clinical trial reports. in the following section, we examine not only contributions within the semeval nli4ct task @xcite but also wider advancements in the field. ## system overview in this section, we explain our methodologies to address the complexities of the natural language inference for clinical trials (nli4ct) task. we primarily focus on the utilization of large language models such as t5 @xcite and bert-based architectures @xcite , to implement innovative techniques to manage the challenge of the long premise lengths and token size limitations. as mentioned in section 2, deberta is a top-performing model for nli tasks. however, it has a limitation in clinical nli such as the task in nli4ct due to its restricted context length, making it difficult to include both the premise and the statement. we therefore propose to fine-tune t5 to summarize long clinical premises, which can then allow us to leverage deberta's discriminative power for clinical nli with extended premises. we call our model tldr for t5-generated clinical-language summaries for deberta report analysis. our full pipeline can be seen in figure 1 . in this section, we delve into the fine-tuning of these models for our clinical nli task and employ specialized summarization strategies using variants of the t5 model, balancing between zero-shot and fine-tuned approaches. * ## experimental setup in this section, we provide a comprehensive outline of our experimental methods, setting the stage for a detailed discussion on each technique employed. the fine-tuning of each module of our model was done on the training set and evaluated on the development set. upon selecting the best performing model from the development set, we then evaluated our model on the held-out test. the performance reported in section 5 is on this held-out test set. importantly, deberta has a maximum input size of 512 tokens. therefore, the combined length of the shortened premise and statement pair is required to fit into this size (the average statement length in the training data was 110 tokens). the description of the different premise shortening techniques we employed is outlined below. ## conclusion in this paper, we introduced tldr (t5-generated clinical-language summaries for deberta report analysis) tailored for clinical nli tasks, with a focus on nli4ct 2024. our investigation reveals that strategies incorporating scifive for distinct premise summarization and fine-tuning for summarization to better align with entailed statements about the premises markedly improve handling of clinical language and reasoning complexities. despite the challenges posed by lengthy premises in clinical reports, our tldr methods, particularly those employing advanced summarization through fine-tuning, consistently demonstrated superior performance over simpler methods. this underscores the importance of model adaptation and the strategic selection of summarization techniques in enhancing the accuracy and reliability of nli tasks within the clinical domain. looking ahead, a promising avenue for future work involves the use of the best encoder-decoder transformer summarization model for each specific section of clinical reports. this approach could potentially improve the overall performance of nli where we saw fine-tuned scifive was particularly better in some sections of the clinical report and t5-based summaries were better at some other sections. continued exploration and refinement of these models are essential to further advance the field of nli in clinical applications. xiaofeng zhao, min zhang, miaomiao ma, chang su, yilun liu, minghan wang, xiaosong qiao, jiaxin guo, yinglu li, and wenbing ma. 2023. hw-tsc at semeval-2023 task 7: exploring the natural language inference capabilities of chatgpt and pre-trained language model for clinical trial. in proceedings of the the 17th international workshop on semantic evaluation (semeval-2023), pages 1603-1608. yuxuan zhou, ziyu jin, meiwei li, miao li, xien liu, xinxin you, and ji wu. 2023. thifly research at semeval-2023 task 7: a multi-granularity system for ctr-based textual entailment and evidence retrieval. in proceedings of the 17th international workshop on semantic evaluation (semeval-2023), pages 1681-1690, toronto, canada. association for computational linguistics.